Fluent bit getting killed with "flex scanner jammed" - reading Keycloak metrics

[diegosarina]

Bug Report

Fluent bit getting killed with "flex scanner jammed" while reading metrics from Keycloak.

To Reproduce

• Docker compose with Keycloak v25 running inside and port metric enabled to 9000
• Add a prometheus_scrape input for Keycloak /metrics ui

  [INPUT]
  name prometheus_scrape
  host keycloak
  port 9000
  tag keycloak
  metrics_path /metrics
  scrape_interval 10s

• Reading from prometheus directly works as expected
• Here is the trace route from fluentbit

______ _                  _    ______ _ _           _____  __
|  ___| |                | |   | ___ (_) |         |____ |/  |
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __   / /`| |
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / /   \ \ | |
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /.___/ /_| |_
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/ \____(_)___/

[2024/10/01 13:03:08] [ info] [fluent bit] version=3.1.7, commit=c6e902a43a, pid=1
[2024/10/01 13:03:08] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/10/01 13:03:08] [ info] [cmetrics] version=0.9.5
[2024/10/01 13:03:08] [ info] [ctraces ] version=0.5.5
[2024/10/01 13:03:08] [ info] [input:forward:forward.0] initializing
[2024/10/01 13:03:08] [ info] [input:forward:forward.0] storage_strategy='memory' (memory only)
[2024/10/01 13:03:08] [ info] [input:forward:forward.0] listening on 0.0.0.0:24224
[2024/10/01 13:03:08] [ info] [input:prometheus_scrape:prometheus_scrape.1] initializing
[2024/10/01 13:03:08] [ info] [input:prometheus_scrape:prometheus_scrape.1] storage_strategy='memory' (memory only)
[2024/10/01 13:03:08] [ info] [input:prometheus_scrape:prometheus_scrape.2] initializing
[2024/10/01 13:03:08] [ info] [input:prometheus_scrape:prometheus_scrape.2] storage_strategy='memory' (memory only)
[2024/10/01 13:03:08] [ info] [input:prometheus_scrape:prometheus_scrape.3] initializing
[2024/10/01 13:03:08] [ info] [input:prometheus_scrape:prometheus_scrape.3] storage_strategy='memory' (memory only)
[2024/10/01 13:03:08] [ info] [input:prometheus_scrape:prometheus_scrape.4] initializing
[2024/10/01 13:03:08] [ info] [input:prometheus_scrape:prometheus_scrape.4] storage_strategy='memory' (memory only)
[2024/10/01 13:03:08] [ info] [input:prometheus_scrape:prometheus_scrape.5] initializing
[2024/10/01 13:03:08] [ info] [input:prometheus_scrape:prometheus_scrape.5] storage_strategy='memory' (memory only)
[2024/10/01 13:03:08] [ info] [output:prometheus_exporter:prometheus_exporter.0] listening iface=0.0.0.0 tcp_port=2021
[2024/10/01 13:03:08] [ info] [output:loki:loki.1] configured, hostname=XXX.XXXXX.XXX:<port>
[2024/10/01 13:03:08] [ info] [sp] stream processor started
flex scanner jammed

I really appreciate any insight in this situation :) Diego

ALSO: related issue: https://github.com/fluent/fluent-bit/issues/8152

[edsiper]

@diegosarina I tried the payload provided in #8152 and I could not reproduce the issue. Would you please provide a full copy of the Prometheus metrics that Fluent Bit is unable to process ?

curl http://host:port/metrics

[diegosarina]

Hi @edsiper, sorry for the delay. Here you can find attached a TXT with the information that you requested

keycloak.txt

Please, let me know if you need anything else.