Metrics scrapped using input prometheus scrape doesn't show when using filter stdout

kazeborja commented 5 months ago

Bug Report

Describe the bug When I gather metrics using the input prometheus scrape and try to output them using a stdout filter nothing appears, it works with the stdout output.

Following few experiments, I'm not sure the prometheus metrics follow the same structure as other flows and I'm not sure if this should work at all, but couldn't find any documentation referring to it.

To Reproduce

Steps to reproduce the problem:

fluent-bit -i prometheus_scrape -p 'host=0.0.0.0' -p 'port=9103' -p 'scrape_interval=10s' -p 'metrics_path=/metrics' -p 'tag=prometheus' -F stdout -m '*' -o null

Expected behavior Prometheus metrics to appear in the stdout

Your Environment

Version used: 3.0.2
Operating System and version: Alma9
Filters and plugins: stdout

Additional context What I'm actually trying to do is to rewrite_tag prometheus metrics based on the metric name (was trying to output them to understand the internal format), but this seemed even more a basic issue.

patrick-stephens commented 5 months ago

Are you sure it's actually scraping some metrics? Might be easier to test/repro with a docker compose stack to run up something reporting them too.

You should verify any firewall rules, etc. as well.

patrick-stephens commented 5 months ago

I tried it by scraping Fluent Bit's own metrics just to confirm and seems fine to me:

[SERVICE]
    http_server on

[INPUT]
    name prometheus_scrape
    host 0.0.0.0 
    port 2020
    tag test 
    metrics_path /api/v1/metrics/prometheus 
    scrape_interval 10s

[OUTPUT]
    name stdout
    match *

docker run --rm -it -v $PWD/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro fluent/fluent-bit:3.0.6
Digest: sha256:16b5f6e92b6e551c8758639b47a659c0e9f82f33c1de2d94c1f5c2e933ebe4be
Status: Downloaded newer image for fluent/fluent-bit:3.0.6
Fluent Bit v3.0.6
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

___________.__                        __    __________.__  __          ________  
\_   _____/|  |  __ __   ____   _____/  |_  \______   \__|/  |_  ___  _\_____  \ 
 |    __)  |  | |  |  \_/ __ \ /    \   __\  |    |  _/  \   __\ \  \/ / _(__  < 
 |     \   |  |_|  |  /\  ___/|   |  \  |    |    |   \  ||  |    \   / /       \
 \___  /   |____/____/  \___  >___|  /__|    |______  /__||__|     \_/ /______  /
     \/                     \/     \/               \/                        \/ 

[2024/06/04 15:10:48] [ info] [fluent bit] version=3.0.6, commit=9af65e2c36, pid=1
[2024/06/04 15:10:48] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/06/04 15:10:48] [ info] [cmetrics] version=0.9.0
[2024/06/04 15:10:48] [ info] [ctraces ] version=0.5.1
[2024/06/04 15:10:48] [ info] [input:prometheus_scrape:prometheus_scrape.0] initializing
[2024/06/04 15:10:48] [ info] [input:prometheus_scrape:prometheus_scrape.0] storage_strategy='memory' (memory only)
[2024/06/04 15:10:48] [ info] [output:stdout:stdout.0] worker #0 started
[2024/06/04 15:10:48] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2024/06/04 15:10:48] [ info] [sp] stream processor started
2024-06-04T15:10:58.118000128Z fluentbit_input_bytes_total{name="prometheus_scrape.0"} = 0
2024-06-04T15:10:58.118000128Z fluentbit_input_records_total{name="prometheus_scrape.0"} = 0
2024-06-04T15:10:58.118000128Z fluentbit_output_dropped_records_total{name="stdout.0"} = 0
2024-06-04T15:10:58.118000128Z fluentbit_output_errors_total{name="stdout.0"} = 0
2024-06-04T15:10:58.118000128Z fluentbit_output_proc_bytes_total{name="stdout.0"} = 0
2024-06-04T15:10:58.118000128Z fluentbit_output_proc_records_total{name="stdout.0"} = 0
2024-06-04T15:10:58.118000128Z fluentbit_output_retried_records_total{name="stdout.0"} = 0
2024-06-04T15:10:58.118000128Z fluentbit_output_retries_failed_total{name="stdout.0"} = 0
2024-06-04T15:10:58.118000128Z fluentbit_output_retries_total{name="stdout.0"} = 0
2024-06-04T15:10:58.119031129Z fluentbit_uptime = 10
2024-06-04T15:10:58.119031129Z process_start_time_seconds = 1717513848
2024-06-04T15:10:58.119031129Z fluentbit_build_info{version="3.0.6",edition="Community"} = 1

kazeborja commented 5 months ago

Hey, thanks for having a look!

Yes, as stated the stdout output works fine (that's how I know I actually have metrics :D ). It's more when trying to do anything with them in the filters (like using the stdout filter which seems the most basic one, it won't print anything) I've also tried with Lua scripting trying to see if any will go over them and no luck, that's what made me wonder if these Prometheus scrape metrics are somehow special... (Hence the issue)

patrick-stephens commented 5 months ago

Classic filters do not work on metrics only logs unfortunately, you have to use processors.

kazeborja commented 5 months ago

Thanks! that makes total sense, sorry if I skipped it, but it's that mentioned somewhere in the documentation?

patrick-stephens commented 5 months ago

Thanks! that makes total sense, sorry if I skipped it, but it's that mentioned somewhere in the documentation?

Nowhere I can see obviously which is a problem so please submit a docs PR: https://github.com/fluent/fluent-bit-docs

kazeborja commented 5 months ago

Will try to find the correct page on the docs to add that :D

github-actions[bot] commented 2 months ago

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

github-actions[bot] commented 2 months ago

This issue was closed because it has been stalled for 5 days with no activity.

fluent / fluent-bit

Metrics scrapped using input prometheus scrape doesn't show when using filter stdout #8909

Bug Report