Fluent-bit metrics timestamps not updated

[vainiusd]

Bug Report

Describe the bug Fluent-bit own metric timestamps are updated only if the metric is updated. This makes some metric datapoint too old to consume into metric storage services.

To Reproduce Fluent-bit metrics input enabled. File or any other output for the fluent-bit metrics. Example uses file and opentelemetry outputs.

• Example output from file:

  2024-01-24T09:41:10.441988883Z fluentbit_output_proc_records_total{name="example-otel-output"} = 0
  2024-01-24T09:41:10.441988883Z fluentbit_output_proc_bytes_total{name="example-otel-output"} = 6147328
  2024-01-24T09:31:06.794159432Z fluentbit_output_errors_total{name="example-otel-output"} = 0
  2024-01-24T09:31:06.794159432Z fluentbit_output_retries_total{name="example-otel-output"} = 0
  2024-01-24T09:31:06.794159432Z fluentbit_output_retries_failed_total{name="example-otel-output"} = 0
  2024-01-24T09:31:06.794159432Z fluentbit_output_dropped_records_total{name="example-otel-output"} = 0
  2024-01-24T09:31:06.794159432Z fluentbit_output_retried_records_total{name="example-otel-output"} = 0
  2024-01-24T09:41:21.188470385Z fluentbit_output_upstream_total_connections{name="example-otel-output"} = 0
  2024-01-24T09:41:10.441924567Z fluentbit_output_upstream_busy_connections{name="example-otel-output"} = 0

• Steps to reproduce the problem:

Expected behavior All metric timestamps are renewed when metric scrape by fluent-bit metrics input is done.

2024-01-24T09:41:21.188470385Z fluentbit_output_proc_records_total{name="example-otel-output"} = 0
2024-01-24T09:41:21.188470385Z fluentbit_output_proc_bytes_total{name="example-otel-output"} = 6147328
2024-01-24T09:41:21.188470385Z fluentbit_output_errors_total{name="example-otel-output"} = 0
2024-01-24T09:41:21.188470385Z fluentbit_output_retries_total{name="example-otel-output"} = 0
2024-01-24T09:41:21.188470385Z fluentbit_output_retries_failed_total{name="example-otel-output"} = 0
2024-01-24T09:41:21.188470385Z fluentbit_output_dropped_records_total{name="example-otel-output"} = 0
2024-01-24T09:41:21.188470385Z fluentbit_output_retried_records_total{name="example-otel-output"} = 0
2024-01-24T09:41:21.188470385Z fluentbit_output_upstream_total_connections{name="example-otel-output"} = 0
2024-01-24T09:41:21.188470385Z fluentbit_output_upstream_busy_connections{name="example-otel-output"} = 0

Your Environment

• Version used: 2.2.2 and older ones
• Configuration:
• Environment name and version (e.g. Kubernetes? What version?): Linux and Windows

This was also very well described in issue #7650

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.

[dansimov04012022]

Hi @edsiper , can you please tell me if there are any updates on that? Currently facing the same issue.

Also noticed that even if metrics from:

curl -s http://127.0.0.1:2020/api/v1/metrics/prometheus

have a current timestamp, fluent-bit continues sending them with the 3-day old timestamp.

Great thanks in advance.