PrometheusDuplicateTimestamps errors with log_to_metrics filter starting in fluent-bit 3.1.5

[reneeckstein]

Bug Report

Describe the bug After upgrading from fluent-bit 3.1.4 to 3.1.5 all our k8s clusters start reporting PrometheusDuplicateTimestamps errors Prometheus metric rate(prometheus_target_scrapes_sample_duplicate_timestamp_total}[5m]) > 0 is increasing. Prometheus is logging a lot of warnings like this:

ts=2024-09-23T16:22:32.820Z caller=scrape.go:1754 level=warn component="scrape manager" scrape_pool=serviceMonitor/platform-logging/fluent-bit/1 target=http://10.67.3.197:2021/metrics msg="Error on ingesting samples with different value but same timestamp" num_dropped=32
ts=2024-09-23T16:22:38.237Z caller=scrape.go:1754 level=warn component="scrape manager" scrape_pool=serviceMonitor/platform-logging/fluent-bit/1 target=http://10.67.4.81:2021/metrics msg="Error on ingesting samples with different value but same timestamp" num_dropped=1876
ts=2024-09-23T16:22:39.697Z caller=scrape.go:1754 level=warn component="scrape manager" scrape_pool=serviceMonitor/platform-logging/fluent-bit/1 target=http://10.67.13.208:2021/metrics msg="Error on ingesting samples with different value but same timestamp" num_dropped=4
ts=2024-09-23T16:22:41.643Z caller=scrape.go:1754 level=warn component="scrape manager" scrape_pool=serviceMonitor/platform-logging/fluent-bit/1 target=http://10.67.3.110:2021/metrics msg="Error on ingesting samples with different value but same timestamp" num_dropped=7

To Reproduce

• Steps to reproduce the problem:
  • Deploy fluent-bit with the below tail input config as a daemonset into a k8s cluster using version 3.1.4 to see container logs and metrics to validate success.
  • Update fluent-bit image to 3.1.5 (or newer, <= 3.1.8) and verify /metrics endpoint on port 2021

Expected behavior No duplicate metrics on the additional endpoint /metrics for log_to_metrics feature usually on port 2021, no warnings in Prometheus logs, no PrometheusDuplicateTimestamps errors.

Screenshots

Your Environment

• Version used: 3.1.5 (or higher) tested until 3.1.8, issue is still present.
• Configuration: (Helm chart values)

  
  serviceMonitor:
  enabled: true
  interval: 10s
  scrapeTimeout: 10s
  additionalEndpoints:
  - port: log-metrics
  path: /metrics
  interval: 10s
  scrapeTimeout: 10s

extraPorts:

• port: 2021 containerPort: 2021 protocol: TCP name: log-metrics

config: service: | [SERVICE] Flush 1 Daemon Off Log_Level info Parsers_File parsers.conf Parsers_File custom_parsers.conf HTTP_Server On HTTP_Listen 0.0.0.0 HTTP_Port {{ .Values.service.port }}

inputs: | [INPUT] Name tail Tag kube. Alias tail_container_logs Path /var/log/containers/.log multiline.parser docker, cri DB /var/log/flb_kube.db DB.locking true Mem_Buf_Limit 32MB Skip_Long_Lines On

filters: | [FILTER] Name kubernetes Alias kubernetes_all Match kube.* Merge_Log On Keep_Log Off K8S-Logging.Parser On K8S-Logging.Exclude On Annotations Off Buffer_Size 1MB Use_Kubelet true

[FILTER]
    name               log_to_metrics
    match              kube.*
    tag                log_counter_metric
    metric_mode        counter
    metric_name        kubernetes_messages
    metric_description This metric counts Kubernetes messages
    kubernetes_mode    true

outputs: | [OUTPUT] name prometheus_exporter match log_counter_metric host 0.0.0.0 port 2021

* Environment name and version (e.g. Kubernetes? What version?):
  * EKS; Kubernetes 1.30
* Server type and version:
* Operating System and version:
  * EKS on Bottlerocket OS 1.22.0 (aws-k8s-1.30)   Kernel version 6.1.106  containerd://1.7.20+bottlerocket
* Filters and plugins:
  * kubernetes, log_to_metrics

**Additional context**
It is just very annoying when every k8s cluster with this common configuration reports PrometheusDuplicateTimestamps errors

[edsiper]

@reneeckstein are you facing the same issue with v3.1.8 ? (we have some fixes in place for a similar problem)

[reneeckstein]

@edsiper Yes we are facing the same issue in fluent-bit v3.1.8. I'm looking forward for v3.1.9 I noticed to metrics-related commits on master branch.