fluent / fluent-bit

Fast and Lightweight Logs and Metrics processor for Linux, BSD, OSX and Windows
https://fluentbit.io
Apache License 2.0
5.86k stars 1.58k forks source link

out_telemetry: "invalid temporality and type combination" error #6303

Closed flobz closed 2 years ago

flobz commented 2 years ago

Bug Report

Describe the bug The out_telemetry doesn't set an aggregation type for counter metrics. As a results those metrics are discarded because prometheus metric format needs it. The code that check if aggregation type is set: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/9810bb011e1edc835c8f8ae329884a183bf5586d/pkg/translator/prometheusremotewrite/metrics_to_prw.go#L66

To Reproduce Exemple of workflow who leads to this issue:

             fluent-bit                                     open-telemetry-collector
(in_node_exporter_metrics -> out_telemetry) -> (otlp_receiver -> prometheusremotewrite) -> mimir
or 
(in_node_exporter_metrics -> out_telemetry) -> (mimir)

Steps to reproduce the problem: Use this fluent-bit config:

[SERVICE]
    Flush     1
    Daemon    off
    Log_Level info

[INPUT]
    Name                 node_exporter_metrics
    Tag                  node_metrics
    Scrape_interval      10

[OUTPUT]
    Name  stdout
    Match_Regex node_metrics

[OUTPUT]
    Name                 opentelemetry
    Match                node_metrics
    Host                 127.0.0.1
    Port                 4319
    metrics_uri          /v1/metrics
    Log_response_payload True
    Tls                  Off

And this open-telemetry-collector config:

receivers:
  otlp:
      protocols:
          http:
            endpoint: 0.0.0.0:4319

processors:
  batch:

exporters:
  prometheusremotewrite:
    endpoint: http://127.0.0.1:9009/api/v1/push
    resource_to_telemetry_conversion:
      enabled: true
  logging:
     loglevel: info
extensions:
  health_check:
  pprof:

service:
  extensions: [health_check,pprof]
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheusremotewrite,logging]
  telemetry:
    logs:
     level: info

Fix:

https://github.com/fluent/cmetrics/pull/166 https://github.com/fluent/fluent-bit/pull/6302

flobz commented 2 years ago

partial fix: b174c93ded814144c635b366af6e10864b670048

edsiper commented 2 years ago

https://github.com/fluent/fluent-bit/pull/6302 merged