open-telemetry / opentelemetry-collector-contrib

Contrib repository for the OpenTelemetry Collector
https://opentelemetry.io
Apache License 2.0
2.9k stars 2.27k forks source link

Otel Exponential Histogram Metrics are getting dropped while sending using Prometheusremotwrite exporter #34917

Open Shindek77 opened 2 weeks ago

Shindek77 commented 2 weeks ago

Component(s)

exporter/prometheusremotewrite

Describe the issue you're reporting

Actually, We are getting Metrics data which type is Exponential Histogram, But when we are trying to send that metric data to backend tools like Mimir, VictoriaMetrics etc using promtheusremotewrite exporter, It getting dropped even though as per one below issue and PR prometheusremotewriteexporter support conversion of OTel Exponential Histogram to Prometheus Native Histogram. https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/17370 https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/16207

code:

exporters:
   prometheusremotewrite/vm-test:
        endpoint: http://victoria-metrics-cluster-vminsert.metrics-ns.svc.cluster.local:8480/insert/1200/prometheus
        resource_to_telemetry_conversion:
          enabled: true
        timeout: 30s
        tls:
          insecure: true
service:
   extensions:
    - health_check
    pipelines:
      metrics:
        exporters:
        -  prometheusremotewrite/vm-test
        processors:
        - batch
        receivers:
        - otlp

Please let me know why they are not getting converted...is there parameter that I need to set??

github-actions[bot] commented 2 weeks ago

Pinging code owners:

ArthurSens commented 1 week ago

I believe this issue is similar to https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/33703, just for a different exporter.

The problem is that OTLP Histograms don't translate well to Prometheus classic histogram. We might want to take a look in translating it to Prometheus Native Histograms instead, while keeping an eye into implications (e.g. Prometheus doesn't support Native Histograms in text format yet)

Shindek77 commented 1 week ago

Hello @ArthurSens , Thanks for your reply....But as per PR https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/17370 its already support conversion of OTel Exponential Histogram to Prometheus Native Histogram. Then in our case why its not happening when we are using prometheusremotewrite exporter to send data to VictoriaMetrics in backend as follows: exporters: prometheusremotewrite/vm-test: endpoint: http://victoria-metrics-cluster-vminsert.metrics-ns.svc.cluster.local:8480/insert/1200/prometheus resource_to_telemetry_conversion: enabled: true timeout: 30s tls: insecure: true or is there any other flag that we need to set on prometheusremotewrite exporter for this conversion