Otel Exponential Histogram Metrics are getting dropped while sending using Prometheusremotwrite exporter

[Shindek77]

Component(s)

exporter/prometheusremotewrite

Describe the issue you're reporting

Actually, We are getting Metrics data which type is Exponential Histogram, But when we are trying to send that metric data to backend tools like Mimir, VictoriaMetrics etc using promtheusremotewrite exporter, It getting dropped even though as per one below issue and PR prometheusremotewriteexporter support conversion of OTel Exponential Histogram to Prometheus Native Histogram. https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/17370 https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/16207

code:

exporters:
   prometheusremotewrite/vm-test:
        endpoint: http://victoria-metrics-cluster-vminsert.metrics-ns.svc.cluster.local:8480/insert/1200/prometheus
        resource_to_telemetry_conversion:
          enabled: true
        timeout: 30s
        tls:
          insecure: true
service:
   extensions:
    - health_check
    pipelines:
      metrics:
        exporters:
        -  prometheusremotewrite/vm-test
        processors:
        - batch
        receivers:
        - otlp

Please let me know why they are not getting converted...is there parameter that I need to set??

[github-actions[bot]]

Pinging code owners:

• exporter/prometheusremotewrite: @Aneurysm9 @rapphil @dashpole

  See Adding Labels via Comments if you do not have permissions to add labels yourself.

[ArthurSens]

I believe this issue is similar to https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/33703, just for a different exporter.

The problem is that OTLP Histograms don't translate well to Prometheus classic histogram. We might want to take a look in translating it to Prometheus Native Histograms instead, while keeping an eye into implications (e.g. Prometheus doesn't support Native Histograms in text format yet)

[Shindek77]

Hello @ArthurSens , Thanks for your reply....But as per PR https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/17370 its already support conversion of OTel Exponential Histogram to Prometheus Native Histogram. Then in our case why its not happening when we are using prometheusremotewrite exporter to send data to VictoriaMetrics in backend as follows: exporters: prometheusremotewrite/vm-test: endpoint: http://victoria-metrics-cluster-vminsert.metrics-ns.svc.cluster.local:8480/insert/1200/prometheus resource_to_telemetry_conversion: enabled: true timeout: 30s tls: insecure: true or is there any other flag that we need to set on prometheusremotewrite exporter for this conversion

[github-actions[bot]]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/prometheusremotewrite: @Aneurysm9 @rapphil @dashpole

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dashpole]

I believe this will be solved by PRW 2.0, which is tracked in https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/33661. I don't think PRW 1.0 supports exponential histograms (in https://prometheus.io/docs/specs/remote_write_spec/, the value is always a double).

[bboreham]

While the Prometheus Remote-Write Specification 1.0 does not mention native histograms, the data structures were added in Prometheus code two years ago, merged to main and released behind a feature flag.

It does not need Remote-Write 2.0.

[bboreham]

Can someone explain why https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/17370, which at least in the title does what is asked for, is not working for OP ?

[dashpole]

My bad. I forgot we implemented that.

It getting dropped even though as per one below issue and PR prometheusremotewriteexporter support conversion of OTel Exponential Histogram to Prometheus Native Histogram.

@Shindek77 are you seeing any errors logged, or are they dropped silently?

[Shindek77]

My bad. I forgot we implemented that.

It getting dropped even though as per one below issue and PR prometheusremotewriteexporter support conversion of OTel Exponential Histogram to Prometheus Native Histogram.

@Shindek77 are you seeing any errors logged, or are they dropped silently?

@dashpole , on Opentelemetry side we was not getting any error but on vminsert of VictoriaMetrics we was getting error as below as it was not supporting Exponential Histogram Metrics type: {"ts":"2024-11-18T06:40:37.212Z","level":"warn","caller":"VictoriaMetrics/lib/protoparser/opentelemetry/stream/streamparser.go:89","msg":"unsupported type for metric \"my_exponential_histogram\""}

[dashpole]

Ah, you probably should follow along here then: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3733. If you just want to silence the errors, you can filter out the metrics in question using the filter processor, or (I think) the transform processor.