Prometheus Receiver cannot read common exemplars

[tcolgate]

Component(s)

receiver/prometheus

What happened?

Description

Steps to Reproduce

• Create an application that exposes a histogram with exemplars, via the prometheus metrics endpoint. The exemplar should include a label with (not limited to), a ., e.g. any of the common otel labels such as url.scheme
• Ensure model.NameValidationScheme = model.UTF8Validation is set, promhttp.HandlerOpts.EnableOpenMetrics= true` on the handler.
• Configure the collector to scrape prometheus (you can force the OpenMetrics scrape_protocol)

Expected Result

• The exemapler is exposed on /metrics, prometheus quotes the dotted label key, e.g.:

  metric_milliseconds_bucket{n="1"} 0 # {"url.scheme"="abc"} 1.0

• The collector should scrape the application, read the exemplar, and pass this on through the pipeline

Actual Result

• the scrape fails with an error, debug shows that it fails when reading the quote

Cause

This appears to be a bug in the prometheus scrape code that is not catering for the quoted labels (raised prometheus/prometheus#15259) Using the standard otel http instrumentation, it's hard to avoid the dotted form of hte quoted labels.

It is possible to just strings.Replace the dots in the opentelemetry-go client library's prometheus exporter, but this is not a general fix for UTF8 label keys.

Collector version

679374a8656d39b5a38515c7d5fdaf708a999e4c

Environment information

go version go1.23.2 linux/amd64

OpenTelemetry Collector configuration

receivers:     
  prometheus:
        config:
          global:
            evaluation_interval: 1m
            scrape_interval: 1m
            scrape_protocols:
            - OpenMetricsText1.0.0
            - OpenMetricsText0.0.1
            scrape_timeout: 10s

Log output

WILL UPDATE ASAP

Additional context

No response

[github-actions[bot]]

Pinging code owners:

• receiver/prometheus: @Aneurysm9 @dashpole

  See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dashpole]

Thanks for the report. Lets follow-up on the prometheus issue.