Limit prometheus exemplar label characters to conform to prometheus limits

[jack-berg]

I ran the agent using java auto-instrument. Everything seemed to be working fine, but I didn't see any exemplar in grafana. I checked the error below through the prometheus log.

err="label length for exemplar exceeds maximum of 128 UTF-8 characters"

When I checked the metric, there were too many labels attached to the exemplar.

# client_address="127.0.0.6",network_peer_address="127.0.0.6",network_peer_port="55579",server_address="10.3.17.168",server_port="8081",span_id="049178b29912fdb4",trace_id="0669315b30dbe08683c19ed9bd24068b",url_path="/actuator/health",user_agent_original="kube-probe/1.29+"}

Is there a way to remove the Exemplar's Label?

Originally posted by @myeongseok-rpls in https://github.com/open-telemetry/opentelemetry-java-instrumentation/discussions/12411

As discussed in the issue, prometheus has a limit on the allowed characters in exemplars. The java prometheus exporter should not produce exemplars which violate this limit. Currently, the limit appears to be hard coded to 128 characters.

[harshitrjpt]

hi @jack-berg , I am working on this by validating the exemplar label runes length in Otel2PrometheusConverter#convertExemplar. (hopefully that's the right direction, do correct me if wrong)

However, when i tried writing the test case by creating a sample Metric data with exemplar, I noticed that io.prometheus.metrics.expositionformats.PrometheusTextFormatWriter#writeCounter is not writing the exemplar labels to the output stream. Do you think this is a bug?

I could not find any tests validating whether exemplar objects's labels are being validated anywhere during otel 2 prometheus conversion.

[harshitrjpt]

To use exemplars we need to use OpenMetrics exposition format writer. ref: https://github.com/prometheus/docs/blob/main/content/docs/instrumenting/exposition_formats.md#exemplars-experimental

[jack-berg]

To use exemplars we need to use OpenMetrics exposition format writer. ref: https://github.com/prometheus/docs/blob/main/content/docs/instrumenting/exposition_formats.md#exemplars-experimental

Yup.

@fstab is this actually an issue with prometheus/client_java? Should the OpenMetricTextFormatWriter limits the characters in exemplar labels to 128 like the golang implementation?