Limit prometheus exemplar label characters to conform to prometheus limits

jack-berg commented 2 weeks ago

I ran the agent using java auto-instrument. Everything seemed to be working fine, but I didn't see any exemplar in grafana. I checked the error below through the prometheus log.

err="label length for exemplar exceeds maximum of 128 UTF-8 characters"

When I checked the metric, there were too many labels attached to the exemplar.

# client_address="127.0.0.6",network_peer_address="127.0.0.6",network_peer_port="55579",server_address="10.3.17.168",server_port="8081",span_id="049178b29912fdb4",trace_id="0669315b30dbe08683c19ed9bd24068b",url_path="/actuator/health",user_agent_original="kube-probe/1.29+"}

Is there a way to remove the Exemplar's Label?

Originally posted by @myeongseok-rpls in https://github.com/open-telemetry/opentelemetry-java-instrumentation/discussions/12411

As discussed in the issue, prometheus has a limit on the allowed characters in exemplars. The java prometheus exporter should not produce exemplars which violate this limit. Currently, the limit appears to be hard coded to 128 characters.

harshitrjpt commented 1 week ago

hi @jack-berg , I am working on this by validating the exemplar label runes length in Otel2PrometheusConverter#convertExemplar. (hopefully that's the right direction, do correct me if wrong)

However, when i tried writing the test case by creating a sample Metric data with exemplar, I noticed that io.prometheus.metrics.expositionformats.PrometheusTextFormatWriter#writeCounter is not writing the exemplar labels to the output stream. Do you think this is a bug?

I could not find any tests validating whether exemplar objects's labels are being validated anywhere during otel 2 prometheus conversion.

harshitrjpt commented 1 week ago

To use exemplars we need to use OpenMetrics exposition format writer. ref: https://github.com/prometheus/docs/blob/main/content/docs/instrumenting/exposition_formats.md#exemplars-experimental

jack-berg commented 1 week ago

To use exemplars we need to use OpenMetrics exposition format writer. ref: https://github.com/prometheus/docs/blob/main/content/docs/instrumenting/exposition_formats.md#exemplars-experimental

Yup.

@fstab is this actually an issue with prometheus/client_java? Should the OpenMetricTextFormatWriter limits the characters in exemplar labels to 128 like the golang implementation?

open-telemetry / opentelemetry-java

Limit prometheus exemplar label characters to conform to prometheus limits #6770