search

open-telemetry / opentelemetry-java-instrumentation

OpenTelemetry auto-instrumentation and instrumentation libraries for Java
https://opentelemetry.io
Apache License 2.0
1.9k stars 828 forks source link

JVM Semantic Metrics clash with Micrometer Bridged JVM metrics #11122

Open ryanohnemus opened 5 months ago

ryanohnemus commented 5 months ago

Describe the bug

When you opt in to the new jvm semantic metrics (via env OTEL_SEMCONV_STABILITY_OPT_IN=jvm), 2 metrics end up being created of the same name but with different descriptions, sources, and types which cause some downstream issues in exporters like gcp exporter and metrics are then dropped.

The following metrics are being reported to my log on each interval:

instrumentationScopeInfo=InstrumentationScopeInfo{name=io.opentelemetry.runtime-telemetry-java8, version=1.33.1-alpha, schemaUrl=null, attributes={}}, name=jvm.memory.committed, description=Measure of memory committed., unit=By, type=LONG_SUM, data=ImmutableSumData{points=[ImmutableLongPointData{startEpochNanos=1712936148357263000, epochNanos=1712936163369111000, attributes={jvm.memory.pool.name="G1 Survivor Space", jvm.memory.type="heap"}, value=6291456, exemplars=[]}, ImmutableLongPointData{startEpochNanos=1712936148357263000, epochNanos=1712936163369111000, attributes={jvm.memory.pool.name="G1 Eden Space", jvm.memory.type="heap"}, value=52428800, exemplars=[]}, ImmutableLongPointData{startEpochNanos=1712936148357263000, epochNanos=1712936163369111000, attributes={jvm.memory.pool.name="G1 Old Gen", jvm.memory.type="heap"}, value=54525952, exemplars=[]}, ImmutableLongPointData{startEpochNanos=1712936148357263000, epochNanos=1712936163369111000, attributes={jvm.memory.pool.name="CodeHeap 'profiled nmethods'", jvm.memory.type="non_heap"}, value=12648448, exemplars=[]}, ImmutableLongPointData{startEpochNanos=1712936148357263000, epochNanos=1712936163369111000, attributes={jvm.memory.pool.name="Compressed Class Space", jvm.memory.type="non_heap"}, value=7405568, exemplars=[]}, ImmutableLongPointData{startEpochNanos=1712936148357263000, epochNanos=1712936163369111000, attributes={jvm.memory.pool.name="CodeHeap 'non-nmethods'", jvm.memory.type="non_heap"}, value=2555904, exemplars=[]}, ImmutableLongPointData{startEpochNanos=1712936148357263000, epochNanos=1712936163369111000, attributes={jvm.memory.pool.name="Metaspace", jvm.memory.type="non_heap"}, value=52297728, exemplars=[]}, ImmutableLongPointData{startEpochNanos=1712936148357263000, epochNanos=1712936163369111000, attributes={jvm.memory.pool.name="CodeHeap 'non-profiled nmethods'", jvm.memory.type="non_heap"}, value=3932160, exemplars=[]}], monotonic=false, aggregationTemporality=CUMULATIVE}}

instrumentationScopeInfo=InstrumentationScopeInfo{name=io.opentelemetry.micrometer-1.5, version=null, schemaUrl=null, attributes={}}, name=jvm.memory.committed, description=The amount of memory in bytes that is committed for the Java virtual machine to use, unit=bytes, type=DOUBLE_GAUGE, data=ImmutableGaugeData{points=[ImmutableDoublePointData{startEpochNanos=1712936148357263000, epochNanos=1712936166999000000, attributes={area="nonheap", id="Compressed Class Space"}, value=7471104.0, exemplars=[]}, ImmutableDoublePointData{startEpochNanos=1712936148357263000, epochNanos=1712936166999000000, attributes={area="nonheap", id="CodeHeap 'non-profiled nmethods'"}, value=3997696.0, exemplars=[]}, ImmutableDoublePointData{startEpochNanos=1712936148357263000, epochNanos=1712936166999000000, attributes={area="heap", id="G1 Old Gen"}, value=5.6623104E7, exemplars=[]}, ImmutableDoublePointData{startEpochNanos=1712936148357263000, epochNanos=1712936166999000000, attributes={area="nonheap", id="CodeHeap 'profiled nmethods'"}, value=1.2713984E7, exemplars=[]}, ImmutableDoublePointData{startEpochNanos=1712936148357263000, epochNanos=1712936166999000000, attributes={area="heap", id="G1 Eden Space"}, value=5.24288E7, exemplars=[]}, ImmutableDoublePointData{startEpochNanos=1712936148357263000, epochNanos=1712936166999000000, attributes={area="nonheap", id="CodeHeap 'non-nmethods'"}, value=2555904.0, exemplars=[]}, ImmutableDoublePointData{startEpochNanos=1712936148357263000, epochNanos=1712936166999000000, attributes={area="nonheap", id="Metaspace"}, value=5.2625408E7, exemplars=[]}, ImmutableDoublePointData{startEpochNanos=1712936148357263000, epochNanos=1712936166999000000, attributes={area="heap", id="G1 Survivor Space"}, value=4194304.0, exemplars=[]}]}}

This also affects javaagents 2.0+ when you've opted in to OTEL_INSTRUMENTATION_MICROMETER_ENABLED=true

Steps to reproduce

I am using the opentelemetry-javaagent 1.33.1 and attaching to my application. I am using the following env settings: 

OTEL_LOGS_EXPORTER=none
OTEL_METRIC_EXPORT_INTERVAL=15000
OTEL_METRICS_EXPORTER=logging
OTEL_SEMCONV_STABILITY_OPT_IN=jvm
OTEL_TRACES_EXPORTER=none

Expected behavior

I'm not sure of what should be the expected behavior, but I'm hoping there's a solution so there's no metric clash on the name and types.

Should I be configuring the micrometer bridge in a different way so it's not including these metrics somehow or disabling instrumentation feature via OTEL_INSTRUMENTATION_[something]_ENABLED=false?

Actual behavior

see above

Javaagent or library instrumentation version

1.33.1

Environment

JDK: OS:

Additional context

No response

laurit commented 5 months ago

Currently your best option is not to enable both of these instrumentations at the same time.