search

aws-observability / aws-otel-java-instrumentation

AWS Distro for OpenTelemetry Java Instrumentation Library
https://aws-otel.github.io/
Apache License 2.0
73 stars 55 forks source link

http.server.duration has exceeded the maximum allowed cardinality #644

Closed avictor0826 closed 9 months ago

avictor0826 commented 10 months ago

Describe the bug When the v1.31.1 aws-opentelemetry-agent.jar was used to instrument the metrics, we are seeing the logs filled with the below error

2023-12-12T06:40:54.892696483Z stderr F [otel.javaagent 2023-12-12 06:40:54:892 +0000] [http-nio-8081-exec-3] WARN io.opentelemetry.sdk.metrics.internal.state.DefaultSynchronousMetricStorage - Instrument http.server.duration has exceeded the maximum allowed cardinality (1999)

Steps to reproduce Change the javaagent to v1.31.1 (download) and run the application.

What did you expect to see? The errors were not supposed to be seen in the logs.

What did you see instead? The warnings are being printed that few of the metrics (to be specific http.server.duration) has exceeeded the limit of the allowed cardinality

Additional context The traces are supposed to have a high level of cardinality whereas the metrics are suppose to have a lower cardinality. I was able to see similar issue descriptions and fixes in the opentelemetry-go packages - here where it discusses that high cardinality metrics such as http.client_ip are included in this Instrument. There are fixes in the semconv packages too. I can guess that is a problem with the dependency that this javaagent is using but not able to pinpoint it because of the limited time I have to troubleshoot this. It would be of great help if someone else can check this out..

We are now using the v1.31.0 agent.jar file which is working without any trouble

bryan-aguilar commented 10 months ago

Hi @avictor0826,

Thanks for the report. Can you additional info such as:

  1. How you are enabling the ADOT Java Agent?
  2. Any additional related otel environment variables in your environment?
  3. Could you provide a small example that reproduces this issue?
bryan-aguilar commented 9 months ago

Hi @avictor0826,

We release v1.31.2 which removes additional attribute attached to http metrics. Could you give it a try?

Thanks!

avictor0826 commented 9 months ago

@bryan-aguilar that did work.. 

Picked up JAVA_TOOL_OPTIONS: -javaagent:/opt/aws-opentelemetry-agent.jar
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
[otel.javaagent 2024-01-05 12:21:14:107 +0000] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.31.2-aws

Tried the latest 1.32.0 and that also had no errors.

Picked up JAVA_TOOL_OPTIONS: -javaagent:/opt/aws-opentelemetry-agent.jar
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
[otel.javaagent 2024-01-05 10:54:50:806 +0000] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.32.0-aws

thanks for the fix and appreciate your help..