DataDog / dd-trace-java

Datadog APM client for Java
https://docs.datadoghq.com/tracing/languages/java
Apache License 2.0
573 stars 285 forks source link

Is there a way to make coexist `jmx_prometheus_javaagent` and the Datadog APM agent in a same project? #5148

Open guizmaii opened 1 year ago

guizmaii commented 1 year ago

Hi everyone 🙂

I'm trying to add the DD APM to a project already using jmx_prometheus_javaagent

When having both, the project crashes at boot with the following error:

[dd.trace 2023-05-03 16:11:43:485 +0000] [dd-jmx-collector] ERROR datadog.trace.agent.jmxfetch.JMXFetch - Exception in jmx collector thread
java.lang.IllegalStateException: Shutdown in progress
    at java.base/java.lang.ApplicationShutdownHooks.add(ApplicationShutdownHooks.java:66)
    at java.base/java.lang.Runtime.addShutdownHook(Runtime.java:216)
    at org.datadog.jmxfetch.App.run(App.java:154)
    at datadog.trace.agent.jmxfetch.JMXFetch$1.run(JMXFetch.java:131)
    at java.base/java.lang.Thread.run(Thread.java:833)

I tried to disable JMX collection from the Datadog agent with "-Ddd.jmxfetch.enabled=false" but the app is still crashing at boot

Here's the full set of config that I use with the DD agent:

"-XX:FlightRecorderOptions=stackdepth=256",
"-Ddd.logs.injection=true",
"-Ddd.trace.sample.rate=1",
"-Ddd.profiling.enabled=true",                              // https://docs.datadoghq.com/profiler/enabling/java/?tab=datadog
"-Ddd.profiling.allocation.enabled=true",                   // https://docs.datadoghq.com/tracing/profiler/profiler_troubleshooting/#enabling-the-allocation-profiler
"-Ddd.profiling.heap.enabled=true",                         // https://docs.datadoghq.com/tracing/profiler/profiler_troubleshooting/#enabling-the-heap-profiler
"-Ddd.profiling.jfr-template-override-file=comprehensive",  // https://docs.datadoghq.com/tracing/profiler/profiler_troubleshooting/#increase-profiler-information-granularity
// https://docs.datadoghq.com/tracing/trace_collection/library_config/java/
"-Ddd.service.mapping=postgresql:myproject-api-postgresql,kafka:myproject-api-kafka,nginx:myproject-api-nginx",
"-Ddd.integration.okhttp.enabled=false",                    // needs to disable tracing of datadog agent
"-Ddd.integration.throwables.enabled=false",                // https://twitter.com/ghostdogpr/status/1602243328119971841
"-Ddd.integration.opentelemetry.experimental.enabled=true", // https://github.com/DataDog/dd-trace-java/pull/4669
"-Ddd.integration.zio.experimental.enabled=true",           // https://github.com/DataDog/dd-trace-java/pull/4848

I'm using v1.12.1 of the DD APM

nayeem-kamal commented 1 year ago

Hi @guizmaii , the Datadog Java tracer is currently not designed to be run alongside another Java agent and can sometimes cause application crashes when ran with other tracers/agents.

Does this issue still occur if only the Datadog java agent is used?

guizmaii commented 1 year ago

Hey @nayeem-kamal,

Does this issue still occur if only the Datadog java agent is used?

The app is working fine with DD agent only :)