Latest java agent 1.35.1 and .2 breaks APM tracing

DataDog / dd-trace-java

Datadog APM client for Java

https://docs.datadoghq.com/tracing/languages/java

Apache License 2.0

551 stars 278 forks source link

Latest java agent 1.35.1 and .2 breaks APM tracing #7214

Closed kashyap-parikh-ah closed 1 day ago

kashyap-parikh-ah commented 2 weeks ago

Following this direction https://docs.datadoghq.com/tracing/trace_collection/automatic_instrumentation/dd_libraries/java/?tab=curl

It pulls latest version of dd-java-agent.jar

Version 1.35.1 breaks all traces.

mcculls commented 2 weeks ago

Hi @kashyap-parikh-ah can you provide more information, for example tracer debug logs for the previous version you were using and 1.35.2. You can provide these via a support ticket. It would be especially useful to compare the startup part of the debug logs for the previous version compared to 1.35.2

It would be also useful to know:

What kind of application are you attaching the tracer to (library versions etc.)
Full version and vendor string of the JVM you are using
How are you attaching the Java tracer, single step or using -javaagent ?
What traces did you get with the previous version that you don't with v1.35.2?
How are traces broken - is it you don't get traces at all or something else?

tantran-earnin commented 2 weeks ago

Hey @mcculls we opened up a ticket #1747501. We provided some of the details requested in there. For the tracer debug logs, we'll have to find a couple services to turn it on.

As for high level, we instrument using -javaagent, and once they were on 1.35.2, we saw 0 APM metrics/traces in Datadog. We pinned 1.35.0 and they showed up again.

All our discussions moving forward will be in the ticket.

mcculls commented 2 weeks ago

Thanks @tantran-earnin - having debug logs for the same service before and after the upgrade will help us identify if there were any differences in which types got transformed, as well as if there were any unexpected issues during startup.

So far we haven't identified anything in that release which could result in a complete loss of metrics/traces - the debug logs will also help us identify if the tracer is producing the same spans as before, which might then point towards a communication issue between the tracer and the agent.

mcculls commented 2 weeks ago

Also can you see if the issue goes away when you use the same version for dd-trace-api and the Java tracer?

tantran-earnin commented 2 weeks ago

Matching the dd-trace-api with dd-java-agent didn't seem to fix. Will try to set in trace debug mode for both 1.35.0 and 1.35.2.

tantran-earnin commented 2 weeks ago

I've reverted matching of the dd-trace-api. I redeployed both 1.35.0 and 1.35.2 with debug tracer logs. Links to the logs are noted in the ticket.

mcculls commented 1 day ago

Answered via support ticket 1747501