search

DataDog / dd-trace-java

Datadog APM client for Java
https://docs.datadoghq.com/tracing/languages/java
Apache License 2.0
566 stars 283 forks source link

Missing ExecutionSample and ObjectAllocationSample events in Java 21 #7590

Closed sivachandran closed 5 days ago

sivachandran commented 1 week ago

Hi,

I profiled a simple Java application with dd-trace-dd 1.39.0 using OpenJDK 21. The resultant JFR has less no. of jdk.ExecutionSample and no jdk.ObjectAllocationSample. When I tried to convert the JFR into pprof using async-profiler's converter.jar the resultant profile is empty which confirms the missing events. jfr cli tool also confirms there are no jdk.ObjectAllocationSample and just 76 jdk.ExecutionSample events for 2mins profile.

The problem is only with OpenJDK 21. The same setup with OpenJDK 17 works fine and produces JFR with both of these events.

Attached the JFR for your reference. jfr-profile-openjdk-21-missing-events.jfr.gz

jbachorik commented 1 week ago

Hi @sivachandran - you are right, those events may be missing since we may be using datadog specific events to record CPU, wallclock and allocation profiles, depending on the Java version, architecture etc.

Also, I want to point out that Datadog profiler is not a generic Java profiler (see https://github.com/DataDog/java-profiler?tab=readme-ov-file#disclaimer) and if you want to profile your Java application but not use Datadog UI you should rather use async-profiler (which you seem to be using already, considering that you tried the converter tool) which is a full-fledged command line Java profiler.

sivachandran commented 5 days ago

Appreciate your reply!

I figured if I disable ddprof(DD_PROFILING_DDPROF_ENABLED=false) then I'm getting the jdk.ExecutionSample and jdk.ObjectAllocationSample events. I believe it is because the profiler uses JFR profiler internally. Do you see any problem with such usage? Are there any plan to drop JFR profiler support in the future?

jbachorik commented 5 days ago

No, we are not planning to drop JFR (however, on certain systems, eg. J9, it is not available). Also, there is no problem with such usage, but honestly, I would go for async-profiler if you are not using DD infrastructure anyway. There is no upside to using DD profiler instead of async-profiler as a general CLI profiling tool.

sivachandran commented 5 days ago

I want to stick with DD profiler as its ability to send the profiles to the configured agent nicely fits with my existing architecture. I believe I need to write a wrapper for async-profiler to achieve the same.

jbachorik commented 5 days ago

Yes, I see. As I said, there is no problem with such usage. An alternative to disabling the dd profiler is to make your pipeline aware of datadog.ExecutionSample (CPU profiler), datadog.MethodSample (Wallclock profiler) and datadog.ObjectSample (allocation profiler) events.

sivachandran commented 5 days ago

Will explore the possibilit to support datadog events.

Once again, thank you for your replies.