DataDog / dd-trace-java

Datadog APM client for Java
https://docs.datadoghq.com/tracing/languages/java
Apache License 2.0
586 stars 290 forks source link

Incorrect exec time for child spans when using completable future in spring controller #7825

Open VasaviMadhurima opened 3 weeks ago

VasaviMadhurima commented 3 weeks ago

Hi,

When diagnosing some performance issues, I noticed that the execution times for child spans in traces are incorrect for APIs that are implemented using Spring Async controller feature with Completable futures.

Here is a sample trace: Image Image

Take S3.HeadBucket child span for example - the waterfall view clearly shows that the exec time is 223ms. However, in the span summary we see that exec time is reported as 111ms (exactly half).

Below is a sample code to show how we use the Completable futures:

@GetMapping("/my-service-path")
public CompletableFuture<ResponseEntity<MyResponseObject>> getSomeInfo() {
    return CompletableFuture.supplyAsync(
        TracedSupplier.from(() -> {
            //internalServicelogic
         }),
    executorService
  );
}

Annoyances:

We extensively use Completable futures in spring controller for all our java services. Given this issue, the time spent charts incorrectly represent the time spent in the service v/s the downstream components. This defeats the purpose of using APM tracing to a large extent for us.

VasaviMadhurima commented 3 weeks ago

I can confirm that if we remove the Completable future and make the API implementation synchronous, the exec times of child spans look alright.