search

elastic / elastic-otel-java

Apache License 2.0
10 stars 9 forks source link

Improve inferred spans stack trace #135

Closed SylvainJuge closed 2 months ago

SylvainJuge commented 5 months ago

Found with the Elastiflix demo app when adding a delay to have significant span durations with the sampling profiler.

The inferred spans allow to capture the red spans:

image

However the code.stacktrace for those two spans are not very helpful as they are respectively for the top and bottom red spans:

top span : code.stacktrace ``` java.lang.Throwable at co.elastic.otel.ElasticSpanProcessor.captureStackTrace(ElasticSpanProcessor.java:77) at co.elastic.otel.ElasticSpanProcessor.onEnd(ElasticSpanProcessor.java:53) at io.opentelemetry.sdk.trace.MultiSpanProcessor.onEnd(MultiSpanProcessor.java:52) at io.opentelemetry.sdk.trace.SdkSpan.endInternal(SdkSpan.java:454) at io.opentelemetry.sdk.trace.SdkSpan.end(SdkSpan.java:441) at co.elastic.otel.profiler.CallTree.asSpan(CallTree.java:520) at co.elastic.otel.profiler.CallTree.spanify(CallTree.java:452) at co.elastic.otel.profiler.CallTree.spanify(CallTree.java:460) at co.elastic.otel.profiler.CallTree.spanify(CallTree.java:460) at co.elastic.otel.profiler.CallTree.spanify(CallTree.java:460) at co.elastic.otel.profiler.CallTree.spanify(CallTree.java:460) at co.elastic.otel.profiler.CallTree.spanify(CallTree.java:460) at co.elastic.otel.profiler.CallTree.spanify(CallTree.java:460) at co.elastic.otel.profiler.CallTree.spanify(CallTree.java:460) at co.elastic.otel.profiler.CallTree.spanify(CallTree.java:460) at co.elastic.otel.profiler.CallTree.spanify(CallTree.java:460) at co.elastic.otel.profiler.CallTree.spanify(CallTree.java:460) at co.elastic.otel.profiler.CallTree.spanify(CallTree.java:460) at co.elastic.otel.profiler.CallTree.spanify(CallTree.java:460) at co.elastic.otel.profiler.CallTree.spanify(CallTree.java:460) at co.elastic.otel.profiler.CallTree.spanify(CallTree.java:460) at co.elastic.otel.profiler.CallTree.spanify(CallTree.java:460) at co.elastic.otel.profiler.CallTree.spanify(CallTree.java:460) at co.elastic.otel.profiler.CallTree.spanify(CallTree.java:460) at co.elastic.otel.profiler.CallTree$Root.spanify(CallTree.java:870) at co.elastic.otel.profiler.SamplingProfiler$ActivationEvent.stopProfiling(SamplingProfiler.java:974) at co.elastic.otel.profiler.SamplingProfiler$ActivationEvent.handleDeactivationEvent(SamplingProfiler.java:943) at co.elastic.otel.profiler.SamplingProfiler$ActivationEvent.handle(SamplingProfiler.java:891) at co.elastic.otel.profiler.SamplingProfiler.processActivationEventsUpTo(SamplingProfiler.java:641) at co.elastic.otel.profiler.SamplingProfiler.processTraces(SamplingProfiler.java:560) at co.elastic.otel.profiler.SamplingProfiler.profile(SamplingProfiler.java:423) at co.elastic.otel.profiler.SamplingProfiler.run(SamplingProfiler.java:376) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Thread.java:833) ```
bottom span code.stacktrace ``` at com.movieapi.ApiServlet.getUserFavorites(ApiServlet.java) at com.movieapi.ApiServlet.handlePost(ApiServlet.java) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java) ```
JonasKunz commented 5 months ago

I think the top-span doesn't get a stacktrace from the inferred-spans extension, therefore our SpanProcessor responsible for populating the stack traces for normal spans takes effect.

We should avoid this in the processor by checking for the is_inferred attribute and excluding those spans.

JonasKunz commented 5 months ago

We should avoid this in the processor by checking for the is_inferred attribute and excluding those spans.

Done via #144 .

SylvainJuge commented 2 months ago

Closing this for now as there isn't much we can improve here. The code stacktrace processor contributed upstream allows filtering spans.