[APM] Dependencies Operations - trace cannot be found - with OTel

elastic / kibana

Your window into the Elastic Stack

https://www.elastic.co/products/kibana

Other

19.59k stars 8.21k forks source link

[APM] Dependencies Operations - trace cannot be found - with OTel #193672

Open AlexanderWert opened 1 month ago

AlexanderWert commented 1 month ago

When using OTel data (e.g. the OTel demo), in the Dependencies --> Operations --> [SELECT a specific operation] view the most bottom view always shows The selected trace cannot be found:

elasticmachine commented 1 month ago

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

AlexanderWert commented 3 weeks ago

MiriamAparicio commented 11 hours ago

After some investigation we did with @jennypavlova, these are our findings:

The trace waterfall for dependecies operations is filled querying to span documents using term query for span.destination.service.resource and span.name fields, where transaction.id exist, the results for this query are empty, after this query another one is made to retrieve the transactions for those spans, querying the transaction ids

We assumed that in otel, span documents don't have the transaction.id field and that's the reason why the trace waterfall is empy

@AlexanderWert is it possible to include the transaction.id in span documents from Otel?

AlexanderWert commented 11 hours ago

@AlexanderWert is it possible to include the transaction.id in span documents from Otel?

That's unfortunately not easily possible as it would require to buffer spans (similar to tail-based sampling) which we definitely want to avoid.

AlexanderWert commented 11 hours ago

The trace waterfall for dependecies operations is filled querying to span documents using term query for span.destination.service.resource and span.name fields, where transaction.id exist, the results for this query are empty, after this query another one is made to retrieve the transactions for those spans, querying the transaction ids

@MiriamAparicio Instead of going through the transaction.id couldn't we just do the same with trace.id? And in the second step query for those trace.ids but also transaction.id: * (to only limit to spans that are transactions) and service.name being the same as the one of the corresponding exit span. Would that be possible?

MiriamAparicio commented 9 hours ago

Instead of going through the transaction.id couldn't we just do the same with trace.id?

Yeah, that seems possible, we can try and see if it works