Operations tab in APM dependencies does not display trace information

[rbagd]

Kibana version: 8.11.4

Elasticsearch version: 8.11.4

Server OS version: Ubuntu 22.04

Browser version: Firefox 123.0

Browser OS version: Ubuntu 22.04

Original install method (e.g. download page, yum, from source, etc.): Elastic Operator in Kubernetes

Describe the bug: When browsing Elastic APM and trying to list traces associated to a certain dependency (e.g. database service), those traces are not found, or at least not displayed.

Steps to reproduce:

1. Open Observability -> APM -> Dependencies
2. Select one of the dependency services
3. Click Operations tab (marked as beta)

Expected behavior: Relevant traces for selected dependency and time period should be visible at the bottom. Throughput and latency charts indicate that there are traces for the selected period.

Screenshots (if relevant):

Errors in browser console (if relevant):

There seem to be no HTTP errors per se. In each case, a request similar to this one is fired

GET https://kibana.host/internal/apm/dependencies/operations/spans?dependencyName=mysql&spanName=SELECT&start=2024-03-15T15:43:15.450Z&end=2024-03-15T15:58:15.450Z&environment=ENVIRONMENT_ALL&kuery=

and returns a JSON with { spans: [] }. Similar requests that generate charts, e.g. /internal/apm/dependencies/charts/throughput, return normal JSON responses with data inside. Traces can still be found by searching on span.destination.service.name in other windows, it's just not available in Operations tab.

I also tested v8.12 deployment on a completely separate K8S cluster using Quickstart demo and the result was the same.

[elasticmachine]

Pinging @elastic/apm-ui (Team:APM)

[kpatticha]

@rbagd thanks for opening the ticket. However, I couldn't reproduce it. Can you please provide a HAR with the _inspect enabled.

You can enable the _inspect in Kibana advanced settings -> Inspect ES queries

[rbagd]

@kpatticha, I just created an issue with Elastic support where I provided an attachment with the requested HAR. Case number #01588392.

[kpatticha]

Thanks @rbagd

[rbagd]

I've investigated the issue along with Elastic Support who identified the root cause. In my case, database spans were set to be of Client kind. Here are the findings.

[...] we have identified that the traces are not displaying because the spans lack the transaction.id field. This absence is the reason why the waterfall visualization is not loading, resulting in the message “The selected trace cannot be found” appearing on your end.

To offer you more insight into our process, you can review the logic we use to set the transaction id in the apm-data repository at the following link: apm-data transaction.id logic. According to our current logic, a span will only have a transaction.id field if it is the root of the trace (i.e., there are no upstream spans) or if the OpenTelemetry entry span has a SpanKind of Server or Consumer. In your case, spans have SpanKind equal to Client and are not getting transaction.id assigned.

I also was informed that Elastic will investigate whether the processing logic should be updated.

Should the issue be closed or converted to a feature request?

[elasticmachine]

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

[smith]

@rbagd it's ok to leave it open here as a bug and we'll investigate further what changes are required.