APM UI - Not able to search for child span/transaction labels

[eduhenke]

Note: I've posted this on Elastic Forum. But as it may also be a bug, I've posted this here too

Describe the bug

I expected to be able to filter traces based on their transactions' labels. However as shown below I am not able to do that.

With the following distributed transaction:

When I see the child transaction from the "station-write" service, we can see that there is a "stationId" label:

When I see the parent transaction from the "gateway" service, we can see that there are other labels:

To Reproduce Steps to reproduce the behavior:

Go to "Traces" page, enter in the search box the "labels.stationId" field(we can see that the auto complete works):

Filter by a given value, and see that it doesn't show any transactions

I couldn't find anywhere that said that it is(or is not) possible to search by labels in child spans/transactions. If it isn't possible to search by those labels, is there any way that I can add those labels to the parent transaction, in order for them to be searchable? The gateway service starts the trace, and this "station-write" service creates another transaction inside of that distributed trace(they are in different pods).

Expected behavior I expected to be able to filter traces based on their transactions' labels.

Versions

Kibana version: 7.15.1 Elasticsearch version: 7.15.1 APM Server version: 7.15.1 APM Agent language and version: Node.JS at 3.26.0

[mrclrchtr]

Are there any updates here? We have the same problem and have firmly assumed that we can search for values of labels of spans.

[basepi]

@mrclrchtr you can see a workaround on the discuss thread mentioned.

[mrclrchtr]

Unfortunately, this workaround does not work for us. We have a transaction with many spans that we would like to give labels with different IDs. Based on these IDs we want to be able to search. We do not want to use the Discover search.

I now use another workaround: I create new transactions in the same service with the following code snippet:

fun getCurrentTransactionTraceHeaders(): Map<String, String> {
      val transactionTraceHeaderMap = mutableMapOf<String, String>()
      ElasticApm.currentTransaction().injectTraceHeaders { headerName, headerValue -> transactionTraceHeaderMap[headerName] = headerValue }
      return transactionTraceHeaderMap.toMap()
    }

and

val transaction =  ElasticApm.startTransactionWithRemoteParent { key: String -> getCurrentTransactionTraceHeaders()[key] }

transaction.activate().use {
  transaction.setName("${this.javaClass.simpleName}#startOrResume")
            .setLabel("customId", customId)
}

We can now search for these transactions in the Service overview as well. Intuitively, of course, it would make more sense to be able to search for the spans.