Find Traces Matching Multiple Services

[frankgreco]

Counterpart to UI ticket https://github.com/jaegertracing/jaeger-ui/issues/180

Currently, you can only search spans belonging to a one service. It would be nice if you could find traces belonging to multiple services. Then, you could search tags where error:true on all tags or search for traces which include multiple services.

Implementation plan:

• [ ] define a new Storage Capability property indicating if multi-service search is supported (default: not supported)
• [ ] change the UI to receive this flag via existing capabilities mechanism and allow / disallow not specifying service name in the search panel
• [ ] implement multi-service search in memory store (should be trivial)
• [ ] implement multi-service search in Elasticsearch / Opensearch (probably the easiest across real backends)
• [ ] try to implement it in Badger (difficulty: unknown)
• [ ] for Cassandra, introduce another flag to allow / disallow multi-service searches. The motivation is that with the current indexing scheme the only way to support multi-service search is by reading all service names and iterating across all of them (which I think what Zipkin implementation did). This might work for small deployments, but would be useless for large companies like Uber, so they would likely opt to disable it.
  • Another possibility is to use secondary index in Cassandra. The current schema is not designed for that, so difficulty: unknown.

[vprithvi]

Related: https://github.com/jaegertracing/jaeger/issues/238

[yurishkuro]

Currently, service name is required because of the limitation of Cassandra implementation, where we do not have general indexing and instead build indices manually on top of kv-store, such that service name is always a prefix of the lookup key. There is a way to work around that by first querying all service names and then executing a search query against each of them. Such approach will very wildly inefficient for systems with large number of services.

On the other hand, dropping the requirement for service name in Elasticsearch is trivial, it has no special role. I am not sure if Badger implementation has a similar limitation as Cassandra.

An alternative approach that we discussed in some other tickets is to allow the UI to understand different capabilities of the storage backend and present slightly different search panels, like not requiring service name for ES, but requiring it for Cassandra.

[VuiDJi]

Have you got some news? This would be a very useful feature

[mempirate]

This would be very useful for our use cases as well!

[alex-kormukhin]

This feature is stopper for us for migration from zipkin (see also).

[AndrWeisR]

Without this feature, how do you track a request that is handled by several services? You should be able to search by tag, such as a transaction reference, and have that tag found in every service that the transaction has been through.

[yurishkuro]

@AndrWeisR if you know a specific tag you may also know which service writes it to the trace, and you search for that service.

[yurishkuro]

I updated the description with an action plan, if someone wants to contribute the implementation.