[APM] Support for message queues

[sorenlouv]

TODO:

• [x] 1. Add kafka and other messaging backends to be displayed on the service map (global and service-specific)
• [ ] 2. Add kafka and other messaging backends to be displayed on the service dependencies list
• [ ] 3. Add kafka and other messaging backends to be displayed on the global backends list
• [ ] 4. Collapse multiple topics into one (kafka/* to kafka) similar to how we collapse external HTTP resources.

Sample data

A sample application for setting up kafka can be found here

1a. Global Service Map

Problem: Map is missing kafka, but connects services directly like if they are talking over HTTP or gRPC.

Current

Expected

1b. Service-specific Service Map

Problem: The kafka topic is showing up on the service specific service map, but the other service isn’t showing up.

Expected

2. Dependencies list

Problem: Span by type is missing messaging details, and downstream services is not showing the kafka backend.

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[dannycroft]

@dgieselaar can this be closed now or can some of the TODO items be checked off?

[dgieselaar]

@dannycroft don't think so, we just have a POC at this point.

[alex-fedotyev]

Totally forgot to add a requirement to make sure queues are showing up in the global APM -> Dependencies list, like below (as at the time when this issue was originally created we had no global dependencies yet):

[dannycroft]

@alex-fedotyev and I spoke about the dependencies list and there is an option to deliver the dependencies list in the next release if confidence is low on achieving it this release. When this ticket gets picked up we'll see if anything has changed. Alex has some ideas about releasing this in a staged appoach that might help with scope here, if needed

@dgieselaar some good information here if you've not seen it yet 👍🏻

[felix-lessoer]

@dannycroft Any updates on this one?

[smith]

@felix-lessoer Danny is no longer assigned to this. @gbamparop is currently the team lead for APM UI and might be a better person to contact about this.

[gbamparop]

@felix-lessoer this is in our radar but we are not planning to work on it for 8.8. We will post an update if something changes. Please upvote or reach to us if you have any other queries to help with prioritisation.

[elasticmachine]

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

[smith]

New Relic article about tracing in Kafka: https://newrelic.com/blog/how-to-relic/distributed-tracing-with-kafka

[AlexanderWert]

Point 1. from above has been addressed in https://github.com/elastic/kibana/pull/153784

[botelastic[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.