Enhance APM UI Tracing Features for RAG Workflows

[neptunian]

Summary

This issue is a list of enhancements aimed at improving tracing in the APM UI, particularly for RAG applications. The need for these improvements has emerged as the Security team has been integrating tracing functionalities within the Security AI assistant.

Issues

• [ ] https://github.com/elastic/kibana/issues/172411 Create a Waterfall Embeddable to improve in-app UX
• [ ] https://github.com/elastic/kibana/issues/172413 Enhance the span details to include additional metadata, such as token counts, which will provide valuable insights directly within the traces.
• [ ] https://github.com/elastic/kibana/issues/172412 Introduce the ability to collapse certain spans to clean up the UI and focus on the most relevant data, reducing clutter from common but less critical information like ES authentication spans.
• [ ] https://github.com/elastic/kibana/issues/172415 Although less critical once the embeddable waterfall component is in place, adding custom renderers could highlight specific documents retrieved within the RAG pipeline, enhancing the detail available in span explorations.
• [ ] https://github.com/elastic/kibana/issues/171368 Trace explorer bug

Related

• https://github.com/elastic/apm-agent-nodejs/issues/3734 Needed to capture the outgoing request from Kibana to the LLM provider, otherwise we can't see what the LLM is actually being sent

[elasticmachine]

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

[smith]

@roshan-elastic