search

grafana / pyroscope

Continuous Profiling Platform. Debug performance issues down to a single line of code
https://grafana.com/oss/pyroscope/
GNU Affero General Public License v3.0
9.98k stars 598 forks source link

Trace exemplars #3289

Open kolesnikovae opened 4 months ago

kolesnikovae commented 4 months ago

We already have an integration with distributed tracing solutions that enables traces to profiles navigation. We should extend the scenario to enable the other way around: profiles to traces.

From the user perspective this could look like trace exemplars in metrics: image

Given a query expression (series labels + call site filter), we could retrieve top K samples with tracing attribution (by count/sum) and place them in the timeline. The links would include trace_id:span_id pairs, which allows to navigate users straight to the trace span view. In turn, in the trace span view, there would show up a profile for this specific span.

image

This will require a few adjustments of the storage schema and SDKs (we only collect root span_id and span_name as of now).

It's possible that the trace referred by a profile was not sampled by the trace collector (tail sampling). This should be addressed in the collector

knylander-grafana commented 4 months ago

I like the idea you have in this issue. If we do this, we'll want to add doc to cover concepts and any required configurations.