grafana / grafana

The open and composable observability and data visualization platform. Visualize metrics, logs, and traces from multiple sources like Prometheus, Loki, Elasticsearch, InfluxDB, Postgres and many more.
https://grafana.com
GNU Affero General Public License v3.0
63.79k stars 11.94k forks source link

Correlations: Handle explore editor in Traces list view #79997

Open gelicia opened 8 months ago

gelicia commented 8 months ago

When running a search in tempo, it returns a table of traces to choose from. Selecting one pulls up the trace view for that trace.

Data links were implemented in that trace view, so we added correlations there in the original delivery of the feature. However, data links were never meant to be in the trace search table. Data links do work in the table visualization, but require the link to be configured to the data frame correctly. This is how the links to traces and spans currently work with the trace search result table.

When you see the trace search results table and hit Add to Correlation, it generates the temporary data links for the table and those show up. You can even click it and it will pull up the correlation details helper (blue box) with the information about that row. However, since that table row is not something directly returned by a data source, saving a correlation here effectively builds a correlation that cannot be displayed.

A correlation can be built in the editor on a specific trace, and that will work as expected.

TLDR:

We will need to brainstorm what we want to do in this scenario. I'm wondering if we want to have an explore state that's like "disableCorrelationHelper" where if there's a string, the Add Correlation button is disabled with the string as a tool tip. We can then let datasources pass that up to state if needed.

ifrost commented 5 months ago

More details from the conversation with Bojan:

  • The way tempo returns search results makes it hard to match data frames with correct correlation (the refId of the results doesn’t match the refId of the search query). The workaround is to change the ref id to traces (more details in the video).
  • The above workaround won’t make the correlation show up in Trace Details because the service name is stored in a different field (search returns it as traceService while trace details returns it as serviceName). This makes it impossible to reuse exactly the same correlation. The workaround is to create another correlation for the trace details view (more details in the video)
video https://github.com/grafana/grafana/assets/745532/32a4709b-3219-4d08-8262-0d2f64aa340e