Same queries result in different results

[tg3rh]

Describe the bug When quickly refreshing (running a search query in the explore view) I am getting a non deterministic view of the traces, resulting in different traces (different timestamps) being shown.

To Reproduce Steps to reproduce the behavior:

1. Navigate to the Explore section in Grafana
2. Navigate to Query type: Search
3. Hit Run query a couple of times
4. Observe different timestamp (+- 5min) are being show in the list

Expected behavior Expecting to receive the latest traces on the top when refreshing the query.

Environment:

• Grafana Cloud

Additional Context Attaching screenshots (queries run after each other)

[joe-elliott]

Tempo returns the first spans it finds in the supplied time range matching the criteria. We would like to change this to return the most recent spans which would make the results stable.

Some discussion here: https://github.com/grafana/tempo/issues/2659#issuecomment-2110381426

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity. Please apply keepalive label to exempt this Issue.

[proffalken]

@joe-elliott - have you been able to make any progress on this?

[joe-elliott]

I have. Here is my current branch diffed against main:

https://github.com/grafana/tempo/compare/main...joe-elliott:ordered-results

I believe I overcame the last of the major perf regressions yesterday. Results will definitely be a bit slower to come in b/c Tempo has to search more data to make sure that it has the most recent results, but I have mitigated it as much as possible.

It's been slow going. I have not had a ton of time to focus on it and I also went down a few paths I ended up not liking before finding this one. Having said that I think it's close and I'm hoping to get a PR up next week.

Remaining todo:

• Restore/cleanup tests
• Docs
• Balance some internal parameters. I have some hardcoded values that are balanced against our internal testing cluster that I either need to make configurable or find ways to dynamically adjust.

[PietroPasotti]

Not sure this is related, but the same happens to me when hitting the search API directly via http. Would you like me to open this as a different issue @joe-elliott?

[joe-elliott]

No, this is the same issue