search

grafana / tempo

Grafana Tempo is a high volume, minimal dependency distributed tracing backend.
https://grafana.com/oss/tempo/
GNU Affero General Public License v3.0
3.76k stars 489 forks source link

Grafana Cloud Tempo: Default Start Time view for traces in Tempo does not order by most recent #3109

Open liamoddell opened 8 months ago

liamoddell commented 8 months ago

Describe the bug Using the Tempo data sources view within Explore on Grafana Cloud, the Start Time of shown traces does not sort as most recent when changing timeframe. For example, if I choose the last 6 hours worth of data in Tempo and the current time is 12:47pm, the first traces we see would be from 6am or 3am that same day and not from the X number of shown traces up to the current time period.

To Reproduce Steps to reproduce the behavior:

  1. Access Tempo via Grafana Cloud
  2. Have constant trace data over the last 6 hours
  3. Change the timerange to Last 1/2/6 hours.
  4. See that the Traces are not the most recent

Expected behavior If I sort by the last 1/2/6/12 hours, the first traces that show should be the most recent without having to manually sort by StartTime.

Environment:

Additional Context

mapno commented 8 months ago

Hi @liamoddell. Tempo search is non-deterministic, which means that Tempo will return the first results that it finds, without a preference to more recent traces.

Related issue: https://github.com/grafana/tempo/issues/2476

joe-elliott commented 8 months ago

I'd like to move towards deterministic search. We did not at first because search was slow. Parquet search is much faster now. That plus the streaming endpoint makes me think we can go to deterministic.

Also, for future questions I'd consider using support. This is kind of a generic question so it's easy to answer, but if your question was specific to your instance it would be easier to answer through that channel.

gouthamve commented 8 months ago

I think this is not very intuitive behavior and borders on being a bug.

With logs, you see the most recent logs first and the expectation is that its the same with Tempo. But with the current behaviour makes it seem like the trace ingestion stopped because we are seeing quite old traces. This is even worse with explore view because it opens with a default 6hr view. So you could see traces that are 6hrs old and none of the recent traces.

github-actions[bot] commented 6 months ago

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity. Please apply keepalive label to exempt this Issue.

mghildiy commented 5 months ago

I can give it a try.

joe-elliott commented 5 months ago

@mghildiy I am currently in the middle of a frontend rewrite/refactor. Now is not a great time to make changes there.

One of the goals of the refactor is to make adding functionality such as this easier.

rolfik commented 4 months ago

Was this ever resolved? We still have problems viewing recent/current traces in Grafana loaded from Tempo. We see older traces instead of newer ones which is odd.

joe-elliott commented 4 months ago

It has not been, but I intend to tackle it as part of the refactor I'm working on now:

https://github.com/grafana/tempo/pull/3400

elburnetto-intapp commented 2 months ago

Hey @joe-elliott, is there any further updates on this issue at all?

joe-elliott commented 1 month ago

Unfortunately, I was unable to address this while working on the refactor. Perhaps as TraceQL metrics wind down I will find time to revisit.