No query results for older telemetry

[rolfik-mycronic]

I import older logs/traces to loki/tempo via OTEL Collector. In Grafana, logs are searchable. Links from log records to trace records show a trace from tempo well, but I see no traces when using TraceQL even for simplest {} case. When I sent new traces from an app, it is shown in tempo search, but previous older traces never, whatever time interval is used, which covers also old traces times.

I have Aspire Dashboard in parallel and can see traces there.

Something similar is described at https://community.grafana.com/t/inconsistent-empty-search-results-no-traces-returning-after-some-time/124735

I want to see old traces when querying it.

Environment:

• Grafana stack runs in containers on Windows 10 pc with Docker Desktop WSL.

• tempo 2.4.1 (I have tried 2.5.0 with same results)

  • configuration attached
  • http://localhost:3200/metrics result
    • tempo_distributor_spans_received_total{tenant="single-tenant"} 256389

• Deployment tool: docker compose

Query logs:

Working with traceid:

2024-07-25 11:16:56 level=info ts=2024-07-25T09:16:56.193891372Z caller=handler.go:149 tenant=single-tenant method=GET traceID=36d1ebaffb5692d2 url=/api/traces/1d999f68f39c6c412492158a1444c840 duration=12.368019ms response_size=6826 status=200

No result for {} query: 2024-07-25 11:28:10 level=info ts=2024-07-25T09:28:10.536375108Z caller=searchsharding.go:254 msg="sharded search query request stats and SearchMetrics" tenant=single-tenant query="end=1721899690&limit=20&q={}&spss=3&start=1690277290" duration_seconds=3.277639ms request_throughput=0 total_requests=1 started_requests=1 cancelled_requests=0 finished_requests=1 totalBlocks=0 inspectedBytes=0 inspectedTraces=0 totalBlockBytes=0

[joe-elliott]

This indicates that the query frontend is only creating a single job to search the ingesters:

total_requests=1

This means that the query frontend is unaware of any blocks in object storage. Can you check object storage to see if you have any blocks? Also look for errors in the query frontend related to polling.

[rolfik-mycronic]

But as I wrote, I can query newer telemetry when added to older one which cannot be queried. This is my storage:

[joe-elliott]

What is the value of tempodb_blocklist_length as reported by your query frontend?