grafana / loki

Like Prometheus, but for logs.
https://grafana.com/loki
GNU Affero General Public License v3.0
23.97k stars 3.46k forks source link

query_ingesters_within didn't work as expected #14984

Open yukinami opened 4 days ago

yukinami commented 4 days ago

Describe the bug query_ingesters_within didn't work as expected

To Reproduce Steps to reproduce the behavior:

  1. set reject_old_samples: false, query_ingesters_within=168h',max_chunk_age=2h`
  2. Started Loki 3.2.1
  3. Started Promtail 3.1.2 to tail logs with settings
    
    server:
    http_listen_port: 9080
    grpc_listen_port: 0

positions: filename: ./positions.yaml

clients:

scrape_configs:


3. Query: `{job="training"} term`

I imported some logs which were logged 4 days ago, and I hoped to query the logs immediately after I imported. I read this [blog](https://grafana.com/blog/2024/01/04/the-concise-guide-to-loki-how-to-work-with-out-of-order-and-older-logs)
According to my understanding, the logs of 4 days ago I justed imported has not yet be flushed, and with the default 'query_ingesters_within=2h' settings, the query searching logs for thoese logs will directly go to the store, which would match nothing.
SoI set query_ingesters_within=168h and the query (searching logs of 4 days ago)should be executed on the ingester (where the logs data exist currently). But still I return nothing. Only If I set both `max_chunks_age= 168h` and  `query_ingesters_within= 168h`, It would return data. 

**Expected behavior**
With setting `query_ingesters_within= 168h`, loki should return historical data immediately.

**Environment:**
 - Infrastructure: bare-metal
 - Deployment tool: mannully

**Screenshots, Promtail config, or terminal output**