grafana / loki

Like Prometheus, but for logs.
https://grafana.com/loki
GNU Affero General Public License v3.0
23.96k stars 3.46k forks source link

feat: add per-tenant time sharding for long out-of-order ingestion #14711

Closed na-- closed 2 weeks ago

na-- commented 2 weeks ago

What this PR does / why we need it:

This adds support for automatically splitting incoming log streams in the distributor by injecting a __time_shard__ label. The value of that label is bounded by the ingester.max_chunk_age/2 value, which should allow the ingesters to accept all logs without rejecting them as too far behind here: https://github.com/grafana/loki/blob/c0856bf4a66fd589137576b87a80f5531992660f/pkg/ingester/stream.go#L424-L428

Special notes for your reviewer:

Checklist