search

thanos-io / thanos

Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project.
https://thanos.io
Apache License 2.0
13.08k stars 2.1k forks source link

Allow setting min and max time on Querier #5638

Open fpetkovski opened 2 years ago

fpetkovski commented 2 years ago

Is your proposal related to a problem?

We currently run receivers with a 3d retention period. There are various recording rules that run agains them and use data from the entire 3d range period.

We also upload blocks to S3 which are used for showing metrics on dashboards. The data in S3 is deduplicated and compacted, so it can often be faster to query. Unfortunately, we can only run store gateways with an overlap with receivers which would lead to deduplicating 4 independent replicated streams, making querying even slower.

Describe the solution you'd like

I would like to be able to set a max-time on a Thanos querier just like I can on the store-gateway and sidecar. With this solution, I can use one querier against receivers with for the full 3d retention period. I can also have another querier with a max-time of 6 hours, and use Store Gateway for the rest of the data.

Describe alternatives you've considered

I don't see a good solution right now, other than to reduce receiver retention to a shorter interval and use store gateway for recording rules as well.

bwplotka commented 2 years ago

Thanks for idea!

Why you can't use receiver min-time and store gateway max-time for this case? I am still missing what case this is not satisfying 🤔

We can totally and Querier min/max, just time sharding on sidecar, store and receivers were usually enough.

fpetkovski commented 2 years ago

Hm, do receivers support min-time? I couldn't find anything in the flags: https://thanos.io/tip/components/receive.md/#flags

BooNny95 commented 2 years ago

Hm, do receivers support min-time? I couldn't find anything in the flags: https://thanos.io/tip/components/receive.md/#flags

On receivers you may change tsdb retention time. I suppose that query service on sidecar and receiver check how long you have data in your tsdb.

stale[bot] commented 1 year ago

Hello 👋 Looks like there was no activity on this issue for the last two months. Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗 If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

MichaHoffmann commented 9 months ago

@fpetkovski is the issue still relevant? Community member asked for guidances and I wondered that!

houtaru commented 3 months ago

I encounter a similar problem on querier and sidecar. There's also a similar proposal #4027 before.