search

thanos-io / thanos

Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project.
https://thanos.io
Apache License 2.0
13.08k stars 2.09k forks source link

Blocking certain queries in Thanos #7579

Open rkdutta opened 2 months ago

rkdutta commented 2 months ago

Is your proposal related to a problem?

I want to block certain patterns of queries which are very expensive and lead to OOM errors.

Describe the solution you'd like

It would be great if we could provide a config containing a list of queries or patterns we would like to block. I see grafana mimir has support for this but so far I did not find any such option in Thanos. https://grafana.com/docs/mimir/latest/configure/configure-blocked-queries/#configure-queries-to-block

Describe alternatives you've considered

(Write your answer here.)

Additional context

(Write your answer here.)

harry671003 commented 2 months ago

This feature was also implemented recently in Cortex. https://github.com/cortexproject/cortex/pull/6005 It's just a matter of porting the changes to Thanos.

rkdutta commented 2 months ago

ok thanks for the quick response, I am not sure if there is a separate thread about porting the changes to Thanos.

harry671003 commented 2 months ago

It's not being worked on currently. I was merely pointing out that we can copy most of the Cortex code as is into Thanos.

Thanks for submitting this feature request.

rkdutta commented 2 months ago

Thinking from user behavior..

If you go to explorer page in Grafana, select a thanos datasource and then you specify the labels before you set the metric, this creates a very open-ended query to the backend and leads to OOM. I have seen people doing this many times. I don't know yet how to block this behavioral pattern.

Maybe it's a good idea to have a solution that does not allow querying if the metric name is missing or if the minimum number of labels is less than an acceptable number. Just thought of sharing my 2 cents.