Create a configurable list of TraceQL queries that are immediately 400'ed

[joe-elliott]

Create a configurable list of regexes that are applied to every TraceQL query received by Tempo. If one of the regexes matches the query should be immediately 400ed instead of executed. This is to help in situations where a tenant is executing a query that is negatively impacting Tempo (e.g. OOMing or panicing queriers).

The config can be added here:

https://github.com/grafana/tempo/blob/2870e82df7045b41c166da4a624ae8d68c0e6394/modules/frontend/config.go#L34

Ideally we would create a new middleware and add it here:

https://github.com/grafana/tempo/blob/2870e82df7045b41c166da4a624ae8d68c0e6394/modules/frontend/frontend.go#L113

to reject the queries.

[ishanjain28]

Hey, I would like to work on this. How do I read the query from *http.Request ?

[maliciousbucket]

Hi I've sort of started having a go this, just have a couple of questions:

• Does the body of the response returned from a match need to be modified?
• Is the middleware applied to all of endpoints? Thanks

[joe-elliott]

Hey, I would like to work on this. How do I read the query from *http.Request ?

Here's an example of parsing the http.Request to get the query and other params: https://github.com/grafana/tempo/blob/e706a6a468a8f889e5400ad71524d5b98810266e/modules/frontend/search_sharder.go#L68

Does the body of the response returned from a match need to be modified?

Perhaps the body could be something like: "Query is temporarily blocked by your administrator."

Is the middleware applied to all of endpoints?

Should be applied to TraceQL search: https://github.com/grafana/tempo/blob/e706a6a468a8f889e5400ad71524d5b98810266e/modules/frontend/frontend.go#L95

and TraceQL metrics: https://github.com/grafana/tempo/blob/e706a6a468a8f889e5400ad71524d5b98810266e/modules/frontend/frontend.go#L128