grafana / grafana

The open and composable observability and data visualization platform. Visualize metrics, logs, and traces from multiple sources like Prometheus, Loki, Elasticsearch, InfluxDB, Postgres and many more.
https://grafana.com
GNU Affero General Public License v3.0
60.72k stars 11.61k forks source link

Alerting: Do not retry rule evaluations with "input data must be a wide series but got type long" style errors #87343

Closed alexweav closed 1 week ago

alexweav commented 2 weeks ago

What is this feature?

Usually, this error happens with SQL or tabular datasources that return a dataframe that can't be converted to an Alerting-style series.

Therefore, it's in the same vein of problem as alerting's invalidEvalResultFormatError which we already consider as non retryable. This one just happens earlier in the stack. Consider it non-retryable as well.

Why do we need this feature?

Prevents unnecessary retries on an error that usually indicates that the query needs to be fixed.

Who is this feature for?

Users with this configuration problem who are using alerting evaluation retries - send fewer unnecessary queries.

Which issue(s) does this PR fix?:

n/a

Special notes for your reviewer:

Please check that: