search

grafana / grafana

The open and composable observability and data visualization platform. Visualize metrics, logs, and traces from multiple sources like Prometheus, Loki, Elasticsearch, InfluxDB, Postgres and many more.
https://grafana.com
GNU Affero General Public License v3.0
64.99k stars 12.13k forks source link

Alerting: alert state for provisioned alerts is reset to `Normal (Updated)` on Grafana restart #93200

Open equinsuocha opened 2 months ago

equinsuocha commented 2 months ago

What happened?

We manage grafana alert rules in a state repository and mount alert rule provisioning files to grafana pods as configmaps. The problem we are facing, is that every time grafana is restarted, all alert rules states are reset to Normal (Updated).

The issue seems to be quite similar to https://github.com/grafana/grafana/issues/61917

Thread in Grafana Labs Community slack: here

This also seems to cause severe database pollution for us, the alert_rule_version table has already grown 3.8 million entries and every restart takes more and more cpu time.

Example |id |rule_org_id|rule_uid |rule_namespace_uid|rule_group |parent_version|restored_from|version|created |title |condition|data |interval_seconds|no_data_state|exec_err_state|for |annotations |labels |rule_group_idx|is_paused|notification_settings| |------|-----------|------------------------------------|------------------|--------------------------------|--------------|-------------|-------|-------------------|--------------------------------|---------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------|-------------|--------------|-------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------|--------------|---------|---------------------| |492608|49 |d4fb2f40-bd03-4fa5-a5f0-d3fd81ac8fde|bdstisda7lv5sd |company_actions_lock queue alert|0 |0 |1 |2024-07-25 15:37:57|company_actions_lock queue alert|A |[{"refId":"A","queryType":"","relativeTimeRange":{"from":0,"to":0},"datasourceUid":"__expr__","model":{"conditions":[{"evaluator":{"params":[12],"type":"gt"},"operator":{"type":"and"},"query":{"params":["SCRUBBED"]},"reducer":{"type":"min"}},{"evaluator":{"params":[12],"type":"gt"},"operator":{"type":"or"},"query":{"params":["SCRUBBED"]},"reducer":{"type":"min"}},{"evaluator":{"params":[12],"type":"gt"},"operator":{"type":"or"},"query":{"params":["SCRUBBED"]},"reducer":{"type":"min"}},{"evaluator":{"params":[12],"type":"gt"},"operator":{"type":"or"},"query":{"params":["SCRUBBED"]},"reducer":{"type":"avg"}},{"evaluator":{"params":[12],"type":"gt"},"operator":{"type":"or"},"query":{"params":["SCRUBBED"]},"reducer":{"type":"avg"}},{"evaluator":{"params":[12],"type":"gt"},"operator":{"type":"or"},"query":{"params":["SCRUBBED"]},"reducer":{"type":"avg"}},{"evaluator":{"params":[12],"type":"gt"},"operator":{"type":"or"},"query":{"params":["SCRUBBED"]},"reducer":{"type":"avg"}},{"evaluator":{"params":[12],"type":"gt"},"operator":{"type":"or"},"query":{"params":["SCRUBBED"]},"reducer":{"type":"avg"}}],"intervalMs":1000,"maxDataPoints":43200,"refId":"A","type":"classic_conditions"}},{"refId":"SCRUBBED","queryType":"","relativeTimeRange":{"from":43200,"to":0},"datasourceUid":"prom-live","model":{"datasource":{"type":"prometheus","uid":"prom-live"},"editorMode":"code","exemplar":true,"expr":"sum(insights_consistency_checker_lock_migration_queue_size{region=\"SCRUBBED\"}) by (region)","format":"time_series","interval":"1","intervalFactor":1,"intervalMs":1000,"legendFormat":"SCRUBBED","maxDataPoints":43200,"range":true,"refId":"SCRUBBED"}},{"refId":"SCRUBBED","queryType":"","relativeTimeRange":{"from":43200,"to":0},"datasourceUid":"prom-live","model":{"datasource":{"type":"prometheus","uid":"prom-live"},"editorMode":"code","exemplar":true,"expr":"sum(insights_consistency_checker_lock_migration_queue_size{region=\"SCRUBBED\"}) by (region)","format":"time_series","interval":"1","intervalFactor":1,"intervalMs":1000,"legendFormat":"SCRUBBED","maxDataPoints":43200,"range":true,"refId":"SCRUBBED"}},{"refId":"SCRUBBED","queryType":"","relativeTimeRange":{"from":43200,"to":0},"datasourceUid":"prom-live","model":{"datasource":{"type":"prometheus","uid":"prom-live"},"editorMode":"code","exemplar":true,"expr":"sum(insights_consistency_checker_lock_migration_queue_size{region=\"SCRUBBED\"}) by (region)","format":"time_series","interval":"1","intervalFactor":1,"intervalMs":1000,"legendFormat":"SCRUBBED","maxDataPoints":43200,"range":true,"refId":"SCRUBBED"}},{"refId":"SCRUBBED","queryType":"","relativeTimeRange":{"from":43200,"to":0},"datasourceUid":"prom-live","model":{"datasource":{"type":"prometheus","uid":"prom-live"},"editorMode":"code","exemplar":true,"expr":"sum(insights_consistency_checker_lock_migration_queue_size{region=\"SCRUBBED\"}) by (region)","format":"time_series","interval":"1","intervalFactor":1,"intervalMs":1000,"legendFormat":"SCRUBBED","maxDataPoints":43200,"range":true,"refId":"SCRUBBED"}},{"refId":"SCRUBBED","queryType":"","relativeTimeRange":{"from":43200,"to":0},"datasourceUid":"prom-live","model":{"datasource":{"type":"prometheus","uid":"prom-live"},"exemplar":true,"expr":"sum(insights_consistency_checker_lock_migration_queue_size{region=\"SCRUBBED\"}) by (region)","format":"time_series","interval":"1","intervalFactor":1,"intervalMs":1000,"legendFormat":"SCRUBBED","maxDataPoints":43200,"refId":"SCRUBBED"}},{"refId":"SCRUBBED","queryType":"","relativeTimeRange":{"from":43200,"to":0},"datasourceUid":"prom-live","model":{"datasource":{"type":"prometheus","uid":"prom-live"},"editorMode":"code","exemplar":true,"expr":"sum(insights_consistency_checker_lock_migration_queue_size{region=\"SCRUBBED\"}) by (region)","format":"time_series","interval":"1","intervalFactor":1,"intervalMs":1000,"legendFormat":"SCRUBBED","maxDataPoints":43200,"range":true,"refId":"SCRUBBED"}},{"refId":"SCRUBBED","queryType":"","relativeTimeRange":{"from":43200,"to":0},"datasourceUid":"prom-live","model":{"datasource":{"type":"prometheus","uid":"prom-live"},"editorMode":"code","exemplar":true,"expr":"sum(insights_consistency_checker_lock_migration_queue_size{region=\"SCRUBBED\"}) by (region)","format":"time_series","interval":"1","intervalFactor":1,"intervalMs":1000,"legendFormat":"SCRUBBED","maxDataPoints":43200,"range":true,"refId":"SCRUBBED"}},{"refId":"SCRUBBED","queryType":"","relativeTimeRange":{"from":43200,"to":0},"datasourceUid":"prom-live","model":{"datasource":{"type":"prometheus","uid":"prom-live"},"editorMode":"code","exemplar":true,"expr":"sum(insights_consistency_checker_lock_migration_queue_size{region=\"SCRUBBED\"}) by (region)","format":"time_series","interval":"1","intervalFactor":1,"intervalMs":1000,"legendFormat":"SCRUBBED","maxDataPoints":43200,"range":true,"refId":"SCRUBBED"}}]|60 |KeepLast |KeepLast |3600000000000|{"__alertId__":"10309","__dashboardUid__":"WFyEThFWz","__panelId__":"60","description":"SCRUBBED","summary":"{{ range $k, $v := $values -}}\n{{ if (match \"A[0-9]+\" $k) -}}\n{{ $k }}: {{ $v.Value }} {{ $v.Labels }}{{ end }}\n{{ end }}"}|{"__alert_subscribers":"","__contacts__":"SCRUBBED","owner":"SCRUBBED","priority":"p3","rule_uid":"d4fb2f40-bd03-4fa5-a5f0-d3fd81ac8fde"}|0 |0 |null | |492609|49 |d4fb2f40-bd03-4fa5-a5f0-d3fd81ac8fde|bdstisda7lv5sd |company_actions_lock queue alert|1 |0 |2 |2024-07-25 15:37:57|company_actions_lock queue alert|A |[{"refId":"A","queryType":"","relativeTimeRange":{"from":0,"to":0},"datasourceUid":"__expr__","model":{"conditions":[{"evaluator":{"params":[12],"type":"gt"},"operator":{"type":"and"},"query":{"params":["SCRUBBED"]},"reducer":{"type":"min"}},{"evaluator":{"params":[12],"type":"gt"},"operator":{"type":"or"},"query":{"params":["SCRUBBED"]},"reducer":{"type":"min"}},{"evaluator":{"params":[12],"type":"gt"},"operator":{"type":"or"},"query":{"params":["SCRUBBED"]},"reducer":{"type":"min"}},{"evaluator":{"params":[12],"type":"gt"},"operator":{"type":"or"},"query":{"params":["SCRUBBED"]},"reducer":{"type":"avg"}},{"evaluator":{"params":[12],"type":"gt"},"operator":{"type":"or"},"query":{"params":["SCRUBBED"]},"reducer":{"type":"avg"}},{"evaluator":{"params":[12],"type":"gt"},"operator":{"type":"or"},"query":{"params":["SCRUBBED"]},"reducer":{"type":"avg"}},{"evaluator":{"params":[12],"type":"gt"},"operator":{"type":"or"},"query":{"params":["SCRUBBED"]},"reducer":{"type":"avg"}},{"evaluator":{"params":[12],"type":"gt"},"operator":{"type":"or"},"query":{"params":["SCRUBBED"]},"reducer":{"type":"avg"}}],"intervalMs":1000,"maxDataPoints":43200,"refId":"A","type":"classic_conditions"}},{"refId":"SCRUBBED","queryType":"","relativeTimeRange":{"from":43200,"to":0},"datasourceUid":"prom-live","model":{"datasource":{"type":"prometheus","uid":"prom-live"},"editorMode":"code","exemplar":true,"expr":"sum(insights_consistency_checker_lock_migration_queue_size{region=\"SCRUBBED\"}) by (region)","format":"time_series","interval":"1","intervalFactor":1,"intervalMs":1000,"legendFormat":"SCRUBBED","maxDataPoints":43200,"range":true,"refId":"SCRUBBED"}},{"refId":"SCRUBBED","queryType":"","relativeTimeRange":{"from":43200,"to":0},"datasourceUid":"prom-live","model":{"datasource":{"type":"prometheus","uid":"prom-live"},"editorMode":"code","exemplar":true,"expr":"sum(insights_consistency_checker_lock_migration_queue_size{region=\"SCRUBBED\"}) by (region)","format":"time_series","interval":"1","intervalFactor":1,"intervalMs":1000,"legendFormat":"SCRUBBED","maxDataPoints":43200,"range":true,"refId":"SCRUBBED"}},{"refId":"SCRUBBED","queryType":"","relativeTimeRange":{"from":43200,"to":0},"datasourceUid":"prom-live","model":{"datasource":{"type":"prometheus","uid":"prom-live"},"editorMode":"code","exemplar":true,"expr":"sum(insights_consistency_checker_lock_migration_queue_size{region=\"SCRUBBED\"}) by (region)","format":"time_series","interval":"1","intervalFactor":1,"intervalMs":1000,"legendFormat":"SCRUBBED","maxDataPoints":43200,"range":true,"refId":"SCRUBBED"}},{"refId":"SCRUBBED","queryType":"","relativeTimeRange":{"from":43200,"to":0},"datasourceUid":"prom-live","model":{"datasource":{"type":"prometheus","uid":"prom-live"},"editorMode":"code","exemplar":true,"expr":"sum(insights_consistency_checker_lock_migration_queue_size{region=\"SCRUBBED\"}) by (region)","format":"time_series","interval":"1","intervalFactor":1,"intervalMs":1000,"legendFormat":"SCRUBBED","maxDataPoints":43200,"range":true,"refId":"SCRUBBED"}},{"refId":"SCRUBBED","queryType":"","relativeTimeRange":{"from":43200,"to":0},"datasourceUid":"prom-live","model":{"datasource":{"type":"prometheus","uid":"prom-live"},"exemplar":true,"expr":"sum(insights_consistency_checker_lock_migration_queue_size{region=\"SCRUBBED\"}) by (region)","format":"time_series","interval":"1","intervalFactor":1,"intervalMs":1000,"legendFormat":"SCRUBBED","maxDataPoints":43200,"refId":"SCRUBBED"}},{"refId":"SCRUBBED","queryType":"","relativeTimeRange":{"from":43200,"to":0},"datasourceUid":"prom-live","model":{"datasource":{"type":"prometheus","uid":"prom-live"},"editorMode":"code","exemplar":true,"expr":"sum(insights_consistency_checker_lock_migration_queue_size{region=\"SCRUBBED\"}) by (region)","format":"time_series","interval":"1","intervalFactor":1,"intervalMs":1000,"legendFormat":"SCRUBBED","maxDataPoints":43200,"range":true,"refId":"SCRUBBED"}},{"refId":"SCRUBBED","queryType":"","relativeTimeRange":{"from":43200,"to":0},"datasourceUid":"prom-live","model":{"datasource":{"type":"prometheus","uid":"prom-live"},"editorMode":"code","exemplar":true,"expr":"sum(insights_consistency_checker_lock_migration_queue_size{region=\"SCRUBBED\"}) by (region)","format":"time_series","interval":"1","intervalFactor":1,"intervalMs":1000,"legendFormat":"SCRUBBED","maxDataPoints":43200,"range":true,"refId":"SCRUBBED"}},{"refId":"SCRUBBED","queryType":"","relativeTimeRange":{"from":43200,"to":0},"datasourceUid":"prom-live","model":{"datasource":{"type":"prometheus","uid":"prom-live"},"editorMode":"code","exemplar":true,"expr":"sum(insights_consistency_checker_lock_migration_queue_size{region=\"SCRUBBED\"}) by (region)","format":"time_series","interval":"1","intervalFactor":1,"intervalMs":1000,"legendFormat":"SCRUBBED","maxDataPoints":43200,"range":true,"refId":"SCRUBBED"}}]|600 |KeepLast |KeepLast |3600000000000|{"__alertId__":"10309","__dashboardUid__":"WFyEThFWz","__panelId__":"60","description":"SCRUBBED","summary":"{{ range $k, $v := $values -}}\n{{ if (match \"A[0-9]+\" $k) -}}\n{{ $k }}: {{ $v.Value }} {{ $v.Labels }}{{ end }}\n{{ end }}"}|{"__alert_subscribers":"","__contacts__":"SCRUBBED","owner":"SCRUBBED","priority":"p3","rule_uid":"d4fb2f40-bd03-4fa5-a5f0-d3fd81ac8fde"}|0 |0 |null | |495844|49 |d4fb2f40-bd03-4fa5-a5f0-d3fd81ac8fde|bdstisda7lv5sd |company_actions_lock queue alert|2 |0 |3 |2024-07-25 15:39:33|company_actions_lock queue alert|A |[{"refId":"A","queryType":"","relativeTimeRange":{"from":0,"to":0},"datasourceUid":"__expr__","model":{"conditions":[{"evaluator":{"params":[12],"type":"gt"},"operator":{"type":"and"},"query":{"params":["SCRUBBED"]},"reducer":{"type":"min"}},{"evaluator":{"params":[12],"type":"gt"},"operator":{"type":"or"},"query":{"params":["SCRUBBED"]},"reducer":{"type":"min"}},{"evaluator":{"params":[12],"type":"gt"},"operator":{"type":"or"},"query":{"params":["SCRUBBED"]},"reducer":{"type":"min"}},{"evaluator":{"params":[12],"type":"gt"},"operator":{"type":"or"},"query":{"params":["SCRUBBED"]},"reducer":{"type":"avg"}},{"evaluator":{"params":[12],"type":"gt"},"operator":{"type":"or"},"query":{"params":["SCRUBBED"]},"reducer":{"type":"avg"}},{"evaluator":{"params":[12],"type":"gt"},"operator":{"type":"or"},"query":{"params":["SCRUBBED"]},"reducer":{"type":"avg"}},{"evaluator":{"params":[12],"type":"gt"},"operator":{"type":"or"},"query":{"params":["SCRUBBED"]},"reducer":{"type":"avg"}},{"evaluator":{"params":[12],"type":"gt"},"operator":{"type":"or"},"query":{"params":["SCRUBBED"]},"reducer":{"type":"avg"}}],"intervalMs":1000,"maxDataPoints":43200,"refId":"A","type":"classic_conditions"}},{"refId":"SCRUBBED","queryType":"","relativeTimeRange":{"from":43200,"to":0},"datasourceUid":"prom-live","model":{"datasource":{"type":"prometheus","uid":"prom-live"},"editorMode":"code","exemplar":true,"expr":"sum(insights_consistency_checker_lock_migration_queue_size{region=\"SCRUBBED\"}) by (region)","format":"time_series","interval":"1","intervalFactor":1,"intervalMs":1000,"legendFormat":"SCRUBBED","maxDataPoints":43200,"range":true,"refId":"SCRUBBED"}},{"refId":"SCRUBBED","queryType":"","relativeTimeRange":{"from":43200,"to":0},"datasourceUid":"prom-live","model":{"datasource":{"type":"prometheus","uid":"prom-live"},"editorMode":"code","exemplar":true,"expr":"sum(insights_consistency_checker_lock_migration_queue_size{region=\"SCRUBBED\"}) by (region)","format":"time_series","interval":"1","intervalFactor":1,"intervalMs":1000,"legendFormat":"SCRUBBED","maxDataPoints":43200,"range":true,"refId":"SCRUBBED"}},{"refId":"SCRUBBED","queryType":"","relativeTimeRange":{"from":43200,"to":0},"datasourceUid":"prom-live","model":{"datasource":{"type":"prometheus","uid":"prom-live"},"editorMode":"code","exemplar":true,"expr":"sum(insights_consistency_checker_lock_migration_queue_size{region=\"SCRUBBED\"}) by (region)","format":"time_series","interval":"1","intervalFactor":1,"intervalMs":1000,"legendFormat":"SCRUBBED","maxDataPoints":43200,"range":true,"refId":"SCRUBBED"}},{"refId":"SCRUBBED","queryType":"","relativeTimeRange":{"from":43200,"to":0},"datasourceUid":"prom-live","model":{"datasource":{"type":"prometheus","uid":"prom-live"},"editorMode":"code","exemplar":true,"expr":"sum(insights_consistency_checker_lock_migration_queue_size{region=\"SCRUBBED\"}) by (region)","format":"time_series","interval":"1","intervalFactor":1,"intervalMs":1000,"legendFormat":"SCRUBBED","maxDataPoints":43200,"range":true,"refId":"SCRUBBED"}},{"refId":"SCRUBBED","queryType":"","relativeTimeRange":{"from":43200,"to":0},"datasourceUid":"prom-live","model":{"datasource":{"type":"prometheus","uid":"prom-live"},"exemplar":true,"expr":"sum(insights_consistency_checker_lock_migration_queue_size{region=\"SCRUBBED\"}) by (region)","format":"time_series","interval":"1","intervalFactor":1,"intervalMs":1000,"legendFormat":"SCRUBBED","maxDataPoints":43200,"refId":"SCRUBBED"}},{"refId":"SCRUBBED","queryType":"","relativeTimeRange":{"from":43200,"to":0},"datasourceUid":"prom-live","model":{"datasource":{"type":"prometheus","uid":"prom-live"},"editorMode":"code","exemplar":true,"expr":"sum(insights_consistency_checker_lock_migration_queue_size{region=\"SCRUBBED\"}) by (region)","format":"time_series","interval":"1","intervalFactor":1,"intervalMs":1000,"legendFormat":"SCRUBBED","maxDataPoints":43200,"range":true,"refId":"SCRUBBED"}},{"refId":"SCRUBBED","queryType":"","relativeTimeRange":{"from":43200,"to":0},"datasourceUid":"prom-live","model":{"datasource":{"type":"prometheus","uid":"prom-live"},"editorMode":"code","exemplar":true,"expr":"sum(insights_consistency_checker_lock_migration_queue_size{region=\"SCRUBBED\"}) by (region)","format":"time_series","interval":"1","intervalFactor":1,"intervalMs":1000,"legendFormat":"SCRUBBED","maxDataPoints":43200,"range":true,"refId":"SCRUBBED"}},{"refId":"SCRUBBED","queryType":"","relativeTimeRange":{"from":43200,"to":0},"datasourceUid":"prom-live","model":{"datasource":{"type":"prometheus","uid":"prom-live"},"editorMode":"code","exemplar":true,"expr":"sum(insights_consistency_checker_lock_migration_queue_size{region=\"SCRUBBED\"}) by (region)","format":"time_series","interval":"1","intervalFactor":1,"intervalMs":1000,"legendFormat":"SCRUBBED","maxDataPoints":43200,"range":true,"refId":"SCRUBBED"}}]|600 |KeepLast |KeepLast |3600000000000|{"__alertId__":"10309","__dashboardUid__":"WFyEThFWz","__panelId__":"60","description":"SCRUBBED","summary":"{{ range $k, $v := $values -}}\n{{ if (match \"A[0-9]+\" $k) -}}\n{{ $k }}: {{ $v.Value }} {{ $v.Labels }}{{ end }}\n{{ end }}"}|{"__alert_subscribers":"","__contacts__":"SCRUBBED","owner":"SCRUBBED","priority":"p3","rule_uid":"d4fb2f40-bd03-4fa5-a5f0-d3fd81ac8fde"}|0 |0 |null | |499075|49 |d4fb2f40-bd03-4fa5-a5f0-d3fd81ac8fde|bdstisda7lv5sd |company_actions_lock queue alert|3 |0 |4 |2024-07-25 16:07:52|company_actions_lock queue alert|A |[{"refId":"A","queryType":"","relativeTimeRange":{"from":0,"to":0},"datasourceUid":"__expr__","model":{"conditions":[{"evaluator":{"params":[12],"type":"gt"},"operator":{"type":"and"},"query":{"params":["SCRUBBED"]},"reducer":{"type":"min"}},{"evaluator":{"params":[12],"type":"gt"},"operator":{"type":"or"},"query":{"params":["SCRUBBED"]},"reducer":{"type":"min"}},{"evaluator":{"params":[12],"type":"gt"},"operator":{"type":"or"},"query":{"params":["SCRUBBED"]},"reducer":{"type":"min"}},{"evaluator":{"params":[12],"type":"gt"},"operator":{"type":"or"},"query":{"params":["SCRUBBED"]},"reducer":{"type":"avg"}},{"evaluator":{"params":[12],"type":"gt"},"operator":{"type":"or"},"query":{"params":["SCRUBBED"]},"reducer":{"type":"avg"}},{"evaluator":{"params":[12],"type":"gt"},"operator":{"type":"or"},"query":{"params":["SCRUBBED"]},"reducer":{"type":"avg"}},{"evaluator":{"params":[12],"type":"gt"},"operator":{"type":"or"},"query":{"params":["SCRUBBED"]},"reducer":{"type":"avg"}},{"evaluator":{"params":[12],"type":"gt"},"operator":{"type":"or"},"query":{"params":["SCRUBBED"]},"reducer":{"type":"avg"}}],"intervalMs":1000,"maxDataPoints":43200,"refId":"A","type":"classic_conditions"}},{"refId":"SCRUBBED","queryType":"","relativeTimeRange":{"from":43200,"to":0},"datasourceUid":"prom-live","model":{"datasource":{"type":"prometheus","uid":"prom-live"},"editorMode":"code","exemplar":true,"expr":"sum(insights_consistency_checker_lock_migration_queue_size{region=\"SCRUBBED\"}) by (region)","format":"time_series","interval":"1","intervalFactor":1,"intervalMs":1000,"legendFormat":"SCRUBBED","maxDataPoints":43200,"range":true,"refId":"SCRUBBED"}},{"refId":"SCRUBBED","queryType":"","relativeTimeRange":{"from":43200,"to":0},"datasourceUid":"prom-live","model":{"datasource":{"type":"prometheus","uid":"prom-live"},"editorMode":"code","exemplar":true,"expr":"sum(insights_consistency_checker_lock_migration_queue_size{region=\"SCRUBBED\"}) by (region)","format":"time_series","interval":"1","intervalFactor":1,"intervalMs":1000,"legendFormat":"SCRUBBED","maxDataPoints":43200,"range":true,"refId":"SCRUBBED"}},{"refId":"SCRUBBED","queryType":"","relativeTimeRange":{"from":43200,"to":0},"datasourceUid":"prom-live","model":{"datasource":{"type":"prometheus","uid":"prom-live"},"editorMode":"code","exemplar":true,"expr":"sum(insights_consistency_checker_lock_migration_queue_size{region=\"SCRUBBED\"}) by (region)","format":"time_series","interval":"1","intervalFactor":1,"intervalMs":1000,"legendFormat":"SCRUBBED","maxDataPoints":43200,"range":true,"refId":"SCRUBBED"}},{"refId":"SCRUBBED","queryType":"","relativeTimeRange":{"from":43200,"to":0},"datasourceUid":"prom-live","model":{"datasource":{"type":"prometheus","uid":"prom-live"},"editorMode":"code","exemplar":true,"expr":"sum(insights_consistency_checker_lock_migration_queue_size{region=\"SCRUBBED\"}) by (region)","format":"time_series","interval":"1","intervalFactor":1,"intervalMs":1000,"legendFormat":"SCRUBBED","maxDataPoints":43200,"range":true,"refId":"SCRUBBED"}},{"refId":"SCRUBBED","queryType":"","relativeTimeRange":{"from":43200,"to":0},"datasourceUid":"prom-live","model":{"datasource":{"type":"prometheus","uid":"prom-live"},"exemplar":true,"expr":"sum(insights_consistency_checker_lock_migration_queue_size{region=\"SCRUBBED\"}) by (region)","format":"time_series","interval":"1","intervalFactor":1,"intervalMs":1000,"legendFormat":"SCRUBBED","maxDataPoints":43200,"refId":"SCRUBBED"}},{"refId":"SCRUBBED","queryType":"","relativeTimeRange":{"from":43200,"to":0},"datasourceUid":"prom-live","model":{"datasource":{"type":"prometheus","uid":"prom-live"},"editorMode":"code","exemplar":true,"expr":"sum(insights_consistency_checker_lock_migration_queue_size{region=\"SCRUBBED\"}) by (region)","format":"time_series","interval":"1","intervalFactor":1,"intervalMs":1000,"legendFormat":"SCRUBBED","maxDataPoints":43200,"range":true,"refId":"SCRUBBED"}},{"refId":"SCRUBBED","queryType":"","relativeTimeRange":{"from":43200,"to":0},"datasourceUid":"prom-live","model":{"datasource":{"type":"prometheus","uid":"prom-live"},"editorMode":"code","exemplar":true,"expr":"sum(insights_consistency_checker_lock_migration_queue_size{region=\"SCRUBBED\"}) by (region)","format":"time_series","interval":"1","intervalFactor":1,"intervalMs":1000,"legendFormat":"SCRUBBED","maxDataPoints":43200,"range":true,"refId":"SCRUBBED"}},{"refId":"SCRUBBED","queryType":"","relativeTimeRange":{"from":43200,"to":0},"datasourceUid":"prom-live","model":{"datasource":{"type":"prometheus","uid":"prom-live"},"editorMode":"code","exemplar":true,"expr":"sum(insights_consistency_checker_lock_migration_queue_size{region=\"SCRUBBED\"}) by (region)","format":"time_series","interval":"1","intervalFactor":1,"intervalMs":1000,"legendFormat":"SCRUBBED","maxDataPoints":43200,"range":true,"refId":"SCRUBBED"}}]|600 |KeepLast |KeepLast |3600000000000|{"__alertId__":"10309","__dashboardUid__":"WFyEThFWz","__panelId__":"60","description":"SCRUBBED","summary":"{{ range $k, $v := $values -}}\n{{ if (match \"A[0-9]+\" $k) -}}\n{{ $k }}: {{ $v.Value }} {{ $v.Labels }}{{ end }}\n{{ end }}"}|{"__alert_subscribers":"","__contacts__":"SCRUBBED","owner":"SCRUBBED","priority":"p3","rule_uid":"d4fb2f40-bd03-4fa5-a5f0-d3fd81ac8fde"}|0 |0 |null | |502303|49 |d4fb2f40-bd03-4fa5-a5f0-d3fd81ac8fde|bdstisda7lv5sd |company_actions_lock queue alert|4 |0 |5 |2024-07-25 16:09:30|company_actions_lock queue alert|A |[{"refId":"A","queryType":"","relativeTimeRange":{"from":0,"to":0},"datasourceUid":"__expr__","model":{"conditions":[{"evaluator":{"params":[12],"type":"gt"},"operator":{"type":"and"},"query":{"params":["SCRUBBED"]},"reducer":{"type":"min"}},{"evaluator":{"params":[12],"type":"gt"},"operator":{"type":"or"},"query":{"params":["SCRUBBED"]},"reducer":{"type":"min"}},{"evaluator":{"params":[12],"type":"gt"},"operator":{"type":"or"},"query":{"params":["SCRUBBED"]},"reducer":{"type":"min"}},{"evaluator":{"params":[12],"type":"gt"},"operator":{"type":"or"},"query":{"params":["SCRUBBED"]},"reducer":{"type":"avg"}},{"evaluator":{"params":[12],"type":"gt"},"operator":{"type":"or"},"query":{"params":["SCRUBBED"]},"reducer":{"type":"avg"}},{"evaluator":{"params":[12],"type":"gt"},"operator":{"type":"or"},"query":{"params":["SCRUBBED"]},"reducer":{"type":"avg"}},{"evaluator":{"params":[12],"type":"gt"},"operator":{"type":"or"},"query":{"params":["SCRUBBED"]},"reducer":{"type":"avg"}},{"evaluator":{"params":[12],"type":"gt"},"operator":{"type":"or"},"query":{"params":["SCRUBBED"]},"reducer":{"type":"avg"}}],"intervalMs":1000,"maxDataPoints":43200,"refId":"A","type":"classic_conditions"}},{"refId":"SCRUBBED","queryType":"","relativeTimeRange":{"from":43200,"to":0},"datasourceUid":"prom-live","model":{"datasource":{"type":"prometheus","uid":"prom-live"},"editorMode":"code","exemplar":true,"expr":"sum(insights_consistency_checker_lock_migration_queue_size{region=\"SCRUBBED\"}) by (region)","format":"time_series","interval":"1","intervalFactor":1,"intervalMs":1000,"legendFormat":"SCRUBBED","maxDataPoints":43200,"range":true,"refId":"SCRUBBED"}},{"refId":"SCRUBBED","queryType":"","relativeTimeRange":{"from":43200,"to":0},"datasourceUid":"prom-live","model":{"datasource":{"type":"prometheus","uid":"prom-live"},"editorMode":"code","exemplar":true,"expr":"sum(insights_consistency_checker_lock_migration_queue_size{region=\"SCRUBBED\"}) by (region)","format":"time_series","interval":"1","intervalFactor":1,"intervalMs":1000,"legendFormat":"SCRUBBED","maxDataPoints":43200,"range":true,"refId":"SCRUBBED"}},{"refId":"SCRUBBED","queryType":"","relativeTimeRange":{"from":43200,"to":0},"datasourceUid":"prom-live","model":{"datasource":{"type":"prometheus","uid":"prom-live"},"editorMode":"code","exemplar":true,"expr":"sum(insights_consistency_checker_lock_migration_queue_size{region=\"SCRUBBED\"}) by (region)","format":"time_series","interval":"1","intervalFactor":1,"intervalMs":1000,"legendFormat":"SCRUBBED","maxDataPoints":43200,"range":true,"refId":"SCRUBBED"}},{"refId":"SCRUBBED","queryType":"","relativeTimeRange":{"from":43200,"to":0},"datasourceUid":"prom-live","model":{"datasource":{"type":"prometheus","uid":"prom-live"},"editorMode":"code","exemplar":true,"expr":"sum(insights_consistency_checker_lock_migration_queue_size{region=\"SCRUBBED\"}) by (region)","format":"time_series","interval":"1","intervalFactor":1,"intervalMs":1000,"legendFormat":"SCRUBBED","maxDataPoints":43200,"range":true,"refId":"SCRUBBED"}},{"refId":"SCRUBBED","queryType":"","relativeTimeRange":{"from":43200,"to":0},"datasourceUid":"prom-live","model":{"datasource":{"type":"prometheus","uid":"prom-live"},"exemplar":true,"expr":"sum(insights_consistency_checker_lock_migration_queue_size{region=\"SCRUBBED\"}) by (region)","format":"time_series","interval":"1","intervalFactor":1,"intervalMs":1000,"legendFormat":"SCRUBBED","maxDataPoints":43200,"refId":"SCRUBBED"}},{"refId":"SCRUBBED","queryType":"","relativeTimeRange":{"from":43200,"to":0},"datasourceUid":"prom-live","model":{"datasource":{"type":"prometheus","uid":"prom-live"},"editorMode":"code","exemplar":true,"expr":"sum(insights_consistency_checker_lock_migration_queue_size{region=\"SCRUBBED\"}) by (region)","format":"time_series","interval":"1","intervalFactor":1,"intervalMs":1000,"legendFormat":"SCRUBBED","maxDataPoints":43200,"range":true,"refId":"SCRUBBED"}},{"refId":"SCRUBBED","queryType":"","relativeTimeRange":{"from":43200,"to":0},"datasourceUid":"prom-live","model":{"datasource":{"type":"prometheus","uid":"prom-live"},"editorMode":"code","exemplar":true,"expr":"sum(insights_consistency_checker_lock_migration_queue_size{region=\"SCRUBBED\"}) by (region)","format":"time_series","interval":"1","intervalFactor":1,"intervalMs":1000,"legendFormat":"SCRUBBED","maxDataPoints":43200,"range":true,"refId":"SCRUBBED"}},{"refId":"SCRUBBED","queryType":"","relativeTimeRange":{"from":43200,"to":0},"datasourceUid":"prom-live","model":{"datasource":{"type":"prometheus","uid":"prom-live"},"editorMode":"code","exemplar":true,"expr":"sum(insights_consistency_checker_lock_migration_queue_size{region=\"SCRUBBED\"}) by (region)","format":"time_series","interval":"1","intervalFactor":1,"intervalMs":1000,"legendFormat":"SCRUBBED","maxDataPoints":43200,"range":true,"refId":"SCRUBBED"}}]|600 |KeepLast |KeepLast |3600000000000|{"__alertId__":"10309","__dashboardUid__":"WFyEThFWz","__panelId__":"60","description":"SCRUBBED","summary":"{{ range $k, $v := $values -}}\n{{ if (match \"A[0-9]+\" $k) -}}\n{{ $k }}: {{ $v.Value }} {{ $v.Labels }}{{ end }}\n{{ end }}"}|{"__alert_subscribers":"","__contacts__":"SCRUBBED","owner":"SCRUBBED","priority":"p3","rule_uid":"d4fb2f40-bd03-4fa5-a5f0-d3fd81ac8fde"}|0 |0 |null |

What did you expect to happen?

I expect that only modified alerts change the state.

Did this work before?

unknown, according to this issue - yes

How do we reproduce it?

  1. Create always firing/pending alert
  2. Provision it with alert rule provisioning file
  3. Make sure that alert appeared in grafana with provisioned badge
  4. Wait until it switches to firing/pending state
  5. Restart grafana

Is the bug inside a dashboard panel?

no

Environment (with versions)?

Grafana: 10.4.1 , 10.4.8 , 11.2.0 OS: ubuntu Browser: irrelevant

Grafana platform?

Kubernetes

Datasource(s)?

No response

yuri-tceretian commented 1 month ago

I think I know what's going on. In Grafana we do some optimizations of Prometheus queries, we may change "range" queries to "instant" if the query result does not change. When we do that we deserialize the JSON into a hashmap, fix the query, and serialize it back. During that process, the fields' order likely does not match the original order (because the hashmap does not store items in the original order). That can cause the hash of the query to change, which changes the fingerprint of the rule.

To confirm my theory, try to set the query for the problematic rule to instant, or add some extra reduce expression to the rule query. Either of those options would make Grafana skip the optimization.