After setting up alerts within Grafana, I found that if the Prometheus instance was temporarily unavailable, this would cause the alert to fail permanently with:
description = Error: Could not find datasource Data source not found
This would re-trigger until I went into Grafana, and manually re-tested the rule and saved the alert (with no actual changes to the alert, or data sources).
This issue suggests the cause is specifying deleteDatasources which has the same name as datasource defined after it. I tried this in my deployment, then killed prometheus: I witnessed the alerts re-enter pending and then clear automatically with no other side-effects (e.g. duplication of the data source within Grafana).
I'm not aware of why this entry exists, so I have created this quick PR to remove it. If it's needed, please let me know and I'll look at other workarounds.
After setting up alerts within Grafana, I found that if the Prometheus instance was temporarily unavailable, this would cause the alert to fail permanently with:
This would re-trigger until I went into Grafana, and manually re-tested the rule and saved the alert (with no actual changes to the alert, or data sources).
This issue suggests the cause is specifying
deleteDatasourceswhich has the same name as datasource defined after it. I tried this in my deployment, then killed prometheus: I witnessed the alerts re-enter pending and then clear automatically with no other side-effects (e.g. duplication of the data source within Grafana).I'm not aware of why this entry exists, so I have created this quick PR to remove it. If it's needed, please let me know and I'll look at other workarounds.