Alerts evaluated 2 times and sometimes sending 2 notifications

Azure / azure-managed-grafana

Azure Managed Grafana samples, tips and references.

https://aka.ms/managed-grafana

MIT License

7 stars 4 forks source link

Alerts evaluated 2 times and sometimes sending 2 notifications #13

Open emperorente opened 5 months ago

emperorente commented 5 months ago

Hi team:

We're using managed grafana with version 9.5.17 and Standard plan.

Sometimes we had received the alert twice. I queried the official document and found that when it is configured with HA Mode, it will send the alert with the number of Alertmanager instances. But it doesn't always send twice; occasionally, it sends twice

The Alert rule search window is "now-15m". Rule group evaluation interval is set to "1m"

Could you help to check it?

brahmnes commented 5 months ago

Hi @emperorente, yes, we leverage the Grafana's usage of the gossip protocol for the alert deduplication https://grafana.com/docs/grafana/latest/alerting/set-up/configure-high-availability/#configure-high-availability

How often do you see the alert getting sent twice?

emperorente commented 5 months ago

Hi @brahmnes, about once a week