PagerDuty alerts are stuck if I mute the alert after it's firing but before it's resolved

[freak12techno]

What did you do?

1. Have an Alertmanager setup that has PagerDuty as a receiver
2. Do something that triggers an alert that is routed to PagerDuty receiver
3. Mute an alert
4. Fix it so it'd get resolved
5. Alertmanager doesn't send a notification on a resolved alert as it's muted
6. PagerDuty doesn't know it's fixed so it's stuck there unless I manually go to PagerDuty app and resolve it there
7. Mute expires, so I end up with a case when the firing is no more firing, but PagerDuty thinks it's still firing and bugs me until I resolve it via PagerDuty app

The procedure I describe above is something I do constantly (as in, something breaks, I mute it so it won't annoy me with notifications as I know it's broken and I am fixing it, it's fixed before the mute is expired, and the alert is stuck in PagerDuty). For Telegram it seems ok, as nothing wrong will happen if I won't receive a resolve notification, but for PagerDuty (and I think for other similar services like OpsGenie, but I cannot verify that as I do not use it) it might be annoying as they might have their own set of alerting rules, like pinging a person every hour or sending them a push notification/SMS/call once the alert is firing.

What did you expect to see? Alert is resolved in PagerDuty.

What did you see instead? Under which circumstances? Alert is not resolved and is stuck, although is fixed.

Environment

• System information: Linux 5.15.0-91-generic x86_64

• Alertmanager version:

alertmanager, version 0.26.0 (branch: HEAD, revision: d7b4f0c7322e7151d6e3b1e31cbc15361e295d8d)
  build user:       root@df8d7debeef4
  build date:       20230824-11:11:58
  go version:       go1.20.7
  platform:         linux/amd64
  tags:             netgo

• Prometheus version:

Pretty sure it's not relevant, but:

prometheus, version 2.48.1 (branch: HEAD, revision: 63894216648f0d6be310c9d16fb48293c45c9310)
  build user:       root@71f108ff5632
  build date:       20231208-23:33:22
  go version:       go1.21.5
  platform:         linux/amd64
  tags:             netgo,builtinassets,stringlabels

• Alertmanager configuration file: Something similar to this (I removed some parts that are not relevant, like inhibit rules):

  
  global:
  pagerduty_url: 'https://events.eu.pagerduty.com'

route: receiver: 'telegram' group_wait: 10s group_by: ['host', 'hosting', 'datacenter', 'alertname'] repeat_interval: 1h

routes:

• receiver: 'telegram' match_re: severity: critical|warning continue: true

• receiver: 'pagerduty' group_by: ['...'] match_re: severity: critical continue: true

receivers:

• name: 'telegram' telegram_configs:

  • send_resolved: true message: '{{ template "telegram.text" . }}' bot_token: 'xxxx' chat_id: 123

• name: 'pagerduty' pagerduty_configs:

  • service_key: xxxx

templates:

• /etc/alertmanager/templates/template.tmpl

* Prometheus configuration file:

insert configuration here (if relevant to the issue)

* Logs:

insert Prometheus and Alertmanager logs relevant to the issue here

[TheMeier]

In the provided config there is no send_resolved: true for the pagerduty receiver

[freak12techno]

@TheMeier yes, but it's true by default, according to https://prometheus.io/docs/alerting/latest/configuration/#pagerduty_config Additionally, I have most of the alerts resolving itself fine unless they are muted.

[grobinson-grafana]

This is an unfortunate side effect of how the Alertmanager works. This is a frequent complaint and fixing it requires a lot of effort.

The issue is that the Alertmanager will delete the alert from the group if:

1. The alert is resolved.
2. A notification for the group was just sent (even if the alert was not in that notification because it was muted).

If this happens while a mute time is still active the resolved alert is deleted. That means when the mute time ends it doesn't even know that this alert existed – for example – it could have been deleted 6 hours ago.

A possible fix for this is to read the nflog when checking if an alert can be deleted – however it does not work if the Alertmanager is restarted because alerts are kept in memory and not persisted to disk.