Slack Notifications Include Stale Grouped Alerts

[jnadler]

What did you do? Running 11 node exporters, 3 prometheus instances scraping all 11, and 1 alertmanger all locally in Docker using the latest published images.

Trigger a grouped alert by taking down 7 node exporters, wait for the Slack alert to arrive, resolve some of the members of the group.

What did you expect to see? After group_interval has passed, another Slack alert with the updated, smaller set of grouped alerts.

What did you see instead? Under which circumstances? After group_interval, another Slack alert including all grouped alerts, even the resolved ones. The AlertManager UI shows the correct (smaller, now that some are resolved) set of grouped alerts.

This behavior is easily reproducible but not 100% consistent - possibly racy. The Slack alert accumulates group members when new alerts are added to the group (they are added reliably) but alerts rarely leave the group (the AM UI is always correct - just the Slack alert retains the resolved alerts).

While building a simple repro environment for this issue I think I did occasionally see an alert removed from the Slack alert list, but it's certainly more common for it to be retained in the Slack message.

Environment

• System information:

  Darwin 18.6.0 x86_64

• Alertmanager version:

  0.18.0

• Prometheus version:

  2.11.1

• Alertmanager configuration file:

  
  Alertmanager
  Alerts
  Silences
  Status
  Help
  Status
  Uptime:
  2019-08-02T00:33:45.262Z
  Cluster Status
  Name:
  01DH7VYBSBYDK910FBTQ890H76
  Status:
  ready
  Peers:
  Name: 01DH7VYBSBYDK910FBTQ890H76
  Address: 172.17.0.16:8001
  Version Information
  Branch:
  HEAD
  BuildDate:
  20190708-14:31:49
  BuildUser:
  root@868685ed3ed0
  GoVersion:
  go1.12.6
  Revision:
  1ace0f76b7101cccc149d7298022df36039858ca
  Version:
  0.18.0
  Config
  global:
  resolve_timeout: 5m
  http_config: {}
  smtp_hello: localhost
  smtp_require_tls: true
  slack_api_url: <secret>
  pagerduty_url: https://events.pagerduty.com/v2/enqueue
  hipchat_api_url: https://api.hipchat.com/
  opsgenie_api_url: https://api.opsgenie.com/
  wechat_api_url: https://qyapi.weixin.qq.com/cgi-bin/
  victorops_api_url: https://alert.victorops.com/integrations/generic/20131114/alert/
  route:
  receiver: pandora-alerts
  group_by:
  - alertname
  - service
  routes:
  - receiver: pandora-alerts
  group_by:
  - alertname
  - service
  - cluster
  match:
    from_docc: "true"
  continue: true
  - match:
    team: eng-observability
  routes:
  - receiver: eng-observability-debug
    match:
      severity: debug
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 3h
  receivers:

• name: pandora-alerts slack_configs:

  • send_resolved: true http_config: {} api_url: channel: '#jefftest' username: '{{ template "slack.default.username" . }}' color: '{{ if eq .Status "firing" }}danger{{ else }}good{{ end }}' title: '{{ .GroupLabels.alertname }}' title_link: '{{ template "slack.default.titlelink" . }}' pretext: '{{ template "slack.default.pretext" . }}' text: |- status: {{ .Status }} severity: {{ .CommonLabels.severity }} team: {{ .CommonLabels.team }} source: {{ range .Alerts }}<{{ .GeneratorURL }}|Source>{{ end }} description: {{ range .Alerts }}{{ .Annotations.description }} {{ end }} footer: '{{ template "slack.default.footer" . }}' fallback: '{{ template "slack.default.fallback" . }}' callback_id: '{{ template "slack.default.callbackid" . }}' icon_emoji: '{{ template "slack.default.iconemoji" . }}' icon_url: '{{ template "slack.default.iconurl" . }}'

• name: eng-observability-debug slack_configs:

  • send_resolved: true http_config: {} api_url: channel: '#observability-toilet' username: '{{ template "slack.default.username" . }}' color: '{{ if eq .Status "firing" }}danger{{ else }}good{{ end }}' title: '{{ .GroupLabels.alertname }}' title_link: '{{ template "slack.default.titlelink" . }}' pretext: '{{ template "slack.default.pretext" . }}' text: |- status: {{ .Status }} | severity: {{ .CommonLabels.severity }} | source: {{ range .Alerts }}<{{ .GeneratorURL }}|Prometheus> {{ end }}{{ if .CommonAnnotations.playbook }}| playbook: <{{ .CommonAnnotations.playbook }}|Playbook>{{ end }}{{- if .CommonAnnotations.description }} description: {{ .CommonAnnotations.description }}{{ else if gt (len .Alerts) 0}}{{ if index (index .Alerts 0).Annotations "description" }} description: {{ range .Alerts }}{{ .Annotations.description }} {{ end }}{{ end }}{{ end }}{{ if gt (len .Alerts) 0 }}{{ if index (index .Alerts 0).Annotations "summary" }} summary: {{ end }}{{ end }}{{ range .Alerts }}{{ .Annotations.summary }} {{ end }} footer: '{{ template "slack.default.footer" . }}' fallback: '{{ template "slack.default.fallback" . }}' callback_id: '{{ template "slack.default.callbackid" . }}' icon_emoji: '{{ template "slack.default.iconemoji" . }}' icon_url: '{{ template "slack.default.iconurl" . }}' templates: []

• Prometheus configuration file:

  global:
  scrape_interval: 1m
  scrape_timeout: 1m
  evaluation_interval: 1m
  external_labels:
  replica: prom-1
  alerting:
  alert_relabel_configs:
  - separator: ;
    regex: replica
    replacement: $1
    action: labeldrop
  alertmanagers:
  - static_configs:
      - targets:
          - docker.for.mac.localhost:9193
          - docker.for.mac.localhost:9194
          - docker.for.mac.localhost:9195
    scheme: http
    timeout: 10s
  rule_files:
  - /etc/prometheus/alerts.yml
  scrape_configs:
  - job_name: node-exporter-1
  static_configs:
    - targets: ['docker.for.mac.localhost:9101']
  - job_name: node-exporter-2
  static_configs:
    - targets: ['docker.for.mac.localhost:9102']
  - job_name: node-exporter-3
  static_configs:
    - targets: ['docker.for.mac.localhost:9103']
  - job_name: node-exporter-4
  static_configs:
    - targets: ['docker.for.mac.localhost:9104']
  - job_name: node-exporter-5
  static_configs:
    - targets: ['docker.for.mac.localhost:9105']
  - job_name: node-exporter-6
  static_configs:
    - targets: ['docker.for.mac.localhost:9106']
  - job_name: node-exporter-7
  static_configs:
    - targets: ['docker.for.mac.localhost:9107']
  - job_name: node-exporter-8
  static_configs:
    - targets: ['docker.for.mac.localhost:9108']
  - job_name: node-exporter-9
  static_configs:
    - targets: ['docker.for.mac.localhost:9109']
  - job_name: node-exporter-10
  static_configs:
    - targets: ['docker.for.mac.localhost:9110']
  - job_name: node-exporter-11
  static_configs:
    - targets: ['docker.for.mac.localhost:9111']

alerts.yml

groups:
  - name: example
    rules:
      - alert: NodeIsDown
        expr: up == 0
        for: 2m
        labels:
          severity: debug
          team: eng-observability
        annotations:
          summary: node {{ $labels.instance }}

[simonpasquier]

The notification data includes both firing and resolved alerts. If you want the Slack message to only display the firing ones, you could do: {{ range .Alerts.Firing }}...{{ end }}

[jnadler]

Wow, thanks! I studied the docs before filing this issue and couldn't find this. Might it be helpful if it were doc'd here? https://prometheus.io/docs/alerting/notifications/

[simonpasquier]

You're right. The source is at https://github.com/prometheus/docs/blob/master/content/docs/alerting/notifications.md

[jnadler]

How's this https://github.com/prometheus/docs/pull/1411

[jnadler]

I've confirmed (with 3 HA AlertManagers) that intermittently some alerts do disappear from the .Alerts collection, and thus from Slack alerts based on it.

[simonpasquier]

I suppose that the alert for docker.for.mac.localhost:9111 got resolved? In this case, AlertManager sees that the alert group has changed and it will trigger a new notification at the next group evaluation.

[jnadler]

I'm not getting that result consistently. The substantial majority of the time, when the alert for 9111 gets resolved, at the next group evaluation that alert is still in the .Alerts list.

Intermittently I'm able to trigger behavior where the alert for 9111 is resolved and on the next grouping it's removed from the .Alerts list, as in the example above. I don't have a strong feeling as to what the behavior should be, but it should be reliable and consistent.

[simonpasquier]

Ah I thought that your last screenshot displayed only the firing alerts (eg .Alerts.Firing) but IIUC it still uses .Alerts.

[jnadler]

That's correct, still using .Alerts. I can probably document how to reproduce this if that's helpful - it's just a bunch of scripts that start docker containers. Not a super high priority now that I know about .Alerts.Firing.