search

prometheus / alertmanager

Prometheus Alertmanager
https://prometheus.io
Apache License 2.0
6.66k stars 2.16k forks source link

Not sending resolved on webhook #2784

Open jividijuvent opened 2 years ago

jividijuvent commented 2 years ago

Hi, I have a problem sending resolved messages, I am using webhook, which by default will send resolve, but I tried adding the option, downgrading, but nothing happened, I have logging on my API, which only shows that receives requests on firing, not on resolved. Here is my alert manager configuration:

global:
  resolve_timeout: 5m

route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  receiver: 'webhook'
receivers:
- name: 'webhook'
  webhook_configs:
  - url: 'http://192.168.201.55:8080'
    send_resolved: true

Alretmanager Versions tested: 0.22.1 0.22.2 0.23.0

Prometheus Version: 2.31.1

And here is debug logs for Alertmanager, firing and resolving:

2021-12-08T14:29:11.844477200Z level=debug ts=2021-12-08T14:29:11.843Z caller=dispatch.go:138 component=dispatcher msg="Received alert" alert="Haproxy Backend Failed[a9ecff1][active]"
2021-12-08T14:29:21.844941396Z level=debug ts=2021-12-08T14:29:21.844Z caller=dispatch.go:475 component=dispatcher aggrGroup="{}:{alertname=\"Haproxy Backend Failed\"}" msg=flushing alerts="[Haproxy Backend Failed[a9ecff1][active]]"
2021-12-08T14:29:22.335699730Z level=debug ts=2021-12-08T14:29:22.335Z caller=notify.go:734 component=dispatcher receiver=webhook integration=webhook[0] msg="Notify success" attempts=1
2021-12-08T14:29:51.847121525Z level=debug ts=2021-12-08T14:29:51.846Z caller=dispatch.go:475 component=dispatcher aggrGroup="{}:{alertname=\"Haproxy Backend Failed\"}" msg=flushing alerts="[Haproxy Backend Failed[a9ecff1][active]]"
2021-12-08T14:30:01.847217212Z level=debug ts=2021-12-08T14:30:01.846Z caller=dispatch.go:475 component=dispatcher aggrGroup="{}:{alertname=\"Haproxy Backend Failed\"}" msg=flushing alerts="[Haproxy Backend Failed[a9ecff1][active]]"

2021-12-08T14:30:11.844174601Z level=debug ts=2021-12-08T14:30:11.842Z caller=dispatch.go:138 component=dispatcher msg="Received alert" alert="Haproxy Backend Failed[a9ecff1][resolved]"
2021-12-08T14:30:11.850302323Z level=debug ts=2021-12-08T14:30:11.847Z caller=dispatch.go:475 component=dispatcher aggrGroup="{}:{alertname=\"Haproxy Backend Failed\"}" msg=flushing alerts="[Haproxy Backend Failed[a9ecff1][resolved]]"
2021-12-08T14:30:11.851919082Z level=debug ts=2021-12-08T14:30:11.851Z caller=notify.go:734 component=dispatcher receiver=webhook integration=webhook[0] msg="Notify success" attempts=1
jividijuvent commented 2 years ago

I think I found something when I disable grouping, resolved notify will not fire, just a log with debug level. with this configuration, no resolved notify:

  group_by: ['...']
  group_wait: 10s
  group_interval: 10s

level=debug ts=2021-12-13T12:00:26.859Z caller=dispatch.go:165 component=dispatcher msg="Received alert" alert=HaproxyHasNoAliveBackends[45806bb][resolved]
level=debug ts=2021-12-13T12:00:26.859Z caller=dispatch.go:165 component=dispatcher msg="Received alert" alert=HaproxyHasNoAliveBackends[8877eb7][resolved]

but when using groups, resolved notify works fine:

level=debug ts=2021-12-13T12:06:26.855Z caller=dispatch.go:165 component=dispatcher msg="Received alert" alert=HaproxyHasNoAliveBackends[45806bb][resolved]
level=debug ts=2021-12-13T12:06:26.855Z caller=dispatch.go:165 component=dispatcher msg="Received alert" alert=HaproxyHasNoAliveBackends[8877eb7][resolved]
level=debug ts=2021-12-13T12:06:33.795Z caller=notify.go:735 component=dispatcher receiver=webhook integration=webhook[0] msg="Notify success" attempts=1
level=debug ts=2021-12-13T12:06:33.797Z caller=notify.go:735 component=dispatcher receiver=webhook integration=webhook[0] msg="Notify success" attempts=1