PagerDuty configuration: unable to set severity field using template

Shanti-G commented 2 years ago

What did you do? I am trying to set the severity using templated label with below pagerduty configuration in alertmanager:

  pagerduty_configs:
  - send_resolved: true
    routing_key: xxx
    description: |-
      test-cluster.{{ range .Alerts }}{{ .Annotations.summary }}.
      {{ end }}
    severity: '{{ .Labels.pdseverity }}'

I am setting .Labels.pdseverity on an alert in Prometheus.

What did you expect to see? The alertmanager should be able to trigger a PD alert with its severity set based on the templated label.

What did you see instead? Under which circumstances? PagerDuty v2 API allows severity of alert to be set, default being error. With the above configuration, though an alert is in firing state in Prometheus, I don't get the triggered alert in PD.

However, when I directly set the value, like below, I get to see the incident in PD with the given severity, i.e. warning:

  pagerduty_configs:
  - send_resolved: true
    routing_key: xxx
    description: |-
      test-cluster.{{ range .Alerts }}{{ .Annotations.summary }}.
      {{ end }}
    severity: 'warning'

Would like to know how to set the severity field with a templated value. Thanks in advance.

Environment

Alertmanager version: alertmanager, version 0.21.0 (branch: HEAD, revision: 4c6c03ebfe21009c546e4d1e9b92c371d67c021d) build user: root@dee35927357f build date: 20200617-08:54:02 go version: go1.14.4
Prometheus version: prometheus, version 2.27.1 (branch: HEAD, revision: db7f0bcec27bd8aeebad6b08ac849516efa9ae02) build user: root@fd804fbd4f25 build date: 20210518-14:17:54 go version: go1.16.4 platform: linux/amd64
Logs: No error logs from alertmanager to indicate configuration error.

sbrass commented 2 years ago

You made the mistake of using .Labels, which is part of the Alert structure.

You may want to use .CommonLabels at that point? Check the documentation https://prometheus.io/docs/alerting/latest/notifications/.

Shanti-G commented 2 years ago

Thanks @sbrass for getting back on this. We have the pdseverity defined for each alert at Alert level. Would you be having any suggestion on getting this alert level data here? Thanks.

sbrass commented 2 years ago

The answer depends on your configuration, for example chosen routes and your application case.

I would try to route explicitly by your pdseverity label, i.e., all alerts send to a receiver will share a common label, pdseverity, which you then can access by .CommonLabels.pdseverity.

Alternatively, if you group alerts by cluster, service and/or host and have multiple alerts with possible differing pdseverity, you will want to choose the highest pdseverity to apply as payload severity? I would expect Go Template to be able to handle such a case, where you want to have the highest severity printed. However, I can not come up with a quick example for such an application.

mike2194 commented 6 months ago

isn't the severity not even passed to PD? seems like Severity should be mentioned in notifiers.go

PR#3739

prometheus / alertmanager

PagerDuty configuration: unable to set severity field using template #2757