Notification template no longer working on v0.25.0

chrism417 commented 1 year ago

What did you do?

Upgraded from Alertmanager 0.24.0 to 0.25.0.

What did you expect to see?

No changes in existing functionality.

What did you see instead? Under which circumstances?

Alerts are constantly failing to send to Slack with the following error:

ts=2023-02-02T17:19:19.679Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="sysops-slack/slack[0]: notify retry canceled due to unrecoverable error after 1 attempts: template: :1:12: executing \"\" at <{{template \"common_title\" .}}>: template \"common_title\" not defined"

Environment

Alertmanager version: v0.25.0
Prometheus version: v2.41.0

Alertmanager configuration file:


{{ define "common_title" -}}
[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }} for {{ .CommonLabels.job }}
{{- if gt (len .CommonLabels) (len .GroupLabels) -}}
  {{" "}}(
  {{- with .CommonLabels.Remove .GroupLabels.Names }}
    {{- range $index, $label := .SortedPairs -}}
      {{ if $index }}, {{ end }}
      {{- $label.Name }}="{{ $label.Value -}}"
    {{- end }}
  {{- end -}}
  )
{{- end }}
{{- end }}

{{ define "common_text" -}} {{ with index .Alerts 0 -}} :chart_with_upwards_trend: <{{ .GeneratorURL }}|Graph> {{- if .Annotations.runbook }} :notebook: <{{ .Annotations.runbook }}|Runbook>{{ end }} {{ end }}

Alert details:

{{ range .Alerts -}} Alert: {{ .Annotations.title }}{{ if .Labels.severity }} - {{ .Labels.severity }}{{ end }} Description: {{ .Annotations.description }}{{ .Annotations.message }} Details: {{ range .Labels.SortedPairs }} • {{ .Name }}: {{ .Value }} {{ end }} {{ end }} {{- end }}

{{ define "sysops_slack_severity_channel" -}}#alerts-{{- .CommonLabels.severity -}}{{- end }}

Slack config:

name: 'sysops-slack' slack_configs:
- api_url: 'blah' channel: '#alerts-dev' icon_url: https://avatars3.githubusercontent.com/u/3380462 send_resolved: true title: '{{ template "common_title" . }}' text: '{{ template "common_text" . }}'

simonpasquier commented 1 year ago

can you share the full Alertmanager configuration? On top of my head, I don't know of a change in functionality between 0.24 and 0.25 that would explain your issue.

chrism417 commented 1 year ago

can you share the full Alertmanager configuration? On top of my head, I don't know of a change in functionality between 0.24 and 0.25 that would explain your issue.

Sorry for the delay. Here it is:

global:
  resolve_timeout: 5m
  http_config:
    follow_redirects: true
    enable_http2: true
  smtp_from: alertmanager@yourdomain.com
  smtp_hello: prometheus.yourdomain.com
  smtp_smarthost: nou:587
  smtp_require_tls: false
  pagerduty_url: nou
  opsgenie_api_url: https://api.opsgenie.com/
  wechat_api_url: https://qyapi.weixin.qq.com/cgi-bin/
  victorops_api_url: https://alert.victorops.com/integrations/generic/20131114/alert/
  telegram_api_url: https://api.telegram.org
  webex_api_url: https://webexapis.com/v1/messages
route:
  receiver: sysops-email
  group_by:
  - severity
  - namespace
  - alertname
  - instance
  continue: false
  routes:
  - receiver: sysops-email
    match:
      team: sysops
    continue: false
    routes:
    - receiver: sysops-slack
      match_re:
        alertname: Watchdog
        severity: none
      continue: false
    - receiver: info-blackhole
      match_re:
        severity: info|none
      continue: false
    - receiver: pagertree-infra
      match:
        severity: outage
      continue: true
    - receiver: pagerduty-cloudops
      match:
        severity: outage
      continue: false
    - receiver: sysops-slack
      match_re:
        severity: critical|warning
      continue: true
    - receiver: sysops-email
      match:
        severity: critical
      continue: false
  - receiver: team2-email
    match:
      team: team2
    continue: false
    routes:
    - receiver: info-blackhole
      match_re:
        severity: info
      continue: false
    - receiver: pagertree-team2
      match:
        severity: outage
      continue: true
    - receiver: team2-slack
      match_re:
        severity: outage|critical|warning
      continue: true
    - receiver: team2-email
      match_re:
        severity: outage|critical
      continue: false
  - receiver: info-blackhole
    match:
      team: team1
    continue: false
  - receiver: team3-email
    match:
      team: team3
    continue: false
    routes:
    - receiver: team3-slack
      match_re:
        severity: critical
      continue: false
  group_wait: 10s
  group_interval: 1m
  repeat_interval: 1h
inhibit_rules:
- source_match:
    severity: critical
  target_match_re:
    severity: warning|info
  equal:
  - namespace
  - alertname
- source_match_re:
    severity: critical|major
  target_match_re:
    severity: minor|warning|info
  equal:
  - namespace
  - alertname
- source_match:
    severity: warning
  target_match_re:
    severity: info
  equal:
  - namespace
  - alertname
- source_match_re:
    alertname: InfoInhibitor
  target_match_re:
    severity: info|none
  equal:
  - namespace
receivers:
- name: info-blackhole
- name: sysops-email
  email_configs:
  - send_resolved: false
    to: sysops@yourdomain.com
    from: alertmanager@yourdomain.com
    hello: prometheus.yourdomain.com
    smarthost: nou:587
    headers:
      From: alertmanager@yourdomain.com
      Subject: '{{ template "email.default.subject" . }}'
      To: sysops@yourdomain.com
    html: '{{ template "email.default.html" . }}'
    require_tls: false
- name: team2-email
  email_configs:
  - send_resolved: false
    to: team2@yourdomain.com
    from: alertmanager@yourdomain.com
    hello: prometheus.yourdomain.com
    smarthost: nou:587
    headers:
      From: alertmanager@yourdomain.com
      Subject: '{{ template "email.default.subject" . }}'
      To: team2@yourdomain.com
    html: '{{ template "email.default.html" . }}'
    require_tls: false
- name: team1-email
  email_configs:
  - send_resolved: false
    to: team1@yourdomain.com
    from: alertmanager@yourdomain.com
    hello: prometheus.yourdomain.com
    smarthost: nou:587
    html: '{{ template "email.default.html" . }}'
    require_tls: false
- name: team3-email
  email_configs:
  - send_resolved: false
    to: team3@yourdomain.com
    from: alertmanager@yourdomain.com
    hello: prometheus.yourdomain.com
    smarthost: nou:587
    headers:
      From: alertmanager@yourdomain.com
      Subject: '{{ template "email.default.subject" . }}'
      To: team3@yourdomain.com
    html: '{{ template "email.default.html" . }}'
    require_tls: false
- name: sysops-slack
  slack_configs:
  - send_resolved: true
    http_config:
      follow_redirects: true
      enable_http2: true
    api_url: <secret>
    channel: '#alerts-dev'
    username: '{{ template "slack.default.username" . }}'
    color: '{{ if eq .Status "firing" }}danger{{ else }}good{{ end }}'
    title: '{{ template "common_title" . }}'
    title_link: '{{ template "slack.default.titlelink" . }}'
    pretext: '{{ template "slack.default.pretext" . }}'
    text: '{{ template "common_text" . }}'
    short_fields: false
    footer: '{{ template "slack.default.footer" . }}'
    fallback: '{{ template "slack.default.fallback" . }}'
    callback_id: '{{ template "slack.default.callbackid" . }}'
    icon_emoji: '{{ template "slack.default.iconemoji" . }}'
    icon_url: https://avatars3.githubusercontent.com/u/3380462
    link_names: false
- name: team2-slack
  slack_configs:
  - send_resolved: true
    http_config:
      follow_redirects: true
      enable_http2: true
    api_url: <secret>
    channel: '#team2-alerts'
    username: '{{ template "slack.default.username" . }}'
    color: '{{ if eq .Status "firing" }}danger{{ else }}good{{ end }}'
    title: '{{ template "common_title" . }}'
    title_link: '{{ template "slack.default.titlelink" . }}'
    pretext: '{{ template "slack.default.pretext" . }}'
    text: '{{ template "common_text" . }}'
    short_fields: false
    footer: '{{ template "slack.default.footer" . }}'
    fallback: '{{ template "slack.default.fallback" . }}'
    callback_id: '{{ template "slack.default.callbackid" . }}'
    icon_emoji: '{{ template "slack.default.iconemoji" . }}'
    icon_url: https://avatars3.githubusercontent.com/u/3380462
    link_names: false
- name: team3-slack
  slack_configs:
  - send_resolved: true
    http_config:
      follow_redirects: true
      enable_http2: true
    api_url: <secret>
    channel: '#team3-alerts'
    username: '{{ template "slack.default.username" . }}'
    color: '{{ if eq .Status "firing" }}danger{{ else }}good{{ end }}'
    title: '{{ template "common_title" . }}'
    title_link: '{{ template "slack.default.titlelink" . }}'
    pretext: '{{ template "slack.default.pretext" . }}'
    text: '{{ template "common_text" . }}'
    short_fields: false
    footer: '{{ template "slack.default.footer" . }}'
    fallback: '{{ template "slack.default.fallback" . }}'
    callback_id: '{{ template "slack.default.callbackid" . }}'
    icon_emoji: '{{ template "slack.default.iconemoji" . }}'
    icon_url: https://avatars3.githubusercontent.com/u/3380462
    link_names: false
- name: team1-alerts
  slack_configs:
  - send_resolved: true
    http_config:
      follow_redirects: true
      enable_http2: true
    api_url: <secret>
    channel: '#team1-alerts'
    username: '{{ template "slack.default.username" . }}'
    color: '{{ if eq .Status "firing" }}danger{{ else }}good{{ end }}'
    title: '{{ template "slack.default.title" . }}'
    title_link: '{{ template "slack.default.titlelink" . }}'
    pretext: '{{ template "slack.default.pretext" . }}'
    text: '{{ template "slack.default.text" . }}'
    short_fields: false
    footer: '{{ template "slack.default.footer" . }}'
    fallback: '{{ template "slack.default.fallback" . }}'
    callback_id: '{{ template "slack.default.callbackid" . }}'
    icon_emoji: '{{ template "slack.default.iconemoji" . }}'
    icon_url: https://avatars3.githubusercontent.com/u/3380462
    link_names: false
- name: team1-info
  slack_configs:
  - send_resolved: true
    http_config:
      follow_redirects: true
      enable_http2: true
    api_url: <secret>
    channel: '#team1-info'
    username: '{{ template "slack.default.username" . }}'
    color: '{{ if eq .Status "firing" }}danger{{ else }}good{{ end }}'
    title: '{{ template "common_title" . }}'
    title_link: '{{ template "slack.default.titlelink" . }}'
    pretext: '{{ template "slack.default.pretext" . }}'
    text: '{{ template "common_text" . }}'
    short_fields: false
    footer: '{{ template "slack.default.footer" . }}'
    fallback: '{{ template "slack.default.fallback" . }}'
    callback_id: '{{ template "slack.default.callbackid" . }}'
    icon_emoji: '{{ template "slack.default.iconemoji" . }}'
    icon_url: https://avatars3.githubusercontent.com/u/3380462
    link_names: false
- name: pagertree-team
  webhook_configs:
  - send_resolved: true
    http_config:
      follow_redirects: true
      enable_http2: true
    url: blah
    max_alerts: 0
- name: pagerduty-team
  pagerduty_configs:
  - send_resolved: true
    http_config:
      follow_redirects: true
      enable_http2: true
    service_key: <secret>
    url: nou
    client: '{{ template "pagerduty.default.client" . }}'
    client_url: '{{ template "pagerduty.default.clientURL" . }}'
    description: '{{ template "pagerduty.default.description" .}}'
    details:
      firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
      num_firing: '{{ .Alerts.Firing | len }}'
      num_resolved: '{{ .Alerts.Resolved | len }}'
      resolved: '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
    source: '{{ template "pagerduty.default.client" . }}'
- name: pagertree-team2
  webhook_configs:
  - send_resolved: true
    http_config:
      follow_redirects: true
      enable_http2: true
    url: nou
    max_alerts: 0
- name: team1-pagertree
  webhook_configs:
  - send_resolved: true
    http_config:
      follow_redirects: true
      enable_http2: true
    url: nou
    max_alerts: 0
- name: slack
  slack_configs:
  - send_resolved: true
    http_config:
      follow_redirects: true
      enable_http2: true
    api_url: <secret>
    channel: '@ctenberge'
    username: '{{ template "slack.default.username" . }}'
    color: '{{ if eq .Status "firing" }}danger{{ else }}good{{ end }}'
    title: '{{ template "common_title" . }}'
    title_link: '{{ template "slack.default.titlelink" . }}'
    pretext: '{{ template "slack.default.pretext" . }}'
    text: '{{ template "common_text" . }}'
    short_fields: false
    footer: '{{ template "slack.default.footer" . }}'
    fallback: '{{ template "slack.default.fallback" . }}'
    callback_id: '{{ template "slack.default.callbackid" . }}'
    icon_emoji: '{{ template "slack.default.iconemoji" . }}'
    icon_url: https://avatars3.githubusercontent.com/u/3380462
    link_names: false
templates:
- /etc/alertmanager/config_out/*.tmpl

chrism417 commented 1 year ago

Had to change the templates location in the config to the new directory:

templates:

/etc/alertmanager/config/*.tmpl

prometheus / alertmanager

Notification template no longer working on v0.25.0 #3238