Alert customization

[kaspernissen]

Up until now, we have been using Elasticsearch with ElastAlert as our Alertmanager. ElastAlert provides some capabilities in terms of reporting issues in Slack. We were heavy users of the following:

• 1 webhook - multiple channels we are reporting alerts in multiple slack channels, e.g. every squad has there own channel, and they have to receive the alerts they specify. Instead of creating multiple webhooks, reusing the webhook with the ability to specify channel.
• custom name and avatar we are using this for providing e.g. our supports with custom formatting about issues they need to verify. E.g. provide them with a link to our internal supporting tool and a customer description of the task they have to do. Further a custom avatar that are the same as a "bot" we use in our internal system.
• custom fields Providing options for attaching messages based on data from specific fields.

[pmech]

Thank. The first bullet is related to #22

[anagrius]

Thank you @kaspernissen those are good ideas. I guess each notifier would have some additional options on a per-alert basis. Currently I think the way the slack integration is built, the URL you provide is channel specific. We would need to look into another way of doing it.

[kaspernissen]

As discussed with @pmech I will fill in some more details of what we would like in terms of alert customization.

Our Prometheus alerts are configurable with templating, we would like something similar. An example of an alert definition in Prometheus:

  - alert: KubernetesWorkerNodeCPUUtilizationHIGH
    expr: (sum(irate(node_cpu{job="ec2",k8s_io_role_node="1",mode!="idle"}[5m])) BY
      (instance) * 100) > 175
    for: 5m
    labels:
      service: kubernetes
      severity: critical
    annotations:
      description: '{{$labels.instance}} CPU Utilization is above 175%'
      summary: '{{$labels.instance}} CPU Utilization is very HIGH!'

And the alertmanager configuration:

  slack_configs:
  - channel: '#alerts'
    username: 'Prometheus Alertmanager'
    icon_url: '*.png'
    title: '{{ .CommonLabels.alertname }}'
    title_link: 'https://*.atlassian.net/wiki/display/TECH/{{ .CommonLabels.alertname }}'
    text: "*Status*: {{ .Status }} \n *Summary:* {{ .CommonAnnotations.summary }} \n *Description:* {{ .CommonAnnotations.description }} \n *Severity:* {{ .CommonLabels.severity }} \n *Service:* {{ .CommonLabels.service }} \n *Runbook:* https://*.atlassian.net/wiki/display/TECH/{{ .CommonLabels.alertname }}"
    send_resolved: true

Which results in an alert as follows:

The main things we would like here, are the possibility to add a description to the alert, that will be outputtet in Slack along with what we already see today. In this description, we could provide more context, and attach a link to a specific runbook. Templating, and adding actual values would be really helpful as well.

[anagrius]

@pmech Would you take a look at this and how close we are to closing it with the new ability to assign multiple notifiers?