Open kaspernissen opened 6 years ago
Thank. The first bullet is related to #22
Thank you @kaspernissen those are good ideas. I guess each notifier would have some additional options on a per-alert basis. Currently I think the way the slack integration is built, the URL you provide is channel specific. We would need to look into another way of doing it.
As discussed with @pmech I will fill in some more details of what we would like in terms of alert customization.
Our Prometheus alerts are configurable with templating, we would like something similar. An example of an alert definition in Prometheus:
- alert: KubernetesWorkerNodeCPUUtilizationHIGH
expr: (sum(irate(node_cpu{job="ec2",k8s_io_role_node="1",mode!="idle"}[5m])) BY
(instance) * 100) > 175
for: 5m
labels:
service: kubernetes
severity: critical
annotations:
description: '{{$labels.instance}} CPU Utilization is above 175%'
summary: '{{$labels.instance}} CPU Utilization is very HIGH!'
And the alertmanager configuration:
slack_configs:
- channel: '#alerts'
username: 'Prometheus Alertmanager'
icon_url: '*.png'
title: '{{ .CommonLabels.alertname }}'
title_link: 'https://*.atlassian.net/wiki/display/TECH/{{ .CommonLabels.alertname }}'
text: "*Status*: {{ .Status }} \n *Summary:* {{ .CommonAnnotations.summary }} \n *Description:* {{ .CommonAnnotations.description }} \n *Severity:* {{ .CommonLabels.severity }} \n *Service:* {{ .CommonLabels.service }} \n *Runbook:* https://*.atlassian.net/wiki/display/TECH/{{ .CommonLabels.alertname }}"
send_resolved: true
Which results in an alert as follows:
The main things we would like here, are the possibility to add a description to the alert, that will be outputtet in Slack along with what we already see today. In this description, we could provide more context, and attach a link to a specific runbook. Templating, and adding actual values would be really helpful as well.
@pmech Would you take a look at this and how close we are to closing it with the new ability to assign multiple notifiers?
Up until now, we have been using Elasticsearch with ElastAlert as our Alertmanager. ElastAlert provides some capabilities in terms of reporting issues in Slack. We were heavy users of the following: