Open thanatchakromsang opened 3 years ago
Hmm, how does the stock alertmanager handle this edge case for Slack?
From https://prometheus.io/docs/alerting/latest/notification_examples/ alertmanager seems to use templating to handle how to send commonAnnotations such as summary and description through slack
Example. Alert
groups:
- name: Instances
rules:
- alert: InstanceDown
expr: up == 0
for: 5m
labels:
severity: page
# Prometheus templates apply here in the annotation and label fields of the alert.
annotations:
description: '{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes.'
summary: 'Instance {{ $labels.instance }} down'
Receivers
- name: 'team-x'
slack_configs:
- channel: '#alerts'
# Alertmanager templates apply here.
text: "<!channel> \nsummary: {{ .CommonAnnotations.summary }}\ndescription: {{ .CommonAnnotations.description }}"
So I think It's a matter of using description and summary in the template or not
And by limitation of webhook_configs we can only send webhook alert with stricts JSON format following
{
"version": "4",
"groupKey": <string>, // key identifying the group of alerts (e.g. to deduplicate)
"truncatedAlerts": <int>, // how many alerts have been truncated due to "max_alerts"
"status": "<resolved|firing>",
"receiver": <string>,
"groupLabels": <object>,
"commonLabels": <object>,
"commonAnnotations": <object>,
"externalURL": <string>, // backlink to the Alertmanager.
"alerts": [
{
"status": "<resolved|firing>",
"labels": <object>,
"annotations": <object>,
"startsAt": "<rfc3339>",
"endsAt": "<rfc3339>",
"generatorURL": <string> // identifies the entity that caused the alert
},
...
]
}
So I don't think we can do anything more except create a precaution in README.md or handle description gracefully
I don't even get any throws using the docker hub image. Had to read this issue and add the description field to get the hook to work. Should probably include it in the readme + log or just have an empty string in place when description is missing?
Yeah faced with the same issue as it no error ether in alertmanager/prometheus or webhook just alert not displayed
I found this problem while setting up alertmanager-discord in my cluster and tested that It doesn't send to discord and silent leading to investigation consider following Aleart rules
It does trigger discord webhook with annotation description not empty though
It doesn't have description then when the rules triggered alertmanager firing to webhooks but It does nothing
consider this example alert output from alertmanager to alertmanager-discord. This example doesn't send to discord because It lack annotation
description
but It does send after alert rules to send description thoughMy question is how should we fix this? Should we just thrown an error when no description or send empty string to discord instead?
Or should we just Update README.md to have a required annotation for alertmanager-discord?