Open ki4rbc opened 4 years ago
benjojo
commented
4 years ago Cheers for the input json context,
Right now there is not a way for you to do this, The way I do this in my infra is to inline the interesting data bits into my alerts description
What would you want to do with the locID? Just display it?
ki4rbc
commented
4 years ago Thanks for responding so quickly, The locID is how I know where the alert is coming from it comes from an os env which is then used as a yaml anchor which is later used to create the external_label.
The problem I am running into is that external_labels are not available for processing within alert rules; unless I am missing something.
And yes. I would like locID to show up anywhere in the alert in discord.
ki4rbc
commented
4 years ago Hello again @benjojo, So I got this mod of main.go to work but I'm not entirely happy and it's not something I think I would suggest a fork for.
--- original/alertmanager-discord/main.go 2020-08-31 18:35:49.698522600 -0400
+++ alertmanager-discord/main.go 2020-08-31 22:28:46.702549100 -0400
@@ -121,7 +121,7 @@
}
RichEmbed.Fields = append(RichEmbed.Fields, discordEmbedField{
- Name: fmt.Sprintf("[%s]: %s on %s", strings.ToUpper(status), alert.Labels["alertname"], realname),
+ Name: fmt.Sprintf("[%s]: %s on %s - %s", strings.ToUpper(status), alert.Labels["alertname"], realname, alert.Labels["locID"]),
Value: alert.Annotations.Description,
})
}
Yeah I think for now a fork is the best, I kinda want to see if this is a one off use case or something other people have issues with too
On Tue, Sep 1, 2020 at 6:42 PM Mike McDonough notifications@github.com wrote:
Hello again @benjojo https://github.com/benjojo, So I got this mod of main.go to work but I'm not entirely happy and it's not something I think I would suggest a fork for.
--- original/alertmanager-discord/main.go 2020-08-31 18:35:49.698522600 -0400+++ alertmanager-discord/main.go 2020-08-31 22:28:46.702549100 -0400@@ -121,7 +121,7 @@ }
RichEmbed.Fields = append(RichEmbed.Fields, discordEmbedField{- Name: fmt.Sprintf("[%s]: %s on %s", strings.ToUpper(status), alert.Labels["alertname"], realname),+ Name: fmt.Sprintf("[%s]: %s on %s - %s", strings.ToUpper(status), alert.Labels["alertname"], realname, alert.Labels["locID"]), Value: alert.Annotations.Description, }) }— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/benjojo/alertmanager-discord/issues/8#issuecomment-684989504, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALPK4XC5Q6NI7HOT6YMYDLSDUQATANCNFSM4QRAES7Q .
ki4rbc
commented
4 years ago I found a work around to adding an alert origin identity, locationID in my case. By creating the alertmanager alerting rules is a config map template, I have access to environmental variables. It took a while to figure out I have to escape curly brackets. For example
- alert: Deployments unavailable
expr: 100 * (kube_deployment_status_replicas_available / kube_deployment_status_replicas) != 100
for: 5m
labels:
severity: warn
annotations:
summary: 'Deployments unavailable at Location ID {{ .Values.locationID }}'
description: '{{ "{{" }} $labels.deployment {{ "}}" }}.{{ "{{" }} $labels.namespace {{ "}}" }} at {{ "{{" }} $value | printf "%.2f" {{ "}}" }}% of intent'I now can see where the alerts originate from.
I did see that on some alerts there is no label instance or exported_instance so we can still end up with alerts where realname is NULL.
I've created a fork and will work though a solution to propose.
I am attempting to use an external label,
locID, with Prometheus alerts. I have the external label being sent to alertmanager-discord, but how does one configure what's going to discord; other then modifying main.go?Here is the print-pretty formatted input JSON: