Alerting

[iainhtp]

Currently, alerting has two blocker aspects

Issues

1. You cannot set a message subject
2. You cannot have anything but static text in the message body

Experience

1. Every alarm notification has the Subject: 'AWS Notification Message', From: 'AWS Notifications'
   There is no way to see what the alarm indicates without opening the message or setting up a plethora of inbox rules
2. The message received will be generic and cannot indicate a severity - each alarm would then have to be viewed in Grafana to ascertain response.

Workarounds

1. We could put a JSON block of info into the message body, write a lambda subscriber to the grafana notification topic, to parse and create a new SNS message to another topic for distribution to end users.
   This increases cost by doubling the number of SNS messages and requiring additional compute resources
   This also makes the alert fragile due to the reliance on well formed JSON in the message body
2. There is no workaround to have a query value inserted into the message body - it is not possible with Legacy Grafana Alerting

Request

1. Allow us to set the message subject for message published to SNS topics 1.1. Allow us to include dashboard variables and single element query results in the message subject
2. Allow us to include dashboard variables, panel query aliases and single element query results in the message body
3. Perhaps allow us to post to a lambda directly?

[mhausenblas]

Thank you @iainhtp for the detailed description and at least some of the issues should be addressed with the upcoming unified alerting.

One clarification question, though:

Perhaps allow us to post to a lambda directly?

Not sure I follow, do you mean a direct integration between Grafana alert notification and AWS Lambda?

[iainhtp]

Hi @mhausenblas - yes, being able to invoke a lambda directly as a Grafana Notification Channel would at least remove the requirement for an SNS proxy to lambda.