[Feature request] Dynatrace-like notifications in Slack

[enchobelezirev]

Description

We are having our "health-check" pipeline to monitor our service health on different Cloud landscapes(AWS, GCP, Azure, etc.). When one of the services on some of the landscapes falls, we are getting notification in our Slack channel. So far so good, the problem is that the health-check is triggered every 5 minutes and if the service is again DOWN(on each trigger) we receive another duplicated notification in our Slack channel. As you might think, this could lead to an explosion in our Slack channel when we have outage on one of the landscapes.(Image what will happen when all the other landscapes are DOWN :) )

Feature request

I was thinking whether the Slack notifications could be dynatrace-like notifications. These notifications have the following pattern:

• Send notification when a problem araises - in this case when our health-check concourse job fails
• If there are sequential failures of the same job - do not send notification
• Send RESOLVED notification when the problem is gone - send one notification which is describing that the recently failing job is green again

Problems with the current implementation

What I have observed during my development is that with the current implementation I cannot implement the above requirement. I have tried to do this but my attemp involves a lot of custom coding and too much Concourse complexity which in my opinion is not a responsibility to a developer.

Please, tell me what do you think about this proposal and please, tell me if there is already such resource which could give me the desired behavior.

[jhunt]

This requires state outside of what Concourse can handle.

For this reason, I wrote an edge-triggered notification gateway, a couple of years back. It's called SHOUT!

https://github.com/jhunt/shout