search

prometheus / alertmanager

Prometheus Alertmanager
https://prometheus.io
Apache License 2.0
6.68k stars 2.16k forks source link

Feature request: Slack alert threading #2165

Open MaT1g3R opened 4 years ago

MaT1g3R commented 4 years ago

It would be nice to have firing alerts and resolved alerts to be grouped into a single slack thread to reduce noise and make it easier to identify currently open issues.

https://stackoverflow.com/questions/56667592/aggregate-alertmanager-messages-as-slack-threads

simonpasquier commented 4 years ago

The existing Slack notifier relies on Slack's incoming webhooks which support simple messages only. I'm not sure how this could work without a major overhaul of the code.

MaT1g3R commented 4 years ago

I think we'd have to implement the alerts as an slack app instead for this to work. If this is not a high priority for the maintainers to implement, I can take a poke at it when I have some free time.

simonpasquier commented 4 years ago

Yes I don't think that any of the maintainers has time to look into this. I'd encourage to prototype something using the webhook receiver so you can experiment and move faster.

ALA-DQ commented 4 years ago

As it is written in the section "Post your message as a reply in a thread" of the Slack Incoming Webhooks documentation, it seems to be possible to send "Reply" message in Slack Thread using Incoming WebHook:

You can use Incoming Webhooks to make your message appear as a reply in a thread.

In addition, in the section "Replying to your message" of the Slack message sending documentation, if Alertmanager can handle the ts field for firing alerts and use it in the resolved, we can manage to do feature.

What do you think about this ? Am I totally wrong or is there a way to have this ?

If you guys think there is a possibility in the Alertmanager, and if you do not have the time to do this, maybe I can try to help.

brian-brazil commented 4 years ago

That'd require state from the response of a previous notification to be kept around, which isn't really possible - especially considering clustering. Each notification is independent.