Open poornima-krishnasamy opened 10 months ago
poornima-krishnasamy
commented
9 months ago Another suggestion to setup slack_api_token to send alerts to multiple channels dynamically https://github.com/prometheus/alertmanager/issues/2513#issuecomment-1694033990
ties
commented
6 months ago There seems to be other ways to include slack config dynamically to the Alertmanager: More information here: https://mojdt.slack.com/archives/C57UPMZLY/p1701968614790789
This slack link is not available to outside parties.
I think this is an issue that outside parties also face in their environment. Could you elaborate on the solution you arrived at in the end?
mikebell
commented
5 months ago @ties here are the links pulled from the thread -
https://ralphmcneal.com/prometheus-alertmanager-dynamic-slack-config/
https://engineering.reputation.com/custom-alertmanager-alerts-based-on-service-config-17586342f3a8
ties
commented
5 months ago @ties here are the links pulled from the thread -
https://ralphmcneal.com/prometheus-alertmanager-dynamic-slack-config/
https://engineering.reputation.com/custom-alertmanager-alerts-based-on-service-config-17586342f3a8
Thanks for sharing these. The first approach, combined with checking the required labels on alerts in CI/CD is what we will likely explore.
Background
Currently, when users wants to setup custom alerts for their application, they follow the guide, setup the slack app and raise support ticket to create a AM route. This is a long process and involve multiple teams to get approval.
There seems to be other ways to include slack config dynamically to the Alertmanager: More information here: https://mojdt.slack.com/archives/C57UPMZLY/p1701968614790789
Explore the suggestions and see if that is doable and can fit into our monitoring stack
Questions / Assumptions
Definition of done