When a user clicks close in Alerta the alert could remain in an open state in alertmanager, which would not send a new alert to alerta.
This PR will take the raw_data from the original alert and add/modify the endsAt time to 5 minutes in the past and post it back to the Alertmanager API URL. This will trick the Alertmanager into considering it expired and so if a new update for that alert comes in to Alertmanager it will send it along to Alerta once again which should re-open the alert. This should help keep Alerta and Alertmanager more in sync in case an end user accidentally closes an alert that isn't actually fixed.
Additionally, you can set a flag to tell Alert to prefer the externalUrl for an alert when setting silences. This is handy for a situation where a user has a load balanced Alertmanager API endpoint that mirrors requests to all alertmanagers in the cluster.
Prometheus docs specify that prometheus should send all alerts to all alertmanagers. However, alertmanager seems to sync silences across all nodes in a cluster, so mirroring silence requests will create one silence per alertmanager and sync all of them. Using an appropriately set externalUrl of each alertmanager will only create one and having and externalUrl set differently than the loadbalanced url is also handy for troubleshooting.
Also added the integration with OpsGenie Edge Connector. This allows OpsGenie to send Acknowledge/Close etc into Alerta. When used in conjunction with the OpsGenie plugin which allows Alerta to fire alerts through OpsGenie an end user can now have a good experience Acknowledging/Closing alerts in Alerta through the OpsGenie app.
satterly
commented
2 years ago
This PR will take the raw_data from the original alert and add/modify the
endsAttime to 5 minutes in the past and post it back to the Alertmanager API URL. This will trick the Alertmanager into considering it expired and so if a new update for that alert comes in to Alertmanager it will send it along to Alerta once again which should re-open the alert. This should help keep Alerta and Alertmanager more in sync in case an end user accidentally closes an alert that isn't actually fixed.Additionally, you can set a flag to tell Alert to prefer the externalUrl for an alert when setting silences. This is handy for a situation where a user has a load balanced Alertmanager API endpoint that mirrors requests to all alertmanagers in the cluster. Prometheus docs specify that prometheus should send all alerts to all alertmanagers. However, alertmanager seems to sync silences across all nodes in a cluster, so mirroring silence requests will create one silence per alertmanager and sync all of them. Using an appropriately set externalUrl of each alertmanager will only create one and having and externalUrl set differently than the loadbalanced url is also handy for troubleshooting.