integr8ly / application-monitoring-operator

Operator for installing the Application Monitoring Stack on OpenShift (Prometheus, AlertManager, Grafana)
Apache License 2.0
30 stars 44 forks source link

INTLY-1100 Inhibit duplicate alerts with different severity #26

Closed aidenkeating closed 5 years ago

aidenkeating commented 5 years ago

An alert could escalate it's severity over time. For example, after one minute of a Pod not existing an Alert with the Warning severity could trigger, after five minutes an Alert with the Critical severity would trigger. We shouldn't have both of these active in AlertManager at the same time, instead we should inhibit the less severe alert when the escalated alert triggers.

Verification:

aidenkeating commented 5 years ago

@pb82 @david-martin Mind taking a look?

pb82 commented 5 years ago

:eyes: