An alert could escalate it's severity over time. For example, after
one minute of a Pod not existing an Alert with the Warning severity
could trigger, after five minutes an Alert with the Critical
severity would trigger. We shouldn't have both of these active in
AlertManager at the same time, instead we should inhibit the less
severe alert when the escalated alert triggers.
Verification:
Deploy this branch of the operator
The alertmanager-application-monitoring secret should contain
the inhibit_rules featured in this commit.
Go to the PrometheusRule created in the namespace
It should contain a DeadMansSwitch rule with severity None
Change this to severity Warning
Duplicate it with a severity Critical
You should now see two Alerts in the Prometheus dashboard
You should see one triggered Alert in AlertManager with the higher
severity (Critical).
An alert could escalate it's severity over time. For example, after one minute of a Pod not existing an Alert with the Warning severity could trigger, after five minutes an Alert with the Critical severity would trigger. We shouldn't have both of these active in AlertManager at the same time, instead we should inhibit the less severe alert when the escalated alert triggers.
Verification:
alertmanager-application-monitoring
secret should contain theinhibit_rules
featured in this commit.