Closed aidenkeating closed 5 years ago
All criticals are shown, the warnings are inhibited.
The warning without a critical is shown.
@david-martin @pb82 I'm going to provision these changes onto my cluster and do the first half of the verification steps, as the verification steps are quite long
This is now provisioned on the akeating-4ecc
cluster.
I've set one CronJob to debug mode (enmasse-pv-backup
), so the Job will never end to kick off severity=critical
. I've set the rest of the CronJobs to suspended.
So the verification steps remaining would be:
severity=critical
for enmasse-pv-backup
.-d
(the line under it) in the command to true
severity=warning
fires in
AlertManager. This should take about 2 minutesJobRunningTimeExceeded
enmasse-pv-backup
) alert should now be a severity=critical
severity=warning
severity=warning
alert from the first CronJob (enmasse-pv-backup
).
Currently the inhibition rule for CronJob alerts are vague and encompass other alerts also. We only want to address the issue where JobRunningTimeExceeded has a warning and a critical alert for the same job.
This change restricts the inhibit_rule to just address alerts from JobRunningTimeExceeded.
Verification:
*/1 * * * *
so you don't have to wait too longbackup-monitoring-alerts
PrometheusRule in the namespace:JobRunningTimeExceeded
withseverity=warning
, change300
to120
JobRunningTimeExceeded
withseverity=critical
, change600
to240
-d
(the line under it) in the command totrue
severity=critical
severity=warning