INTLY-1100 Make inhibition rules more explicit for CronJob alerts

[aidenkeating]

Currently the inhibition rule for CronJob alerts are vague and encompass other alerts also. We only want to address the issue where JobRunningTimeExceeded has a warning and a critical alert for the same job.

This change restricts the inhibit_rule to just address alerts from JobRunningTimeExceeded.

Verification:

• Deploy this application-monitoring-operator to a cluster
• Open up the AlertManager console, ensure the config in the Status tab looks correct
• Ensure the backups playbook has been run on the cluster
  • Ensure their schedules are */1 * * * * so you don't have to wait too long
  • CronJobs should exist
  • PrometheusRules should exist
• Edit the backup-monitoring-alerts PrometheusRule in the namespace:
  • For JobRunningTimeExceeded with severity=warning, change 300 to 120
  • For JobRunningTimeExceeded with severity=critical, change 600 to 240
• Pick a CronJob and edit it:
  • Set the value of -d (the line under it) in the command to true
• The CronJob should stay running forever, wait until the warning fires in AlertManager. This should take about 2 minutes
• When the warning fires, edit a second CronJob the same way.
• Wait another 2 minutes
• The first CronJob alert should now be a severity=critical
• The second CronJob alert should be severity=warning
• If you click the show inhibited option you should see the inhibited warning alert from the first CronJob.

[aidenkeating]

After the Change

All criticals are shown, the warnings are inhibited.

The warning without a critical is shown.

[aidenkeating]

@david-martin @pb82 I'm going to provision these changes onto my cluster and do the first half of the verification steps, as the verification steps are quite long

[aidenkeating]

This is now provisioned on the akeating-4ecc cluster.

I've set one CronJob to debug mode (enmasse-pv-backup), so the Job will never end to kick off severity=critical. I've set the rest of the CronJobs to suspended.

So the verification steps remaining would be:

• Open up the AlertManager console, ensure the config in the Status tab looks correct
• Ensure there is one Alert already firing with severity=critical for enmasse-pv-backup.
• Pick a suspended CronJob and edit it:
  • Set it's suspended value to false
  • Set the value of -d (the line under it) in the command to true
• The CronJob should stay running forever, wait until the Alert with severity=warning fires in AlertManager. This should take about 2 minutes
• Ensure there are now two Alerts with the name JobRunningTimeExceeded
• The first CronJob (enmasse-pv-backup) alert should now be a severity=critical
• The second CronJob alert should be severity=warning
• If you click the show inhibited option you should see the inhibited severity=warning alert from the first CronJob (enmasse-pv-backup).