SLO alert label update

[TheoBrigitte]

Goal

We need to be able to change label on SLO alert. e.g. a component ownership changes and team label need to be updated in the alert.

Problem

Changing SLO alert seems not possible currently as this cause prometheus rules evaluation failure today after SLO alert label got changed.

Details

Changes:

• https://github.com/giantswarm/sloth-rules/commit/65e4ee94102c6673a8bff7bf88066e5b93665cb5
• https://github.com/giantswarm/sloth-rules/commit/f77c67dddddedf5cebd99019b89aa8c156610836

This lead to the following

If you change any of those labels, while keeping other values the same, then the expressions in those rules will start to return more than one series (one with the old label, one with the new label), causing the recording rules to fail until the old label ages out.

More details :

• https://github.com/slok/sloth/issues/311 (issue won't be fixed)
• #inc-2023-06-15-prometheusrules-failing-all-around-sloth-rules-fail

[TheoBrigitte]

Solutions

• Instead of renaming label, delete and re-create a new alert with a different name.

[TheoBrigitte]

This happened again today, someone changed the team label on a SLO alert which broke the alert.

We need to make sure people do not do this again:

• Documentation to explain this
• CI check to prevent this