Clearing/Resolved Notification & Notification states

DerPhlipsi commented 7 years ago

Expected Behavior

It should be possible to send an email, when an alert has been marked as resolved.

Also, it should be possible to mark alerts as acknowledged or similar. For example, some alerts should only be marked as resolved by hand, since the values only indicate an alert, but their return to normal (or absence) do not indicate that the alert reason has been fixed. For example, Alerts could have the states (brackets mark additional information like a tooltip).

pending
acknowledged (by person x / by all)
fix underway
backup system active
resolved (by condition / by person x)

Maybe multiple states can also be active like acknowledged by all, fix underway and backup system active, to display what is currently happening regarding this alert.

Current Behavior

Alert are marked silently as resolved. If you want notifications about that, you have to define a second alert that is active if "the values are in a normal state".

Context

See the example in #3664: "An NTP server sometimes struggles with the radio signal of the DCF77 clock signal send from Frankfurt, Germany. It looses the signal and switches to its local quartz clock and sends a syslog message. This should start an alert and send an email to the person controlling the NTP servers as an notification. If the NTP server regains the signal, it sends another syslog message which should mark the alert as resolved and should also be able to send a second notification/email that informs about the resolved alert."

In addition, we have many critical alerts that should only be resolved by hand, since they can result in other alerts. This way, colleagues are able to see if an alert is currently active that could have caused their own alert. This is especially useful if you get woken up at 3 AM by a call of the operations dept. and first have to check around if something else caused your alert.