[Uptime][RAC] Display alert severity level in Observability alerts table

[dominiqueclarke]

Meta issue: elastic/kibana#97951

Prerequisites

• [ ] Migrate from alerts registry to observability rules registry. Issue elastic/kibana#98382

Implementation When we save alerts-as-data to our alerts indices, we need to save the severity of the given evaluation onto the alert document, so that we can display that information in the Alerts table.

Background

Currently, all our alerts have the same severity level. Our goal is to add a differentiation of Critical and Warning alerts. These severity levels will be shown in the alert list and alert details.

Example of how the severity levels will be shown in the alerts list

Please consider, the Infrastructure/Metrics app already has severity levels in place (see alert rule creation flyout).

Going forward we are considering attaching different actions to these severity levels. Example: If the threshold is above 50% send a Slack message to X; if the threshold is above 75% send this to PagerDuty.

[elasticmachine]

Pinging @elastic/uptime (Team:uptime)

[dominiqueclarke]

Need to identify:

• How is severity calculated for monitor status alerts?
  • Monitors are either up or down (healthy or unhealthy). Is there a notion of a severity hierarchy?
• How is severity calculated for certs alerts
  • I believe @andrewvc or @justinkambic may have mentioned that we do have an established definition of severity for certs
• How is severity calculated for anomalies?

[andrewvc]

Per a meeting on this date, we'd like to add a severity dropdown to the alert dialog letting the user pick the severity on a per-alert basis. This is important because severity is usually more about the site / check than anything else.