[Metric Alerts] Warning triggers notificaiton despite no action for Warning status

[JLuse]

Environment

SaaS (https://sentry.io/)

Steps to Reproduce

It has been reported that a metric alert is sending notifications when it’s in a warning status. This happens despite the alert being configured to send notifications only when the alert reaches a critical status. It has occurred for the customer in the past and has come up again. Customer information linked in Jira.

Expected Result

Only send notifications when critical status is met

Actual Result

Notifications sent to customer indicating warning status

Product Area

Alerts

Link

No response

DSN

No response

Version

No response

┆Issue is synchronized with this Jira Improvement by Unito

[getsantry[bot]]

Auto-routing to @getsentry/product-owners-alerts for triage ⏲️

[nholden]

👋 Hi there! I'm the customer bumping into this.

To add some more context, we have an alert that we've configured to notify us when a metric hits a critical threshold. We're notified correctly when the metric hits the critical threshold, but we're also notified when the metric drops below the critical threshold into the "warning" threshold. This is a regular source of confusion because the system is becoming healthier (fewer errors) but we get a big "warning" message, which makes it seems like things are getting worse.

Here are screenshots of a "warning" email and our alert actions configuration. Let me know if there's anything else I can share that would be helpful. Thanks!

[nhsiehgit]

Ah interesting!

Yes - so i believe this is because the alert fires when the state is both entered AND exited (ie. you hit critical, and you leave critical)

agree, this is confusing and not optimal.

I'll create a ticket on our end to address. Thanks for bringing this up!

[nholden]

Thanks, @nhsiehgit!

FWIW: We love the "resolved" emails (when the metric dips below the warning threshold). Would be great if we could continue to get those without getting the "warning" emails.