Circuit breaker on the callback URL when it fails too much

[sastels]

Description

As a Notify team member, I need Notify to be stable and reliable.

WHY are we building? Callback failures can cause system slowdowns / instability.

WHAT are we building? Stop sending to a callback that has been generating errors.

VALUE created by our solution System more stable and reliable

Acceptance Criteria

• [ ] Decide threshold on when we stop using the callback. (5 separate notification callback failures hit the max retry over a 10 minute period)
• [ ] Decide if there's an automatic "try it again" after some time period. (Email to let a user know we've suspended their callbacks)
• [ ] Automate sending a warning email to services who's callbacks are failing (5 failures, not max retry, over 30 minutes)
• [ ] When alarms hit critical - ops lead can manually
• [ ] Add code to implement suspending a service
• [ ] Add code to unsuspend a service
• [ ] Who has access to suspend/resume a callback service

1. TF warning alert - send warning email
2. TF critical alert - send suspension email
3. Ops lead suspend service manually

QA Steps

• [ ] Hook up a failing callback.
• [ ] Send a lot of notifications.
• [ ] Verify that the callback stops being called.
• [ ] Verify that it is called after the specified cooldown period, if applicable.

Related Incident 2024-03-28-notify-is-spitting-out-a-bunch-of-errors

[whabanks]

As part of the work on 1564 I sketched out what this might look in implementation.

[jzbahrai]

Based on the ADR for callbacks - we are not automating most of the callback flows. I am going to shut this ticket as things that are relevant in it are covered by other tickets, things that are not - we will pick up in version 2 of callbacks if needed.