Azure Alerts for Errors - Azure is down

[scleary1cs]

Story

As a developer, I need to know if Azure is down, so that we can begin an incident.

Acceptance Criteria

• [ ] Alert is working in Slack

Tasks

• [x] Create alert in TI alert.tf

Definition of Done

• [x] Documentation tasks completed
  • [x] Documentation and diagrams created or updated
  • [x] ADRs (/adr folder)
  • [x] Main README.md
  • [x] Other READMEs in the repo
  • [x] If applicable, update the ReportStream Setup section in README.md
  • [x] Threat model updated
  • [x] API documentation updated
• [x] Code quality tasks completed
  • [x] Code refactored for clarity and no design/technical debt
  • [x] Adhere to separation of concerns; code is not tightly coupled, especially to 3rd party dependencies
• [x] Testing tasks completed
  • [x] Load tests passed
  • [x] Additional e2e tests created
  • [x] Additional RS e2e assertions created in the rs-e2e project for any new transformations. Includes improvements to the assertion code required to make the new assertions
• [x] Build & Deploy tasks completed
  • [x] Build process updated
  • [x] API(s) are versioned
  • [x] Feature toggles created and/or deleted. Document the feature toggle
  • [x] Source code is merged to the main branch

Notes

• This story came out of #1379, and #1380.

[pluckyswan]

Created a Service Health > Health alerts in internal, currently disabled.

[jherrflexion]

Curious if this would be too noisy if it is looking at every Azure service? Will attempt to test this and convert to Terraform today.

[halprin]

Curious if this would be too noisy if it is looking at every Azure service? Will attempt to test this and convert to Terraform today.

Yeah, we definitely don't want to be looking at services we don't use. And yes, we want all these alert stories done via Terraform.

[jherrflexion]

azure-outage-alert branch created. Internal clickops enabled for testing.

[jherrflexion]

Currently blocked by an error in Terraform "CreateOrUpdate" on the test PR. Attempted to rerun the job a few times and draft another new PR and still received the error.

[pluckyswan]

PR is out.

[jherrflexion]

Addressing PR comments

[halprin]

Had to revert this work because sadly this is failing deploys in staging. You can see the revert PR for the thinking behind this. Because of all of this, moved this story back into In Progress.