Azure Alerts for Errors - CDC TI services are down/offline

[scleary1cs]

Story

As a developer, I need to know if our services are down/offline, so that we can begin an incident.

Pre-conditions

• [ ] Assumptions of prior or future work that's out of scope for this story

Acceptance Criteria

• [ ] Alerts trigger when the TI and/or Ingestion Service are down. After discussion, we decided to base this on whether there are an appropriate number of healthy instances - if instances can't stay up/healthy, the service is not working

Tasks

Engineering

• [x] Alerts are set up that trigger if TI is down
• [ ] Alerts are set up that trigger if Ingestion Service is down

Definition of Done

• [ ] Documentation tasks completed
  • [ ] Documentation and diagrams created or updated
  • [ ] ADRs (/adr folder)
  • [ ] Main README.md
  • [ ] Other READMEs in the repo
  • [ ] If applicable, update the ReportStream Setup section in README.md
  • [ ] TI Threat model or Ingestion Service threat model updated
  • [ ] API documentation updated
• [ ] Code quality tasks completed
  • [ ] Code refactored for clarity and no design/technical debt
  • [ ] Adhere to separation of concerns; code is not tightly coupled, especially to 3rd party dependencies
• [ ] Testing tasks completed
  • [ ] Load tests passed
  • [ ] Additional e2e tests created
  • [ ] Additional RS e2e assertions created in the rs-e2e project for any new transformations. Includes improvements to the assertion code required to make the new assertions
• [ ] Build & Deploy tasks completed
  • [ ] Build process updated
  • [ ] API(s) are versioned
  • [ ] Feature toggles created and/or deleted. Document the feature toggle
  • [ ] Source code is merged to the main branch

Research Questions

• Optional: Any initial questions for research

Decisions

• Optional: Any decisions we've made while working on this story

Notes

• This story came out of #1379, and #1380.

[pluckyswan]

I believe this is covered by #1397

[JohnNKing]

Already complete?

[somesylvie]

1397 is about whether Azure services are down (like if e.g. all web applications are down or all of blob storage is unavailable). This card is about whether our team's services (TI and SFTP Ingestion Service) are down, which can happen even if Azure is fine. This card is about resource health in Azure parlance, while 1397 is about service health and Azure status