Azure Alerts for Errors - 4xx status codes

[scleary1cs]

Story

As a developer, I need CDC-TI 4xx status codes to be alerted on after a threshold, so that we can begin an incident.

Pre-conditions

• [ ] Assumptions of prior or future work that's out of scope for this story

Acceptance Criteria

• [ ] Required outcomes of the story

Tasks

Research

• [ ] Research work needed to complete the story
• [ ] Foundational: High-level research that will support this and future efforts

Engineering

• [ ] Engineering work needed to complete the story
• [ ] Foundational: Technical runway work to support this and future efforts

Definition of Done

• [ ] Documentation tasks completed
  • [ ] Documentation and diagrams created or updated
  • [ ] ADRs (/adr folder)
  • [ ] Main README.md
  • [ ] Other READMEs in the repo
  • [ ] If applicable, update the ReportStream Setup section in README.md
  • [ ] Threat model updated
  • [ ] API documentation updated
• [ ] Code quality tasks completed
  • [ ] Code refactored for clarity and no design/technical debt
  • [ ] Adhere to separation of concerns; code is not tightly coupled, especially to 3rd party dependencies
• [ ] Testing tasks completed
  • [ ] Load tests passed
  • [ ] Additional e2e tests created
  • [ ] Additional RS e2e assertions created in the rs-e2e project for any new transformations. Includes improvements to the assertion code required to make the new assertions
• [ ] Build & Deploy tasks completed
  • [ ] Build process updated
  • [ ] API(s) are versioned
  • [ ] Feature toggles created and/or deleted. Document the feature toggle
  • [ ] Source code is merged to the main branch

Research Questions

• Optional: Any initial questions for research

Decisions

• Optional: Any decisions we've made while working on this story

Notes

• This story came out of #1379, and #1380.

[saquino0827]

For this alert I believe we need the following:

• A metric alert (Azure monitor has metrics that monitor http status codes)
• The alert is stateful (See: https://learn.microsoft.com/en-us/azure/azure-monitor/alerts/alerts-overview#types-of-alerts)
  • Two reasons for the stateful alerts are
    1. Reduce noise of the alert. Stateful alerts won't fire again until the previous alert has been resolved. Stateful Metric alerts only resolve when the criteria for the alert has not been triggered three consecutive times.
    2. The alert will stay in a 'fired' state, if it finds any 4XX http codes. This would mean we would need to fix ALL 4XX issues so the three consecutive resolution checks happen and 4XX codes are not being logged by our application anymore.
• For the interval and lookback period I'm not 100% sure yet and we can discuss this further with the team.

[saquino0827]

We attempted creating a list of variables for the lifecycle ignore changes block.

cdc_tags = 
set(
    "business_steward",
    "center",
    "environment",
    "escid",
    "funding_source",
    "pii_data",
    "security_compliance",
    "security_steward",
    "support_group",
    "system",
    "technical_steward",
    "zone"
 )
  ignore_tags = [for tag in cdc_tags : "tags[\"${tag}\"]"]

With

lifecycle {
    ignore_changes = local.ignore_tags
  }

And we get the following error

 Error: Invalid expression
│
│   on ../../template/alert.tf line 111, in resource "azurerm_monitor_metric_alert" "azure_4XX_alert":
│  111:     ignore_changes = local.ignore_tags
│
│ A static list expression is required.

We'll revert back to the old lifecycle ignore changes list as there doesn't seem to be a better way to do this. We could add it to the provider of azurerm but that would ignore all the tags in every azure resource and I think we use those tags in some resources.