Infrastructure: Validate that feature branch changes won't reprovision resources

[decodingahmed]

As a developer I want to know if my infrastructure changes will cause a resource to be recreated when the terraform is applied So that my changes don't disrupt dev and prod environments when they merged into main branch

Background Currently, when we make an infrastructure change in a PR branch, it creates a brand-new deployment/environment. The PR could contain changes that might cause Terraform to destroy and recreate (e.g. - dns_prefix property for AKS: link), which would cause AKS.

Acceptance Criteria

• [ ] Should not destroy Cosmos DB
• [ ] Should not destroy AKS Cluster
• [ ] TBD after discussion with the team

[decodingahmed]

Possible solution

We might need to do 2 things:

1. In the PR workflow, we could terraform plan (but not apply) the PR changes against dev and prod environments.

2. Protect some resources using the lifecycle arguments (see docs), for example:

   resource "azurerm_kubernetes_cluster" "example" {
   # ...
   
   lifecycle {
   prevent_destroy = true
   ignore_changes = [
     # Ignore changes to tags, e.g. because a management agent
     # updates these based on some ruleset managed elsewhere.
     tags,
   ]
   }
   }