[PRP-268] Correctly allow Github and Terraform to manage ECS task definitions

[rocketnova]

Ticket

https://wicmtdp.atlassian.net/browse/PRP-268

Changes

What was added, updated, or removed in this PR.

• Ignore container_definitions on aws_ecs_task_definition by default
• Add inline comments for how to manually manage ECS service and task definitions redeploys in terraform

Context for reviewers

Testing instructions, background context, more in-depth details of the implementation, and anything else you'd like to call out or ask reviewers. Explain how the changes were verified.

We have a confluence of factors that makes CD a little challenging:

• I would like us to use immutable docker image tags so that we can more easily trace which image has been deployed to which environment at what time
• I would like us to grant the Github Actions AWS IAM role as narrow a set of permissions as possible so that we aren't granting the machine user full permissions to run terraform apply
• I would like Github Actions to be able to build new docker images and deploy them to any of our environments so that we can support continuous deployment
• I would like to be able to manage everything else about infrastructure in terraform so that we have our infrastructure documented as code

To accomplish this, we have:

• Set ECR image repositories to use immutable tags
• Narrowed down our Github Actions role permissions
• Used the aws-actions Github Actions to deploy updated ECS task definitions
• (so far) Added task_definition to the lifecycle ignore_changes block for the ECS service, like so:

# in aws_ecs_service
  lifecycle {
    ignore_changes = [
      desired_count,
      task_definition,
    ]
  }

However, if this is the only change we ignore, the aws_ecs_task_definition will try to update the container definitions to the default image tag if we have previously set it to something else.

This PR adds the following:

# in aws_ecs_task_definition
  lifecycle {
    ignore_changes = [
      container_definitions
    ]
  }

This change will by default make NO changes to the task container definition when running terraform apply.

If you DO need to make changes to the task container definition for some reason, you need to manually and TEMPORARILY uncomment both of these ignore_changes lines and pass in -var="image_tag=<correct_image_tag>". This has been documented inline.

Note: It would be ideal to control this behavior with a variable that you can pass into terraform plan/terraform apply. However, lifecycle cannot handle expression evaluation or variable interpolation. 😞 See https://github.com/hashicorp/terraform/issues/3116

Testing

Screenshots, GIF demos, code examples or output to help show the changes working as expected. ProTip: you can drag and drop or paste images into this textbox.

1. terraform plan with the default should return no changes for the ECS services and ECS task definitions
2. terraform plan with only the task_definition line commented out should show that the task definitions want to update the image tags
3. terraform plan with both the task_definition and container_definitions lines commented out should show that the task definitions want to update the image tags AND the services want to point to the new task definitions