AWS automatic updates cause churn on terraform plan

[mball-agathos]

Description

If AWS updates the images used in the autoscaling groups, it causes churn on the resulting terraform plan. Here is an example of the output with no terraform changes, but after AWS makes an image update behind the scenes:

  # module.ecs-cluster.module.asg.aws_autoscaling_group.idc[0] will be updated in-place
  ~ resource "aws_autoscaling_group" "idc" {
        id                               = "my-autoscaling-group-20230208085230581500000006"
        name                             = "my-autoscaling-group-20230208085230581500000006"
        # (27 unchanged attributes hidden)

      ~ launch_template {
            id      = "lt-03fa51b97f2afa7cf"
            name    = "ecs-node-my-autoscaling-group-20230208085229776700000004"
          ~ version = "19" -> (known after apply)
        }

        # (4 unchanged blocks hidden)
    }

  # module.ecs-cluster.module.asg.aws_launch_template.this[0] will be updated in-place
  ~ resource "aws_launch_template" "this" {
      ~ default_version         = 19 -> (known after apply)
        id                      = "lt-03fa51b97f2afa7cf"
      ~ image_id                = "ami-0d6d367ace19e09dd" -> "ami-0b7778d86d8789cff"
      ~ latest_version          = 19 -> (known after apply)
        name                    = "ecs-node-my-autoscaling-group-20230208085229776700000004"
        tags                    = {
            "AmazonECSManaged" = "true"
        }
        # (11 unchanged attributes hidden)

        # (3 unchanged blocks hidden)
    }

Versions

• Module version [Required]:

• Terraform version: Terraform v1.5.7

• Provider version(s): provider registry.terraform.io/hashicorp/aws v5.23.1

Reproduction Code [Required]

Steps to reproduce the behavior:

1. Instantiate the terraform-aws-modules/autoscaling/aws module.
2. Apply this module
3. Wait for AWS to update the image
4. Note the churn in terraform plan

Expected behavior

This should have a clean plan with no terraform changes

Actual behavior

Several fields in resources used by the terraform-aws-modules/autoscaling/aws module show churn.

Additional context

A really straight-forward solution would be something like adding the fields in the terraform plan to lifecycle { ignore_changes }. I would do that myself, except that Terraform doesn't support adding lifecycle to a module. It would be great if it was possible to just ignore those here so that everyone benefits from seeing reduced churn!

[bryantbiggs]

I don't know if that would make sense - you can specify the AMI ID you wish to use and then only update that when you are ready to update the instances

[mball-agathos]

Is it possible to differentiate between when the AMI is specified directly Terraform and when it's unspecified and AWS picks whatever version it wants? It's this second case where I'm hoping to avoid seeing this churn.

[bryantbiggs]

AWS does not specify "whatever version it wants" for an AMI ID - the AMI ID is the version, its immutable

[mball-agathos]

From reading through the autoscaling_group documentation a little more, it appears that this churn is somewhat by design as a way to make it visible that a new version is available. It would in theory be possible to set launch_template_version to $Latest, but I see this note:

A refresh will not start when version = "$Latest" is configured in the launch_template block. To trigger the instance refresh when a launch template is changed, configure version to use the latest_version attribute of the aws_launch_template resource.

As such, it's necessary to match the aws_autoscaling_group->launch_template->version to that of the corresponding aws_launch_template.

In digging a little further into our config, it looks like we're actually passing in the aws_launch_template->image_id var by getting the latest AWS image_id as defined in this Parameter Store key: /aws/service/ecs/optimized-ami/amazon-linux-2/recommended/image_id.

So basically, whenever AWS updates their latest recommend image ID, terraform reads this value and then updates that in the aws_launch_template, which then causes a new version to get created, which in turn updates the aws_autoscaling_group.

So, in short, it looks like this churn is actually a side-effect of us updating to the latest AWS recommend AMI, which requires a terraform apply for it to take effect. I'll go ahead and close this as "working as intended".

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.