Change in aws_ssm_parameter does not immediately result in dependencies/references being updated.

[leynebe]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

$ tf --version Terraform v1.0.4 on linux_amd64

• provider registry.terraform.io/hashicorp/aws v3.54.0
• provider registry.terraform.io/hashicorp/external v2.1.0
• provider registry.terraform.io/hashicorp/local v2.1.0
• provider registry.terraform.io/hashicorp/tls v3.1.0
• provider registry.terraform.io/mongodb/mongodbatlas v0.9.1

Affected Resource(s)

• aws_ssm_parameter
• aws_ecs_task_definition
• aws_ecs_service

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

resource "aws_ssm_parameter" "tmp" {
  name      = "/services/web-service/tmp"
  overwrite = true
  type      = "String"
  value     = "foobar"
}

resource "aws_ecs_cluster" "cluster" {
  name = "bentest-web-cluster"
}

resource "aws_ecs_service" "web-service" {
  name                               = "web-service"
  cluster                            = aws_ecs_cluster.cluster.arn
  task_definition                    = aws_ecs_task_definition.web-service.arn
  deployment_minimum_healthy_percent = 100
  deployment_maximum_percent         = 200
  launch_type                        = "EC2"
  scheduling_strategy                = "REPLICA"
  desired_count                      = "1"
}

resource "aws_iam_role" "execution-role" {
  name               = "ecs-task-web-service"
  assume_role_policy = data.aws_iam_policy_document.assume-role.json
}

data "aws_iam_policy_document" "assume-role" {
  statement {
    actions = ["sts:AssumeRole"]

    principals {
      type        = "Service"
      identifiers = ["ecs-tasks.amazonaws.com"]
    }
  }
}

resource "aws_ecs_task_definition" "web-service" {
  family                = "web-service"
  execution_role_arn    = aws_iam_role.execution-role.arn
  container_definitions = <<EOF
[{
  "name": "web-service",
  "image": "httpd:latest",
  "essential": true,
  "cpu": 128,
  "memory": 500,
  "memoryReservation": 100,
  "portMappings": [
    {
      "hostPort": 0,
      "protocol": "tcp",
      "containerPort": 80
    }
  ],
  "environment": [
    {
      "name": "SSM_PARAMETER_VERSIONS_TMP",
      "value": "${base64encode(aws_ssm_parameter.tmp.version)}"
    }
  ],
  "secrets": [
    {
      "valueFrom": "/services/web-service/tmp",
      "name": "TMP_SECRET"
    }
  ]
}]
EOF
}

Debug Output

/

Panic Output

/

Expected Behavior

When the aws_ssm_parameter is changed, terraform expects a change. Since there is a reference of the aws_ssm_parameter version in aws_ecs_task_definition I would expect the first apply to contain a recreation of the aws_ecs_task_definition (and a change to the aws_ecs_service) resulting in a redeployment of the underlying web service task.

Actual Behavior

The first plan/apply lists changes to the aws_ssm_parameter:

Terraform will perform the following actions:

  # aws_ssm_parameter.tmp will be updated in-place
  ~ resource "aws_ssm_parameter" "tmp" {
        id        = "/services/web-service/tmp"
        name      = "/services/web-service/tmp"
        tags      = {}
      ~ value     = (sensitive value)
        # (7 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

It is only in the second plan/apply that terraform sees changes in the aws_ecs_task_definition:

Terraform will perform the following actions:

  # aws_ecs_service.web-service will be updated in-place
  ~ resource "aws_ecs_service" "web-service" {
        id                                 = "arn:aws:ecs:eu-west-1:XXX:service/bentest-web-cluster/web-service"
        name                               = "web-service"
        tags                               = {}
      ~ task_definition                    = "arn:aws:ecs:eu-west-1:XXX:task-definition/web-service:1" -> (known after apply)
        # (12 unchanged attributes hidden)

        # (2 unchanged blocks hidden)
    }

  # aws_ecs_task_definition.web-service must be replaced
-/+ resource "aws_ecs_task_definition" "web-service" {
      ~ arn                      = "arn:aws:ecs:eu-west-1:XXX:task-definition/web-service:1" -> (known after apply)
      ~ container_definitions    = jsonencode(
          ~ [ # forces replacement
              ~ {
                  ~ environment       = [
                      ~ {
                          ~ value = "MQ==" -> "Mg=="
                            # (1 unchanged element hidden)
                        },
                    ]
                  - mountPoints       = [] -> null
                  - volumesFrom       = [] -> null
                    # (8 unchanged elements hidden)
                } # forces replacement,
            ]
        )
      ~ id                       = "web-service" -> (known after apply)
      + network_mode             = (known after apply)
      - requires_compatibilities = [] -> null
      ~ revision                 = 1 -> (known after apply)
      - tags                     = {} -> null
      ~ tags_all                 = {} -> (known after apply)
        # (2 unchanged attributes hidden)
    }

Plan: 1 to add, 1 to change, 1 to destroy.

Steps to Reproduce

1. terraform apply
2. Change the aws_ssm_parameter value.
3. terraform apply
4. terraform apply

Important Factoids

/

References

/

Note

• Adding the following dependency code to the aws_ecs_task_definition also does nothing.

  depends_on = [aws_ssm_parameter.tmp]

• Using "${base64encode(aws_ssm_parameter.tmp.value)}" instead of "${base64encode(aws_ssm_parameter.tmp.version)}" does work, but I explicitly do not want sensitive values in the environment variables, which is why I'm using version. Is there any way I can force Terraform to see the version change to be as important as the value change?

[ei-grad]

Using "${base64encode(aws_ssm_parameter.tmp.value)}" instead of "${base64encode(aws_ssm_parameter.tmp.version)}" does work, but I explicitly do not want sensitive values in the environment variables, which is why I'm using version.

Currently there could be a workaround using the triggers block, where it should be good to use the .value of ssm parameter (since/if this parameter value is managed by terraform). Also, it could be a good idea to use the hash of the value.

But the issue is still actual for current aws provider versions, and I think it is a bug (maybe in terraform itself?).

[ei-grad]

This line sounds strange for me - https://github.com/hashicorp/terraform-provider-aws/blob/b524557/internal/service/ecs/service.go#L700

[ei-grad]

But using this workaround with the triggers block I receive "Error: Provider produced inconsistent final plan"

[apparentlymart]

As someone who is familiar with Terraform's design but not so familiar with SSM, the way I'm interpreting this issue is that updating the value of an SSM Parameter causes other attributes of that object (versions here, but maybe also others?) to be implicitly updated as a side-effect, but the provider isn't accurately modeling that behavior and so Terraform can't react to it when planning other resource instances downstream.

However, I do see some logic in the current provider code that seems like it should model this behavior correctly:

https://github.com/hashicorp/terraform-provider-aws/blob/dced8e21ec95ec4dd10cbe2e8e248bd55aece4f6/internal/service/ssm/parameter.go#L125-L142

Specifically the one that works with names.AttrVersion seems to be expressing the rule "if value has changed then version's value will be decided during the apply phase". That should cause the downstream resource shown in the original example to propose the following change during the first plan:

  value = "MQ==" -> (known after apply)

The logic I linked to seems to have been introduced in https://github.com/hashicorp/terraform-provider-aws/pull/22522, whose effect was released in v3.72.0, and so that's a later release than the one that was current when this issue was opened. Therefore it seems like that PR has fixed this issue but this issue remained open because it was later duplicated by https://github.com/hashicorp/terraform-provider-aws/issues/12213.