TaskDefinition replacement on every update

[kennyjwilli]

Pulumi thinks it needs to replace my aws:ecs:TaskDefinition on every up run even though nothing has changed. Looking at the diff output for the containerDefinitions parameter on the TaskDefinition, I can see the only difference is on both my containers, mountPoints, portMappings, user, and volumesFrom currently exist as empty arrays (except user which is "0") and do not exist in the desired TaskDefinition. I am not setting these properties. Below are the before and after of the containerDefinitions parameter. The JSON Diff tool is useful for comparing. This happens every time, however, I have not reproduced this issue outside of our codebase.

As a workaround, I can manually add each of the missing properties. This results in a correct diff on each update (i.e., no changes when no changes are present).

Before

[
  {
    "cpu": 2000,
    "environment": [
      {
        "name": "COMPUTE_CONFIG_URI",
        "value": "<<REDACTED>>"
      },
      {
        "name": "COMPUTE_ENV",
        "value": "<<REDACTED>>"
      },
      {
        "name": "COMPUTE_SERVICE_NAME",
        "value": "<<REDACTED>>"
      },
      {
        "name": "DD_APM_ENABLED",
        "value": "false"
      },
      {
        "name": "DD_ENV",
        "value": "<<REDACTED>>"
      },
      {
        "name": "DD_SERVICE",
        "value": "<<REDACTED>>"
      },
      {
        "name": "DD_SERVICE_NAME",
        "value": "<<REDACTED>>"
      },
      {
        "name": "DD_TRACER_LOG_LEVEL",
        "value": "info"
      },
      {
        "name": "DD_VERSION",
        "value": "<<REDACTED>>"
      },
      {
        "name": "EXTRA_JVM_OPTS",
        "value": "<<REDACTED>>"
      }
    ],
    "essential": true,
    "healthCheck": {
      "command": [
        "bash",
        "-c",
        "test $(find alive.txt -mmin -1)"
      ],
      "interval": 30,
      "retries": 3,
      "startPeriod": 30,
      "timeout": 5
    },
    "image": "<<REDACTED>>",
    "logConfiguration": {
      "logDriver": "awsfirelens",
      "options": {
        "apiKey": "<<REDACTED>>",
        "dd_service": "<<REDACTED>>",
        "dd_source": "<<REDACTED>>",
        "dd_tags": "<<REDACTED>>",
        "Host": "http-intake.logs.datadoghq.com",
        "Name": "datadog",
        "provider": "ecs",
        "TLS": "on"
      }
    },
    "memory": 3584,
    "mountPoints": [
    ],
    "name": "service",
    "portMappings": [
    ],
    "volumesFrom": [
    ]
  },
  {
    "cpu": 0,
    "environment": [
      {
        "name": "DD_APM_ENABLED",
        "value": "false"
      }
    ],
    "essential": true,
    "firelensConfiguration": {
      "options": {
        "config-file-type": "file",
        "config-file-value": "/fluent-bit/configs/parse-json.conf",
        "enable-ecs-log-metadata": "true"
      },
      "type": "fluentbit"
    },
    "image": "906394416424.dkr.ecr.us-west-2.amazonaws.com/aws-for-fluent-bit:latest",
    "logConfiguration": {
      "logDriver": "awslogs",
      "options": {
        "awslogs-group": "lp-processor-log-router-log-group-0f081b1",
        "awslogs-region": "us-west-2",
        "awslogs-stream-prefix": "log_router"
      }
    },
    "memoryReservation": 50,
    "mountPoints": [
    ],
    "name": "log_router",
    "portMappings": [
    ],
    "user": "0",
    "volumesFrom": [
    ]
  }
]

After

[
  {
    "cpu": 2000,
    "environment": [
      {
        "name": "EXTRA_JVM_OPTS",
        "value": "<<REDACTED>>"
      },
      {
        "name": "COMPUTE_CONFIG_URI",
        "value": "<<REDACTED>>"
      },
      {
        "name": "COMPUTE_SERVICE_NAME",
        "value": "<<REDACTED>>"
      },
      {
        "name": "COMPUTE_ENV",
        "value": "<<REDACTED>>"
      },
      {
        "name": "DD_SERVICE_NAME",
        "value": "<<REDACTED>>"
      },
      {
        "name": "DD_TRACER_LOG_LEVEL",
        "value": "info"
      },
      {
        "name": "DD_APM_ENABLED",
        "value": "false"
      },
      {
        "name": "DD_ENV",
        "value": "<<REDACTED>>"
      },
      {
        "name": "DD_SERVICE",
        "value": "<<REDACTED>>"
      },
      {
        "name": "DD_VERSION",
        "value": "<<REDACTED>>"
      }
    ],
    "essential": true,
    "healthCheck": {
      "command": [
        "bash",
        "-c",
        "test $(find alive.txt -mmin -1)"
      ],
      "interval": 30,
      "retries": 3,
      "startPeriod": 30,
      "timeout": 5
    },
    "image": "<<REDACTED>>",
    "logConfiguration": {
      "logDriver": "awsfirelens",
      "options": {
        "apiKey": "<<REDACTED>>",
        "dd_service": "<<REDACTED>>",
        "dd_source": "<<REDACTED>>",
        "dd_tags": "<<REDACTED>>",
        "Host": "http-intake.logs.datadoghq.com",
        "Name": "datadog",
        "provider": "ecs",
        "TLS": "on"
      }
    },
    "memory": 3584,
    "name": "service"
  },
  {
    "cpu": 0,
    "environment": [
      {
        "name": "DD_APM_ENABLED",
        "value": "false"
      }
    ],
    "essential": true,
    "firelensConfiguration": {
      "options": {
        "config-file-type": "file",
        "config-file-value": "/fluent-bit/configs/parse-json.conf",
        "enable-ecs-log-metadata": "true"
      },
      "type": "fluentbit"
    },
    "image": "906394416424.dkr.ecr.us-west-2.amazonaws.com/aws-for-fluent-bit:latest",
    "logConfiguration": {
      "logDriver": "awslogs",
      "options": {
        "awslogs-group": "lp-processor-log-router-log-group-0f081b1",
        "awslogs-region": "us-west-2",
        "awslogs-stream-prefix": "log_router"
      }
    },
    "memoryReservation": 50,
    "name": "log_router"
  }
]

Diff

6c6
<         "name": "COMPUTE_CONFIG_URI",
---
>         "name": "EXTRA_JVM_OPTS",
10c10
<         "name": "COMPUTE_ENV",
---
>         "name": "COMPUTE_CONFIG_URI",
18,26c18
<         "name": "DD_APM_ENABLED",
<         "value": "false"
<       },
<       {
<         "name": "DD_ENV",
<         "value": "<<REDACTED>>"
<       },
<       {
<         "name": "DD_SERVICE",
---
>         "name": "COMPUTE_ENV",
38c30,34
<         "name": "DD_VERSION",
---
>         "name": "DD_APM_ENABLED",
>         "value": "false"
>       },
>       {
>         "name": "DD_ENV",
42c38,42
<         "name": "EXTRA_JVM_OPTS",
---
>         "name": "DD_SERVICE",
>         "value": "<<REDACTED>>"
>       },
>       {
>         "name": "DD_VERSION",
73,79c73
<     "mountPoints": [
<     ],
<     "name": "service",
<     "portMappings": [
<     ],
<     "volumesFrom": [
<     ]
---
>     "name": "service"
108,115c102
<     "mountPoints": [
<     ],
<     "name": "log_router",
<     "portMappings": [
<     ],
<     "user": "0",
<     "volumesFrom": [
<     ]
---
>     "name": "log_router"
117c104
< ]
---
> ]
\ No newline at end of file

[lukehoban]

Can you share anything about your code? I cannot reproduce this with a simple test.

const nginx = new awsx.ecs.FargateService("nginx", {
    taskDefinitionArgs: {
        containers: {
            nginx: {
                image: "nginx",
            },
        },
    }
});

$ pulumi up
Previewing update (dev)

View Live: https://app.pulumi.com/lukehoban/tasktest/dev/previews/e581eff0-4f11-42a8-a0be-4b83bd244556

     Type                 Name          Plan     
     pulumi:pulumi:Stack  tasktest-dev           

Resources:
    25 unchanged

[kennyjwilli]

Sure. See this example code: https://gist.github.com/kennyjwilli/18920da9c2d880270dc2f92d1f6b1908. Running with these deps.

"@pulumi/pulumi": "2.16.2",
"@pulumi/kubernetes": "2.7.5",
"@pulumi/eks": "0.20.0",
"@pulumi/awsx": "0.23.0",
"@pulumi/aws": "3.22.0",```

[blampe]

@kennyjwilli I know it's been a while and this might no longer be relevant, but I also ran into this. You can see from the diff you posted that the keys were initially all sorted, and Pulumi is trying to update them to match the unsorted order defined in your code.

So... our workaround at the time was to sort the container's environment variables by name, something like

environemnt = environment.sort((a, b) => a.name.compare(b.name))

This is definitely not ideal. We didn't dig in enough to figure out exactly what was wrong. Our theory at the time was that Pulumi might be applying some JSON canonicalization during update but not preview. Edit: I remember now that JSON.stringify was also involved, which isn't guaranteed stable output and could also be contributing to the discrepancy.

[lukehoban]

I spent some time looking at this this afternoon.

I am fairly certain that replacements will only be proposed if there is an actual change. If there is no change at all, the ordering of env vars won't matter. The underlying provider takes care of taking env var ordering into account as part of deciding whether there is a diff or not. And if the provider says there is not a diff (after taking into account canonicalization which includes canonical reordernig of env vars), then a replacement will not be proposed.

In the original example above for example, I see that the second container removed the user: "0"from its definition - which is treated as a real diff by the canonicalization logic in the upstream provider. Specially - passing the before and after above through the canonicalization logic there - I get this - indicating these are viewed as a real diff on containerDefintions, which requires a replacement.

[{"cpu":2000,"environment":[{"name":"COMPUTE_CONFIG_URI","value":"<<REDACTED>>"},{"name":"COMPUTE_ENV","value":"<<REDACTED>>"},{"name":"COMPUTE_SERVICE_NAME","value":"<<REDACTED>>"},{"name":"DD_APM_ENABLED","value":"false"},{"name":"DD_ENV","value":"<<REDACTED>>"},{"name":"DD_SERVICE","value":"<<REDACTED>>"},{"name":"DD_SERVICE_NAME","value":"<<REDACTED>>"},{"name":"DD_TRACER_LOG_LEVEL","value":"info"},{"name":"DD_VERSION","value":"<<REDACTED>>"},{"name":"EXTRA_JVM_OPTS","value":"<<REDACTED>>"}],"essential":true,"healthCheck":{"command":["bash","-c","test $(find alive.txt -mmin -1)"],"interval":30,"retries":3,"startPeriod":30,"timeout":5},"image":"<<REDACTED>>","logConfiguration":{"logDriver":"awsfirelens","options":{"Host":"http-intake.logs.datadoghq.com","Name":"datadog","TLS":"on","apiKey":"<<REDACTED>>","dd_service":"<<REDACTED>>","dd_source":"<<REDACTED>>","dd_tags":"<<REDACTED>>","provider":"ecs"}},"memory":3584,"name":"service"},{"environment":[{"name":"DD_APM_ENABLED","value":"false"}],"essential":true,"firelensConfiguration":{"options":{"config-file-type":"file","config-file-value":"/fluent-bit/configs/parse-json.conf","enable-ecs-log-metadata":"true"},"type":"fluentbit"},"image":"906394416424.dkr.ecr.us-west-2.amazonaws.com/aws-for-fluent-bit:latest","logConfiguration":{"logDriver":"awslogs","options":{"awslogs-group":"lp-processor-log-router-log-group-0f081b1","awslogs-region":"us-west-2","awslogs-stream-prefix":"log_router"}},"memoryReservation":50,"name":"log_router","user":"0"}]
[{"cpu":2000,"environment":[{"name":"COMPUTE_CONFIG_URI","value":"<<REDACTED>>"},{"name":"COMPUTE_ENV","value":"<<REDACTED>>"},{"name":"COMPUTE_SERVICE_NAME","value":"<<REDACTED>>"},{"name":"DD_APM_ENABLED","value":"false"},{"name":"DD_ENV","value":"<<REDACTED>>"},{"name":"DD_SERVICE","value":"<<REDACTED>>"},{"name":"DD_SERVICE_NAME","value":"<<REDACTED>>"},{"name":"DD_TRACER_LOG_LEVEL","value":"info"},{"name":"DD_VERSION","value":"<<REDACTED>>"},{"name":"EXTRA_JVM_OPTS","value":"<<REDACTED>>"}],"essential":true,"healthCheck":{"command":["bash","-c","test $(find alive.txt -mmin -1)"],"interval":30,"retries":3,"startPeriod":30,"timeout":5},"image":"<<REDACTED>>","logConfiguration":{"logDriver":"awsfirelens","options":{"Host":"http-intake.logs.datadoghq.com","Name":"datadog","TLS":"on","apiKey":"<<REDACTED>>","dd_service":"<<REDACTED>>","dd_source":"<<REDACTED>>","dd_tags":"<<REDACTED>>","provider":"ecs"}},"memory":3584,"name":"service"},{"environment":[{"name":"DD_APM_ENABLED","value":"false"}],"essential":true,"firelensConfiguration":{"options":{"config-file-type":"file","config-file-value":"/fluent-bit/configs/parse-json.conf","enable-ecs-log-metadata":"true"},"type":"fluentbit"},"image":"906394416424.dkr.ecr.us-west-2.amazonaws.com/aws-for-fluent-bit:latest","logConfiguration":{"logDriver":"awslogs","options":{"awslogs-group":"lp-processor-log-router-log-group-0f081b1","awslogs-region":"us-west-2","awslogs-stream-prefix":"log_router"}},"memoryReservation":50,"name":"log_router"}]

It's plausible that this addition of user: "0" itself should not trigger a diff - but I'm not certain. The documentation on the precise defaults for user is not readily clear in ECS docs.

All that said, there is a second issue which makes this more confusing. Once the diff (and replacement) are identified, Pulumi then renders what it believes the diff to be back to the user. When it does that, it does not account for the canonicalization that was used to determine whether a diff was required or not. So it renders the diff in terms of the original string inputs - which may have had a variety of differences that don't matter to the canonicalized diff logic - but there will be at least one difference which does matter. This one difference will be hard to find amongst the noise though.

Notably, the upstream provider has specific logic to handle the noise of env var reordering in this exact condition:

                StateFunc: func(v interface{}) string {
                    // Sort the lists of environment variables as they are serialized to state, so we won't get
                    // spurious reorderings in plans (diff is suppressed if the environment variables haven't changed,
                    // but they still show in the plan if some other property changes).
                    orderedCDs, _ := expandContainerDefinitions(v.(string))
                    containerDefinitions(orderedCDs).OrderEnvironmentVariables()
                    unnormalizedJson, _ := flattenContainerDefinitions(orderedCDs)
                    json, _ := structure.NormalizeJsonString(unnormalizedJson)
                    return json
                },

And Pulumi does invoke that state func - which leads to the following in the Pulumi stat file:

{
                "urn": "urn:pulumi:dev::ecsreplace::awsx:ecs:FargateService$awsx:ecs:FargateTaskDefinition$aws:ecs/taskDefinition:TaskDefinition::service",
                "custom": true,
                "id": "service-cf4359e8",
                "type": "aws:ecs/taskDefinition:TaskDefinition",
                "inputs": {
                    "__defaults": [
                        "skipDestroy"
                    ],
                    "containerDefinitions": "[{\"cpu\":512,\"environment\":[{\"name\":\"FOO\",\"value\":\"BAR\"},{\"name\":\"BAZ\",\"value\":\"BAR2\"},{\"name\":\"ZAB\",\"value\":\"BAR3\"}],\"essential\":true,\"image\":\"153052959999.dkr.ecr.us-west-2.amazonaws.com/repo-c7a2477:0d5bf3701e4847eaa6faff0ff8c8e7c26b34dc32fc8bd50b7aceb04b174d81e6\",\"memory\":128,\"portMappings\":[{\"containerPort\":80,\"hostPort\":80}],\"name\":\"container\",\"logConfiguration\":{\"logDriver\":\"awslogs\",\"options\":{\"awslogs-group\":\"service-81e1d63\",\"awslogs-region\":\"us-west-2\",\"awslogs-stream-prefix\":\"container\"}}}]",
                    "cpu": "512",
                    "executionRoleArn": "arn:aws:iam::153052959999:role/service-execution-d3a7c84",
                    "family": "service-cf4359e8",
                    "memory": "1024",
                    "networkMode": "awsvpc",
                    "requiresCompatibilities": [
                        "FARGATE"
                    ],
                    "skipDestroy": false,
                    "taskRoleArn": "arn:aws:iam::153052959999:role/service-task-4338deb"
                },
                "outputs": {
                    "__meta": "{\"schema_version\":\"1\"}",
                    "arn": "arn:aws:ecs:us-west-2:153052959999:task-definition/service-cf4359e8:1",
                    "containerDefinitions": "[{\"cpu\":512,\"environment\":[{\"name\":\"BAZ\",\"value\":\"BAR2\"},{\"name\":\"FOO\",\"value\":\"BAR\"},{\"name\":\"ZAB\",\"value\":\"BAR3\"}],\"essential\":true,\"image\":\"153052959999.dkr.ecr.us-west-2.amazonaws.com/repo-c7a2477:0d5bf3701e4847eaa6faff0ff8c8e7c26b34dc32fc8bd50b7aceb04b174d81e6\",\"logConfiguration\":{\"logDriver\":\"awslogs\",\"options\":{\"awslogs-group\":\"service-81e1d63\",\"awslogs-region\":\"us-west-2\",\"awslogs-stream-prefix\":\"container\"}},\"memory\":128,\"mountPoints\":[],\"name\":\"container\",\"portMappings\":[{\"containerPort\":80,\"hostPort\":80,\"protocol\":\"tcp\"}],\"volumesFrom\":[]}]",
                    "cpu": "512",
                    "ephemeralStorage": null,
                    "executionRoleArn": "arn:aws:iam::153052959999:role/service-execution-d3a7c84",
                    "family": "service-cf4359e8",
                    "id": "service-cf4359e8",
                    "inferenceAccelerators": [],
                    "ipcMode": "",
                    "memory": "1024",
                    "networkMode": "awsvpc",
                    "pidMode": "",
                    "placementConstraints": [],
                    "proxyConfiguration": null,
                    "requiresCompatibilities": [
                        "FARGATE"
                    ],
                    "revision": 1,
                    "runtimePlatform": null,
                    "skipDestroy": false,
                    "tags": {},
                    "tagsAll": {},
                    "taskRoleArn": "arn:aws:iam::153052959999:role/service-task-4338deb",
                    "volumes": []
                },
               ...
            },

Note that the outputs is stored in canonicalized order, but the inputs are not. Even though Pulumi only uses the outputs for future diffs, it does use the inputs for presenting the diff to users. And those inputs do not include the processing of the StateFunc. This is different than Terraform, and means that the visual display of the diff still contains this env var reordering noise, even though it would not for the equivalent program in Terraform. So there is a unique-to-Pulumi diff rendering problem, but I am fairly sure there is not a unique-to-Pulumi replacement issue.

[Buthrakaur]

Few years passed but I'm facing the same issue - I tried to sort the environment variables by name as a workaround but it didn't help. Pulumi creates a new task definition which triggers ECS re-deployment with every pulumi up even though I didn't change anything in the code. When I download JSON files for the last 2 task definition revisions from the AWS console and make a diff there's no real difference:

git diff --no-index ecs-task-revision54.json ecs-task-revision55.json
@@ -1,5 +1,5 @@
 {
-    "taskDefinitionArn": "arn:aws:ecs:us-east-1:665047452217:task-definition/xxx-api-ecs-task:54",
+    "taskDefinitionArn": "arn:aws:ecs:us-east-1:665047452217:task-definition/xxx-api-ecs-task:55",
     "containerDefinitions": [
         {
             "name": "app",
@@ -152,7 +152,7 @@
     "family": "xxx-api-ecs-task",
     "executionRoleArn": "arn:aws:iam::xxx:role/xxx-api-ecs-task-execution-role-1c48f41",
     "networkMode": "awsvpc",
-    "revision": 54,
+    "revision": 55,
     "volumes": [],
     "status": "ACTIVE",
     "requiresAttributes": [
@@ -191,7 +191,7 @@
     "compatibilities": [
         "EC2"
     ],
-    "registeredAt": "2024-08-20T13:56:03.120Z",
+    "registeredAt": "2024-08-20T14:00:26.220Z",
     "registeredBy": "arn:aws:iam::xxx",
     "tags": [
         {

I added logging into my pulumi code for the resulting TaskDefinitionArgs.containerDefinitions JSON string value and there is no difference too.

I'm using pulumi.output to produce the task definition JSON - could that make a difference?

const containerDefinitions = pulumi.output(
    pulumi.all([app_db_connstr.id, app_analytics_db_connstr.id, app_jwt_token_security_key.id])
).apply(([app_db_connstr, app_analytics_db_connstr, app_jwt_token_security_key]) => {
    const containerDefinition: aws.ecs.ContainerDefinition = {
        name: ecsTaskContainerName,
        image: stackConfig.require('ecs_task_image'),
        memory: stackConfig.requireNumber('ecs_task_memory'),
        cpu: stackConfig.requireNumber('ecs_task_cpu'),
        essential: true,
        portMappings: [{ containerPort: ecsTaskContainerPort }],
        secrets: [
            {name: 'ConnectionStrings:DB', valueFrom: app_db_connstr},
            {name: 'ConnectionStrings:Analytics', valueFrom: app_analytics_db_connstr},
            {name: 'Security:JwtTokenSecurityKey', valueFrom: app_jwt_token_security_key},
        ],
        environment: [
            {
                "name": "ASPNETCORE_ENVIRONMENT",
                "value": stackConfig.require('app_net_environment'),
            },
            {
                name: 'ASPNETCORE_URLS',
                value: 'http://*:80',
            },
            // 10+ more here
        ],
        logConfiguration: {
            logDriver: "awslogs",
            options: {
                "awslogs-group": appLogGroupName, 
                "awslogs-region": region,
                "awslogs-stream-prefix": stackConfig.require('app_log_stream_prefix'),
            }
        },
        healthCheck: {
            command: ["CMD-SHELL", `curl -f http://localhost${stackConfig.require('app_healthcheck_url')} || exit 1`],
            interval: 30,
            timeout: 5,
            retries: 3,
            startPeriod: 30,
        },
    };

    containerDefinition.environment = containerDefinition.environment?.sort((a, b) => `${a.name}`.localeCompare(`${b.name}`))

    const jsonString = JSON.stringify([containerDefinition]);

    console.log('ECS task definition', jsonString); // to diff the JSONs between runs => no difference

    return jsonString;
});

const ecsTaskDefinition = new aws.ecs.TaskDefinition(name('xxx-api-ecs-task'), {
    networkMode: 'awsvpc',
    family: name('xxx-api-ecs-task'),
    containerDefinitions,
    executionRoleArn: ecsTaskExecutionRole.arn
}, { provider });

Does anyone have more insight into this or any potential workaround I could try? Is this actually a bug in Pulumi or terraform? aving ECS trigger re-deploy of the whole service with every pulumi up is quite a significant issue :(

[Buthrakaur]

my fault - it was actually caused by a duplicate environment variable 🤦‍♂️

[myhr-chman]

Wanted to share another case we encountered today, in case it helps 🙂 .

If environment variable resolves to null (or undefined) in the container definition, AWS doesn't add the null-ified variable to the task definition, hence always needing a replacement.

new aws.ecs.TaskDefinition(
  'example-task-definition',
  {
    family: 'example-service',
    containerDefinitions: JSON.stringify([
      {
        name: 'example-service',
        image: '<<REDACTED>>',
        environment: [{
          name: 'EXAMPLE_VARIABLE',
          value: null
        }],
      },
    ]),
  }
)

Ref 1. Not quite working minimal example of null-valued variable.

[t0yv0]

Thanks for this repro! If this reproduces still this can be fixable by improving diff normalizers for TaskDefinition in pulumi-aws provider (or upstream TF provider if applicable). Moving to pulumi-aws I think this is where the fix needs to happen.

[flostadler]

I was able to repro it. Upstream is not correctly handling environment variables that have a null value. This could be done in the StateFunc of the container_defintions. Right now that one is just sorting the env variables but it should also filter out nulls: https://github.com/hashicorp/terraform-provider-aws/blob/465d0690625e474010378042164d42595e617d1a/internal/service/ecs/container_definitions.go#L146

The intended behavior can be copied from the AWS Batch integration for ECS, that one is already filtering out nulls: https://github.com/hashicorp/terraform-provider-aws/blob/465d0690625e474010378042164d42595e617d1a/internal/service/batch/ecs_properties.go#L80-L95

What's notable is that this also happens for environment variables set to an empty string

[flostadler]

I opened an upstream issue to track this: https://github.com/hashicorp/terraform-provider-aws/issues/39533