Timeout deleting `aws_ecs_capacity_provider` if it falls into `DELETE_FAILED` state due to internal AWS failure

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

terraform CLI: v1.1.7
AWS provider: v4.8.0

Affected Resource(s)

aws_ecs_capacity_provider

Terraform Configuration Files

resource "aws_ecs_capacity_provider" "this" {
  name  = "elastio-vault-black--d-_jack-1-shard-variable-s0-capacity-provider"

  auto_scaling_group_provider {
    auto_scaling_group_arn = aws_autoscaling_group.this.arn
    managed_termination_protection = "ENABLED"
    managed_scaling {
      status          = "ENABLED"
      target_capacity = 100
    }
  }
}

The configuration of the aws_autoscaling_group is probably not relevant, but it is rather huge in our application and requires a lot of other resources to be defined. If you want to reproduce the exact same stack, you can get the full sources of our stack in .tar format from our public s3 bucket.

Debug Output

We run terraform destroy in automation without TF_LOG=debug. So here are the best logs we can provide

Part of terraform logs + automation script error output

``` module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 1m0s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 1m10s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 1m20s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 1m30s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 1m40s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 1m50s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 2m0s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 2m10s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 2m20s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 2m30s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 2m40s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 2m50s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 3m0s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 3m10s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 3m20s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 3m30s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 3m40s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 3m50s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 4m0s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 4m10s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 4m20s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 4m30s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 4m40s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 4m50s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 5m0s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 5m10s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 5m20s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 5m30s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 5m40s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 5m50s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 6m0s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 6m10s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 6m20s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 6m30s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 6m40s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 6m50s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 7m0s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 7m10s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 7m20s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 7m30s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 7m40s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 7m50s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 8m0s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 8m10s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 8m20s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 8m30s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 8m40s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 8m50s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 9m0s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 9m10s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 9m20s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 9m30s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 9m40s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 9m50s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 10m0s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 10m10s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 10m20s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 10m30s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 10m40s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 10m50s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 11m0s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 11m10s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 11m20s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 11m30s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 11m40s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 11m50s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 12m0s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 12m10s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 12m20s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 12m30s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 12m40s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 12m50s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 13m0s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 13m10s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 13m20s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 13m30s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 13m40s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 13m50s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 14m0s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 14m10s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 14m20s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 14m30s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 14m40s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 14m50s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 15m0s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 15m10s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 15m20s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 15m30s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 15m40s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 15m50s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 16m0s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 16m10s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 16m20s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 16m30s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 16m40s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 16m50s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 17m0s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 17m10s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 17m20s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 17m30s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 17m40s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 17m50s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 18m0s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 18m10s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 18m20s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 18m30s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 18m40s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 18m50s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 19m0s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 19m10s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 19m20s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 19m30s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 19m40s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 19m50s elapsed] module.vault.module.inner.module.provisioner[0].module.capacity_providers.aws_ecs_capacity_provider.scalez_capacity_provider[1]: Still destroying... [id=arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capa...-1-shard-variable-s0-capacity-provider, 20m0s elapsed] ╷ │Error: error waiting for ECS Capacity Provider (arn:aws:ecs:us-east-2:ACCOUNT_ID_REDACTED:capacity-provider/elastio-vault-black--d-_jack-1-shard-variable-s0-capacity-provider) to delete: timeout while waiting for resource to be gone (last state: 'ACTIVE', timeout: 20m0s) │ │ ╵ Releasing state lock. This may take a few moments... 0: Failed to destroy vault resources 1: Command failed 2: Child process (pid Some(20828)) has returned an error exit code `exit status: 1`, command: Cmd( 2: CmdShared { 2: bin: "/home/elastio/.cache/elastio-terraform-1.1.7/terraform", 2: args: [ 2: "destroy", 2: "-lock-timeout", 2: "5m", 2: "-auto-approve", 2: "-input=false", 2: "-backup=-", 2: "-var", 2: "aws_region=us-east-2", 2: "-var", 2: "stack_env=ci", 2: "-var", 2: "acc_cfn_stack_region=us-east-2", 2: "-var", 2: "acc_cfn_stack_name=elastio-account-level-stack", 2: "-var", 2: "git_branch=staging", 2: "-var", 2: "ci_run_number=3379", 2: "-var", 2: "vault_name=black.-_jack-1", 2: "-var", 2: "vpc_id=VPC_ID_REDACTED", 2: ], 2: env: { 2: "TF_IN_AUTOMATION": "1", 2: }, 2: stdin: Null, 2: current_dir: Some( 2: "/home/elastio/.local/share/.elastio/elastio-terraform/deployment/vault", 2: ), 2: }, 2: ) ```

Expected Behavior

AWS provider should retry deleting aws_ecs_capacity_provider when it enters DELETE_FAILED update status with

{
    "updateStatusReason": "The capacity provider cannot be deleted due to an internal error. Try again."
}

Actual Behavior

The deletion times out after 20 minutes and the capacity provider is left in hanging in DELETE_FAILED state.

Unfortunately, we delete the terraform state even when errors happens during terraform destroy on our CI. I manually created a terraform source file with this specific capacity provider and imported it into the state. I however used the latest terraform provider version at the time of this writing instead (4.17.1). Deleting this provider with terraform destroy succeded, however I think that retrying the delete with 4.8.0 would have also deleted it.

The output of

aws ecs describe-capacity-providers --include=TAGS

after the timed out terraform destroy:

Output of describe-capacity-providers

```json { "capacityProviders": [ { "capacityProviderArn": "arn:aws:ecs:us-east-2:ACCOUNT_REDACTED:capacity-provider/FARGATE", "name": "FARGATE", "status": "ACTIVE", "tags": [] }, { "capacityProviderArn": "arn:aws:ecs:us-east-2:ACCOUNT_REDACTED:capacity-provider/FARGATE_SPOT", "name": "FARGATE_SPOT", "status": "ACTIVE", "tags": [] }, { "capacityProviderArn": "arn:aws:ecs:us-east-2:ACCOUNT_REDACTED:capacity-provider/elastio-vault-black--d-_jack-1-shard-variable-s0-capacity-provider", "name": "elastio-vault-black--d-_jack-1-shard-variable-s0-capacity-provider", "status": "ACTIVE", "autoScalingGroupProvider": { "autoScalingGroupArn": "arn:aws:autoscaling:us-east-2:ACCOUNT_REDACTED:autoScalingGroup:181b0c14-0482-45e0-8bc2-eb1acb9e07ee:autoScalingGroupName/elastio-vault-black--d-_jack-1-shard-variable-s0-autoscaling-group", "managedScaling": { "status": "ENABLED", "targetCapacity": 100, "minimumScalingStepSize": 1, "maximumScalingStepSize": 10000, "instanceWarmupPeriod": 300 }, "managedTerminationProtection": "ENABLED" }, "updateStatus": "DELETE_FAILED", "updateStatusReason": "The capacity provider cannot be deleted due to an internal error. Try again.", "tags": [] } ], "failures": [] } ```

Steps to Reproduce

It's difficult to reproduce this problem because it requires that the capacity provider fails to be deleted due to an inter error in AWS. However, I'll write the scenario either way

Deploy a sample ECS service with a capacity provider as specified in the code above with terraform apply
Run terraform destroy
Cross your fingers to make sure that AWS fails to delete the capacity provider and puts it into DELETE_FAILED state
Observe that deleting the capacity provider times out

hashicorp / terraform-provider-aws