search

hashicorp / terraform-provider-aws

The AWS Provider enables Terraform to manage AWS resources.
https://registry.terraform.io/providers/hashicorp/aws
Mozilla Public License 2.0
9.8k stars 9.15k forks source link

Terraform doesn't destroy instance profile from iam role #22111

Open yalattas opened 2 years ago

yalattas commented 2 years ago

Community Note

Terraform CLI and Terraform AWS Provider Version

Affected Resource(s)

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

terraform {
    required_providers {
    aws = {
        source  = "hashicorp/aws"
        version = "~> 3.27"
        }
    }
    required_version = ">= 0.14.9"
}
resource "aws_iam_instance_profile" "beanstalk_ec2_profile_role"{
    name = format("%s-%s-beanstalk-ec2-profile-role", var.app_name, var.purpose_tag)
    role = aws_iam_role.beanstalk_ec2_role.name
}
resource "aws_iam_role" "beanstalk_ec2_role"{
    name = format("%s-%s-beanstalk-ec2-role", var.app_name, var.purpose_tag)
    # attach with AWS managed policies
    managed_policy_arns = [
        data.aws_iam_policy.beanstalk_web_tier_policy.arn,
        data.aws_iam_policy.beanstalk_multi_conatiner_policy.arn,
        data.aws_iam_policy.beanstalk_worker_tier_policy.arn
    ]
    # associate as trusted entity with EC2 (allowing it to be associated with EC2)
    assume_role_policy = jsonencode(
        {
            "Version": "2008-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Principal": {
                        "Service": "ec2.amazonaws.com"
                    },
                    "Action": "sts:AssumeRole"
                }
            ]
        }
    )
}

Debug Output

Panic Output

Expected Behavior

Terraform is destroy all resources except instance profile. It should remove associated profile using the following command from CLI or via AWS API

aws iam delete-instance-profile --delete-instance-profile <profile_name>

Actual Behavior

Here are the results from AWS CLI after destroying the infrastructure

"InstanceProfiles": [
        {
            "Path": "/",
            "InstanceProfileName": "demo-spot-beanstalk-ec2-profile-role",
            "InstanceProfileId": "****************RB",
            "Arn": "arn:aws:iam::<account_ID>:instance-profile/demo-spot-beanstalk-ec2-profile-role",
            "CreateDate": "2021-12-08T13:00:52+00:00",
            "Roles": []
        }
]

Steps to Reproduce

related to beanstalk roles

  1. write terraform script to create the following. Make sure to have a fixed name for both roles
    • aws_iam_role
    • aws_iam_instance_profile
  2. terraform apply
  3. terraform destroy
  4. terraform apply

    Important Factoids

References

justinretzolk commented 2 years ago

Hey @yalattas 👋 Thank you for taking the time to file this. So that we have all of the necessary information in order to reproduce, can you supply debug logs as well?

aarowman commented 2 years ago

running into this same issue as well

ARashitov commented 2 years ago

@justinretzolk Hey! I running into the same issue, can provide inputs for this task.

Terraform configs

terraform {
  required_version = ">= 1.0.8, < 2.0.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "= 4.9.0"
    }
  }
}

Terraform destroy & Terraform apply

(research_kedro_microservice) ➜  aws_batch git:(iss-3-aws-infrastructure) ✗ make destroy
terraform destroy
aws_iam_role.ecs_instance_role: Refreshing state... [id=test-research-kedro-microservice-aws-iam-role]
aws_iam_role_policy_attachment.ecs_instance_role: Refreshing state... [id=test-research-kedro-microservice-aws-iam-role-20220614120738864000000001]
aws_iam_instance_profile.ecs_instance_role: Refreshing state... [id=test-research-kedro-microservice-aws-iam-instance-profile]

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply":

  # aws_iam_role.ecs_instance_role has been changed
  ~ resource "aws_iam_role" "ecs_instance_role" {
        id                    = "test-research-kedro-microservice-aws-iam-role"
      ~ managed_policy_arns   = [
          + "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role",
        ]
        name                  = "test-research-kedro-microservice-aws-iam-role"
        tags                  = {
            "environment"  = "test"
            "project_name" = "research-kedro-microservice"
            "region"       = "us-east-2"
        }
        # (8 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan may include actions to undo or respond to these changes.

────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  # aws_iam_instance_profile.ecs_instance_role will be destroyed
  - resource "aws_iam_instance_profile" "ecs_instance_role" {
      - arn         = "arn:aws:iam::946627858531:instance-profile/test-research-kedro-microservice-aws-iam-instance-profile" -> null
      - create_date = "2022-06-14T12:07:38Z" -> null
      - id          = "test-research-kedro-microservice-aws-iam-instance-profile" -> null
      - name        = "test-research-kedro-microservice-aws-iam-instance-profile" -> null
      - path        = "/" -> null
      - role        = "test-research-kedro-microservice-aws-iam-role" -> null
      - tags        = {
          - "environment"  = "test"
          - "project_name" = "research-kedro-microservice"
          - "region"       = "us-east-2"
        } -> null
      - tags_all    = {
          - "environment"  = "test"
          - "project_name" = "research-kedro-microservice"
          - "region"       = "us-east-2"
        } -> null
      - unique_id   = "AIPA5YZ3K2BR6ZSWECQDF" -> null
    }

  # aws_iam_role.ecs_instance_role will be destroyed
  - resource "aws_iam_role" "ecs_instance_role" {
      - arn                   = "arn:aws:iam::946627858531:role/test-research-kedro-microservice-aws-iam-role" -> null
      - assume_role_policy    = jsonencode(
            {
              - Statement = [
                  - {
                      - Action    = "sts:AssumeRole"
                      - Effect    = "Allow"
                      - Principal = {
                          - Service = "ec2.amazonaws.com"
                        }
                    },
                ]
              - Version   = "2012-10-17"
            }
        ) -> null
      - create_date           = "2022-06-14T12:07:37Z" -> null
      - force_detach_policies = false -> null
      - id                    = "test-research-kedro-microservice-aws-iam-role" -> null
      - managed_policy_arns   = [
          - "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role",
        ] -> null
      - max_session_duration  = 3600 -> null
      - name                  = "test-research-kedro-microservice-aws-iam-role" -> null
      - path                  = "/" -> null
      - tags                  = {
          - "environment"  = "test"
          - "project_name" = "research-kedro-microservice"
          - "region"       = "us-east-2"
        } -> null
      - tags_all              = {
          - "environment"  = "test"
          - "project_name" = "research-kedro-microservice"
          - "region"       = "us-east-2"
        } -> null
      - unique_id             = "AROA5YZ3K2BR45A6YXLU6" -> null

      - inline_policy {}
    }

  # aws_iam_role_policy_attachment.ecs_instance_role will be destroyed
  - resource "aws_iam_role_policy_attachment" "ecs_instance_role" {
      - id         = "test-research-kedro-microservice-aws-iam-role-20220614120738864000000001" -> null
      - policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role" -> null
      - role       = "test-research-kedro-microservice-aws-iam-role" -> null
    }

Plan: 0 to add, 0 to change, 3 to destroy.

Changes to Outputs:
  - aws_batch_env_subnets_ids = [
      - "subnet-0ffc295914ee8f62e",
      - "subnet-0644a889bb64f3e2c",
    ] -> null
  - aws_batch_subnets_ids     = [
      - "subnet-0ffc295914ee8f62e",
      - "subnet-0644a889bb64f3e2c",
      - "subnet-018a09b8964cddeb1",
      - "subnet-0c9ffb7ee119d4539",
      - "subnet-050fde753140abc0b",
      - "subnet-04d08097f9b5e19a7",
    ] -> null
  - ce_instance_type          = [
      - "c5a.large",
      - "c5a.xlarge",
    ] -> null
  - docker_image_id           = "946627858531.dkr.ecr.us-east-2.amazonaws.com/research-kedro-microservice:latest" -> null
  - ecr_repository_url        = "946627858531.dkr.ecr.us-east-2.amazonaws.com/research-kedro-microservice" -> null
  - env                       = "test" -> null
  - project_name              = "research-kedro-microservice" -> null

Do you really want to destroy all resources in workspace "test"?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

aws_iam_role_policy_attachment.ecs_instance_role: Destroying... [id=test-research-kedro-microservice-aws-iam-role-20220614120738864000000001]
aws_iam_instance_profile.ecs_instance_role: Destroying... [id=test-research-kedro-microservice-aws-iam-instance-profile]
aws_iam_role_policy_attachment.ecs_instance_role: Destruction complete after 1s
aws_iam_instance_profile.ecs_instance_role: Destruction complete after 1s
aws_iam_role.ecs_instance_role: Destroying... [id=test-research-kedro-microservice-aws-iam-role]
aws_iam_role.ecs_instance_role: Destruction complete after 1s

Destroy complete! Resources: 3 destroyed.
(research_kedro_microservice) ➜  aws_batch git:(iss-3-aws-infrastructure) ✗ make apply 
terraform apply -auto-approve

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_batch_compute_environment.aws_batch_ce will be created
  + resource "aws_batch_compute_environment" "aws_batch_ce" {
      + arn                             = (known after apply)
      + compute_environment_name        = "test-research-kedro-microservice-batch-ce"
      + compute_environment_name_prefix = (known after apply)
      + ecs_cluster_arn                 = (known after apply)
      + id                              = (known after apply)
      + service_role                    = (known after apply)
      + state                           = "ENABLED"
      + status                          = (known after apply)
      + status_reason                   = (known after apply)
      + tags                            = {
          + "environment"  = "test"
          + "project_name" = "research-kedro-microservice"
          + "region"       = "us-east-2"
        }
      + tags_all                        = {
          + "environment"  = "test"
          + "project_name" = "research-kedro-microservice"
          + "region"       = "us-east-2"
        }
      + type                            = "MANAGED"

      + compute_resources {
          + desired_vcpus      = (known after apply)
          + ec2_key_pair       = "aws-us-east-2"
          + instance_role      = (known after apply)
          + instance_type      = [
              + "c5a.large",
              + "c5a.xlarge",
            ]
          + max_vcpus          = 16
          + min_vcpus          = 2
          + security_group_ids = [
              + "sg-0982f0b060b95781a",
            ]
          + subnets            = [
              + "subnet-0644a889bb64f3e2c",
              + "subnet-0ffc295914ee8f62e",
            ]
          + tags               = {
              + "environment"  = "test"
              + "project_name" = "research-kedro-microservice"
              + "region"       = "us-east-2"
            }
          + type               = "SPOT"

          + ec2_configuration {
              + image_id_override = (known after apply)
              + image_type        = (known after apply)
            }
        }
    }

  # aws_iam_instance_profile.ecs_instance_role will be created
  + resource "aws_iam_instance_profile" "ecs_instance_role" {
      + arn         = (known after apply)
      + create_date = (known after apply)
      + id          = (known after apply)
      + name        = "test-research-kedro-microservice-aws-iam-instance-profile"
      + path        = "/"
      + role        = "test-research-kedro-microservice-aws-iam-role"
      + tags        = {
          + "environment"  = "test"
          + "project_name" = "research-kedro-microservice"
          + "region"       = "us-east-2"
        }
      + tags_all    = {
          + "environment"  = "test"
          + "project_name" = "research-kedro-microservice"
          + "region"       = "us-east-2"
        }
      + unique_id   = (known after apply)
    }

  # aws_iam_role.aws_batch_service_role will be created
  + resource "aws_iam_role" "aws_batch_service_role" {
      + arn                   = (known after apply)
      + assume_role_policy    = jsonencode(
            {
              + Statement = [
                  + {
                      + Action    = "sts:AssumeRole"
                      + Effect    = "Allow"
                      + Principal = {
                          + Service = "batch.amazonaws.com"
                        }
                    },
                ]
              + Version   = "2012-10-17"
            }
        )
      + create_date           = (known after apply)
      + force_detach_policies = false
      + id                    = (known after apply)
      + managed_policy_arns   = (known after apply)
      + max_session_duration  = 3600
      + name                  = "test-research-kedro-microservice-aws-iam-role"
      + name_prefix           = (known after apply)
      + path                  = "/"
      + tags                  = {
          + "environment"  = "test"
          + "project_name" = "research-kedro-microservice"
          + "region"       = "us-east-2"
        }
      + tags_all              = {
          + "environment"  = "test"
          + "project_name" = "research-kedro-microservice"
          + "region"       = "us-east-2"
        }
      + unique_id             = (known after apply)

      + inline_policy {
          + name   = (known after apply)
          + policy = (known after apply)
        }
    }

  # aws_iam_role.ecs_instance_role will be created
  + resource "aws_iam_role" "ecs_instance_role" {
      + arn                   = (known after apply)
      + assume_role_policy    = jsonencode(
            {
              + Statement = [
                  + {
                      + Action    = "sts:AssumeRole"
                      + Effect    = "Allow"
                      + Principal = {
                          + Service = "ec2.amazonaws.com"
                        }
                    },
                ]
              + Version   = "2012-10-17"
            }
        )
      + create_date           = (known after apply)
      + force_detach_policies = false
      + id                    = (known after apply)
      + managed_policy_arns   = (known after apply)
      + max_session_duration  = 3600
      + name                  = "test-research-kedro-microservice-aws-iam-role"
      + name_prefix           = (known after apply)
      + path                  = "/"
      + tags                  = {
          + "environment"  = "test"
          + "project_name" = "research-kedro-microservice"
          + "region"       = "us-east-2"
        }
      + tags_all              = {
          + "environment"  = "test"
          + "project_name" = "research-kedro-microservice"
          + "region"       = "us-east-2"
        }
      + unique_id             = (known after apply)

      + inline_policy {
          + name   = (known after apply)
          + policy = (known after apply)
        }
    }

  # aws_iam_role_policy_attachment.aws_batch_service_role will be created
  + resource "aws_iam_role_policy_attachment" "aws_batch_service_role" {
      + id         = (known after apply)
      + policy_arn = "arn:aws:iam::aws:policy/service-role/AWSBatchServiceRole"
      + role       = "test-research-kedro-microservice-aws-iam-role"
    }

  # aws_iam_role_policy_attachment.ecs_instance_role will be created
  + resource "aws_iam_role_policy_attachment" "ecs_instance_role" {
      + id         = (known after apply)
      + policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role"
      + role       = "test-research-kedro-microservice-aws-iam-role"
    }

Plan: 6 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + aws_batch_env_subnets_ids = [
      + "subnet-0ffc295914ee8f62e",
      + "subnet-0644a889bb64f3e2c",
    ]
  + aws_batch_subnets_ids     = [
      + "subnet-0ffc295914ee8f62e",
      + "subnet-0644a889bb64f3e2c",
      + "subnet-018a09b8964cddeb1",
      + "subnet-0c9ffb7ee119d4539",
      + "subnet-050fde753140abc0b",
      + "subnet-04d08097f9b5e19a7",
    ]
  + ce_instance_type          = [
      + "c5a.large",
      + "c5a.xlarge",
    ]
  + docker_image_id           = "946627858531.dkr.ecr.us-east-2.amazonaws.com/research-kedro-microservice:latest"
  + ecr_repository_url        = "946627858531.dkr.ecr.us-east-2.amazonaws.com/research-kedro-microservice"
  + env                       = "test"
  + project_name              = "research-kedro-microservice"
aws_iam_role.ecs_instance_role: Creating...
aws_iam_role.aws_batch_service_role: Creating...
aws_iam_role.aws_batch_service_role: Creation complete after 1s [id=test-research-kedro-microservice-aws-iam-role]
aws_iam_role_policy_attachment.aws_batch_service_role: Creating...
aws_iam_role_policy_attachment.aws_batch_service_role: Creation complete after 1s [id=test-research-kedro-microservice-aws-iam-role-20220614121327577500000001]
╷
│ Error: failed creating IAM Role (test-research-kedro-microservice-aws-iam-role): EntityAlreadyExists: Role with name test-research-kedro-microservice-aws-iam-role already exists.
│       status code: 409, request id: 7f006469-b72a-4c32-8355-298c2df936c6
│ 
│   with aws_iam_role.ecs_instance_role,
│   on security.tf line 1, in resource "aws_iam_role" "ecs_instance_role":
│    1: resource "aws_iam_role" "ecs_instance_role" {
│ 
╵
make: *** [Makefile:8: apply] Error 1
(research_kedro_microservice) ➜  aws_batch git:(iss-3-aws-infrastructure) ✗

security.tf file

resource "aws_iam_role" "ecs_instance_role" {
  name = "${local.env}-${local.project_name}-aws-iam-role"

  assume_role_policy = <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
    {
        "Action": "sts:AssumeRole",
        "Effect": "Allow",
        "Principal": {
            "Service": "ec2.amazonaws.com"
        }
    }
    ]
}
EOF

  tags = merge(
    local.vpc_tags,
    {environment = terraform.workspace}
  )
}

resource "aws_iam_role_policy_attachment" "ecs_instance_role" {
  role       = aws_iam_role.ecs_instance_role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role"
}

resource "aws_iam_instance_profile" "ecs_instance_role" {

  name = "${local.env}-${local.project_name}-aws-iam-instance-profile"
  role = aws_iam_role.ecs_instance_role.name

  tags = merge(
    local.vpc_tags,
    {environment = terraform.workspace}
  )
}

resource "aws_iam_role" "aws_batch_service_role" {
  name = "${local.env}-${local.project_name}-aws-iam-role"

  assume_role_policy = <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
    {
        "Action": "sts:AssumeRole",
        "Effect": "Allow",
        "Principal": {
        "Service": "batch.amazonaws.com"
        }
    }
    ]
}
EOF

  tags = merge(
    local.vpc_tags,
    {environment = terraform.workspace}
  )
}

resource "aws_iam_role_policy_attachment" "aws_batch_service_role" {
  role       = aws_iam_role.aws_batch_service_role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSBatchServiceRole"
}
giltayar1 commented 1 year ago

Hey, I have got the same problem... Would you happen to have any time estimate for a fix? (I also can provide more info if needed)

thedini commented 1 year ago

Is the workaround to manually remove the roles after destroy since they are no longer tracked in the state?

smacswain commented 1 year ago

https://docs.aws.amazon.com/cli/latest/reference/iam/delete-instance-profile.html worked I guess. I don't like changing state files or forcing one off destroys.

ARashitov commented 1 year ago

@smacswain @thedini @giltayar1 @aarowman @justinretzolk @yalattas

Guys! I've met recently this problem again at my current job and found a potential solution that may work for you (At least worked out for me).

I've had some terraform code written by my collegues before and needed to extend module. I was able to get this error again. Unfortenately, i don't know the reason of error, however my collegue provided following solution that worked out to me:

  1. Remove resource from terraform state by command: terraform state rm aws_iam_role.<resource_name> <IAM_role_name>
  2. Import state again: terraform import aws_iam_role.<resource_name> <IAM_role_name>

(For more details https://developer.hashicorp.com/terraform/cli/commands/import)

If you meet such error please try this solution for yourself and let me know whether it is general solution or a corner case

josh-m-sharpe commented 10 months ago

Can confirm its an issue on 4.67.0.

Some anecdotal things I noticed in fixing our issue:

1) if you remove iam_instance_profile from an aws_resource terraform won't do anything. If you set it to null terraform also won't do anything. 2) but if you remove it from ec2 console terraform will notice and add it back 3) If you edit a aws_iam_instance_profile and remove the optional role nothing happens on the AWS side. terraform makes no changes

I ultimately fixed this by commenting out my aws_iam_role, aws_iam_instance_profile all related aws_iam_role_policy_attachments and running apply - they weren't attached or doing anything correct anyways.

When I added them all back at once they got applied correctly.

As an aside, there appear to be a naming issue in the instance_profile/iam_role world as you fiddle in ec2 console. (e.g. stupid seemingly aws-buggy stuff like, if an IAM role is correctly attached on an ec2 instance the label will be "IAM Role" and it'll show you the role's name as a link to the IAM Role. But if it is somehow mis-attached (due to this terraform bug) it'll show a different label (I think) "IAM Instance Profile" and sow the name of that object - and that there are no roles attached to it. It's super janky - but this feels like it's all AWS-side.