AWS Provider 4.9 AWS IAM instance profile not updated on manually changed AWS IAM role

bashoKa commented 2 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

terraform -v Terraform v1.1.9 on darwin_arm64

provider registry.terraform.io/hashicorp/aws v4.9.0
provider registry.terraform.io/hashicorp/random v3.1.2

Affected Resource(s)

aws_iam_instance_profile
aws_iam_role

25646
32671
34099
1270
838

Terraform Configuration Files

resource "aws_iam_instance_profile" "brt" {
  name_prefix = "aws-brt"
  role        = aws_iam_role.brt.name
}

resource "aws_iam_role" "brt" {
  name               = "aws-brt"
  assume_role_policy = data.aws_iam_policy_document.brt_instance_assume_role.json
}

Actual Behavior

Manual deletion of the AWS IAM role
Manual rebuilding of the AWS IAM role
Instance profile will not be updated to the new role on terraform apply
Warning in AWS console: "The instance profile xxxx-profile associated with the environment has no role. Please attach a role to the instance profile."
Changing the AWS IAM role name to some random value && terraform apply
Changing the role name back to original value && terraform apply will update the instance profile

Steps to Reproduce

Manual deletion of the AWS IAM role
Manual rebuilding of the AWS IAM role
Instance profile will not be updated to the new role on terraform apply
Warning in AWS console: "The instance profile xxxx-profile associated with the environment has no role. Please attach a role to the instance profile."
Changing the AWS IAM role name to some random value && terraform apply
Changing the role name back to original value && terraform apply will update the instance profile

Important Factoids

References

Similar bug happened already before:

https://github.com/hashicorp/terraform-provider-aws/issues/1270

Thanks guys for your investigation and please let me know if you need further details.

justinretzolk commented 2 years ago

Hey @bashoKa 👋 Thank you for taking the time to raise this! So that we have all of the necessary information in order to investigate, would it be possible to supply debug logs (redacted as needed) as well?

bashoKa commented 2 years ago

Hi

Sure, just running it with debug mode?

best

justinretzolk commented 2 years ago

Hey @bashoKa 👋 That's correct. That can be done by setting an environment variable of TF_LOG=TRACE.

bashoKa commented 2 years ago

One thing I noticed preparing the trace: I wasn't able to attach the created role to an ec2 instance: Value (arn:aws:iam::123:instance-profile/awsbrt) for parameter iamInstanceProfile.arn is invalid. Invalid IAM Instance Profile ARN But this happens right away after the creation of role and profile

macropin commented 2 years ago

We've hit this again. These are the steps to manually reattach a role to an instance profile:

# list all the instance profiles in the account
$ aws iam list-instance-profiles

# reattach the role required to the instance profile
$ aws iam add-role-to-instance-profile --instance-profile-name "instance1.foo.example.com_20220617071515949800000001" --role-name  instanceRole

I think this is a simple lifecycle issue caused by a boundary evaluation bug (> should be >=). Can someone please look at the code I linked in #25646 and confirm?

vishwa-trulioo commented 2 years ago

I can confirm that this is issue there on provider 4.27.0 as well.

My observation is, The following error shows up if the desired capacity has a deviation at the time when running the TF apply. But, if the desired capacity has no diff, then it won't throw errors. However, this issue doesn't happen on 4.14.0.

Errors:

...
  # module.myasg.aws_autoscaling_group.this[0] will be updated in-place
  ~ resource "aws_autoscaling_group" "this" {
      ~ desired_capacity          = 2 -> 1
        id                        = "myasg"
        name                      = "myasg"
        # (22 unchanged attributes hidden)
      ~ launch_template {
            id      = "lt-05c1f947e191ff630"
            name    = "mylaunchtpl"
          ~ version = "55" -> "56"
        }
        # (9 unchanged blocks hidden)
    }
Plan: 0 to add, 2 to change, 0 to destroy.
module.myasg.aws_autoscaling_group.this[0]: Modifying... [id=myasg]
╷
│ Error: waiting for Auto Scaling Group (myasg) capacity satisfied: 13 errors occurred:
│   * Scaling activity (bb960aa2-4fc7-3683-7214-19e7d4b2d771): Failed: Value (myasg-role) for parameter iamInstanceProfile.name is invalid. Invalid IAM Instance Profile name. Launching EC2 instance failed.
│   * Scaling activity (18f60aa2-414a-69e4-d9bb-497515b792ab): Failed: Value (myasg-role) for parameter iamInstanceProfile.name is invalid. Invalid IAM Instance Profile name. Launching EC2 instance failed.
│   * Scaling activity (02960aa2-39b1-9321-93fa-456708f879ab): Failed: Value (myasg-role) for parameter iamInstanceProfile.name is invalid. Invalid IAM Instance Profile name. Launching EC2 instance failed.

This was reported in the past and fixed. Here are the ticket numbers. But, it seems it needs a bit more tuning.

Nuru commented 1 year ago

Just hit this bug in provider version 5.9.0

circa10a commented 1 year ago

Hitting this bug in v4.67.0

JoshuaRowland commented 12 months ago

This issue exists in version 5.4.0.

github-actions[bot] commented 6 months ago

[!WARNING] This issue has been closed, meaning that any additional comments are hard for our team to see. Please assume that the maintainers will not see them.

Ongoing conversations amongst community members are welcome, however, the issue will be locked after 30 days. Moving conversations to another venue, such as the AWS Provider forum, is recommended. If you have additional concerns, please open a new issue, referencing this one where needed.

github-actions[bot] commented 6 months ago

This functionality has been released in v5.43.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

github-actions[bot] commented 5 months ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

hashicorp / terraform-provider-aws