Add waiter for AWS eventual consistency on IAM role attachment operations

atsalolikhin-spokeo commented 3 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

This is related to #7646

This is an intermittent issue that AWS Support confirmed is caused by eventual consistency.

When we bring up an EMR cluster, we create a service role and attach an existing service role policy to it. The EMR cluster uses this service role.

Intermittently, the cluster creation fails due to the role not having enough permissions.

Let's say we attach arn:aws:iam::1234567890:policy/servicerole-policy to arn:aws:iam::1234567890:role/cluster7-servicerole.

The servicerole-policy provides ec2:DescribeImages.

The error is:

Error: Service role arn:aws:iam::1234567890:role/cluster7-servicerole has insufficient EC2 permissions: DescribeImages.

Terraform output showing the IAM role attachment was created before initiating the cluster launch (we have depends_on in our Terraform code, the aws_emr_cluster depends on the aws_iam_role):

aws_iam_role.primary: Creation complete after 1s [id=cluster7-servicerole]
aws_iam_role_policy_attachment.primary: Creating...
aws_iam_role_policy_attachment.primary: Creation complete after 2s [id=cluster7-servicerole-20210901102623505400000001]
aws_emr_cluster.primary: Creating...
aws_emr_cluster.primary: Still creating... [10s elapsed]
...
Error: Error waiting for EMR Cluster state to be "WAITING" or "RUNNING": TERMINATED_WITH_ERRORS: VALIDATION_ERROR: Service role arn:aws:iam::1234567890:role/cluster7-servicerole has insufficient EC2 permissions: DescribeImages.

  on emr.tf line 5, in resource "aws_emr_cluster" "primary":
   5: resource "aws_emr_cluster" "primary" {

New or Affected Resource(s)

aws_emr_cluster
aws_iam_role
aws_iam_role_policy_attachment

Potential Terraform Configuration

References

7646

jvisker commented 2 years ago

I would definitely love this to be fixed. As a work around I have used the time provider to sleep for 5 seconds to help ensure that it exists. https://registry.terraform.io/providers/hashicorp/time/latest/docs/resources/sleep.

github-actions[bot] commented 10 months ago

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

atsalolikhin-spokeo commented 10 months ago

This is still an issue.

jpatallah commented 4 months ago

This is still an issue.

hashicorp / terraform-provider-aws