pulumi / pulumi-aws

An Amazon Web Services (AWS) Pulumi resource package, providing multi-language access to AWS
Apache License 2.0
442 stars 155 forks source link

Creating multiple `RolePolicyAttachment` results in attachments getting dropped. #4235

Open EvanBoyle opened 1 month ago

EvanBoyle commented 1 month ago

My working code is below. Notice that I had to add an explicit depends on for the third resource. Without this, I'm seeing the first policy attachment for SNS get dropped consistently. Not great as it causes your underlying app in ECS to start failing in strange and unexpected ways.

// Create an IAM role for the ECS task
const taskRole = new aws.iam.Role("ecsTaskRole", {
  assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal({ Service: "ecs-tasks.amazonaws.com" }),
});

// Attach the AmazonSNSFullAccess policy to the role
const snsPolicyAttachment = new aws.iam.RolePolicyAttachment("ecsTaskSNSRolePolicyAttachment", {
  role: taskRole.name,
  policyArn: aws.iam.ManagedPolicy.AmazonSNSFullAccess,
});

// Attach the AmazonKinesisFullAccess policy to the role
const kinesisPolicyAttachment = new aws.iam.RolePolicyAttachment(
  "ecsTaskKinesisRolePolicyAttachment",
  {
    role: taskRole.name,
    policyArn: aws.iam.ManagedPolicy.AmazonKinesisFullAccess,
  },
  { dependsOn: snsPolicyAttachment },
);
EvanBoyle commented 1 month ago

cc @t0yv0 in case the bugs I open are still going into the void.

corymhall commented 1 month ago

@EvanBoyle thanks for raising this with us. I've tried a couple of times and haven't been able to reproduce this. It looks like it is not possible to concurrently attach a policy to a role, but upstream has code to specifically handle that scenario. My hunch is that it may have been a perfect race condition or an error in the IAM service that failed to recognize the concurrent operation and send an error code.

If this issue occurs regularly or if you find a way to reliably reproduce this please let us know.

EvanBoyle commented 1 month ago

@corymhall sorry, should've included more detailed repro steps

  1. run the program and create only the SNS attachement
  2. add the Kinesis attachment and run the program again
  3. Run a refresh and see that one of the attachements has been deleted (verify the policies attached to the role in the AWS console as well)
EvanBoyle commented 1 month ago

I'd be surprised if this is a one off race condition. We saw this in both our development and production stacks.

corymhall commented 1 month ago

I'm still not able to reproduce that behavior. Do you know what version of pulumi-aws you are using?

EvanBoyle commented 1 month ago

I see references to both 6.33.1 and 6.32.0 in my state file.