Error in assuming access role

AreebSiddiqui commented 1 year ago

What happened?

I am trying to create an AppRunner service, and for that, I am creating an IAM Role that has an AmazonEC2ContainerRegistryFullAccess policy attached to it. But when I try to deploy it gives me the following error:

Error:

aws:apprunner:Service (node-app): error: 1 error occurred:

error creating App Runner Service (node-app): InvalidRequestException: Error in assuming access role arn:aws:iam::402228412873:role/apprunner-role-83c64cban

Expected Behavior

It should create an AppRunner Service.

Steps to reproduce

  const policy = {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": {
          "Service": "build.apprunner.amazonaws.com"
        },
        "Action": "sts:AssumeRole"
      }
    ]
  };

  // Base Role
  const role = await new aws.iam.Role("apprunner-role", {
    assumeRolePolicy: JSON.stringify(policy)
  });

  const ec2ContainerRegistry = new aws.iam.RolePolicyAttachment("AmazonEC2ContainerRegistryFullAccess", {
    role: role,
    policyArn: "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryFullAccess"
  });

const nodeapp = new aws.apprunner.Service("node-app", {
    serviceName: "node-app",
    sourceConfiguration: {
    autoDeploymentsEnabled: false,
    authenticationConfiguration: {
      accessRoleArn: role.arn
    },
    imageRepository: {
      imageConfiguration: {
        port: "8080",
      },
      imageIdentifier: repositoryUrl,
      imageRepositoryType: "ECR_PUBLIC",
    },
  },
  tags: {
    Name: "example-apprunner-service",
  },
});

Output of `pulumi about`

CLI
Version 3.57.1 Go Version go1.20.1 Go Compiler gc

Plugins NAME VERSION aws 5.31.0 awsx 1.0.2 docker 4.0.0 docker 3.6.1 nodejs unknown

Host
OS ubuntu Version 22.04 Arch x86_64

This project is written in nodejs: executable='/home/areebpersonal/.nvm/versions/node/v16.17.1/bin/node' version='v16.17.1'

Current Stack: mareebsiddiqui/midrun-infra/dev

Found no resources associated with dev

Found no pending operations associated with dev

Backend
Name pulumi.com URL https://app.pulumi.com/AreebSiddiqui User AreebSiddiqui Organizations AreebSiddiqui, mareebsiddiqui

Dependencies: NAME VERSION @aws-sdk/client-codebuild 3.289.0 @pulumi/aws 5.31.0 @pulumi/awsx 1.0.2 @pulumi/docker 4.0.0 @pulumi/pulumi 3.57.1 @types/express 4.17.17 dotenv 16.0.3 express 4.18.2 @types/node 16.18.14

Pulumi locates its logs in /tmp by default

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

danielrbradley commented 1 year ago

Hi @AreebSiddiqui, can you see the role and attachment being created as you've described it in your Pulumi program?

From the error, it looks like the role might just be mis-configured for the apprunner service context. Did you take this from an example somewhere? I'd also suggest taking a look at the AWS docuentation for the apprunner service identity management to see if this program aligns with the IAM setup they suggest.

AreebSiddiqui commented 1 year ago

hi @danielrbradley, thanks for the reply. Yes, the role and the association are working fine. I can see the role is being created and the policy is attached to it. I am basically creating this role for Apprunner to have access to ECR repository.

AreebSiddiqui commented 1 year ago

@danielrbradley I have updated policy and attached to the role. Still getting that exact same erorr:

Error in assuming access role arn:aws:iam::12345:role/apprunner-role-new-a88f0c

const policy = {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": {
          "Service": "build.apprunner.amazonaws.com"
        },
        "Action": "sts:AssumeRole"
      }
    ]
  };

  const role = await new aws.iam.Role("apprunner-role-new", {
    assumeRolePolicy: JSON.stringify(policy)
  });

  const ec2ContainerRegistry = new aws.iam.RolePolicyAttachment("AWSAppRunnerServicePolicyForECRAccess", {
    role: role,
    policyArn: "arn:aws:iam::aws:policy/service-role/AWSAppRunnerServicePolicyForECRAccess"
  });

const nodeapp = new aws.apprunner.Service("node-app", {
    serviceName: "node-app",
    sourceConfiguration: {
    autoDeploymentsEnabled: false,
    authenticationConfiguration: {
      accessRoleArn: role.arn
    },
    imageRepository: {
      imageConfiguration: {
        port: "8080",
      },
      imageIdentifier: repositoryUrl,
      imageRepositoryType: "ECR",
    },
  },
  tags: {
    Name: "example-apprunner-service",
  },
});

danielrbradley commented 1 year ago

What we need to establish is if there's an issue with the way Pulumi creates the role or service, or if this is an issue about how AWS expects you to configure the role.

Having created the role via Pulumi, are you able to create and attach the role to a service via the console? Does it give the same error?
If you use the console to create the role along with the service then review the auto-created role assume policy does it match to what you're specifying in your program?

rhyek commented 1 year ago

Just to add more evidence: program:

const appRunnerEcrRole = new aws.iam.Role('webApiAppRunnerEcrRole', {
  assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal({
    Service: 'build.apprunner.amazonaws.com',
  }),
});

new aws.iam.RolePolicyAttachment('webApiAppRunnerEcrRoleAttachment', {
  policyArn: aws.iam.ManagedPolicy.AWSAppRunnerServicePolicyForECRAccess,
  role: appRunnerEcrRole.name,
});

const appRunner = new aws.apprunner.Service('webApiAppRunner', {
  serviceName: 'web-api',
  sourceConfiguration: {
    imageRepository: {
      imageIdentifier: process.env.IMAGE_URI!,
      imageRepositoryType: 'ECR',
    },
    authenticationConfiguration: {
      accessRoleArn: appRunnerEcrRole.arn,
    },
    autoDeploymentsEnabled: false,
  },
  instanceConfiguration: {
    cpu: '512',
    memory: '1 GB',
  },
});

export const serviceUrl = appRunner.serviceUrl;

error:

  Diagnostics:
    aws:apprunner:Service (webApiAppRunner):
      error: 1 error occurred:
        * error creating App Runner Service (web-api): InvalidRequestException: Error in assuming access role arn:aws:iam::***:role/webApiAppRunnerEcrRole-fec7ad1

@pulumi/aws: v5.41.0

pdemagny commented 1 year ago

Hi, I'm facing this issue exactly too.

When trying to create the ECR Role and then the AppRunner Service(s) for the first time, it errors out with this error: InvalidRequestException: Error in assuming access role arn:aws:iam::000000000000:role/redacted-role-name.

When I pulumi up a second time it works. Forcing with a single DependsOn doesn't help. The only workaround I found is to do a double DependsOn with an intermediate resource ...

This doesn't work:

newBlobrAppRunnerECRPullRole, err := blobriam.New(
    blobriam.WithContext(ctx),
    blobriam.WithDebug(blobrEnableDebug),
    blobriam.WithRoleArgs(
        blobriam.RoleArgs{
            Name: blobrEnvironmentPrefix + "apprunner-ecr-pull-role",
            AssumeRolePolicy: pulumi.JSONMarshal(map[string]interface{}{
                "Version": "2012-10-17",
                "Statement": []interface{}{
                    map[string]interface{}{
                        "Effect": "Allow",
                        "Action": []string{"sts:AssumeRole"},
                        "Principal": map[string]interface{}{
                            "Service": []string{
                                "build.apprunner.amazonaws.com",
                            },
                        },
                    },
                },
            }),
            Tags: commonAwsResourceTags,
            InlinePolicies: pulumi.JSONMarshal(map[string]interface{}{
                "Version": "2012-10-17",
                "Statement": []interface{}{
                    map[string]interface{}{
                        "Effect": "Allow",
                        "Action": []string{
                            "ecr:GetAuthorizationToken",
                            "ecr:BatchCheckLayerAvailability",
                            "ecr:GetDownloadUrlForLayer",
                            "ecr:BatchGetImage",
                            "ecr:DescribeImages",
                        },
                        "Resource": "*",
                    },
                },
            }),
        },
    ),
)
if err != nil {
    return err
}
blobrAppRunnerECRPullRole, err := newBlobrAppRunnerECRPullRole.CreateIamRole()
if err != nil {
    return err
}

blobrFrontAppRunnerService, err := apprunner.NewService(ctx, blobrEnvironmentPrefix+"apprunner-front-svc", &apprunner.ServiceArgs{
    ServiceName: utils.Ps(blobrEnvironmentPrefix + "apprunner-front-svc"),
    SourceConfiguration: &apprunner.ServiceSourceConfigurationArgs{
        AutoDeploymentsEnabled: utils.Pb(false),
        AuthenticationConfiguration: &apprunner.ServiceSourceConfigurationAuthenticationConfigurationArgs{
            AccessRoleArn: blobrAppRunnerECRPullRole.Arn,
        },
        ImageRepository: &apprunner.ServiceSourceConfigurationImageRepositoryArgs{
            ImageConfiguration: &apprunner.ServiceSourceConfigurationImageRepositoryImageConfigurationArgs{
                Port: utils.Ps("80"),
            },
            ImageIdentifier:     utils.Ps("redacted repository name ;)"),
            ImageRepositoryType: utils.Ps("ECR"),
        },
    },
    InstanceConfiguration: &apprunner.ServiceInstanceConfigurationArgs{
        Cpu:    utils.Ps("256"),
        Memory: utils.Ps("512"),
    },
    Tags: commonAwsResourceTags,
},
    []pulumi.ResourceOption{
        pulumi.DependsOn([]pulumi.Resource{
            blobrAppRunnerECRPullRole,
        }),
    }...)
if err != nil {
    return err
}

This works:

newBlobrAppRunnerECRPullRole, err := blobriam.New(
    blobriam.WithContext(ctx),
    blobriam.WithDebug(blobrEnableDebug),
    blobriam.WithRoleArgs(
        blobriam.RoleArgs{
            Name: blobrEnvironmentPrefix + "apprunner-ecr-pull-role",
            AssumeRolePolicy: pulumi.JSONMarshal(map[string]interface{}{
                "Version": "2012-10-17",
                "Statement": []interface{}{
                    map[string]interface{}{
                        "Effect": "Allow",
                        "Action": []string{"sts:AssumeRole"},
                        "Principal": map[string]interface{}{
                            "Service": []string{
                                "build.apprunner.amazonaws.com",
                            },
                        },
                    },
                },
            }),
            Tags: commonAwsResourceTags,
            InlinePolicies: pulumi.JSONMarshal(map[string]interface{}{
                "Version": "2012-10-17",
                "Statement": []interface{}{
                    map[string]interface{}{
                        "Effect": "Allow",
                        "Action": []string{
                            "ecr:GetAuthorizationToken",
                            "ecr:BatchCheckLayerAvailability",
                            "ecr:GetDownloadUrlForLayer",
                            "ecr:BatchGetImage",
                            "ecr:DescribeImages",
                        },
                        "Resource": "*",
                    },
                },
            }),
        },
    ),
)
if err != nil {
    return err
}
blobrAppRunnerECRPullRole, err := newBlobrAppRunnerECRPullRole.CreateIamRole()
if err != nil {
    return err
}

blobrMongoDBIPAccessList, err := mongodbatlas.NewProjectIpAccessList(ctx, blobrEnvironmentPrefix+"mongodb-ip-access-list-0", &mongodbatlas.ProjectIpAccessListArgs{
    CidrBlock: utils.Ps("redacted ip cidr ;)"),
    Comment:   utils.Ps("redacted comment ;)"),
    ProjectId: utils.Ps(blobrMongoDBProjectId),
},
    []pulumi.ResourceOption{
        pulumi.DependsOn([]pulumi.Resource{
            blobrAppRunnerECRPullRole,
        }),
    }...)
if err != nil {
    return err
}

blobrFrontAppRunnerService, err := apprunner.NewService(ctx, blobrEnvironmentPrefix+"apprunner-front-svc", &apprunner.ServiceArgs{
    ServiceName: utils.Ps(blobrEnvironmentPrefix + "apprunner-front-svc"),
    SourceConfiguration: &apprunner.ServiceSourceConfigurationArgs{
        AutoDeploymentsEnabled: utils.Pb(false),
        AuthenticationConfiguration: &apprunner.ServiceSourceConfigurationAuthenticationConfigurationArgs{
            AccessRoleArn: blobrAppRunnerECRPullRole.Arn,
        },
        ImageRepository: &apprunner.ServiceSourceConfigurationImageRepositoryArgs{
            ImageConfiguration: &apprunner.ServiceSourceConfigurationImageRepositoryImageConfigurationArgs{
                Port: utils.Ps("80"),
            },
            ImageIdentifier:     utils.Ps("redacted repository name ;)"),
            ImageRepositoryType: utils.Ps("ECR"),
        },
    },
    InstanceConfiguration: &apprunner.ServiceInstanceConfigurationArgs{
        Cpu:    utils.Ps("256"),
        Memory: utils.Ps("512"),
    },
    Tags: commonAwsResourceTags,
},
    []pulumi.ResourceOption{
        pulumi.DependsOn([]pulumi.Resource{
            blobrMongoDBIPAccessList,
        }),
    }...)
if err != nil {
    return err
}

Hope this helps fixing this issue (or working around it for those affected).

flostadler commented 1 day ago

I think that this is caused by eventual consistency. IAM has a single global control plane in us-east-1 for the commercial partition, changes usually propagate within ~2 seconds to the data planes. So there's a chance that the role wasn't yet propagated by the time apprunner tried to assume it.

You can try working around this by using a short sleep (e.g. 3s) in the dependency chain between the role and service: https://www.pulumi.com/registry/packages/time/api-docs/sleep/

I've also seen this happen with ECS before.

pulumi / pulumi-aws