aws-cdk-lib.aws_codepipeline: Internal Failure (GetRolePolicy returns NoSuchEntityException)

[chrisammon3000]

Describe the bug

Trying to implement a multi-region pipeline for Lambda Layers following this blog: https://aws.amazon.com/blogs/compute/deploying-aws-lambda-layers-automatically-across-multiple-regions/

It worked last month but the latest deployment is returning Internal Failure.

cdk deploy returns the message:

8:09:13 PM | CREATE_FAILED        | AWS::CodePipeline::Pipeline | hgraphPipelineDEB06F2A
Internal Failure

CloudTrail log shows that all the GetRolePolicy calls for the codepipeline.Pipeline resource are returning NoSuchEntityException:

// cloudtrail event
...
    "userAgent": "cloudformation.amazonaws.com",
    "errorCode": "NoSuchEntityException",
    "errorMessage": "The role policy with name hgraphPipelineDistributedistributeuswest1CodePipelineActionRoleDefaultPolicy260032F1 cannot be found.",
    "requestParameters": {
        "roleName": "LambdaLayerPipelineStack-hgraphPipelineDistri-1AQJ1DDB2H06",
        "policyName": "hgraphPipelineDistributedistributeuswest1CodePipelineActionRoleDefaultPolicy260032F1"
    },
...

CDK resource:

const lambdaLayerBuilderPipeline = new codepipeline.Pipeline(this, `${config.repoName}Pipeline`, {
    crossAccountKeys: false,
    stages: [
        {
            stageName: 'Source',
            actions: [sourceAction]
        },
        {
            stageName: 'Build',
            actions: [buildAction]
        },
        {
            stageName: 'Distribute',
            actions: parallel,
        }
    ]
});

When reviewing the error events, it looks like the role policies are not being created for the actions in the stage steps for at least Build and Distribute.

Expected Behavior

The role policies should be created automatically.

Current Behavior

Role policies are not being created.

Reproduction Steps

export default class LambdaLayerPipelineStack extends Stack {
    constructor(scope: Construct, id: string, props: LambdaLayerPipelineStackProps) {
        super(scope, id, props);

        const oauthTokenGitHub = SecretValue.secretsManager('github-token', { jsonField: 'token' })

        const layerUpdaterRole = this.createLambdaRole();
        const distributor = new NodejsFunction(this, 'LayerDistributor', {
            entry: './lambda/distributor/lambda/layer-distributor.ts',
            role: layerUpdaterRole,
            functionName: 'LambdaLayerDistributor',
            description: 'Distributes Lambda layers into multiple regions from a single ZIP archive.',
            timeout: Duration.seconds(15),
            memorySize: 512,
        });

        const codeBuildProjectIamRole = new iam.Role(this, 'CodeBuildProjectIamRole', {
            assumedBy: new iam.ServicePrincipal('codebuild.amazonaws.com'),
            inlinePolicies: {
                CodeBuildProjectIamRolePolicy: new iam.PolicyDocument({
                    statements: [
                        new iam.PolicyStatement({
                            resources: ['*'],
                            actions: [
                                'codeartifact:GetAuthorizationToken',
                                'codeartifact:GetRepositoryEndpoint',
                                'codeartifact:ReadFromRepository'
                            ],
                            effect: iam.Effect.ALLOW,
                        }),
                        new iam.PolicyStatement({
                            resources: ['*'],
                            actions: [
                                'sts:GetServiceBearerToken',
                            ],
                            effect: iam.Effect.ALLOW,
                            conditions: {
                                StringEquals: {
                                    'sts:AWSServiceName': 'codeartifact.amazonaws.com'
                                }
                            }
                        }),
                    ],
                })
            }
        });

        // Everything below this can be iterated over
        // iterate over each config using for loop
        for (const config of props.config) {
            const project = this.createCodeBuild(codeBuildProjectIamRole, config.repoName);
            const sourceOutput = new codepipeline.Artifact();
            const sourceAction = new codepipelineActions.GitHubSourceAction({
                actionName: 'GitHub_Source',
                owner: config.repoOwner,
                repo: config.repoName,
                output: sourceOutput,
                branch: 'main',
                oauthToken: oauthTokenGitHub,
                trigger: codepipelineActions.GitHubTrigger.WEBHOOK
            });

            const buildOutput = new codepipeline.Artifact();
            const buildAction = new codepipelineActions.CodeBuildAction({
                actionName: 'CodeBuild',
                project,
                input: sourceOutput,
                outputs: [buildOutput]
            });

            // Create action per specified region
            const parallel = config.regionCodesToDistribute.map((region) => new codepipelineActions.LambdaInvokeAction({
                actionName: `distribute-${region}`,
                lambda: distributor,
                inputs: [buildOutput],
                userParameters: {
                    region,
                    layerPrincipal: props.layerPrincipal,
                    organizationId: config.organizationId,
                    repoOwner: config.repoOwner,
                    repoName: config.repoName,
                    repoDescription: config.repoDescription,
                    layerRuntimes: config.layerRuntimes,
                }
            }));

            const lambdaLayerBuilderPipeline = new codepipeline.Pipeline(this, `${config.repoName}Pipeline`, {
                crossAccountKeys: false,
                stages: [
                    {
                        stageName: 'Source',
                        actions: [sourceAction]
                    },
                    {
                        stageName: 'Build',
                        actions: [buildAction]
                    },
                    {
                        stageName: 'Distribute',
                        actions: parallel,
                    }
                ]
            });

            new codepipeline.CfnWebhook(this, `${config.repoName}WebhookResource`, {
                authentication: 'GITHUB_HMAC',
                authenticationConfiguration: {
                    secretToken: process.env.GITHUB_TOKEN,
                },
                filters: [
                    {
                        jsonPath: '$.ref',
                        matchEquals: 'refs/heads/main',
                    },
                ],
                targetAction: sourceAction.actionProperties.actionName,
                targetPipeline: lambdaLayerBuilderPipeline.pipelineName,
                targetPipelineVersion: 1, // Number(lambdaLayerBuilderPipeline.pipelineVersion.toString()),
                registerWithThirdParty: true,
            });
        }
        new LambdaLayerArnSsmParamUpdater (this, 'LambdaLayerArnSsmParamUpdater');
    }

Possible Solution

When reviewing the error events, it looks like the role policies are not being created for the actions in the stage steps for at least Build and Distribute. Not sure if this is a bug or user error.

Additional Information/Context

No response

CDK CLI Version

2.66.0 (build c96c17d)

Framework Version

No response

Node.js Version

v19.4.0

OS

macOS M2

Language

Typescript

Language Version

Version 4.9.4

Other information

No response

[pahud]

Hi @abk7777

Unfortunately I can hardly reproduce this with your given code as I can't just run the given code from my IDE. Do you happen to have a simplified repo that I can just clone and deploy in my environment to reproduce this error?

[github-actions[bot]]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.