(aws-codepipeline): Asset from BucketDeployment not deployed to cross-account bucket

rantoniuk commented 2 years ago

Describe the bug

This is going to be tricky to describe without all the code, but I'll do my best.

I have a pipeline that is using Pipeline from CodePipeline, CDKv2, current version. Since all components were upgraded to CDKv2, I'm assuming it's using newStyleSynthesis by default and I can see some hints of that in the generated CloudFormation template (checking for Bootstrap version, referencing the bootstrap S3 asset bucket).

Now, there is a deployment step defined like this:

            actions: [
                new CloudFormationCreateUpdateStackAction({
                    runOrder: 1,
                    actionName: 'DeployCF',
                    stackName: props.deployedStackName,
                    adminPermissions: false,
                    role: this.integrationDevOpsRole,
                    deploymentRole: this.integrationChangeSetRole,
                    parameterOverrides: {
                        s3LambdaLayerCodeBucketName: lambdaLayerArtifact.bucketName,
                        s3LambdaLayerCodeBucketKey: lambdaLayerArtifact.objectKey,
                        s3LambdaCodeBucketName: lambdaArtifact.bucketName,
                        s3LambdaCodeBucketKey: lambdaArtifact.objectKey,
                    },
                    extraInputs: [lambdaArtifact, lambdaLayerArtifact],
                    templatePath: cdkNoMonitoring.artifact.atPath(`${templatePath}`),
                    cfnCapabilities: [CfnCapabilities.NAMED_IAM, CfnCapabilities.AUTO_EXPAND],

In the cdkNoMonitoring Stack there is a definition of:

        new BucketDeployment(this, 'TemplatesDeploy', {
            sources: [Source.asset(path.join(__dirname, 'templates'))],
            destinationBucket: aBucket,
            destinationKeyPrefix: 'templates',
        });

and that step fails in the cross-account target AWS account deployment, because:

Error occurred while GetObject. S3 Error Code: NoSuchKey. S3 Error Message: The specified key does not exist. (Service: AWSLambdaInternal; Status Code: 400; Error Code: InvalidParameterValueException;

And indeed, when I check the cdkNoMonitoring CloudFormation template, I can see that CDK generates a reference to the S3 asset bucket on the pipeline account correctly:

  TemplatesDeployAwsCliLayer21CBB3EE:
    Type: AWS::Lambda::LayerVersion
    Properties:
      Content:
        S3Bucket:
          Fn::Sub: cdk-XXX-assets-${AWS::AccountId}-${AWS::Region}
        S3Key: XXX.zip
      Description: /opt/awscli/aws
    Metadata:
      aws:cdk:path: ...
      aws:asset:path: asset.02927fd0ce5bb130cbc8d11f17469e74496526efe5186a9ab36e8a8138e9a557.zip
      aws:asset:is-bundled: false
      aws:asset:property: Content

If I check the artefact of the cdkNoMonitoring build, I can see that the asset.02927fd0ce5bb130cbc8d11f17469e74496526efe5186a9ab36e8a8138e9a557 is there.

However, that asset is not being indeed deployed to the cdk-XXX-assets-${AWS::AccountId}-${AWS::Region} and if I understand correct, that's the moment when that asset should be placed in that cross-account S3 bucket.

I know I could do probably the magic with parameterOverrides but I thought that CDKv2 and new bootstrap solved this usecase out of the box, doesn't it?

Am I missing some step? property?

Expected Behavior

as above

Current Behavior

as above

Reproduction Steps

as above

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.26.0 (build a409d63)

Framework Version

2.27.0

Node.js Version

14.19.3

OS

MacOS

Language

Typescript

Language Version

No response

Other information

No response

rantoniuk commented 2 years ago

Just found my old report that mentioned exactly the scenario - #13940 - which should be now automatically fixed by CDKv2 migration but it seems it's not.

peterwoodworth commented 2 years ago

Which part of this exactly is cross-account?

rantoniuk commented 2 years ago

Sorry this was indeed not visible from the code above, so here is how the pipeilne is constructed (I extracted the relevant part of the classes so hopefully it's complete, but shout if anything is unclear):

export class DynamicPipelineConstruct extends Construct {

 this.pipeline = new Pipeline(this, 'pipeline', {
      pipelineName: `${id}-pipeline`,
      artifactBucket: this.artifactBucket,
      role: this.devOpsPipelineRole,
    });

    ....
}

export class CdkBuildConstruct extends Construct {
    artifact: Artifact;
    action: CodeBuildAction;

    constructor(scope: Construct, id: string, props: CdkConstructProps) {
        super(scope, id);

        let buildCommand = 'npm run -- cdk synth "*" -c dev_mode="false"';

        this.artifact = new Artifact(`${props.assetPrefix}-${id}`);
        const cdkBuild = new PipelineProject(this, id, {
            projectName: `${props.assetPrefix}-${id}`,
            environment: { buildImage: LinuxBuildImage.STANDARD_5_0 },
            encryptionKey: props.encryptionKey,
            role: props.role,
            buildSpec: BuildSpec.fromObject({
                version: '0.2',
                phases: {
                    install: {
                        'runtime-versions': {
                            nodejs: 14,
                        },
                        commands: [
                            'npm install -g aws-cdk',
                            'cdk --version',
                            'aws codeartifact login ...,
                            'cd cdk',
                            'npm ci',
                        ],
                    },
                    build: {
                        commands: ['npm run build', buildCommand],
                    },
                },
                artifacts: {
                    'base-directory': `cdk/cdk.out`,
                    files: ['**/*'],
                },
            }),
        });

        this.action = new CodeBuildAction({
            runOrder: props.runOrder,
            actionName: id,
            project: cdkBuild,
            input: props.sourceInput,
            outputs: [this.artifact],
            role: props.role,
        });
    }

export class ServicesPipelineConstruct extends DynamicPipelineConstruct {
         const templatePath = `${props.deployedStackName}.template.json`;

        const cdkbuild = new CdkBuildConstruct(this, 'cdk', {
            runOrder: 1,
            encryptionKey: this.encryptionKey,
            assetPrefix: this.assetPrefix,
            sourceInput: this.sourceOutput,
            role: this.pipeline.role,
        });

}
        const lambdaArtifact = new Artifact(`${this.assetPrefix}-lambda-artifact`);
        const lambdaBuild = new PipelineProject(this, 'LambdaBuild', {
            projectName: `${this.assetPrefix}-lambda-build`,
            environment: { buildImage: LinuxBuildImage.STANDARD_5_0 },
            buildSpec: BuildSpec.fromSourceFilename('lambda/buildspec.yaml'),
            encryptionKey: this.encryptionKey,
            role: this.pipeline.role,
        });

        const lambdaCodebuildAction = new CodeBuildAction({
            runOrder: 1,
            actionName: `Lambda`,
            project: lambdaBuild,
            input: this.sourceOutput,
            outputs: [lambdaArtifact],
            role: this.pipeline.role,
        });

        this.pipeline.addStage({
            stageName: 'Build',
            actions: [
                lambdaCodebuildAction,
                cdkbuild.action,
            ],
        });

       // here the cross-account deployment step that fails because cdk assets are not present in the shared bucket
        this.pipeline.addStage({
            stageName: 'Deployment-INTEGRATION',
            actions: [
                new CloudFormationCreateUpdateStackAction({
                    runOrder: 1,
                    actionName: 'DeployCF',
                    stackName: props.deployedStackName,
                    adminPermissions: false,
                    role: this.integrationDevOpsRole,
                    deploymentRole: this.integrationChangeSetRole,
                    parameterOverrides: {
                        s3LambdaCodeBucketName: lambdaArtifact.bucketName,
                        s3LambdaCodeBucketKey: lambdaArtifact.objectKey,
                    },
                    extraInputs: [lambdaArtifact],
                    templatePath: cdkNoMonitoring.artifact.atPath(`${templatePath}`),
                    cfnCapabilities: [CfnCapabilities.NAMED_IAM, CfnCapabilities.AUTO_EXPAND],
                })
         ]});
}

The mentioned BucketDeployment is added in the CDK stack of the app itself, so not in the pipeline stack above, but inside of the source code that is built with the CdkBuildConstruct.

rantoniuk commented 2 years ago

Coming to think about it after weekend, maybe I am missing a deployment step for the shared assets part?

the cdkNoMonitoring is building the CDK assets and collecting them as artefact to an S3 bucket but never deploys it actually to the shared bucket
the deployment-integration step expects it to be there

I don't see it anywhere in the docs, is there a special step needed to actually deployed the shared asset to the central devops account bootstrap shared bucket that will later be re-used by the pipeline during deployment in other accounts?

(loud thinking) Probably this would work if I add explicitly something like this to deploy the shared asset to the bootstrap s3 bucket (but shouldn't it be done automatically?):

        const bootstrapBucket = Bucket.fromBucketAttributes(this, 'cdkBucket', {
            bucketArn: Fn.importValue(DefaultStackSynthesizer.DEFAULT_FILE_ASSETS_BUCKET_NAME),
            encryptionKey: this.encryptionKey,
        });

        const deployCdkAssetsAction = new S3DeployAction({
            actionName: 'S3Deploy',
            input: cdkNoMonitoring.artifact,
            bucket: bootstrapBucket,
            runOrder: 2,
        });

On the other hand, since the pipeline already has an this.artifactBucket, it should probably automatically deploy the needed shared assets to this or to the bootstrap-created shared bucket and download them from there? Maybe I could even replace the Pipeline this.artifactBucket to use bootstrap's bucket instead of a custom bucket? Of course, the lambda artefact is downloaded from this.artifactBucket -> lambdaArtifact.bucketName + objectKey but why is this manual override of parameters needed if it could be automatically "inferred" from the CDK shared S3 bucket?

Summing my thoughts up to one sentence:

It seems that Pipeline's internal CDK is storing the artefacts in this.artifactBucket instead of CDK's bootstrap bucket where they would be automatically visible for cross-account deployment.

rantoniuk commented 2 years ago

OK, I'm that close to making it work as I described above, but before I describe my findings and potential bugs, let me ask a question to clarify if I'm not missing something.

After bootstrapping the "central" environment, I see that the cdk bootstrap bucket does not have any policies attached to it. Similarly, on the target (deployment) account, after provisioning it with the command:

CDK_NEW_BOOTSTRAP=1 cdk bootstrap aws://TARGET/eu-west-1 --trust DEVOPS --cloudformation-execution-policies arn:aws:iam::aws:policy/AdministratorAccess

I see that on the ....-deploy-role.... there are policies that refer to grant the READ access to cdk-assets bucket:

{
            "Action": [
                "s3:GetObject*",
                "s3:GetBucket*",
                "s3:List*"
            ],
            "Resource": [
                "arn:aws:s3:::cdk-hnb659fds-assets-TARGET-eu-west-1",
                "arn:aws:s3:::cdk-hnb659fds-assets-TARGET-eu-west-1/*"
            ],
            "Effect": "Allow",
            "Sid": "CliStagingBucket"
        },

but the TARGET here is actually wrong as I would expect DEVOPS there as being the central repository trusted account where the pipeline actually runs and builds the artefacts.

So am I confusing something here? I understood that with the cross-account deployment scenario, the central account's bucket would serve as the artefact repository, but from those policies (or rather of lack of cross-account policies to permit reading from the central bucket) it looks to me that the artefacts produced by the pipeline are somehow expected to be pushed to the asset buckets of the target deployment environment.

If that is the case, could you point me to some documentation where this is described and to the relevant code that is actually doing the push of the artefact during pipeline build from the central account to the target account?

On top of that, the same is with the KMS key: the bucket in the DEVOPS account is provisioned to use the alias/s3 key that is said not to support cross-account deployment, so why in a cross-account scenario a CMK is not created by default and appropriate cross-account policies are not added to it?

I have a feeling the new bootstrap is still not supporting cross-account deployment out-of-the-box (but I'm sure I can force it to do so, but first I'd like to check if I'm not reinventing the wheel :D)

rantoniuk commented 2 years ago

@peterwoodworth, looking at the documentation, I'm getting even more confused:

at first, it states:

Older versions of the bootstrap template created a Customer Master Key (CMK) in each bootstrapped environment by default. To avoid charges for the CMK, re-bootstrap these environments using --no-bootstrap-customer-key. The current default is to not use a CMK to avoid these charges.

then later in the table, for modern (v1,v2) it says "Customer-managed key" for Bucket encryption.

But the CMK is not created and if I check the CDKToolkit CloudFormation stack that was upgraded with CDK_NEW_BOOTSTRAP cdk bootstrap, I can see the following:

FileAssetKeyArn -> value: AWS_MANAGED_KEY

and if I check the cdk-hnb659fds-assets-... bucket properties, I see it is using alias/s3 key, not a CMK.

Obviously, that alias/s3 key is missing appropriate policies to grant permissions to encrypt/decrypt from other cross-accounts and I cannot add them via CDK, because it's not a CMK.

What's wrong here? is the bootstrap buggy to not provision the CMK+S3 bucket correctly with the CMK? Or do I have to manually create the CMK and pass the --bootstrap-kms-key-id to cdk bootstrap?

peterwoodworth commented 2 years ago

Hey, thanks for all this info @rantoniuk. I'll take a look at this on monday 🙂

peterwoodworth commented 2 years ago

Sorry I haven't gotten to this yet, will do my best by tomorrow

rantoniuk commented 2 years ago

@peterwoodworth I would be grateful for any reply at least on my assumptions as I don't know which way to go.

One way is that I hack my way through this by manually fixing bootstrap issues and by changing the used bucket and encryption key and summarise it in a bullet-pointed list of bugs for bootstrap.

However if you confirm my suspicions or bad assumptions even in a one sentence summary then I can either redesign the solution to comply with bootstrap's way of working if it's actually correct :)

rantoniuk commented 2 years ago

I decided to give it a go again, I'll describe what I achieved. To sum up the scenario:

previous setup (old bootstrap, cdkv1, old API) used 3 codebuild projects: one to build the lambda code itself from TS, one to build the Lambda layer (dependencies) and one to build the CDK. All of them had their independent artifacts published (see above the code example) and the CF deployment part was done with the hacky parameterOverrides thing - which is exactly what I want to get rid of in this new approach.

The problem with the old approach is that BucketDeployment was not supported in the pipeline, because it did not see the assets.

The new setup is:

layout

| cdk -> directory with CDK
| code -> directory with Lambda code
| code/layer -> nodejs dependencies

The CDK code was rewritten to this:

        new BucketDeployment(this, 'bd', {
            sources: [Source.asset(path.join(__dirname, 'static'))],
            destinationBucket: someBucket,
            destinationKeyPrefix: 'static',
        });

        const nodeLayer = new LayerVersion(this, 'lv', {
            code: Code.fromAsset(path.join(__dirname, '../../lambda/layer/')),
        });

        const lambdaCode = Code.fromAsset(path.join(__dirname, '../../lambda/dist/src/'));

Now we get to the problem description:

The above works perfectly fine locally: after doing cdk synth, in the cdk.out directory I see all assets.XXXXX and assets.YYYY.zip - this is great for the developer so he can test this without the pipeline on the DEV env.

The updated pipeline definition looks like this:

there is only one CodeBuild now (because dist/src must be built first so that assets are available for cdk synth):

new PipelineProject(this, id, {
        encryptionKey: props.encryptionKey,
        role: props.role,
        buildSpec: BuildSpec.fromObject({
            version: '0.2',
            phases: {
                install: {
                    'runtime-versions': { nodejs: 14 },
                    commands: [
                        'aws --version',
                        'aws codeartifact login --tool ...',
                        'npx cdk --version',
                        'cd cdk && npm ci && cd $CODEBUILD_SRC_DIR',
                        'cd lambda && npm ci && cd $CODEBUILD_SRC_DIR',
                        'cd lambda/layer/nodejs && npm ci && cd $CODEBUILD_SRC_DIR',
                    ],
                },
                build: {
                    commands: [
                        // lambda build needs to be first as cdk builds needs the produced assets
                        'cd lambda && npm run build && cd $CODEBUILD_SRC_DIR',
                        // nodejs doesn't need build command, it's node_modules dependencies only
                        "cd cdk && npx cdk synth '*' && cd $CODEBUILD_SRC_DIR",
                    ],
                },
            },
            artifacts: {
                // files: ['cdk/cdk.out/**/*', 'lambda/dist/src/**/*', 'lambda/layer/**/*'],
                'base-directory': 'cdk/cdk.out',
                files: ['**/*'],
            },
        }),
    });

The result of the above when the pipeline runs is that it produces ONE artifact with all the assets from cdk.out zipped and puts it correctly in the cdk-hnb659fds-assets-PIPELINE-eu-west-1 bucket.

The next step in the pipeline is to deploy:

                new CloudFormationCreateUpdateStackAction({
                    runOrder: 1,
                    actionName: 'DeployCF',
                    stackName: props.deployedStackName,
                    adminPermissions: false,
                    role: this.integrationDeployRole,
                    deploymentRole: this.integrationCfnExecRole,
                    // getting rid of those as they shouldn't be needed anymore
                    // parameterOverrides: {
                    //     s3LambdaLayerCodeBucketName: lambdaLayerArtifact.bucketName,
                    //     s3LambdaLayerCodeBucketKey: lambdaLayerArtifact.objectKey,
                    //     s3LambdaCodeBucketName: lambdaArtifact.bucketName,
                    //     s3LambdaCodeBucketKey: lambdaArtifact.objectKey,
                    // },
                    // extraInputs: [lambdaArtifact, lambdaLayerArtifact],
                    templatePath: buildNoMonitoring.artifact.atPath(`${templatePath}`),
                    cfnCapabilities: [CfnCapabilities.NAMED_IAM, CfnCapabilities.AUTO_EXPAND],
                }),

The deployment fails on S3 No such object, so let's see what's happening:

the cdk.out./stack.template.json shows that it expects the following asset (reminder: it works locally)

"SourceBucketNames": [
 {
  "Fn::Sub": "cdk-hnb659fds-assets-${AWS::AccountId}-${AWS::Region}"
 }
],
"SourceObjectKeys": [
 "104259a481cc94463b7076232bf0765081abb91e80e40f340db5e5c848df05cf.zip"
],

This obviously will not work, because:

104259a481cc94463b7076232bf0765081abb91e80e40f340db5e5c848df05cf.zip was not published in the previous step as a separate file, it was zipped in the global artifact produced by the CodeBuild. However, since the step refers templatePath: buildNoMonitoring.artifact.atPath(${templatePath}), it means it has unzipped that artifact so it has it right there, the problem is that it's looking for it in S3 instead of locally unzipped files.
additionally (not sure about this), but wouldn't the ${AWS::AccountId} refer to the TARGET AccountId instead of the PIPELINE AccountId?

If I would be able to somehow cdk publish the resulting assets.* artefacts then I think we would be home - but shouldn't that be done out of the box by CDK? Or maybe is my build or artefact definition wrong?

Hopefully that brings a refresher to the problem and gives you the full insight.

I have resolved all S3 access issues and KMS problems (with the custom CMK) and stumbled on some bugs, but I will report them separately in order not to overload this thread.

Note: I'm not using cdkv2 Modern API, nor new Stage/addApplicationStage() etc.

Thanks a lot for help on this.

rantoniuk commented 2 years ago

Hi @peterwoodworth,

I created a [reproduction code](https://github.com/rantoniuk/aws-cdk-repro] for this so you can easily see the concept and reproduce it easily. Of course, you'll need to provision the roles and buckets, but other than this it should work out of the box.

Here is what is happening here:

When developer tests locally, he deploys via standard cdk deploy - this should work out of the box for you and deploy the TestStack and Lambda.
When we go into "pipeline mode" with cdk -c pipeline=true deploy on the DEVOPS account, a pipeline is deployed that does the following (already with some forced adaptations to use the modern bootstrap shared roles as you'll see in the code):
- fetch the source.zip (that repository) from the source S3 bucket
- run the CodeBuild to: transform typescript into JS and produce an artifact
- save that artifact into DEVOPS account shared cdk-hnb.... bucket - that's the first place to enforce usage of modern bootstrap's provisioned resources - it's not using it out of the box
- deploy that in to the TEST account by passing the artifact.

What is working:

the artifact IS accessible from the shared bucket because it sees the TestStack.template.json

What is not working:

the assets that are expected by both, Lambda and BucketDeployment are not found:

Error occurred while GetObject. S3 Error Code: NoSuchKey. S3 Error Message: The specified key does not exist. (Service: AWSLambdaInternal; Status Code: 400; Error Code: InvalidParameterValueException;

However, if you look into cdk.out/TestStack.assets.json, you'll see they are properly referenced to get fetched from the bootstrap's bucket. I suspect though that CloudFormationCreateUpdateStackAction tries to find the assets in the target's account bucket, rather the shared account bucket.

tennantje commented 2 years ago

Found your issue, and I am having a similar problem https://github.com/aws/aws-cdk/discussions/21819

Will review the thread above.

rantoniuk commented 2 years ago

@peterwoodworth were you able to reproduce using the code attached?

rantoniuk commented 2 years ago

I just found #9917 and I think this is the whole reason this is not working out of the box...

gmournos commented 1 year ago

Hi,

CDK Pipelines (https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.pipelines/README.html) supports bucketdeployment and also other dynamically linked assets, like lambda functions, nested stacks etc; it is very simple to use. It links dynamically the assets in an automatically generated step in the beginning of the pipeline. Bucket deployment contributes several assets (probably one for the bucket content, one for the upload function and one for the cloudfront invalidation one).

complexPipeline

I am attaching a screenshot of the assets of a rather complex stack to clarify this. It contains 4 nested stacks, each of which has one bucket deployment, some dynamically bundled lambda functions with the NodeJsFunction etc...

I hope this helps

pahud commented 3 months ago

Hi @rantoniuk

I am looking into this issue now.

Please let me confirm this again:

As the code you provided is a little bit out-of-dated.

Can you share your code snippet that still relevant today so I can copy and paste into my IDE and cdk deploy it?

And:

Is this still the error message?

Error occurred while GetObject. S3 Error Code: NoSuchKey. S3 Error Message: The specified key does not exist. (Service: AWSLambdaInternal; Status Code: 400; Error Code: InvalidParameterValueException;

github-actions[bot] commented 3 months ago

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

github-actions[bot] commented 3 months ago

Comments on closed issues and PRs are hard for our team to see. If you need help, please open a new issue that references this one.

aws / aws-cdk