search

aws / aws-cdk

The AWS Cloud Development Kit is a framework for defining cloud infrastructure in code
https://aws.amazon.com/cdk
Apache License 2.0
11.6k stars 3.9k forks source link

CDK deploys partial stack after error, only on CodeBuild #23610

Closed adam-nielsen closed 1 year ago

adam-nielsen commented 1 year ago

Describe the bug

If a certain type of error takes place, when run locally CDK aborts as expected. However when run inside AWS CodeBuild, CDK stops constructing the stack upon error, but then deploys the partially constructed stack.

This results in any resources after the error point being torn down as they are "missing" from the stack.

Expected Behavior

When run inside AWS CodeBuild, the CDK stack should terminate with an error and nothing should get deployed, as happens when running CDK locally.

Current Behavior

Any error that throws an exception seems to trigger the issue. I see it with aws-s3-deployment.BucketDeployment and with NodeJsFunction. When run in CodeBuild, a message about an unhandled rejected promise is shown, then the incomplete stack is deployed.

(node:266) UnhandledPromiseRejectionWarning: Error: Failed to bundle asset myproject-stack/react-site/deploy/Asset1/Stage, bundle output is located at /codebuild/output/src931507575/src/git-codecommit.ap-southeast-2.amazonaws.com/v1/repos/myproject/deploy/cdk.out/asset.8e0d7bc582372b2a377a084834924b7cabcf66e2d275a9be0376387ae1c2380c-error: Error: docker exited with status 1
    at AssetStaging.bundle (/codebuild/output/src931507575/src/git-codecommit.ap-southeast-2.amazonaws.com/v1/repos/myproject/deploy/node_modules/aws-cdk-lib/core/lib/asset-staging.js:2:646)
    at AssetStaging.stageByBundling (/codebuild/output/src931507575/src/git-codecommit.ap-southeast-2.amazonaws.com/v1/repos/myproject/deploy/node_modules/aws-cdk-lib/core/lib/asset-staging.js:1:4506)
    at stageThisAsset (/codebuild/output/src931507575/src/git-codecommit.ap-southeast-2.amazonaws.com/v1/repos/myproject/deploy/node_modules/aws-cdk-lib/core/lib/asset-staging.js:1:1867)
    at Cache.obtain (/codebuild/output/src931507575/src/git-codecommit.ap-southeast-2.amazonaws.com/v1/repos/myproject/deploy/node_modules/aws-cdk-lib/core/lib/private/cache.js:1:242)
    at new AssetStaging (/codebuild/output/src931507575/src/git-codecommit.ap-southeast-2.amazonaws.com/v1/repos/myproject/deploy/node_modules/aws-cdk-lib/core/lib/asset-staging.js:1:2262)
    at new Asset (/codebuild/output/src931507575/src/git-codecommit.ap-southeast-2.amazonaws.com/v1/repos/myproject/deploy/node_modules/aws-cdk-lib/aws-s3-assets/lib/asset.js:1:736)
    at Object.bind (/codebuild/output/src931507575/src/git-codecommit.ap-southeast-2.amazonaws.com/v1/repos/myproject/deploy/node_modules/aws-cdk-lib/aws-s3-deployment/lib/source.js:1:1200)
    at /codebuild/output/src931507575/src/git-codecommit.ap-southeast-2.amazonaws.com/v1/repos/myproject/deploy/node_modules/aws-cdk-lib/aws-s3-deployment/lib/bucket-deployment.js:1:2895
    at Array.map (<anonymous>)
    at new BucketDeployment (/codebuild/output/src931507575/src/git-codecommit.ap-southeast-2.amazonaws.com/v1/repos/myproject/deploy/node_modules/aws-cdk-lib/aws-s3-deployment/lib/bucket-deployment.js:1:2876)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:266) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 15)
(node:266) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

✨  Synthesis time: 72.73s

Note the final line "Synthesis time" should not appear as this indicates success when there was actually an error. I see the same with with NodeJsFunction:

(node:1291) UnhandledPromiseRejectionWarning: Error: Failed to bundle asset myproject-stack/myproject-api/Code/Stage, bundle output is located at /codebuild/output/src955587944/src/git-codecommit.ap-southeast-2.amazonaws.com/v1/repos/myproject/deploy/cdk.out/bundling-temp-f5278548a9cacc313c59e0540aa9cffbfe88a61e700150bcaccd293ccf7e38cd-error: Error: docker exited with status 1
    at AssetStaging.bundle (/codebuild/output/src955587944/src/git-codecommit.ap-southeast-2.amazonaws.com/v1/repos/myproject/deploy/node_modules/aws-cdk-lib/core/lib/asset-staging.js:2:646)
    at AssetStaging.stageByBundling (/codebuild/output/src955587944/src/git-codecommit.ap-southeast-2.amazonaws.com/v1/repos/myproject/deploy/node_modules/aws-cdk-lib/core/lib/asset-staging.js:1:4506)
    at stageThisAsset (/codebuild/output/src955587944/src/git-codecommit.ap-southeast-2.amazonaws.com/v1/repos/myproject/deploy/node_modules/aws-cdk-lib/core/lib/asset-staging.js:1:1867)
    at Cache.obtain (/codebuild/output/src955587944/src/git-codecommit.ap-southeast-2.amazonaws.com/v1/repos/myproject/deploy/node_modules/aws-cdk-lib/core/lib/private/cache.js:1:242)
    at new AssetStaging (/codebuild/output/src955587944/src/git-codecommit.ap-southeast-2.amazonaws.com/v1/repos/myproject/deploy/node_modules/aws-cdk-lib/core/lib/asset-staging.js:1:2262)
    at new Asset (/codebuild/output/src955587944/src/git-codecommit.ap-southeast-2.amazonaws.com/v1/repos/myproject/deploy/node_modules/aws-cdk-lib/aws-s3-assets/lib/asset.js:1:736)
    at AssetCode.bind (/codebuild/output/src955587944/src/git-codecommit.ap-southeast-2.amazonaws.com/v1/repos/myproject/deploy/node_modules/aws-cdk-lib/aws-lambda/lib/code.js:1:4628)
    at new Function (/codebuild/output/src955587944/src/git-codecommit.ap-southeast-2.amazonaws.com/v1/repos/myproject/deploy/node_modules/aws-cdk-lib/aws-lambda/lib/function.js:1:2803)
    at new NodejsFunction (/codebuild/output/src955587944/src/git-codecommit.ap-southeast-2.amazonaws.com/v1/repos/myproject/deploy/node_modules/aws-cdk-lib/aws-lambda-nodejs/lib/function.js:1:1171)
    at new MyProjectStack (/codebuild/output/src955587944/src/git-codecommit.ap-southeast-2.amazonaws.com/v1/repos/myproject/deploy/lib/myproject-stack.ts:150:22)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:1291) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 15)
(node:1291) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

✨  Synthesis time: 81.1s

Again, Synthesis time appearing indicates success, and after this CDK goes on to deploy the incomplete stack.

In both cases an unhandled promise rejection is to blame, however this does not happen when the stack is deployed from my local machine. The only difference is on my local machine I am using an untested Node version (19) but it works, yet CodeBuild is running supported versions Node 14 or 16 and it fails with the rejected promises.

Reproduction Steps

It is hard to make a small example for CodeBuild deploying a CDK stack. Is there a working demo somewhere I can modify? I imagine sticking a throw in the middle of a CDK stack then running cdk deploy inside CodeBuild will reproduce the problem.

Possible Solution

Catch the unhandled promise and make CDK exit with an error instead of trying to deploy the failed partial stack.

Additional Information/Context

It is really problematic that it deploys half a stack. In my case I had CloudFront constructs later in the stack after the error point which were lost when the exception was thrown, so CloudFormation tore down the CloudFront distribution as it was missing from the stack. I then couldn't deploy again because CloudFront complained the CNAME pointed to a different CloudFront distribution. I had to remove the CNAME entirely, deploy CloudFront, then add the CNAME back again with the new CloudFront distribution ID before the system was working again.

CDK CLI Version

2.59.0 (build b24095d)

Framework Version

No response

Node.js Version

14/19

OS

Linux

Language

Typescript

Language Version

N/A

Other information

No response

adam-nielsen commented 1 year ago

I have narrowed this down to NodeJS 14.x by the looks of it.

If I use the STANDARD_5_0 CodeBuild image with NodeJS 14, then I see this deploy-on-failure issue.

But if I switch to NodeJS 16, which also requires switching CodeBuild image (in my case I switched to AMAZON_LINUX_2_4) then the same failure causes CDK to abort the process as expected.

I'm going to leave this issue open as CDK officially supports Node 14 (until the end of April 2023), however a workaround for the time being is to switch to Node 16 instead where the errors are correctly captured.

pahud commented 1 year ago

Hi @adam-nielsen

Do you have any existing github repo with minimal working sample that I can check out and reproduce in my account?

github-actions[bot] commented 1 year ago

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

adam-nielsen commented 1 year ago

I don't have an example I can supply to reproduce it unfortunately, but it's probably not worth looking into as Node 14 won't be supported by CDK two months from now, so there are probably more important things to fix than this.

joelgarrett commented 1 year ago

Just had the same issue happen to me today. Using Node 16 and CDK version 2.50.0. Are there any ideas on how to get CodeBuild to abort on failures and not proceed to teardown resources?

(node:283) UnhandledPromiseRejectionWarning: Error: docker exited with status 1
--
at dockerExec (/codebuild/output/src368625325/src/deploy/node_modules/.pnpm/aws-cdk-lib@2.50.0_constructs@10.1.267/node_modules/aws-cdk-lib/core/lib/bundling.js:4:45)
at Function.fromBuild (/codebuild/output/src368625325/src/deploy/node_modules/.pnpm/aws-cdk-lib@2.50.0_constructs@10.1.267/node_modules/aws-cdk-lib/core/lib/bundling.js:1:3494)
at new ContentDistribution (/codebuild/output/src368625325/src/deploy/src/lib/constructs/content-distribution.ts:59:28)
at new WebUiService (/codebuild/output/src368625325/src/deploy/src/lib/constructs/web-ui-service.ts:58:33)
at new ApplicationStack (/codebuild/output/src368625325/src/deploy/src/lib/stacks/application-stack.ts:191:29)
at new ApplicationStage (/codebuild/output/src368625325/src/deploy/src/lib/stages/application-stage.ts:15:5)
at new PipelineStack (/codebuild/output/src368625325/src/deploy/src/lib/stacks/pipeline-stack.ts:87:7)
at createProductionStacks (/codebuild/output/src368625325/src/deploy/src/bin/deploy.ts:118:3)
at createStacks (/codebuild/output/src368625325/src/deploy/src/bin/deploy.ts:130:5)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:283) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
joelgarrett commented 1 year ago

Ok, I was able to resolve the issue that occurred by increasing the instance size I was using for the CDK synth step in my CodePipeline. I have also added the NodeJS option --unhandled-rejections=strict to make the default behavior of unhandled promise rejections is to throw.

adam-nielsen commented 1 year ago

@joelgarrett I think yours was a different issue too. I wanted the resources torn down on failure but instead it left them in place, with some resources broken.

adam-nielsen commented 1 year ago

Closing this now Node 14 is no longer officially supported.

github-actions[bot] commented 1 year ago

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.