lambda: corrupt zip archive asset produced when using node v15.6

[machielg]

Update from the CDK team

When using local assets with the lambda module using the Code.fromAsset() API, deployment of new assets (during cdk deploy) will fail with the error - "Uploaded file must be a non-empty zip"

We have confirmed this issue manifests when using node engine v15.6. The root cause is being tracked by node - https://github.com/nodejs/node/issues/37027. For the time being, use a node engine < 15.6.

If you have encountered this issue, you need to do three things to resolve this -

1. Downgrade to a node version < 15.6. You can see what you're current node version is by running node --version.
2. Delete the cached asset in the cdk.out/ directory. You could safely delete the entire directory, and the correct assets and templates will be re-generated during the next run of cdk synth or cdk deploy.
3. Delete the previously uploaded asset from the staging bucket. The S3 key of the asset will be prefixed with asset/ and have the hash of the asset. The hash of the asset can be found in your application's CloudFormation template. The staging bucket can be obtained by running the following command -

   aws cloudformation describe-stack-resources --stack-name CDKToolkit --logical-resource-id StagingBucket  --query 'StackResources[].PhysicalResourceId'

---

Original issue filed below

When doing

const fillFn = new lambda.Function(this, "Zip2GzipSQSFill", {
    code: lambda.Code.fromAsset("lib/python"),
    runtime: lambda.Runtime.PYTHON_3_7,
    handler: "mylambda.handler"
    }
);

my file is here (relative to root of the CDK project): lib/python/mylambda.py

its contents:

def handler(event, context):
    print("HELLO AWS")

I get the error:

Uploaded file must be a non-empty zip (Service: AWSLambdaInternal; Status Code: 400; Error Code: InvalidParameterValueException; Request ID: fcfaf553-70d3-40b8-85d2-a15f6c3bcef0; Proxy: null)

Reproduction Steps

run cdk deploy

What did you expect to happen?

the Lambda is created in AWS

What actually happened?

An zip file is uploaded to the CDK staging bucket, there is a zip file there with my python file, but that file has no contents

Environment

• CDK CLI Version : 1.85.0 or 1.83.0
• Framework Version: 1.83.0
• Node.js Version: 15.6.0
• OS : MacOS
• Language (Version): TypeScript (4.1.3)

Other

I'm suspecting it's a nodejs/library problem, in that some library is producing this invalid zip file but I have no evidence of this.

---

This is :bug: Bug Report

[joel-aws]

@machielg -- could you update the title, maybe to "Uploaded Zips of Functions are Empty".

I'm running into the same issue and want to ensure it's visible.

[joel-aws]

Running into the same issue now with aws_s3_deployment on 1.84.0 and 1.85.0:

//EDIT:

I can see the asset was bundled locally (e.g., asset.90b...), however, when browsing to the s3://cdk-hnb659fds-assets-*-*/assets/ prefix, I can see that the zip corresponding to asset.90b has the correct files in it, but they are entirely empty.

OS: Catalina

[hoegertn]

I had the same issue today with a customer using the serverless framework. So it might be something outside CDK, also.

[joel-aws]

I had the same issue today with a customer using the serverless framework. So it might be something outside CDK, also.

I think you're right... Although I just tried it with us-east-1 and us-west-1 to no avail; so I don't think it's with AWS infra either. Curious where the issue is, then.

[buddhabuddy]

Very similar issue, zip downloaded shows CRC errors but files are present and 7zip can recover them.

Error is "Uploaded file must be a non-empty zip"

[DrCake]

I didn't have issues with deploying the aws_s3_deployment function itself on 1.83, but did find that the files uploaded to S3 were all empty. Updating to 1.85 I now get the error in https://github.com/aws/aws-cdk/issues/12536#issuecomment-761182906

[hoegertn]

What versions of Node do y'all use? As a wild guess, I think it might be a node 15 issue.

[joel-aws]

What versions of Node do y'all use? As a wild guess, I think it might be a node 15 issue.

v15.6.0

[machielg]

I updated to node 15.6.0 just yesterday when also this bug started so it’s a likely suspect. Downgrading to node 14 didn’t seem to help though. I’m planning to run it in a docker container to isolate the library responsible.

[DrCake]

What versions of Node do y'all use? As a wild guess, I think it might be a node 15 issue.

I’m having the same issue on Node 12

[machielg]

I tested with several node versions, I can't get it to work with any version (10, 14, 15).

It's clear that zipping of my asset dir with the python code isn't triggered, the zip file with the code isn't created.

I just updated all packages and it seems to work again.

[joel-aws]

I just updated all packages and it seems to work again.

Which packages did you update and using what commands?

[machielg]

I just updated all packages and it seems to work again.

Which packages did you update and using what commands?

I got an update for @types/node to version 14.14.21 which was released very recently.

[danilobuerger]

I had the same issue after upgrading to 15.6.0. Downgrading (and deleting cdk.out/.cache and deleting the zip files on the s3 bucket) worked!

[machielg]

I had the same issue after upgrading to 15.6.0. Downgrading (and deleting cdk.out/.cache and deleting the zip files on the s3 bucket) worked!

Perhaps you can list your relevant package versions?

[danilobuerger]

@machielg Which package version?

[danilobuerger]

@machielg They have nothing to do with it... as I wrote downgrading node itself and deleting the caches is enough.

[machielg]

@danilobuerger I removed the cache and s3 assets but that didn’t work for me so I’m trying to find out what fixed it

[danilobuerger]

@machielg you probably did not clear the caches.

[danilobuerger]

@machielg please read above, I mentioned it in my first comment here....

[therealsebo]

I had the same issue after upgrading to 15.6.0. Downgrading (and deleting cdk.out/.cache and deleting the zip files on the s3 bucket) worked!

Thx @danilobuerger ! Downgrading to Node@14, deleting the cdk dir and Remove Zip from S3 worked for me

[danilobuerger]

@therealsebo Awesome! It seems to me that everything including 15.5.1 is working.

[PierreKiwi]

I am also facing this issue. I managed to make it work by clearing the asset in S3 and deleting cdk.out (not sure what .cache is) but it s failing again.

[machielg]

I am also facing this issue. I managed to make it work by clearing the asset in S3 and deleting cdk.out (not sure what .cache is) but it s failing again.

it was working for me yesterday, but this morning it failed again. The zip file created by CDK contains all the files, but they are all empty. The issue seems to be coming from the archiver lib: https://github.com/archiverjs/node-archiver/issues/491

[machielg]

Downgrading to Node v15.5.1 and doing a proper clean seems to fix the problem.

[PierreKiwi]

Thanks @machielg. I downgraded to Node v14.15.4 and to node-archiver 5.1.0. So far it is looking OK now.

[cmckni3]

Ran into this issue today with node v12.20.0 and CDK version 1.84.0. Downgraded to CDK version 1.74.0 to get it working again.

[cmckni3]

I am also facing this issue. I managed to make it work by clearing the asset in S3 and deleting cdk.out (not sure what .cache is) but it s failing again.

This worked for me as well. Frustrating. Downgrading CDK retains automation capability.

[MHacker9404]

if it helps with triage - I'm not seeing this issue with Lambda::NodejsFunction (typescript) 1.85.0, Node 15.5.0.

[cmckni3]

@MHacker9404 @machielg why aren't you using an active LTS version of node.js?

[MHacker9404]

Because this is on my personal machine....Gimme a few minutes - I'll switch to v14 - and see if I have the issue.... v14.15.4 I was able to upload both lambdas in my stack.

[cmckni3]

CDK 1.85.0 works for me on node v12.20.x.

[machielg]

@MHacker9404 @machielg why aren't you using an active LTS version of node.js?

Good question, brew installs 15.x by default.

Is it possible for aws-cdk to specify the compatible/supported version(s)? So it doesn't install on node 15.x or gives a warning

[MHacker9404]

I use NVM for Windows - so I have 3-4 different versions installed....

[cmckni3]

I use nvm on macOS. I have 3 major versions installed, 10/12/14.

[kodemaniak]

I have this problem also with Node 14.15.4 and cdk 1.85.0. With 1.74.0 it works. Unfortunately 1.74.0 is missing a feature I require, so this is not a solution for me.

[jmpalomar]

@MHacker9404 @machielg why aren't you using an active LTS version of node.js?

Good question, brew installs 15.x by default.

Is it possible for aws-cdk to specify the compatible/supported version(s)? So it doesn't install on node 15.x or gives a warning

You can do it: brew install node@14 && brew link node@14 --overwrite

Downgrading to node 14.15.4 and cleaning cdk.out and s3 zip files made the trick for me.

[brennick]

Having the same issue on a new M1 MacBook Air. Older versions (<15.x) of Node don't work on the new Macs and v15.6.0 is giving me this error when doing cdk deploy

EDIT: I was able to get it working. Steps below...

1. Use nvm for version management
2. Create copy of iTerm app, rename to Rosetta iTerm and update to Open using Rosetta
3. Open Rosetta iTerm, and run nvm install v12.12.0
4. Use Rosetta iTerm to do cdk deploy commands

[saksham]

Downgrading to node version 14 (from 15.6.0) worked for me for CDK version 1.85.0. I was getting same error saying Uploaded file must be a non-empty zip (Service: AWSLambdaInternal; Status Code: 400; Error Code: InvalidParameterValueException; Request ID: 2993dbcd-474d-466b-a4d6-7abfd49af443; Proxy: null) when trying to use aws_cdk.aws_lambda_python.PythonFunction with a lambda folder with poetry. The asset in cdk.out and the bucket had the correct zip file, but cloudformation failed to acknowledge that the file was there.

[tobukc]

Downgraded to node version 14 but still not working for me. The asset in the bucket was a non-empty zip.

[rudyhadoux]

Same here with serverless deploy.

serverless create --template aws-nodejs-typescript --name lambda1 fails.

sh-3.2# node -v v15.6.0 sh-3.2# sls -v Framework Core: 2.19.0 Plugin: 4.4.2 SDK: 2.3.2 Components: 3.5.1

[duntonr]

I was also experiencing this issue but finally got things working again with NodeJS 12.20.1 and CDK 1.85.0. I suspect it should also work for the latest 14.x LTS.

During initial troubleshooting, I tried several combinations of Node and CDK versions but no to avail, would always get the non-empty zip file error during deployment. I had been clearing out the cdk.out folder but I was NOT clearing the zip files in the CDK bootstrap bucket on S3. Once I did that, it worked.

As may other commenters already pointed out, you NEED to clear BOTH your local cdk.out AND the offending zip files in the CDK bootstrap S3 bucket on AWS (or just deleted everything in the bootstrap bucket which is what I did).

It looks like the root issue is that the zip file names are simple hash checks and CDK is optimized to not bother reuploading the "same file" but "same" is detected just by filename, not by size or modification date. As such, once you have this issue, your bootstrap buckets are "poisoned" for lack of a better term and won't work again unless you either A) clear the offending files in the S3 bucket, or just delete it all or B) modified your actual code enough to result in a new hash being generated (I suspect this could be a side effect for some troubleshooting).

[kodemaniak]

I can confirm what @duntonr wrote. After deleting the zip files from the bucket and deleting the cdk.out directory, deployment works using NodeJS 14.15.4.

[cmckni3]

Issue has returned for me on node v12.20.0 and CDK version 1.85.0.

[machielg]

I think the best solution is for the cdk cli to pin the archiver dependency to 5.1.0. The latest version seems to have timing issues.

[cmckni3]

I think the best solution is for the cdk cli to pin the archiver dependency to 5.1.0. The latest version seems to have timing issues.

CDK 1.86.0 has timing issues?

[cmckni3]

I have a deployment running with CDK 1.86.0 right now.

[machielg]

I think the best solution is for the cdk cli to pin the archiver dependency to 5.1.0. The latest version seems to have timing issues.

CDK 1.86.0 has timing issues?

I mean archiver 5.2

[harrysolovay]

Confirmed––deleting the contents of the staging bucket solves the issue.

[mauricioharley]

I was struggling with his until found this GitHub issue. It worked for me after downgrading Node.JS to 14.5.4 with the commands provided by @jmpalomar.

I'm currently using CDK 1.86.0 (core and modules).