lambda: corrupt zip archive asset produced when using node v15.6

[machielg]

Update from the CDK team

When using local assets with the lambda module using the Code.fromAsset() API, deployment of new assets (during cdk deploy) will fail with the error - "Uploaded file must be a non-empty zip"

We have confirmed this issue manifests when using node engine v15.6. The root cause is being tracked by node - https://github.com/nodejs/node/issues/37027. For the time being, use a node engine < 15.6.

If you have encountered this issue, you need to do three things to resolve this -

1. Downgrade to a node version < 15.6. You can see what you're current node version is by running node --version.
2. Delete the cached asset in the cdk.out/ directory. You could safely delete the entire directory, and the correct assets and templates will be re-generated during the next run of cdk synth or cdk deploy.
3. Delete the previously uploaded asset from the staging bucket. The S3 key of the asset will be prefixed with asset/ and have the hash of the asset. The hash of the asset can be found in your application's CloudFormation template. The staging bucket can be obtained by running the following command -

   aws cloudformation describe-stack-resources --stack-name CDKToolkit --logical-resource-id StagingBucket  --query 'StackResources[].PhysicalResourceId'

---

Original issue filed below

When doing

const fillFn = new lambda.Function(this, "Zip2GzipSQSFill", {
    code: lambda.Code.fromAsset("lib/python"),
    runtime: lambda.Runtime.PYTHON_3_7,
    handler: "mylambda.handler"
    }
);

my file is here (relative to root of the CDK project): lib/python/mylambda.py

its contents:

def handler(event, context):
    print("HELLO AWS")

I get the error:

Uploaded file must be a non-empty zip (Service: AWSLambdaInternal; Status Code: 400; Error Code: InvalidParameterValueException; Request ID: fcfaf553-70d3-40b8-85d2-a15f6c3bcef0; Proxy: null)

Reproduction Steps

run cdk deploy

What did you expect to happen?

the Lambda is created in AWS

What actually happened?

An zip file is uploaded to the CDK staging bucket, there is a zip file there with my python file, but that file has no contents

Environment

• CDK CLI Version : 1.85.0 or 1.83.0
• Framework Version: 1.83.0
• Node.js Version: 15.6.0
• OS : MacOS
• Language (Version): TypeScript (4.1.3)

Other

I'm suspecting it's a nodejs/library problem, in that some library is producing this invalid zip file but I have no evidence of this.

---

This is :bug: Bug Report

[chegger]

Using cdk 1.86 + node:lts docker image DOES work for me

Using cdk 1.86 + node:latest did NOT work.

[proligde]

Same here - I could avoid the problem for me by rolling back to lts:

sudo npm install -g n
sudo n lts

[cmckni3]

I have a deployment running with CDK 1.86.0 right now.

Same problem. Had to empty S3 bucket. Downgraded back to CDK 1.74.0 for now. I am tired of wasting time on this issue.

[robertd]

Same issue here... lts + cleanup did the trick. Luckily our CI/CD pipeline is using node:lts-alpine anyway so issue was only observed on dev boxes.

[NetaNir]

Hi everyone!

Sorry about this :/.

The issue was not automatically assigned to one of the core team members, hence the delay in response. If this ever happens, I asked that you kindly tag one of the core team on the issue, here is the list of GitHub logins: https://github.com/aws/aws-cdk/blob/4871f4a1503bc2d82440e204e1c5b05f2ef26b7b/.mergify.yml#L9

Im going to look into this now.

[viventra]

--- UPDATE: I downgraded NodeJS to LTS (14.15.4) and the cdk deploy succeeded. The error below is with NodeJS v15.6.0 only (for me)

@NetaNir, I am encountering the following error during a cdk deploy: I upgraded to NodeJS v15.6.0 and cdk toolkit v1.86.0 (build a10cfcf) yesterday.

13/23 | 9:06:07 AM | CREATE_FAILED        | AWS::Lambda::Function       | ApimethodsHandler (ApimethodsHandlerD82FED6A) Uploaded file must be a non-empty zip (Service: AWSLambdaInternal; Status Code: 400; Error Code: InvalidParameterValueException; Request ID: c9b1645c-d3b4-4647-bcb2-26b250a22f10; Proxy: null)
        new Function (C:\Users\vx\AppData\Local\Temp\jsii-kernel-jWF0YQ\node_modules\@aws-cdk\aws-lambda\lib\function.js:133:26)
        \_ C:\Users\vx\AppData\Local\Temp\iuq34vgl.n2n\jsii-runtime.js:3556:49
        \_ Kernel._wrapSandboxCode (C:\Users\vx\AppData\Local\Temp\iuq34vgl.n2n\jsii-runtime.js:4040:16)
        \_ Kernel._create (C:\Users\vx\AppData\Local\Temp\iuq34vgl.n2n\jsii-runtime.js:3556:26)
        \_ Kernel.create (C:\Users\vx\AppData\Local\Temp\iuq34vgl.n2n\jsii-runtime.js:3300:21)
        \_ KernelHost.processRequest (C:\Users\vx\AppData\Local\Temp\iuq34vgl.n2n\jsii-runtime.js:13119:28)
        \_ KernelHost.run (C:\Users\vx\AppData\Local\Temp\iuq34vgl.n2n\jsii-runtime.js:13057:14)
        \_ Immediate._onImmediate (C:\Users\vx\AppData\Local\Temp\iuq34vgl.n2n\jsii-runtime.js:13060:37)
        \_ processImmediate (node:internal/timers:463:21)

The code that invokes the Lambda installation is in C#:

            // 2. Add the Lambda functions
            var Apimethods = new Function(this, "ApimethodsHandler", new FunctionProps
            {
                Runtime = Runtime.NODEJS_12_X, // execution environment
                Code = Code.FromAsset("resources"), // Code loaded from the "resources" directory
                Handler = "apimethods.handler" // file is "apimethods", function is "handler"
            });

When I run cdk deploy, the cdk.out folder has the apimethods.js Lambda source file under assets folder, the cdktoolkit-staging-xxx S3 bucket contains a 0 byte ZIP file under assets folder and I get the error as mentioned above.

I tried clearing the cdk.out folder and emptying the cdktoolkit-staging-xxx bucket as others have pointed out. The error occurs even after that. I would prefer a solution that doesn't require a NodeJS downgrade.

[christophgysin]

This seems to be caused by: https://github.com/archiverjs/node-archiver/issues/491, which identified the root cause as https://github.com/nodejs/node/issues/37027

[ckhicks]

Having the same issue on a new M1 MacBook Air. Older versions (<15.x) of Node don't work on the new Macs and v15.6.0 is giving me this error when doing cdk deploy

EDIT: I was able to get it working. Steps below...

1. Use nvm for version management
2. Create copy of iTerm app, rename to Rosetta iTerm and update to Open using Rosetta
3. Open Rosetta iTerm, and run nvm install v12.12.0
4. Use Rosetta iTerm to do cdk deploy commands

@brennick I gave this a try on my M1 Mac Mini and have yet to see a successful deployment, unfortunately.

Running:

• macOS v11.1 (M1)
• iTerm v3.4.3 via Rosetta
• Node v12.12.0
• AWS CDK v1.86.0

When clearing all cache folders, then executing a clean install, build, and deploy, I get the same empty zip warning and also log the following:

 ❌  MercuryApiStack failed: Error: The stack named [snip] failed to deploy: UPDATE_ROLLBACK_COMPLETE
    at Object.waitForStackDeploy (/Users/ckhicks/.nvm/versions/node/v12.12.0/lib/node_modules/aws-cdk/lib/api/util/cloudformation.ts:307:11)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at Object.deployStack (/Users/ckhicks/.nvm/versions/node/v12.12.0/lib/node_modules/aws-cdk/lib/api/deploy-stack.ts:283:26)
    at CdkToolkit.deploy (/Users/ckhicks/.nvm/versions/node/v12.12.0/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:180:24)
    at initCommandLine (/Users/ckhicks/.nvm/versions/node/v12.12.0/lib/node_modules/aws-cdk/bin/cdk.ts:202:9)
The stack named MercuryApiStack failed to deploy: UPDATE_ROLLBACK_COMPLETE

Thanks to @NetaNir and team working on this!

[cmckni3]

For now, I added a version resolution to my package.json. Yarn locked the offending version of node-archiver, archiver, even after downgrading AWS CDK.

package.json:

  "resolutions": {
    "archiver": "~5.1.0"
  }

[cmckni3]

For now, I added a version resolution to my package.json. Yarn locked the offending version of node-archiver, archiver, even after downgrading AWS CDK.

package.json:

  "resolutions": {
    "archiver": "~5.1.0"
  }

Worked after clearing cdk.out and S3 bucket.

[ckhicks]

@cmckni3 Glad you got it working - I just tried each of those steps after downgrading to CDK 1.74.0 and still get the same errors. :/

[cmckni3]

@cmckni3 Glad you got it working - I just tried each of those steps after downgrading to CDK 1.74.0 and still get the same errors. :/

I kept CDK at 1.86.0. Make sure you run an install after updating package.json.

[ckhicks]

@cmckni3 Glad you got it working - I just tried each of those steps after downgrading to CDK 1.74.0 and still get the same errors. :/

I kept CDK at 1.86.0. Make sure you run an install after updating package.json.

Still no dice on my M1 Mac - thanks though! I'll keep at it. ⛏️

EDIT: Success! It was an ID10T error on my part...can confirm that CDK 1.86.0 works on an M1 Mac when limiting node-archiver as suggested. 👍

[scubbo]

I have downgraded all aws-cdk packages to 1.74.0, set archiver to ~5.1.0 as cmckni3 suggested, deleted uploaded s3 files, run rm -rf cdk.out/*, and run npm install - and I'm still getting this same issue (confirmed that the uploaded files are non-empty, and I get a "bad CRC" error when unzipping).

I've also tried all of the above except downgrading from 1.86.0 to 1.74.0 - same error.

Mac, 16-inch 2019, OS 10.15.7

[ebh]

@scubbo, are you using a docker container? I was running cdk in the node:latest container and had to, in addition to setting archiver to ~5.1.0 & deleting all objects in the S3 bucket, switch to using node:lts.

[scubbo]

Nope, no docker container. In fact, I just tried creating a fresh project in case I had some weird configuration leftover, and it's still erroring in the same way:

$ git init testProject
Initialized empty Git repository in /Users/scubbo/Code/testProject/.git/
$ cd testProject
$ cdk init app --language typescript
...
$ rm package-lock.json
####
# ...in IDE, edit package.json and create rudimentary app, then...
####
$ cat package.json
{
  "name": "test_project",
  "version": "0.1.0",
  "bin": {
    "test_project": "bin/test_project.js"
  },
  "scripts": {
    "build": "tsc",
    "watch": "tsc -w",
    "test": "jest",
    "cdk": "cdk"
  },
  "devDependencies": {
    "@aws-cdk/assert": "1.74.0",
    "@types/jest": "^26.0.10",
    "@types/node": "10.17.27",
    "aws-cdk": "1.74.0",
    "jest": "^26.4.2",
    "ts-jest": "^26.2.0",
    "ts-node": "^9.0.0",
    "typescript": "~3.9.7"
  },
  "dependencies": {
    "@aws-cdk/core": "1.74.0",
    "@aws-cdk/aws-lambda": "1.74.0",
    "source-map-support": "^0.5.16"
  },
  "resolutions": {
    "archiver": "~5.1.0"
  }
}
$ cat lambda/index.py
def handler(event, context):
  print('hello world')
$ cat lib/test_project-stack.ts
import * as cdk from '@aws-cdk/core';
import {AssetCode, Function, Runtime} from '@aws-cdk/aws-lambda';

export class TestProjectStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const backend = new Function(this, 'backend-lambda', {
      code: new AssetCode("lambda/"),
      handler: "index.handler",
      runtime: Runtime.PYTHON_3_8
    })

  }
}
$ aws s3 ls s3://cdktoolkit-stagingbucket-<id>/assets/
[no contents]
$ touch .npmignore && rm -rf node_modules  # to avoid an "ENOENT" error
$ npm install
[...]
$ cdk deploy
[Same "Uploaded file must be a non-empty zip" error]
$ aws s3 ls s3://cdktoolkit-stagingbucket-<id>/assets/
2021-01-25 22:55:03        184 83674[...].zip
$ aws s3 cp s3://cdktoolkit-stagingbucket-<id>/assets/83674[...].zip /tmp/zipfile.zip
$ unzip -l /tmp/zipfile.zip
Archive:  /tmp/zipfile.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  01-01-1980 00:00   index.py
---------                     -------
        0                     1 file
$ unzip /tmp/zipfile.zip
Archive:  /tmp/zipfile.zip
  inflating: index.py                 bad CRC 5c545933  (should be 00000000)
$ cat index.py
def handler(event, context):
  print('hello world')

[nija-at]

I was able to reproduce this using Node v15.6 and was absent in Node v15.5 and lower. Until we investigate further and find a different resolution, please use node engine < 15.6.0

[nija-at]

I am able to confirm that downgrading archiver from 5.2.0 to 5.1.0 does not resolve this problem. This seems tied to the node release 15.6.0.

[christophgysin]

@nija-at Just in case you missed it, the root cause is https://github.com/nodejs/node/issues/37027.

And a small nit regarding the changed title: The generated zip files are not actually empty, just corrupt (bad CRC). It's simply the error message displayed by CDK that reads "Uploaded Zips of Functions are Empty".

[gibbster]

I tried downgrading node and am still facing this issue:

node --version
v15.5.1

I also cleared out the CDK staging bucket, and rm -rf node_modules package-lock.json && npm i before redeploying.

FWIW, here's the dependencies I have in my package.json:

  "devDependencies": {
    "@aws-cdk/assert": "1.86.0",
    "@types/jest": "^26.0.10",
    "@types/node": "10.17.27",
    "aws-cdk": "1.86.0",
    "jest": "^26.4.2",
    "ts-jest": "^26.2.0",
    "ts-node": "^9.0.0",
    "typescript": "~3.9.7"
  },
  "dependencies": {
    "@aws-cdk/aws-apigateway": "^1.86.0",
    "@aws-cdk/aws-batch": "^1.86.0",
    "@aws-cdk/aws-cloudfront": "1.86.0",
    "@aws-cdk/aws-cloudfront-origins": "1.86.0",
    "@aws-cdk/aws-cognito": "^1.86.0",
    "@aws-cdk/aws-dynamodb": "^1.86.0",
    "@aws-cdk/aws-ec2": "^1.86.0",
    "@aws-cdk/aws-ecr": "^1.86.0",
    "@aws-cdk/aws-ecs": "^1.86.0",
    "@aws-cdk/aws-events": "^1.86.0",
    "@aws-cdk/aws-events-targets": "^1.86.0",
    "@aws-cdk/aws-lambda": "^1.86.0",
    "@aws-cdk/aws-lambda-nodejs": "^1.86.0",
    "@aws-cdk/aws-rds": "^1.86.0",
    "@aws-cdk/core": "1.86.0",
    "source-map-support": "^0.5.16"
  },
  "resolutions": {
    "archiver": "~5.1.0"
  }

[nija-at]

@gibbster - you also need to delete the cdk.out folder. The asset is cached in there.

[nija-at]

@christophgysin - thanks for pointing this out. I've adjusted the title.

[christophgysin]

@gibbster I think I also needed to delete the asset in the S3 staging bucket. IIRC the hash is calculated over the inputs, not the resulting zip. So the corrupt zip and the valid zip might have the same hash!

[gibbster]

Thanks @christophgysin and @nija-at - clearing the cdk.out folder did the trick!

[scubbo]

I can confirm that, after downgrading my Node version to 15.5.1 (for any other Node newbies who were confused about how to do this, it doesn't seem to be possible via brew install node@15.5.1 - I had to use n on my Mac), my deploy succeeds with 1.86.0 CDK dependencies (no need for 1.74.0), and without a pinned archiver version.

[alex-hladun]

Still getting error with Node 15.5 and clearing bucket (using CircleCI so no cdk.out).

Was able to get it working with 14.15.4 and CDK 1.77.0.

[alst-monosolutions]

Experiencing the issue with Node 10.17, CDK 1.74 as well 1.78

[machielg]

I think the main culprit is archive.js version 5.2.0. @nija-at would it be possible for the CDK cli to pin the version of archive js to 5.1.0 until the bug has been solved?

[vinnayb29]

I have tried node 15.6.0, 15.5.0. Cleared cdk.out still facing the issue

[nija-at]

@machielg - Last I checked, the combination of Node15.6+archiverjs-5.1.0 still manifested the bug.

[jorgeprz10]

tried node 14 with 1.87 while clearing out the cdk.out/cache. It worked... the first time with a brand new bootstrap. afterwards it fails regardless of emptying the staging bucket and cdk.out.

[shaftoe]

I've not been able to fix it even following the above suggestions of downgrading both Node.js and cdk.

My downgraded env:

• macOS 10.15.7, Node.js 14.15.4 installed via homebrew
• cdk version 1.79.0 installed via npm + tarball following official instructions

I could also reproduce the bug with all packages up to date, i.e. Node.js 15.7.0 and CDK 1.87.1 (build 9eeaa93)

[Tantalon]

There's three things needed to work around it:

• Downgrade node to 15.5 or earlier
• Delete your cdk.out directory
• Empty your cdk staging S3 bucket

[lekkerdriveway]

+1 to this. I was having this same issue. Was able to resolve with downgrade of node to v14.15.4, delete cdk.out and empty the cdk staging bucket. I kept aws-cdk version 1.87.1 (build 9eeaa93) didn't seem to cause a problem.

[fabriciolelis]

I use node versions 14.15.4 and 15.7.0 with CDK versions 1.87.1 and 1.74.0 I deleted cdk.out, the files on the bucket And the problem persists.

[JBirdVegas]

I'm unable to get the fix to work

% npm ls -gp --depth=0 | awk -F/ '/node_modules/ && !/\/npm$/ {print $NF}' | xargs npm -g rm
% rm -rf cdk.out
% node --version
v14.15.4
% cdk --version
1.87.1 (build 9eeaa93)
% cdk deploy
...
Uploaded file must be a non-empty zip (Service: AWSLambdaInternal; Status Code: 400; Error Code: InvalidParameterValueException; Request ID: 84803229-94d2-4273-8b12-4a4b647cc4d4; Proxy: n
ull)
...

S3 bucket is empty

[machielg]

anyone know how to downgrade the archive.js package in the aws cdk installation?

[cmckni3]

Problem has returned for me on node.js 12.20.0, AWS CDK 1.86.0, and node-archiver 5.1.0.

I know the AssetCode and Code classes in CDK changed recently. 🤔

[katrinabrock]

Tried fix/workaround in description, still not able to get it to work. Tried

• cdk version 1.87.1, node version 15.5.1
• cdk version 1.77.0, node version 14.15.4

Each time deleted cdk.out and s3://cdktoolkit-stagingbucket-XXXXX/assets. Also, removed leftover docker images like

for i in `docker image list|awk '{print $1}'|grep cdk`; do docker image rm $i; done

for good measure.

I'm on OS X 10.15.7.

I haven't attempted modifying archiver version independently.

[machielg]

Problem has returned for me on node.js 12.20.0, AWS CDK 1.86.0, and node-archiver 5.1.0.

I know the AssetCode and Code classes in CDK changed recently. 🤔

I also remember seeing some await statements being removed 🤔

[nija-at]

Interestingly, I've tried to reproduce this again and I'm only seeing this error for Node 15.6, while using the CDK 1.87.1.

I'll continue to see if there are other variables that might be causing this.

For context, this is the CDK stack I'm deploying - https://github.com/aws/aws-cdk/blob/master/packages/%40aws-cdk/aws-lambda/test/integ.assets.lit.ts - with the below versions, which deploys fine.

⇒  cdk --version
1.87.1 (build 9eeaa93)
⇒  node --version
v14.15.1

When I run the same thing (cdk deploy after clearing both caches) with only the node version changed to 15.6.0, I am able to reproduce this error.

UPDATE: When the node version is changed to 15.7.0, the error is still showing up. So we'll have to stay in node versions below 15.6.0.

[fabriciolelis]

I've tested with node version 14.15.1 and I usedcdk deploy --force and the problem not solved.

[pregno]

I don't know if I can be useful, but I'm sharing my experience with this issue.

I was using latest node version and I encountered this issues. I rolled back to the node lts using nvm, removed cdk.out, removed cdk bootstrap folder content on s3 and then it started to work again even with the latest CDK version

[cmckni3]

Problem has returned for me on node.js 12.20.0, AWS CDK 1.86.0, and node-archiver 5.1.0.

I know the AssetCode and Code classes in CDK changed recently. 🤔

Issue occurred on CDK 1.74.0 again with node-archiver 5.1.0.

Trying with archiver 5.0.2.

[NickTheSecurityDude]

I'm getting this same error using aws_s3_deployment

Uploaded file must be a non-empty zip (Service: AWSLambdaInternal; Status Code: 400; Error Code: InvalidParameterValueException; Request ID: e9bf23f9-####-####-aee7-4f2929 25506c; Proxy: null)

My python code is: deployment_.BucketDeployment(self,"Web Files", destination_bucket=bucket, sources=[deployment_.Source.asset(os.path.join(dirname, "files/web_files.zip"))] )

I'm using node 15.7.0: % node -v v15.7.0

CDK 1.87.1: % cdk --version 1.87.1 (build 9eeaa93)

[ckhicks]

Hit this again when doing a clean install - took me a minute to recall my original settings but these do work!

• macOS v11.1 (M1)
• iTerm v3.4.3 via Rosetta
• Node v12.12.0
• AWS CDK v1.86.0 (CLI & in package.json - make sure you remove the ^ operator!)
• Limit archiver to 5.1.0 in package.json

Long and frustrating, but it works. 😓

[cmckni3]

The problem keeps resurfacing for me. I upgraded crc32-stream to see if it helps. 🤞

[rsilvestre]

The same here with

• osx 11.1 (20C69)
• cdk 1.88.0 (build f65009b)
• node v15.8.0

[katrinabrock]

I tried running CDK in docker in hopes of iterating more quickly and finding reproducible working versions, but only ran into a different issue (https://github.com/aws/aws-cdk/issues/8799).

I've posted what I did so far here: https://github.com/katrinabrock/aws-cdk-dind-py-lambda .

[cmckni3]

The problem keeps resurfacing for me. I upgraded crc32-stream to see if it helps. 🤞

Upgrading crc32-stream by specifying 4.0.2 in resolutions worked for me. No issues with AWS CDK 1.86.0 for 2 days.