(aws-cdk/core): `cdk deploy` fails to publish asset in node v16.14.0

[rnag]

What is the problem?

I posted a question on Stackoverflow with more details about the issue, though I'm honestly at a loss at what could be going wrong. From initial investigation, it looks like completeMultipartUpload fails in node v16.14.0 (LTS) due to a missing ETag in one of the parts. I have literally no clue what any of this means, so I apologize beforehand.

Also to clarify I'm using nvm to manage different node versions currently. I'm switching between node v16 and v14 and the latter works without issue each time with cdk deploy. I honestly feel like it's something wrong with my environment, though I did do an npm -g install aws-cdk and ensured that I'm using the latest CDK version in both cases; in any case I'm at a loss to explain the issue I'm noticing during deployment.

Reproduction Steps

1. Install LTS node version from https://nodejs.org/
2. Set up sample CDK app which has a large enough asset for cdk to opt for multi-part upload
3. Deploy app using cdk:

   $ cdk deploy -vv

What did you expect to happen?

The cdk deploy command should have returned a success. The deployment is successful under the same conditions when using node v14.18.1.

What actually happened?

I got obscure CDK error which I'm unable to understand or debug (shown below):

[50%] fail: The XML you provided was not well-formed or did not validate against our published schema

CDK CLI Version

4.15.0 (build 151055e)

Framework Version

aws-cdk@2.15.0

Node.js Version

v16.14.0

OS

Mac OS Big Sur v11.6.4 (Intel)

Language

Typescript

Language Version

Typescript (4.6.2) Rust (rustc 1.58.1)

Other information

https://stackoverflow.com/questions/71396902/cdk-deploy-fails-to-publish-asset-in-node-v16-14-0

Here's the CDK app I'm building in case it's relevant: https://github.com/rnag/rust.aws-cdk-lambda/tree/main/cdk-examples/rust-standalone

[peterwoodworth]

I'm not able to reproduce this @rnag - I'm uploading a multi part asset on node v16.14.0 successfully with cdk deploy.

I suspect there's something gone slightly wrong somewhere for you when setting up or configuring your project, but i've never seen this kind of error before so I don't have any leads for you unfortunately.

Are you able to reproduce this on a fresh project?

[rnag]

Yeah, its kinda frustrating for me because I have no idea what is going wrong. I've used cdkpreviously in node v14 with no issues, but for some reason with the LTS version (v16.14) on mac I am running into this strange error during deploy.

I've tried a bunch of things already, like uninstalling and re-installing the LTS version with nvm:

nvm uninstall --lts
rm -rf ${NVM_DIR}/.cache/bin/node-v16.14.0-darwin-x64
nvm cache clear --force && npm cache clear --force && npm cache rm --force && npm cache verify 
nvm install --lts
npm -g install aws-cdk 
cdk deploy

For some reason I really can't understand, I'm continuing to run into the same issue in node v16.14 on Mac OS:

[33%] fail: The XML you provided was not well-formed or did not validate against our published schema

I confirmed I did a fresh reinstall of the node version, and npm -g list only shows the following packages installed:

├── aws-cdk@2.15.0
├── corepack@0.10.0
└── npm@8.3.1

Currently I'm still at a loss to figure out what's going wrong, because deploying the same asset works in node v14 for me. I will do some more debugging and hopefully will have a solution.

[rnag]

Ok, just an update but I found something really interesting. Using nvm I installed a bunch of different node versions, including v16.10, 16.12, 16.13.2, and also LTS (v16.14). I also ran npm -g install aws-cdk and confirmed that I'm using the same cdk version in all cases (2.15.0). For some reason, I ran into the exact same deployment issue with cdk whenever I made a small code change and tried to redeploy.

But when I installed node v16.3.0 with nvm (which I believe is the earliest recommended node v16 version that cdk supports) and tried to run a cdk deploy, for some reason it magically worked, each time. It was the exact same when I was trying to deploy with v14 too.

So I've confirmed that cdk deploy with node versions v14.18.1 and v16.3.0 both work. I still have no clue why deployment fails, at least in my case, for any node version above v16.10, but at this point I am just glad that I'm able to get cdk deploy working with any node 16 version. So I agree this issue can be closed, as I can probably chalk it up to general weirdness and something wrong with my Mac OS environment. I still have no idea what the cause of the error was, but at least now I am happy that cdk deploy with a stable node v16 version works for me at least. 👍

[peterwoodworth]

I'm glad you got it working at all rnag 🙂

FWIW, I keep my system at v16.7.0 unless I have a specific reason to switch (no particular reason I stick to this version, it was probably the most recent version when I got this machine lol) and have never run into any issues.

I'll keep this issue in mind, especially the fact that you're running into issues specifically at 16.10. Thanks for the information you've provided after troubleshooting this, I'm sure it will help others in the future

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

[peterwoodworth]

Reopening because customers are still running into this.

According to the comment on the stack overflow post, it looks like the etag isn't getting set properly

MultipartUpload: {
    Parts: [
      { ETag: null, PartNumber: 1 },
      { ETag: '"7ec273b6eeff48c007f37b4f43953657"', PartNumber: 2 },
      [length]: 2
    ]
},

I'm not sure where in our codebase this is handled, I couldn't find anything related to this when searching the codebase. Something to investigate

[93jeffboggess]

I am also experiencing this same issue. A downgrade to 16.3 resolved the immediate issue, though that version is going out of support in June of this year.

[corymhall]

For anyone still seeing this issue ( @93jeffboggess ) can you let me know what version of the CDK CLI you are using? There was a similar issue in the aws-sdk which we incorporated in CDK v2.44.0.

[github-actions[bot]]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[lihanghe]

We are still seeing the same issue both in CDK version v2.96.2 and v2.104.0, with both tried under Node versions v16.18.1 and v18.12.1. Changing to Node version v16.3.0 resolved the issue, but the version is out of support now.

  MultipartUpload: {
    Parts: [
      { ETag: null, PartNumber: 1 },
      { ETag: '"7b113fd8681c3e4c8d1007b210671309"', PartNumber: 2 },
      { ETag: null, PartNumber: 3 },
      { ETag: '"dafe87db1b3ad8e101d39850731cd05e"', PartNumber: 4 },
      [length]: 4
    ]
  },

[khushail]

Reopening this issue as Customer is facing issue with latest node version