search

andrewrk / node-s3-client

high level amazon s3 client for node.js
MIT License
1k stars 303 forks source link

Error: Downloaded size does not match Content-Length #225

Open mirajehossain opened 4 years ago

mirajehossain commented 4 years ago

I am facing Error: Downloaded size does not match Content-Length when I call downloadDir I am downloading the s3 folder with various .ts files with various size (5MB - 300MB)

Error: Downloaded size does not match Content-Length at MultipartETag.<anonymous> (/home/xxx/yyy/node_modules/s3/lib/index.js:482:25) at MultipartETag.emit (events.js:198:13) at MultipartETag.EventEmitter.emit (domain.js:448:20) at endReadableNT (_stream_readable.js:1145:12) at process._tickCallback (internal/process/next_tick.js:63:19)

How do I fix it? Thanks.

cunneen commented 4 years ago

EDIT : See my follow-up comment below. What follows is the original comment.

I had this issue. I switched to @auth0/s3 instead of this library and the issue went away.

npm uninstall s3 --save
npm install @auth0/s3 --save

Then to use:

const s3 = require('@auth0/s3');
mirajehossain commented 4 years ago

@cunneen thank you very much.

cunneen commented 4 years ago

I just thought I'd update this: although the auth0 version seemed to fix it for one case, I still ran into this issue. Further investigation led me to a different library whose Readme contains the following explanation for its existence:

Why would I use this instead of node-s3-client?

In order to compare files, node-s3-client attempts to reimplement the hashing algorithm used by the AWS API in creating an s3 object's etag. My experience is that is doesn't always work (especially with large files).

I'm therefore giving up on this library and its forks, at least for now, and investigating alternatives.