search

archiverjs / node-archiver

a streaming interface for archive generation
https://www.archiverjs.com
MIT License
2.8k stars 220 forks source link

Corrupt ZIP archive when streaming in 200+ files #602

Open nick-george opened 2 years ago

nick-george commented 2 years ago

Hi there,

I've been troubleshooting an issue where archiver appears to be generating corrupt archives. Using version 5.3.1 on node v16.13.1.

We're streaming in files that have been retrieved from ssh2-sftp-client.

This library seems to work fine with very large archives built from a few large files. It also seems to work fine for archives up to 199 files. However, when I have 200 files or more, the archive gets corrupted. By diffing the hexdumps of one archive that has 199 files and another that has 200, I can see the archive with 200 files is missing the "End of central directory record" (EOCD). See below for the bytes that are missing from my archive with 200 files (note the first four bytes below are the last part of the last filename in the archive). 

0039b9b0  65 2e 70 70 50 4b 05 06  00 00 00 00 c6 00 c6 00  |e.ppPK..........|
0039b9c0  f4 3c 00 00 c0 7c 39 00  00 00                    |.<...|9...|
0039b9ca

Otherwise, the generated files are pretty much identical (except for one less file being present in the "good" archive).

Are you aware of any file count limit for this library?

Many thanks, Nick

nick-george commented 2 years ago

I've just tested again with TAR instead of ZIP, and I'm able to create archives with > 3000 files with no issue. I'm using exactly the same code (with TAR instead of ZIP) and the same sets of test files.

Cheers, Nick

hugofialho commented 1 year ago

Same problem here, i tried to zip 605 small files(~6Kb each). When i decompress the ZIP all content are with 0Kb size. After change to TAR everything works fine.

kanjieater commented 11 months ago

confirmed I'm running into file corruption for my zip comic archives as well. I'm not sure that switching to tar will be a viable solution for me