search

sindresorhus / gulp-zip

ZIP compress files
MIT License
270 stars 47 forks source link

Cannot zip files 1GB or larger #58

Closed RoryDungan closed 8 years ago

RoryDungan commented 9 years ago

Trying to zip a file 1GB or larger fails with the following error:

RangeError: File size is greater than possible Buffer: 0x3FFFFFFF bytes

This is easily reproducible by creating a 1GB file ("truncate -s 1024M output.file" is probably the quickest way to test this Linux/OSX/Cygwin) and running gulp-zip on that directory. Multiple files just under 1GB work just fine, so it's definitely the size of each input file, not the combined size.

sindresorhus commented 9 years ago

Yeah, looks like you're hitting the Node.js Buffer limit.

Might be able to work around that by using the gulp stream mode here instead of concating the resulting stream: https://github.com/sindresorhus/gulp-zip/blob/0772beae388e356489709f39444acefdb40e1950/index.js#L58

But I don't have time to look into it right now, so PR welcome.

Docs: https://github.com/gulpjs/gulp/blob/master/docs/writing-a-plugin/dealing-with-streams.md

RoryDungan commented 9 years ago

Yeah, after this I had a look at a couple of other Node.js zip libraries and it definitely is to do with the Node buffer since they all also have the same problem. Thanks for that link though, I'll have a look into that.