search

Touffy / client-zip

A client-side streaming ZIP generator
MIT License
351 stars 22 forks source link

[FEATURE] Support ZIP64 #14

Closed Touffy closed 3 years ago

Touffy commented 3 years ago

As discussed on the roadmap and in #12, ZIP64 support is a highly desirable improvement for client-zip as it allows the creation of ZIP files larger than 4GB and containing files larger than 4GB themselves.

Once implemented, ZIP64 output would be the default for client-zip. That is consistent with other ZIP libraries, and the overhead is negligible. Making it the default is a slightly breaking change, but ZIP64 justifies a major version bump anyway.

Update 1:

While BigInts themselves — flawed though they may be — are finally available in browsers, the BigInt64 ArrayBuffer features have not yet landed in all browsers. But they can be polyfilled quite easily (actually I only need one: DataView.setBigUint64).

So that's what I'm going to do. The plan is to release client-zip 2.0 soon, with ZIP64. However, it will only work in browsers that support BigInts.

Update 2:

I just got the wrong byte order for the Zip64 extra field signature…

Update 3:

After some hexdumping and reverse-engineering of Zip64 files, I got client-zip to make valid ones too (or at least, files that 7zip and unzip don't complain about).

ricky11 commented 3 years ago

Wow. Thanks for your prompt action.

I will be testing this out soon. i like that zip64 will only be used when its needed

What would you suggest for a fallback for ie 11! and safari 11.x I know its sad.

Touffy commented 3 years ago

Nothing you can really do in older browsers. JSZip supports them, but can't stream, so you will run into memory (and speed!) issues very quickly with your large files. You'll have to give up on them or fall back to server-side zipping.

Touffy commented 3 years ago

And yeah, even the latest Safari will need the same fallback because of the ServiceWorker streaming bug.

The only compatibility issue you can work around easily would be the lack of BigInts in browsers like Chrome and Firefox from 2018-2019. That can be polyfilled. You don't even need arbitrary BigInts, just 64-bit.

owpac commented 3 years ago

Do you have any idea on the release date for Zip64 support ? Really interested in it.

Also, after some research on the subject, I found this package DownZip which seems to support Zip64 format. I'm sharing it here just in case, if it can help you. Most interesting thing for you is maybe some of their notes about Zip64 format that could help you either to improve or confirm your reverse-engineering work on the format!

Touffy commented 3 years ago

Actually, it works now — as far as I can tell. I was waiting for some feedback from other users to release version 2.0.0 but you can get version 2.0.0-rc2 already with Zip64 (and tell me if you happen to make a corrupted archive with it).

Thanks for the link. I'll make sure I haven't missed something.

owpac commented 3 years ago

To give you some feedbacks, I tested version 1.3.0 and 2.0.0-rc2 with a 5,63Go file. I'm on a Macbook Pro M1, with an implementation of client-zip with Stream Saver. I'm also using the default Mac unarchiver.

On v2.0.0-rc2, everything seems to work perfectly thanks to your integration of Zip64 format. On v1.3.0, I successfully downloaded my file in a zip, but I can't extract it without an error which is normal since Zip32 can't support file over 4Go.