search

jimmywarting / StreamSaver.js

StreamSaver writes stream to the filesystem directly asynchronous
https://jimmywarting.github.io/StreamSaver.js/example.html
MIT License
3.97k stars 413 forks source link

Zip64 #114

Open DannyZB opened 5 years ago

DannyZB commented 5 years ago

Quote from Zip saver: " // NOTE: My zip library can't generate zip's over 4gb and has no compresseion // it was built solo for the purpus of saving multiple files in browser "

Are you talking about not having zip64 support or about some other limitations(Like blob size?) A Zip64 generator for unlimited size would be extremely useful.

What needs to be done to support that? is it doable?

jimmywarting commented 5 years ago

Are you talking about not having zip64 support

Yes

or about some other limitations(Like blob size?)

Nope, everything you write is written to the hard drive in a streaming mannar. so there is no blob size limit

A Zip64 generator for unlimited size would be extremely useful.

What needs to be done to support that? is it doable?

Yes it would, it's also doable. I didn't know how first but i have recently learn how to parse a zip64 file. The next thing would be to generate one in the same way... By forcing it to be a zip64 format from the beginning

(some zip application can pre determine if the zip entry should be in good old 32 bit or 64 bit format by looking at the file size, but StreamSaver is all about saving a stream with an unknown size so it must always use 64 bit to represent the file size of files etc)

DannyZB commented 5 years ago

The one issue I can anticipate is Zip64 implementations are a mess. I have implemented a Zip64 generator in PHP in the past, had issues making it work with some file managers.

Most of the work on this will be testing.

As for other implementations, is https://github.com/Stuk/jszip compatible with StreamSaver? That would be a good match for both if not a lot of work is required.

https://github.com/Stuk/jszip/blob/9fb481ac2a294f9c894226ea2992919d9d6a70aa/lib/zipEntries.js

It seems Zip64 support is clearly available there

jimmywarting commented 5 years ago

As for other implementations, is https://github.com/Stuk/jszip compatible with StreamSaver? That would be a good match for both if not a lot of work is required.

Somewhat yes. but i wrote my streaming zip version for a reason. jszip where too bloated and to slow. uses to much memory and the development have been put on halt. To get some kind of streaming with streamsaver you need to dig into this issue: https://github.com/Stuk/jszip/issues/343

the only advantage you get by using jszip is compression, but then you can't stream everything as good

https://github.com/Stuk/jszip/blob/9fb481ac2a294f9c894226ea2992919d9d6a70aa/lib/zipEntries.js

It seems Zip64 support is clearly available there

I have digged into jszip a lot and what you linked to seems only to be about parsing zip64 files i don't believe jszip ever when to writing support for generating zip64 files since having 4gb+ data in memory was a limitation in the first place

DannyZB commented 5 years ago

Right. Just read the topic, seems like an ill-maintained project. How much work do you expect for Zip64 support? did you look over the spec?

jimmywarting commented 5 years ago

Tried to. Was hard so i did it reverce to try and read a zip64 File first to understand the format. Shouldn't require so many changes Urs more about getting it right

DannyZB commented 5 years ago

Imo the spec is incomplete. The actual work time required is around 40 hours on this.

EdvardasDlugauskas commented 4 years ago

Has there been any progress on the zip64 generator? Does any library support it?

Would be really nice to use this to zip multiple azure blob files into a big zip stream without having to download the files first

jimmywarting commented 4 years ago

none yet

jimmywarting commented 4 years ago

got disappointed that mac default archiver don't support zip64 😞

DannyZB commented 4 years ago

Don't worry about the Mac one. They use an old implemention of unzip on the OS level. Any mac user that downloads large archives should know to use Unarchiver, no way around that.

I dug through some old stuff, found this: https://github.com/maennchen/ZipStream-PHP/blob/master/src/ZipStream.php

It is pretty short and maps out perfectly what goes into Zip64 It's PHP, but the shortest example I found of this implemented.

NikxDa commented 4 years ago

Any update on this? @jimmywarting

bencmbrook commented 4 years ago

Think we're getting close here https://github.com/transcend-io/conflux/compare/jimmywarting/zip64. Happy to work together on this Jimmy

BioEvo commented 4 years ago

Think we're getting close here https://github.com/transcend-io/conflux/compare/jimmywarting/zip64. Happy to work together on this Jimmy

Is it ready for deployment now?

DannyZB commented 3 years ago

Is there a prototype? Do you guys need help finishing it?

christianwengert commented 3 years ago

I am still not yet fully mastering the streams thing, but for me it looks like https://github.com/archiverjs/node-archiver/issues could solve this problem as the archiver has seemingly zip64 support and streams, would that be something worth to try out?