search

rust-lang / flate2-rs

DEFLATE, gzip, and zlib bindings for Rust
https://docs.rs/flate2
Apache License 2.0
893 stars 160 forks source link

support 'deflate64'? #365

Closed xudesheng closed 1 year ago

xudesheng commented 1 year ago

I'm using the zip 0.66 crate on top of flate2. It seems like deflate64 is not supported in flate2. Windows will use the deflate64 algorithm to zip files when the file size is 2G or more.

Is there any plan to support deflate64?

Byron commented 1 year ago

I have created an example as well and can add files larger than 4GB without problems. Could you add a program that reproduces the issue here?

oyvindln commented 1 year ago

DEFLATE64 is not supported by any of the underlying libraries (zlib, zlib_ng, miniz_oxide - though zlib has some separate source code for decompressing it in it's contrib section). zip can handle >2G files fine with the normal deflate algorithm. The ZIP64 extension to the zip format is what's used for that to handle the 32-bit size issue from the original spec.

That's is not the same thing DEFLATE64. Apparently the zip compressor in windows uses DEFLATE64 on >2G files still for whatever reason otherwise there wouldn't be much use for for libraries/apps dealing with the .zip format to have support for it as it doesn't seem to give all that much advantage over standard DEFLATE and there are better and more modern algorithms like LZMA etc if you want something better than DEFLATE.

I don't know how much it differs from standard DEFLATE and how much it would take to e.g implement it in miniz_oxide. Since it's still not in zlib after all these years I wouldn't expect to see it in there any time soon. If support is added in one of the underlying libraries it would make sense adding the glue logic to swap method to flate2 as well but it's kinda borderline in scope here. If it differs enough that it would make more sense as a separate librarly to zlib/miniz_oxide it would probably not make sense to include with flate2.

xudesheng commented 1 year ago

@Byron @oyvindln Thank you for your response! @Byron My issue is about something other than adding large files. I need to read the content from the zip file which the Windows user generates to be used in my tool. My tool works fine if the zip file is not using the Deflate64 algorithm, even when the size is more significant than 2G.

However, it seems like Windows would automatically use the Deflate64 algorithm when the user is trying to compress files larger than 2G. I am curious to know whether the Windows user has the option to choose the algorithm or not ( I guess not).

@oyvindln I fully understood why the Deflate64 algorithm is not part of the underline zlib. But the headache is that Windows is so popular; on the other side, Windows makes this algorithm the default (transparent to the end-user) even though the benefit is very marginal. So, I have to consider how to support it in my tool. I agree with you that keeping it in zlib/miniz_oxide is better.

robjtede commented 1 year ago

This popped up today: https://crates.io/crates/deflate64

xudesheng commented 1 year ago

This popped up today: https://crates.io/crates/deflate64

Hope the zip-rs crate can leverage this and support deflate64 when unzip.

Byron commented 1 year ago

Thanks for sharing! I think in order to resolve this particular problem, it's more likely that zip-rs will make use of the deflate64 crate than flate2 to learn how to decode such streams.

I am following the new issue over on zip-rs to stay in the loop though.