madler / zlib

A massively spiffy yet delicately unobtrusive compression library.
http://zlib.net/
Other
5.71k stars 2.45k forks source link

Record window_bit to support window size check in GZIP header #823

Closed yyang52 closed 1 year ago

yyang52 commented 1 year ago

ZLIB library supports history buffers of different sizes by setting the windowBits parameter, and ZLIB header has a field to record the window bit, which would be checked when decompressing. But seems like for GZIP format, it does not store such info (specifically, window bit). Though we know that in most cases, the window size would be set to 32KB by default, there would still be some scenarios where the memory efficiency is very limited, or the software stack is relatively old and does not have a large memory capacity.

For those cases, the user may want to set the window bits to a small number to save some memory. But there is no way to check window size when decompressing GZIP files, which might cause some issues when we compress with a larger window size but only have the capacity to decompress with a smaller window size.

I wonder why GZIP does not have such a field to store window bit, or if there is any possibility to add that for the convenience of de/compressing with different window bit? Thanks!

minchai23 commented 1 year ago

There is no field that can store windowbits in gzip format.

george-gu-2021 commented 1 year ago

There is no field that can store windowbits in gzip format.

Thanks for the feedback! Is there any workaround for that if decompressor side has to know the history buffer size?

madler commented 1 year ago

You could put it in an extra field in the gzip header. Or you could put it in the file name.