search

inikep / lizard

Lizard (formerly LZ5) is an efficient compressor with very fast decompression. It achieves compression ratio that is comparable to zip/zlib and zstd/brotli (at low and medium compression levels) at decompression speed of 1000 MB/s and faster.
Other
644 stars 40 forks source link

Question: is seems block format does not allows progressive decoding? #9

Closed xcrh closed 7 years ago

xcrh commented 7 years ago

I'm not realy sure if this is issue at all, but I've got idea one may sometimes want to do a progressive block decoding. Say, specs say block could be fairly large and for e.g. networking it implies considerable latency, well over of network packet size. So I could imagine someone may want to try progressive block decoding to reduce latency a bit.

However, as far as I understand, current stream format assumes receiver have to receive complete block before being able to decode it reasonably, because some data required to begin decoding could be quite close to end of block. I'm not really sure if it actually a problem if such uses were not primary goal and it is somewhat advanced/uncommon usage scenario. I'm also not sure how it could affect entropy coding, IIRC you've told it is easier the way you do it right now.

inikep commented 7 years ago

When you are using lz5.exe there is -B option that divides input data into blocks of a given size. Then the subsequent blocks can be decompressed independently or progressively (if -BD is used). lz5.exe uses LZ5F encapsulation described in lz5frame.h and https://github.com/inikep/lz5/blob/lz5_v2.0/lz5_Frame_format.md.