xcrh
commented
8 years ago Not like if I care about text books much, but when compressing (previously uncompressed) x86-64 Linux kernel, LZ5 proven to be quite interesting tradeoff.
It compresses much better than LZ4 does, though somewhat slower to decompress. But still fast. It also beats LZO, both in terms of ratio and decompression speed, actually improving both at once. Thanks to @Inikep hints I've also gave a shot to some "custom" levels and chopped off extra ~120K without losing decompression speed, first by setting match finding to much more aggresive settings and then also giving a try to larger dictionary, which improves things a bit on 22M chunk of data. Though it easy to make compression slow/memory hog while gain is modest vs sharp increase in resources consumptions. But it can be okay if one needs relatively small amount of precompressed data. In fact, it beats Crush level 2 in many cases. Which isn't that bad and is bit-aligned, not byte aligned. So it slower to decompress.
TBH I have some trouble finding proper "competitors" to LZ5. Most "fast LZ" things are just nowhere close in terms of ratio (and it is possible to get a bit extra at cost of compression speed). Closest in terms of ratio was LZO1Z-999. But it lost by noticeable margin. And was slower to decompress. Slower things usually go for entropy coding and/or at least bit-aligned streams, it tends to kill decompression speed, typically 2x or worse.
In case of kernel it is a like this: first you read it (slow storage gains from decrease of read amount). Then you decompress it (here you can lose all gains if decompression is slow). Stroger things like LZMA introduce noticeable delay, like 1 second on more or less fast PC or several times more on weaker systems (e.g. ARM boards). Next major competitor in terms of ratio is zlib, but it decompresses like 2x slower...
If you want to see what the best one can get from "just LZ" without "real" entropy coding, take a look on https://github.com/alef78/lzoma - on mentioned kernel it got me between Brotli level 6 and 7. But unlike Btrotli it lacks entropy coding, which makes it rather impressive. Just ultimate match finding and some tweaks of data format, which do not slow down decompression too much. But there is price. It is very slow and hogs memory a lot during compress...
inikep
Sanmayce
akdjka
rdp
mappu
Hi, inikep, happy new 2016!
Your lz5 fills the gap between lz4 and Zstd, nice. However, your next statement is valid mainly for mixed/binary data, yes? Or to be more precise, for compressors using up to 4MB dictionary, yes?
For textual e.g. English texts it is not in my quick tests, my latest&last Nakamichi 'Goldenboy' yields better ratio while being faster in decompression (try Skylake):
Note: I used for all files below '>lz5 -18 infile' command line.
Note: The results were obtained on my laptop with Core 2 Q9550s, Win7 64bit.
As a final note, I wonder why Hamid didn't implement LzTurbo 19 with bigger window as well?! I fully expect his state-of-the-art parser to kick our asses right away, hee-hee!