search

PSeitz / lz4_flex

Fastest pure Rust implementation of LZ4 compression/decompression.
MIT License
460 stars 31 forks source link

support LZ4_decompress_faster using statistics #61

Open sundy-li opened 1 year ago

sundy-li commented 1 year ago

ClickHouse has implemented a fast lz4 decompress method using performance statistics.

Maybe it's a good approach to have it in lz4_flex crate.

PSeitz commented 1 year ago

Thanks for the hint, that's pretty interesting.

They seem to use two parameters <size_t copy_amount, bool use_shuffle>, while copy_amount are 8, 16 and 32.

lz4_flex uses a wild_copy of 16 currently, but depending on the architecture, more or less might be good. wild_copy will copy blocks of 16 bytes, event though the actual data required is less. That's possible when we are far enough from the end of the data (most of the time). https://github.com/PSeitz/lz4_flex/blob/main/src/block/decompress.rs#L35-L46

I don't know yet what the shuffle is about.

sundy-li commented 1 year ago

I don't know yet what the shuffle is about.

I don't know the detail either but looks like it will use some SIMD instructions to improve copy performance.

https://doc.rust-lang.org/beta/core/arch/x86_64/fn._mm_shuffle_epi8.html https://doc.rust-lang.org/beta/core/arch/x86_64/fn._mm_loadu_si128.html