search

facebook / zstd

Zstandard - Fast real-time compression algorithm
http://www.zstd.net
Other
23.45k stars 2.09k forks source link

ZSTD Decompression Speed vs Compression Level #4091

Closed VipulGarg869 closed 1 week ago

VipulGarg869 commented 3 months ago

I have been benchmarking the decompression speed of ZSTD at various compression levels. The documentation states that the decompression speed doesn't change much with the compression level. While that is roughly the case, I was still able to find that the decompression speed decreases up till compression level 6. Beyond level 6, it increases. What is the reason for this?

Cyan4973 commented 3 months ago

A variation in the number of sequences produced.

Best decompression speed is typically achieved at level 15, though some sources may prefer level 1.

VipulGarg869 commented 3 months ago

Thanks for your reply. How does the compression level affect the number of sequences produced? I believe that the parameters that compression levels affect increase/decrease in a monotonic manner as we increase the compression level. What then leads to the decompression being slowest at level 5/6?

Cyan4973 commented 2 months ago

This is more complex than that, and is difficult to summarize in one sentence. libzstd produces one format, but actually consists of a collection of different algorithms, each with their own speed / ratio trade offs to occupy their slot in the curve. One would have to look at the implementation details of each variant to understand why the nb of produced sequences differ per algorithm, and in the end, it makes sense. But it's way more complex than a simple monotonic increase.