search

rrrlasse / eXdupe

Ultra fast file archiver that supports data deduplication and differential backups
16 stars 2 forks source link

Feature suggestion: switch compression algorithm for level 3? #2

Open M-Gonzalo opened 11 months ago

M-Gonzalo commented 11 months ago

bzip2 is by now long superseded by other algorithms that provide better ratios for a fraction of the cost in terms of CPU, cspeed and dspeed.

Some alternatives are lzma, lzma2, and fast-lzma2, an lzma2 version with a fast match finder based on a radix sort. It can compress about ~2 - 3x times faster than lzma2, use much less memory, and efficiently utilize all CPU threads available for a negligible increase in memory (as opposed to conventional xz algorithms where every thread needs to have its own memory pool). The drawback is some ~1 - 2% less compression for single-threaded applications (but can have a much better ratio with big blocks compressed using several threads). Decompression speed is about the same as lzma2.

https://github.com/conor42/fast-lzma2

rrrlasse commented 11 months ago

You're right and that's definitely on the todo-list :)