search

CauldronDevelopmentLLC / cbang

C! (cbang) is a library for cross-platform C++ development.
GNU Lesser General Public License v2.1
56 stars 39 forks source link

Add mult-threaded lzma compression support. #79

Open jcoffland opened 2 years ago

jcoffland commented 2 years ago

The 7-zip LZMA SDK is public domain and supports multi-threaded compression.

Artoria2e5 commented 7 months ago

I recommend looking into the public domain XZ utils liblzma instead, at https://github.com/tukaani-project/xz. The liblzma part of XZ is also public domain. There are more pairs of eyes on that code base, because every Linux distribution uses it. There's also real source control with commit messages.

In your boost dependency, there already is an iostream filter for lzma that works through liblzma. I have no idea what your boost version is, but boost 1.83.0 evidently can use multithreaded lzma in its implementation (source code). The amount of code required should, as a result, be pretty close to what you have for bzip2 right now: a thin wrapper on top of boost.

Having lzma iostream is a really good idea, because it compresses usually better than bzip2 and decompresses usually faster. bzip2 should really not be a thing in the compress-once-then-distribute-widely use case now that lzma exists.

(There used to be some comment about the chunk size here. I misread the un-named struct members; boost's defaults are fine. I hate un-named struct initializations.)

Artoria2e5 commented 6 months ago

Well I’ll be damned. Some long-time maintainer just put a big backdoor in the last two versions of xz/liblzma and it’s all over the news. At least boost doesn’t rely on these two specific versions… and MT has been present before those two.