ouch-org / ouch

Painless compression and decompression in the terminal
https://crates.io/crates/ouch
Other
2.18k stars 75 forks source link

Multithreaded zstd compression support #688

Open nalabrie opened 1 month ago

nalabrie commented 1 month ago

The Unix zstd tool can compress using multiple threads. Number of threads can be specified with the -T flag and the -T0 flag will "attempt to detect and use the number of physical CPU cores" (from the man page).

I suggest having the option to enable zstd multithreading or perhaps making it the default in a fashion similar to the -T0 flag mentioned above.

In my experience, this makes a huge difference when using a high compression level. I just compressed 555MB of data using ouch c --slow and it took around 10 minutes. Then when using a similar maximum compression zstd command on the same data, it took about 30 seconds (sorry for not timing properly). Resulting file size was roughly the same, but I can't be certain how well I matched the settings used by Ouch's --slow flag.

A quick search of the zstd crate docs used by Ouch shows that it supports multithreading.