Request: Multi-threaded compression

[johnsanc314]

Unless I am completely overlooking something, I see no way to enable multi-threading.

Is there a way to enable this? If not, is it in the roadmap to support?

[oleg-st]

There is currently no way to enable multi-threading. This may be added in the near future

[oleg-st]

In the original ztd library this is achieved using the ZSTD_c_nbWorkers parameter in streaming compression, I can port this.

The code would be like this:

using var input = File.OpenRead("dickens");
using var output = File.OpenWrite("dickens.zst");
using var compressionStream = new CompressionStream(output, level);
compressionStream.SetParameter(ZSTD_cParameter.ZSTD_c_nbWorkers, 4);
input.CopyTo(compressionStream);

@johnsanc314 Is this what you need?

[johnsanc314]

Yes! that's precisely what's needed. I noticed in the code its basically locked down to single thread everywhere, but not sure the implications of allowing multi-threading like you described.

If you can add that it would be a massive performance enhancement.

[oleg-st]

Added multi-threading support in master

[Spacefish]

Haha ran into that literally today, i tried to

compressor.SetParameter(ZstdSharp.Unsafe.ZSTD_cParameter.ZSTD_c_nbWorkers, Environment.ProcessorCount);

but got the: "Unsupported parameter"

thanks for adding it!

[oleg-st]

Released in 0.7.5

[johnsanc314]

Amazing! Thank you for such quick turnaround. It was a long shot even asking, but I'm glad I did. Sounds like several others were wanting this feature as well.