search

facebook / zstd

Zstandard - Fast real-time compression algorithm
http://www.zstd.net
Other
22.77k stars 2.03k forks source link

Integrating the library with an external thread pool #4043

Open Zeblote opened 1 month ago

Zeblote commented 1 month ago

Is your feature request related to a problem? Please describe. The library creates its own thread pool when compressing a large file which then contends with the main application that uses it (especially a problem in i.e. a game engine).

Describe the solution you'd like An API to set a callback for it creating new tasks so you can execute them on your own thread pool when convenient would be nice. Basically fully replacing the pool implementation that comes with the library.

Describe alternatives you've considered Suppose the alternative would be to not use the multi-threading feature within a single task, since we can already run individual compression tasks when convenient.

Cyan4973 commented 1 month ago

So, you're asking for a feature equivalent to "shared threadPool", exposed at: https://github.com/facebook/zstd/blob/v1.5.6/lib/zstd.h#L1812 , but with the ability to inject an externally defined threadPool via callbacks ?

This would be a new feature indeed.

Zeblote commented 1 month ago

Yeah, something that would give control over when to run the tasks (and from which threads) to the user. There's an example of another library doing this here: https://github.com/NVIDIA-Omniverse/PhysX/blob/main/physx/include/task/PxCpuDispatcher.h#L49

It's a C++ library, but a similar pattern could easily be made in C by giving it a function pointer where it should submit new tasks to.