search

facebook / zstd

Zstandard - Fast real-time compression algorithm
http://www.zstd.net
Other
23.75k stars 2.11k forks source link

multi-threaded compression and dictionary #4189

Open sreendra opened 1 week ago

sreendra commented 1 week ago

Is your feature request related to a problem? Please describe. We have an application which is multi threaded and are planning to compress using a digested dictionary. When i used the same ctx for compression, intermittently i am getting Access violation reading location. Then after changing it to non-static, the issue subsided as mentioned in the manual ?

Describe the solution you'd like Can we make it 1 time init and use it across threads ? This saves init time/thread ?

Describe alternatives you've considered NA

Additional context NA

Cyan4973 commented 1 week ago

Dictionaries can be shared between threads.

sreendra commented 1 week ago

Thanks for the info @Cyan4973 .

I have not benchmarked how long it is taking for compression context and decompression context initialization. But assuming it's an overhead on top of every request ?

Is it possible to make it as a static variable to be leveraged across multiple threads ?

Cyan4973 commented 1 week ago

A compression context can be re-employed, but it cannot be shared concurrently across multiple threads.

Re-using a context is generally good for speed and reduced memory pressure. The initialization cost also gets reduced considerably.

sreendra commented 1 week ago

Sure @Cyan4973 ...So i have to live with it is it ? Because my app is a multi threaded scenario.

Thank you for your prompt responses

Cyan4973 commented 3 days ago

Ensure that each thread has its own compression context.

Then they can work in parallel, sharing the same dictionary.

sreendra commented 2 days ago

Sure @Cyan4973 ...Thank you...

Are there any plans in future to even make this context shareable across threads because it saves lot of time along with storage ?