search

byronknoll / cmix

cmix is a lossless data compression program aimed at optimizing compression ratio at the cost of high CPU/memory usage.
http://www.byronknoll.com/cmix.html
GNU General Public License v3.0
601 stars 44 forks source link

Distributed processing [Question] #41

Closed patchthecode closed 5 years ago

patchthecode commented 5 years ago

Since the cost of this awesome compression tool is processing/ram, Is there any way the processing can be done on a distributed system (multiple systems doing the compression) ?

byronknoll commented 5 years ago

Unfortunately after every bit of input, all the data needs to synchronize in order to advance. Synchronizing the data between different systems after every bit has a lot of overhead, so it would make things really slow. This is also the reason that cmix uses only a single thread - multithreading slows things down for the same reason.

patchthecode commented 5 years ago

thanks.