search

madler / pigz

A parallel implementation of gzip for modern multi-processor, multi-core machines.
http://zlib.net/pigz/
2.65k stars 176 forks source link

MPI support #83

Open tomek-szczesny opened 3 years ago

tomek-szczesny commented 3 years ago

Hi, I'm a complete noob in MPI parallel programming and I can't stop the impression that this environment lacks software for daily, CPU-intensive tasks, such as file compression. If and when I get up to a point when I'm comfortable with MPI libraries, I'd gladly dive into implementing this feature. However, I was curious whether the approach of spreading the compression tasks over multiple machines in LAN has been tried in the past, and maybe it's been already proven to be pointless? Any thoughts would be much appreciated.

Code7R commented 3 years ago

Based on my experience with a competing project (create_compressed_fs from cloop-utils) it can be quite beneficial but you need to make sure that the pipeline is constantly filled, the latencies are reduced as far as possible (TCP_NODELAY, TCP_QUICKACK) and the network performs very well in general.

tomek-szczesny commented 3 years ago

I understand why high throughput would be beneficial, but low latency? Hm, not sure why would that help, as long as CPUs are the actual bottleneck of the whole process. Indeed each thread should have its own task buffer to stay busy around the clock.

ghuls commented 3 years ago

MPI parallel compression might not be needed. When using zlib-ng as zlib library, compression speed is 4 times faster: https://github.com/zlib-ng/zlib-ng/issues/908

tomek-szczesny commented 3 years ago

why not both? ;)