search

PetaByet / cdp

Simple & Open Source Server Backups
https://cdp.me
GNU General Public License v2.0
123 stars 39 forks source link

Support multi-threaded parallel compression tools ? #16

Open centminmod opened 9 years ago

centminmod commented 9 years ago

Instead of tar gzip compression, how about support for parallel multi-threaded compression such as pigz (parallel gzip) and lbzip2 and pbzip2 http://vbtechsupport.com/1614/ ?

PetaByet commented 9 years ago

Don't these use much more CPU resources?

alexandreteles commented 9 years ago

Not so much. You will win in the CPU usage time because multi-threaded compression can be +2x faster (based at @centminmod's benchmarks) than single-threaded. But you can always give to the user the option to choose what tool he wants to use to compress the files. I think that multi-threaded compression use is a very good idea.

centminmod commented 9 years ago

yeah use more cpu but faster backups... for most folks speed of backups would be more important than cpu usage.. after all what's the point of keep cpu usage at <100% i.e. <25% 24x7 you're just wasting 75% of cpu processors idling away doing nothing :)

You can also give users more control over how many threads they use for parallel compression i.e. only use 4 threads on 8 cpu thread server to cap usage to 50% total cpus.

Better to keep PHP process duration to a minimal and get the backups done quicker IMHO :)