g2p / bedup

Btrfs deduplication
http://pypi.python.org/pypi/bedup
GNU General Public License v2.0
322 stars 50 forks source link

Speed-up deduplication by parallelization #100

Open dmromanov opened 6 years ago

dmromanov commented 6 years ago

Currently deduplication take a long time (especially on large filesystems). It seems that Bedup is doing file comparison in a single thread. It would be nice if Bedup would utilize more of computer's resources by doing some of it's tasks in parallel. For example, do hashing multi on multiple cores or hashing multiple files simultaneously..