search

adrianlopezroche / fdupes

FDUPES is a program for identifying or deleting duplicate files residing within specified directories.
2.42k stars 186 forks source link

Feature request : parallelize to make it run on multiple cores #103

Open kanihal opened 6 years ago

kanihal commented 6 years ago

I had ~125 GB of data files and it took > 1 day to complete fdupes run since it uses only a single core.

I think it would be very useful if an option of choosing the number of cores to run and parallelize fudpes accordingly.

mlybarger commented 4 years ago

i agree. i have lots of cpu's and would appreciate huge leaps in performance on large dirs (mp3, jpg, mp4 directories)

quixand commented 4 years ago

or possibly reducing the accuracy of the hashing with something like https://github.com/MatthewJohn/shamean ? to speed things up.

FabioPedretti commented 3 years ago

Blake3 is also a hashing function parallelizzable on multi core CPUs: https://github.com/BLAKE3-team/BLAKE3