theophanemayaud / video-simili-duplicate-cleaner

https://theophanemayaud.github.io/video-simili-duplicate-cleaner/
GNU General Public License v3.0
51 stars 5 forks source link

paralelize file discovering #116

Open theophanemayaud opened 1 year ago

theophanemayaud commented 1 year ago

The initial step is finding files. This is currently quite slow, at about 75/s when testing with the intel build on m1 Mac, scanning an external SSD.

This takes up 100% of one core. There must be a way to spread the load over all cores !! Like dividing main folders, having a thread per folder, or something like that.