Using a hashfile is extremely slow

markfasheh / duperemove

Tools for deduping file systems

GNU General Public License v2.0

797 stars 80 forks source link

Using a hashfile is extremely slow #355

Open HK47196 opened 4 hours ago

HK47196 commented 4 hours ago

BTRFS on a 1TiB NVME ssd, arch linux(tried 0.14 from repos, and built the latest git version from AUR).

When using the hashfile option duperemove becomes incredibly slow. I was able to use the same command without the hashfile to dedupe the entire drive in a few minutes, when using a hashfile I let it run for around 40 minutes before killing the process.

Is this working as intended?

bogi788 commented 3 hours ago

I noticed similar behaviour yesterday (also an Arch) - after fixing my old scripts to account for changed command line arguments.