search

markfasheh / duperemove

Tools for deduping file systems
GNU General Public License v2.0
816 stars 81 forks source link

Please add an option to do `Find-dupes` on other machine #227

Closed galaxy001 closed 1 year ago

galaxy001 commented 4 years ago

I need to run whole volume dedup on a NAS with 4G memory. However, the database file is 18 G now. Which makes the 2nd step using swapfile and be to slow to complete.

Would you offer an option to further separate Find-dupes, to enable it run based only on the database file, and output a list of candidate file pairs. Maybe in the fdupes format.

Then, I can copy the list to my NAS, and do step 3.

JackSlateur commented 1 year ago

Hello @galaxy001

Since your report, all operations have seen their hardware requirement lowered With the implementation of batching, 4GB of memory should be able to process all kind of dataset

While there are more improvement to implement, could you check the latest code and reopen this if you still have the issue ?

Best regards,