search

sreedevk / deduplicator

Filter, Sort & Delete Duplicate Files Recursively
MIT License
281 stars 15 forks source link

[Feature] Mass Processing Options --keep-latest --keep-oldest #44

Open sreedevk opened 1 year ago

sreedevk commented 1 year ago

Is your feature request related to a problem? Please describe. The Interactive mode allows the deletion of files one duplicate group at a time. When working with millions of files, this can be tedious.

Describe the solution you'd like In order to automate this process by using deduplicator in scripts, adding options like --keep-latest --keep-oldest can help.

Describe alternatives you've considered adding custom config files that can parse a DSL to decide which files to keep [idea for the future]

lovelock commented 8 months ago

Glad you've considered this problem already!

I did a similar job for my Synology NAS yesterday, which is pretty raw and not so performant. (Actually I hadn't find your awesome job before mine is pushed to github).

Here is my thought about this problem. Once a file is tagged duplicated, there's no need to determine whether the latest or the oldest should be kept, either is OK. So in my implementation, there is a counter to indicate whether the same file is a first found. If the the value of the counter is larger than 1 then the file holds the hash (counter) would be moved to a directory which is set before execution, with the relative path reserved.

Thanks to your work, about 90,000 of unique files out of nearly 300,000 files, with json output. Next I'll have to figure out how to tackle them.

One more thing, your application is so performant that my NAS with N5105 CPU was drained up and could respond nothing when running it.

sreedevk commented 8 months ago

@lovelock Thank you for your kind words. I am glad you enjoyed using deduplicator. That's an excellent suggestion. I am in the process of implementing a TUI for deduplicator (which includes some restructuring of the app). Once that's done, I will start working on all these improvement issues & your input will definitely be of value when I do.