hpc / mpifileutils

File utilities designed for scalability and performance.
https://hpc.github.io/mpifileutils
BSD 3-Clause "New" or "Revised" License
170 stars 68 forks source link

ddup support for mulitple paths? #570

Open markmoe19 opened 8 months ago

markmoe19 commented 8 months ago

Hi,

ddup accepts "one and only one directory" to scan for duplicates. Would it be possible to allow for multiple folders to scan? And, could it do this across multiple file systems?

Our use-case is setup like /lustre/fs1, /lustre/fs2, /lustre/fs3 (for example). We want to look across all 3 file systems for duplicates. Ideally with something like this:

ddup --open-noatime /lustre/fs*

Maybe we could provide /lustre as the path, but we don't really want all the mounts under /lustre and we are not really sure that would work (or want to test it). But, we do want all the folders matching /lustre/fs* to be scanned together for duplicates.

Thanks!