search

pkolaczk / fclones

Efficient Duplicate File Finder
MIT License
1.93k stars 73 forks source link

Dedup on btrfs (and others) #54

Closed ChrisJefferson closed 2 years ago

ChrisJefferson commented 3 years ago

Btrfs supports in place dedup ( https://btrfs.wiki.kernel.org/index.php/Deduplication ), via a syscall. This is completely safe, as checks if the files are identical before deduping.

This would be very Linux specific low level code.

pkolaczk commented 3 years ago

Thanks, this is a great idea!

ilyapopov commented 3 years ago

See also FICLONE ioctl which is supported by a few CoW filesystems (Btrfs, XFS, and also NFS and SMB for server-side clones) Exists since Linux 4.5.

There is also copy_file_range syscall.

See some discussion on relative merits of these here: https://stackoverflow.com/questions/65505765/difference-of-ficlone-vs-ficlonerange-vs-copy-file-range-for-copy-on-write-supp

By the way, cp implements this functionality under --reflink flag, so I suggest command name to be reflink.

th1000s commented 3 years ago

See #74, I'd appreciate some design feedback.

pkolaczk commented 2 years ago

74 and #79 merged. Thank you @th1000s!