search

trapexit / mergerfs-tools

Optional tools to help manage data in a mergerfs pool
ISC License
388 stars 43 forks source link

Deduplicating data on NTFS storage pool #82

Closed slrslr closed 5 years ago

slrslr commented 5 years ago

Hello,

my NTFS filesystem contains some duplicate files (even they have different file names) and they are distributed in various directories.

I am looking for the software that will build a storage pool out of my NTFS HDDs and that SW will not waste space on duplicates but only store that file data once and make kind of links so the files still shows as duplicate on resulting NTFS filesystem, but in reality will be stored only once on physical HDD.

I have ARM CPU architecture and 512MB free RAM

trapexit commented 5 years ago

This isn't a mergerfs issue.

There are lots of dedup'ing tools around. They generally should work fine with mergerfs. rdfind and fdupes are two that come to mind.

slrslr commented 5 years ago

I want to leave duplicates visible (on resulting pooled filesystem), only physical data should be written once. Symlinks is not a solution for me.

trapexit commented 5 years ago

You aren't exactly explaining what you're trying to accomplish. You asked for links. There are hard and soft links. Not sure why one of those doesn't work for you.

slrslr commented 5 years ago

@trapexit I am trying to accomplish storage "pool" out of several drives that can contain duplicate files, but in fact only single file will use physical space. As said, symbolic links are not a solution for me, because the app that i am using is not properly reading file that is a symlink and hardlink i can not currently use because it works within single drive, not within two drives which i am now using, so i am looking for storage pool where i combine drives and lets say daily or weekly i will run some task which will make sure big duplicate files does not take unnecessary physical disk space on my drive pool. Also i need drive pool contents be encrypted when computer is not running.

trapexit commented 5 years ago

If you're merging drives why would you care about hardlinks not working cross drive? Hard links will work just fine in that the file would exist on one drive and the path would be created to match other drives and a hard link created on the drive with the file. The dup file would be deleted.