search

trapexit / mergerfs-tools

Optional tools to help manage data in a mergerfs pool
ISC License
371 stars 42 forks source link

mergerfs.consolidate moving complete folder from 1 drive to another #115

Open chronic669 opened 3 years ago

chronic669 commented 3 years ago

So as I understand it this tool should keep files in a folder together, but it is moving a season folder that is already on 1 drive to another. So I ran the following command to see what it would move

mergerfs.consolidate /mnt/storage-data

The list it generated was long but this is a small part where the files its trying to move make no sense or maybe im misunderstanding this tools purpose. As I understand it if a folders files are already on a single drive then nothing should happen right

rsync -avHAXWES --numeric-ids --progress --relative --remove-source-files '/srv/dev-disk-by-uuid-214b19e8-755d-48c7-8837-22cfe2902767/./media/tvshows/hd/F/Fear The Walking Dead/Season 1/Fear.the.Walking.Dead.S01E01.Pilot.1080p.WEB-DL.DD5.1.H.264-NTb.mkv' /srv/dev-disk-by-uuid-ca6f8fdf-e56c-45ba-9cf4-47234ca1da14/
rsync -avHAXWES --numeric-ids --progress --relative --remove-source-files '/srv/dev-disk-by-uuid-214b19e8-755d-48c7-8837-22cfe2902767/./media/tvshows/hd/F/Fear The Walking Dead/Season 1/Fear.the.Walking.Dead.S01E02.So.Close.Yet.So.Far.1080p.WEB-DL.DD5.1.H.264-NTb.mkv' /srv/dev-disk-by-uuid-ca6f8fdf-e56c-45ba-9cf4-47234ca1da14/
rsync -avHAXWES --numeric-ids --progress --relative --remove-source-files '/srv/dev-disk-by-uuid-214b19e8-755d-48c7-8837-22cfe2902767/./media/tvshows/hd/F/Fear The Walking Dead/Season 1/Fear.the.Walking.Dead.S01E03.The.Dog.1080p.WEB-DL.DD5.1.H.264-NTb.mkv' /srv/dev-disk-by-uuid-ca6f8fdf-e56c-45ba-9cf4-47234ca1da14/
rsync -avHAXWES --numeric-ids --progress --relative --remove-source-files '/srv/dev-disk-by-uuid-214b19e8-755d-48c7-8837-22cfe2902767/./media/tvshows/hd/F/Fear The Walking Dead/Season 1/Fear.the.Walking.Dead.S01E04.Not.Fade.Away.1080p.WEB-DL.DD5.1.H.264-NTb.mkv' /srv/dev-disk-by-uuid-ca6f8fdf-e56c-45ba-9cf4-47234ca1da14/
rsync -avHAXWES --numeric-ids --progress --relative --remove-source-files '/srv/dev-disk-by-uuid-214b19e8-755d-48c7-8837-22cfe2902767/./media/tvshows/hd/F/Fear The Walking Dead/Season 1/Fear.the.Walking.Dead.S01E05.Cobalt.1080p.WEB-DL.DD5.1.H.264-NTb.mkv' /srv/dev-disk-by-uuid-ca6f8fdf-e56c-45ba-9cf4-47234ca1da14/
rsync -avHAXWES --numeric-ids --progress --relative --remove-source-files '/srv/dev-disk-by-uuid-214b19e8-755d-48c7-8837-22cfe2902767/./media/tvshows/hd/F/Fear The Walking Dead/Season 1/Fear.the.Walking.Dead.S01E06.The.Good.Man.1080p.WEB-DL.DD5.1.H.264-NTb.mkv' /srv/dev-disk-by-uuid-ca6f8fdf-e56c-45ba-9cf4-47234ca1da14/
trapexit commented 3 years ago

It's doing that because it doesn't care where the files come from, just where they might go, which is the drive with the most free space. Doing smarter things is more complicated and given the tool is more of a "oh shit, want to clean this up after screwing something up" not much time has been spent on it. Most people have no need for consolidation.

Crowley007 commented 1 year ago

I came across this problem today. I use epff policy for create, to minimize moving large files between disks. But I also like to use mergerfs.balance on the pool as well sometimes, which messes that up, splitting the directories. The goal is to run consolidate after balance, to merge the directories that balance split up to multiple disks, preferably only moving the least amount of data possible. Is this possible with existing tools? I guess the balance could be smarter as well, keeping the directory on only one drive when using path preservation.

Edit: To answer myself, If I only run consolidate on the sub folder that is split, it will work like I expect. My use case has a folder split with one large file on one drive and many small ones on another, so I guess I can write as script that indentifies split folders "smaller than X MB" and run on that. But I think mergerfs.consolidate could do with a feature that would only consolidate folders that need it, ie. "Only consolidate split sub folders" etc.