search

hasse69 / rar2fs

FUSE file system for reading RAR archives
https://hasse69.github.io/rar2fs/
GNU General Public License v3.0
272 stars 25 forks source link

(soft/hard)link support? #175

Open KyleSanderson opened 2 years ago

KyleSanderson commented 2 years ago

hey man, long time.

I've been using rar2fs prolifically in pod deployments, specifically mounting a pv to a container running rar2fs, and then another container using that volume mount within the pod. The issue seems to be when a hardlink is requested within rar2fs it does a full copy, and because it's doing a full copy and can take ages to write on the same volume. I'm thinking forcing a softlink here from hard may be more desirable.

Specifically I'm using sonarr, the source is rar'd, and sonarr imports to its own structure the raw mkv file. Both sides in my environment are perpetually available, and the link request is from a rar2fs volume to the same disk. Other clients access the pv by their own rar2fs instance (nginx, in my case).

hasse69 commented 2 years ago

Hmm, I have now read your issue report several times and I am still not sure I fully understand what it is that is wrong ☹️ rar2fs has no special handling of hard links? What is happening would thus be something that is related to how FUSE works rather than something that is not supported by rar2fs? But then again, it is probably me...

edrock200 commented 2 years ago

I believe what @KyleSanderson is asking is if a file is not rar'd (just passed through to the mount), and an app requests a hardlink, can rar2fs do some sort of pass through hard link to the original file, so its not creating an actual second copy of the data, since hardlink file pointers only use 1x the file data no matter how many hardlinked copies exist.

@KyleSanderson this would be very cumbersome but should work - I'll try to explain as best I can but, lets say you have: Disk1 Disk1 /mnt/source - Where sonarr saves media too and rar2fs source dir Disk1 /mnt/dest - rar2fs destination mount of source

you could use something like mergerfs - e.g. make /mnt/unionfs then do something like mergerfs -o allow_other,use-ino,etc,etc /mnt/source=RW:/mnt/dest=RO /mnt/unionfs

The above should merge those 2 folders into /mnt/unionfs with /mnt/source being read/write and dest being read only. You can obv adjust policies as needed. If you use the first-found option with mergerfs, it should technically seek out the file from source first before dest. If the file exists in source, it will hard link it into storage. If its not in source, aka it's a rar file, it will find it in dest and that will be a full copy over to source.

Assuming your sonarr is in a container, you can simple remap your root path from /mnt/storage to /mnt/unionfs so you don't have to deal with updating the paths for all the shows. Then in your sonarr path mappings, just make sure the download client target path mappings match the paths you have for your root folders. e.g. if your root folder is /mnt/unionfs/TV/ShowXYZ then your download path map should be /mnt/unionfs/downloads/... (or /mnt/source/... if you remap at the container level) with the idea being that everything is nested in unionfs.

Of course, this will only work if the source and destination are on the same disk/storage.

KyleSanderson commented 2 years ago

Hmm, I have now read your issue report several times and I am still not sure I fully understand what it is that is wrong ☹️ rar2fs has no special handling of hard links? What is happening would thus be something that is related to how FUSE works rather than something that is not supported by rar2fs? But then again, it is probably me...

It's probably me haha.

Maybe scripting is easier to show the issue?

mount -t rar2fs.fuse /mnt/src /mnt/unrar
stat /mnt/unrar/.hidden/weird/file.mkv
ln /mnt/unrar/sub/file.mkv /mnt/unrar/.hidden/weird/file.mkv

The ln seems to fail and kicks back to a full copy. Clearly a hardlink here is not desirable because when the filesystem dies, or goes away we'll lose the data because there's no state maintained. When a hardlink is requested, as opposed to failing the operation, I'd like an override to create a symlink transparently instead.

Does that make more sense?

hasse69 commented 2 years ago

Ok, that was more useful. Still not sure about in what way this is related to how rar2fs works though, but I will take a look at it next week sometime when I am back from my current travels.

Note that since this seems to be a pass-through operation it should be FUSE that dictates what happens here. Btw, why did you not try a soft link? Or that would not work across file system borders?

edrock200 commented 2 years ago

I'm gonna go out on a limb here and say because he is using sonarr/radarr and they dont give that option. Probably because soft links arent as forgiving in that once you delete the source file, the link no longer works whereas hard links will continue to work. I believe you are right though, I don't think there is much you can do as it's a fuse operation, which is why I suggested mergerfs to layer the rar2fs source and target with preference being given to the source, so files that are not rar'd get preference and hard linked. For rar'd files, you can't hardlink them as they have to be extracted and that mount point with rar2fs target is considered a different volume.

edrock200 commented 2 years ago

Hmm, I have now read your issue report several times and I am still not sure I fully understand what it is that is wrong ☹️ rar2fs has no special handling of hard links? What is happening would thus be something that is related to how FUSE works rather than something that is not supported by rar2fs? But then again, it is probably me...

It's probably me haha.

Maybe scripting is easier to show the issue?

mount -t rar2fs.fuse /mnt/src /mnt/unrar
stat /mnt/unrar/.hidden/weird/file.mkv
ln /mnt/unrar/sub/file.mkv /mnt/unrar/.hidden/weird/file.mkv

The ln seems to fail and kicks back to a full copy. Clearly a hardlink here is not desirable because when the filesystem dies, or goes away we'll lose the data because there's no state maintained. When a hardlink is requested, as opposed to failing the operation, I'd like an override to create a symlink transparently instead.

Does that make more sense?

A hard link should continue to work if you delete the file from the original directory. All it's doing is deleting that links reference to the data, but not the data itself. But this only works if source and destination are on the same volume. E.g. if you hard link a file across 2 different volumes, it's essentially the same as a copy operation.

hasse69 commented 1 year ago

This is certainly a very interesting topic but it still fails to occur to me what is expected from rar2fs here and that would not be in direct contrast to normal file system semantics? The purpose of rar2fs is to include archived data as if it was already extracted. What happens with files outside of the archives is really something that up until now at least has been considered transparent. I am not sure why that should need to be changed or that is should even change for that matter.

edrock200 commented 1 year ago

This is certainly a very interesting topic but it still fails to occur to me what is expected from rar2fs here and that would not be in direct contrast to normal file system semantics? The purpose of rar2fs is to include archived data as if it was already extracted. What happens with files outside of the archives is really something that up until now at least has been considered transparent. I am not sure why that should need to be changed or that is should even change for that matter.

It doesn't. I wasn't trying to insinuate there is action needed on your part. Just trying to offer clarification and a potential solution to the op's original scenario.