search

trapexit / mergerfs

a featureful union filesystem
http://spawn.link
Other
4.3k stars 174 forks source link

Mergerfs and Sonarr and Hardlink #788

Closed ghost closed 4 years ago

ghost commented 4 years ago

General description

Hi, I have 2 hdds

/mnt/disk1
/mnt/disk2
/mnt/storage
use_ino,cache.files=partial,dropcacheonclose=true,allow_other,category.create=epff

rtorrent download folder for movies is: I already created this folder myself so it should downkload here /mnt/storage/downloads/incomplete/ located in /mnt/disk1/downloads/incomplete/ /mnt/storage/downloads/completed/ located in /mnt/disk1/downloads/completed/
Using Sonarr docker to import or even add new movie to download I have tried many different combination and policies, I have been using EPFF for majority of times, and tried FF last night; same result.

Expected behavior

Hardlink movie after download is finished

/mnt/storage/downloads/completed => /mnt/storage/tv located in /mnt/disk1/tv

Actual behavior

It actually copies, so I have 2 x of the same file, verified it with using : ln -i 2 Different inode

Precise steps to reproduce the behavior

Explicitly list all steps to reproduce. Preferably create a minimal example of the problem using standard command-line tools. The more variables (apps, settings, etc.) that are involved the more difficult it is to debug. Also, please be sure to have read all of the README. It contains a lot of information regarding known system and user issues.

System information

Please provide as much of the following information as possible:

EDITS= Formatting

saltydk commented 4 years ago

Many people use this functionality perfectly fine so it is likely how you setup the mounts in the sonarr docker. If you mount /mnt directly and use the same in your torrent client you'll get around any path mapping bugs in sonarr. My guess is that this is 99% likely to be regarding path mapping but hard to say for sure without logs in sonarr or how you've set it up.

ghost commented 4 years ago

So I am using DockSTARTer (ds) for my apps. I chose /mnt/storage as base folder for ds and it automatically creates all folder for apps inside /mnt/storage and specifies each folder for the apps they offer. With Sonarr I can access 

/mnt/storage/tv 
/mnt/storage/.config
/mnt/downloads

Now I created those exact folder on a specific hard driver before hand and ds just takes over them with permission and uses them. Only when it gets to doing the hardlink, it actually copies the file hence my issue.

saltydk commented 4 years ago

Sonarr uses the path inside of your download client to know what source folder to use, is that path identical to the one used in sonarr?

ghost commented 4 years ago

in rutorrent I have this setup (or rather ds's setup) downloads automatically goes to: /mnt/storage/downloads/incomplete/ which is in /mnt/disk1/downloads/incomplete After its completed it, using AutoTool from rutorrent it transfers to: /mnt/storage/downloads/completed/ which is in /mnt/disk1/downloads/completed

And Sonarr supposedly needs to hardlink whatever I download to: /mnt/storage/tv which is in /mnt/disk1/tv But it copies the files and I end up with 2X of the original file.

In Sonarr settings the hardlink option is on, and download client page, the directory has not been changed since it should use rtorrent's default download folder (so left blank).

trapexit commented 4 years ago

Why do you have action to be ff?

If you aren't using path preservation policy, which ff isn't, then renames will just work. What do the logs say? What does a rename or link attempt from inside the container do?

ghost commented 4 years ago

So at first it was EPFF and it was like that anyway, and then last night I looked at forums and another issue on here, that you yourself suggested to use FF, so I tried that and same result.
Forgive me I am very new to linux, how can I provide the log to you? For clarification, I do not want the rename, I want the hardlink. I want to continue seeding and have a hardlink to tv folder with new nice names for Jellyfin.

trapexit commented 4 years ago

I'm not claiming changing the action policies would change hardlinking. It wouldn'.t That's not why I asked. You are explicitly setting action which defaults to epall. Anything besides "all" or "epall" will lead to weird behaviors and not recommended unless you know exactly what you're doing.

I know you don't want to rename but they work on the same premise.

I can't tell you how to find the logs. I don't know your setup. Docker images are just randomly created things. I've no idea who created the one you are using or how it gets setup. I'd hope they just output to stdout and you can use "docker logs" to see the log. Regardless, if you go into the container and can manually use "ln" then the problem is with how you've setup Sonarr.

ghost commented 4 years ago

oh for the action policies I also added it last night since someone suggested/had it in their setup so I added it, I guess it did nothing.
I think I went to the right place. Portainer > Sonarr> Log was useless, nothing useful so then > Console connect as root> went to download folder and try to ln it to tv folder, and it gave error of Invalid cross-device link

But they are all on same hdd (downloads folder and tv folder, even config folder), what am I doing wrong. The Sonarr image is from the LinuxServer.io one

trapexit commented 4 years ago

Clearly they aren't the same mount. stat the directories and you'll see. What are the volumes you've mounted in? What exactly is it you're doing. What is the command you ran?

ghost commented 4 years ago

Using Portainer console to Sonarr

root@len:/# stat /downloads/
  File: /downloads/
  Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: 2bh/43d Inode: 12785514231972323680  Links: 4
Access: (0775/drwxrwxr-x)  Uid: ( 1000/     abc)   Gid: ( 1000/     abc)
Access: 2020-07-27 07:56:33.608222468 +0100
Modify: 2020-07-16 17:16:40.437966694 +0100
Change: 2020-07-27 07:49:54.462978737 +0100
 Birth: -
root@len:/# stat /tv
  File: /tv
  Size: 12288           Blocks: 24         IO Block: 4096   directory
Device: 2bh/43d Inode: 5619969458148828617  Links: 3
Access: (0775/drwxrwxr-x)  Uid: ( 1000/     abc)   Gid: ( 1000/     abc)
Access: 2020-07-27 14:04:01.853347539 +0100
Modify: 2020-07-27 02:10:02.833271061 +0100
Change: 2020-07-27 13:57:58.177806868 +0100
 Birth: -
root@len:/# stat /downloads/completed/tv/
  File: /downloads/completed/tv/
  Size: 12288           Blocks: 24         IO Block: 4096   directory
Device: 2bh/43d Inode: 13210306293165644578  Links: 65
Access: (0775/drwxrwxr-x)  Uid: ( 1000/     abc)   Gid: ( 1000/     abc)
Access: 2020-07-27 14:03:27.252426187 +0100
Modify: 2020-07-27 00:43:35.841822715 +0100
Change: 2020-07-27 07:49:52.782985759 +0100
 Birth: -
root@len:/#

Using putty connecting to device directly


Last login: Mon Jul 27 14:26:31 2020 from 192.168.1.2
as@len:~$ stat /mnt/storage/tv/
  File: /mnt/storage/tv/
  Size: 12288           Blocks: 24         IO Block: 4096   directory
Device: 2bh/43d Inode: 5619969458148828617  Links: 3
Access: (0775/drwxrwxr-x)  Uid: ( 1000/      as)   Gid: ( 1000/      as)
Access: 2020-07-27 14:04:01.853347539 +0100
Modify: 2020-07-27 02:10:02.833271061 +0100
Change: 2020-07-27 13:57:58.177806868 +0100
 Birth: -
as@len:~$ stat /mnt/storage/downloads/completed/tv/
  File: /mnt/storage/downloads/completed/tv/
  Size: 12288           Blocks: 24         IO Block: 4096   directory
Device: 2bh/43d Inode: 13210306293165644578  Links: 65
Access: (0775/drwxrwxr-x)  Uid: ( 1000/      as)   Gid: ( 1000/      as)
Access: 2020-07-27 14:03:27.252426187 +0100
Modify: 2020-07-27 00:43:35.841822715 +0100
Change: 2020-07-27 07:49:52.782985759 +0100
 Birth: -
as@len:~$

fstab option is this now, original that you specified it in front page: use_ino,cache.files=partial,dropcacheonclose=true,allow_other,category.create=epff

saltydk commented 4 years ago

Show me the paths used inside the sonarr gui (root folder) and the download path shown in your download client.

trapexit commented 4 years ago

What are your volumes? You are showing /downloads and /tv. Those are different volumes / bind mounts? If that's the case then of course linking fails. You never can link cross mounts. Period.

ghost commented 4 years ago

I hope this is what you meant. https://imgur.com/a/Eijb6Mn

saltydk commented 4 years ago

This is what I meant and it means that your problems stems from either not path mapping in sonarr or it not doing it right (which is not unheard of). So either get help from the sonarr team to set it up right or use the mounting method of just mounting /mnt and have identical paths in both clients.

ghost commented 4 years ago 
as@len:~$ stat /mnt/disk1/downloads/completed/tv/
  File: /mnt/disk1/downloads/completed/tv/
  Size: 12288           Blocks: 24         IO Block: 4096   directory
Device: 821h/2081d      Inode: 190382081   Links: 65
Access: (0775/drwxrwxr-x)  Uid: ( 1000/      as)   Gid: ( 1000/      as)
Access: 2020-07-27 14:03:27.252426187 +0100
Modify: 2020-07-27 00:43:35.841822715 +0100
Change: 2020-07-27 07:49:52.782985759 +0100
 Birth: -
as@len:~$ stat /mnt/disk1/tv/
  File: /mnt/disk1/tv/
  Size: 12288           Blocks: 24         IO Block: 4096   directory
Device: 821h/2081d      Inode: 88997889    Links: 3
Access: (0775/drwxrwxr-x)  Uid: ( 1000/      as)   Gid: ( 1000/      as)
Access: 2020-07-27 14:04:01.853347539 +0100
Modify: 2020-07-27 02:10:02.833271061 +0100
Change: 2020-07-27 13:57:58.177806868 +0100
 Birth: -
as@len:~$ stat /mnt/disk2/tv
stat: cannot stat '/mnt/disk2/tv': No such file or directory
as@len:~$
saltydk commented 4 years ago

If you never made a tv folder on disk2 then ofc it won't be there?

ghost commented 4 years ago

If you never made a tv folder on disk2 then ofc it won't be there?

No I made sure that its definitely not trying to hardlink it to other disk. Just to clarify that the folder does not exist. Sonarr and rtorrent are both in docker container.

trapexit commented 4 years ago

What are your Docker volumes setup. Not what the directories are. That is irrelevant. You can not, ever, link or rename across devices. Bind mounts are different devices.

You show /downloads and /tv directories inside the container which implies you have two separate volumes.

ghost commented 4 years ago

Updated the album with Docker Volumes:

https://imgur.com/a/Eijb6Mn

trapexit commented 4 years ago

So it's what I thought. You can't do that. Has nothing to do with mergerfs. This wouldn't ever work.

saltydk commented 4 years ago

Would work with path mapping if just using one disk (as RW in mergerfs) and using something else to move stuff around to balance them (usage wise).

ghost commented 4 years ago

but if downloads folder and tv folder are both on same hdd, why can't I make hardlink within the sonarr docker? they are both on disk1 which is part of /mnt/storage. Did I get this wrong?

saltydk commented 4 years ago

Just because downloads and tv are mounting on the same disk in the host system doesn't mean sonarr knows about it.

saltydk commented 4 years ago

you'd need to path map to the same volume.

trapexit commented 4 years ago

No, that's not why.

It's because volumes are bind mounts. It does not matter if the underlying device is the same. Different mounts are different devices and you can not link or rename across devices.

trapexit commented 4 years ago

It has nothing at all to do with mergerfs or sonarr.

saltydk commented 4 years ago

I use it though, the hardlinking on mergerfs.

saltydk commented 4 years ago

image So once it is the same volume, as far as sonarr is concerned, since my root folder uses the unionfs pathing it works.

trapexit commented 4 years ago

They are different concerns all together. You can link and rename WITHIN a bind mount. You can not link or rename ACROSS bind mounts. Regardless of what is the underlying device.

saltydk commented 4 years ago

Okay but isn't his problem solved by the above?

trapexit commented 4 years ago

Yes, but I'm trying to explain why it doesn't work.

saltydk commented 4 years ago

Fair enough, sorry to bud in :)

ghost commented 4 years ago

so hardlink works in /mnt/disk1 /mnt/disk2 but never in: /mnt/storage
Is this correct? Sorru I am not really tech savy so I am not fully aware the right terminology and stuff

trapexit commented 4 years ago

Please, can you please show exactly what you're doing?

trapexit commented 4 years ago

links absolutely work in mergerfs IFF you have a non-path preserving create policy, ignorepponrename=true, or things work out as described in the docs: https://github.com/trapexit/mergerfs#rename--link

ghost commented 4 years ago

Sonarr>Add show>'the show'>wanted>manual import>selecting the previously downloaded folder of the show>manual import>shows me the episodes>I pick 1 episode to test and select copy which does hardlink since i have it checked in the option of sonarr> click on import
then I go to samba folder I see the file is getting copied, 10mb > 50mb > 100mb >.....> full size of the episode folder structures as mentioned above of course/

I also tried using Radarr to download a new movie from Radarr itself and the same result, it actually copies it once its finished downloading. took 2 x space

trapexit commented 4 years ago

As I said before if you are trying to link between /downloads and /tv it can never work. It has nothing to do with mergerfs. You MUST use a single mount. Change the volumes from /tv and /downloads to a single one. /mnt/storage to /storage or whatever you like.

trapexit commented 4 years ago

No. I mean a single mount. A single "volume" by Docker vocab. Just do as I said above.

Remove the volumes for /tv and /downloads and add /mnt/storage:/storage

ghost commented 4 years ago

YYYYYYYYYYYYYEEEEEEEESSSSSSSSSSSSSSSSSSSSSSSS Ok I removed those volumes and just added /mnt/storage:storage like you suggested, told sonarr the shows folder is /mnt/storage/tv/showname and manual import it from /mnt/storage/downloads
And it worked like a charm, hardlink and used ls -i to verify it. Tested it with multiple shows. Thank you very much. Sorry for my lack of knowledge and trouble I gave you. I really appreciat the time and effort to help me fix this issue even though it happen to be not related to Mergerfs.

trapexit commented 4 years ago

np