Closed xavpitz closed 3 weeks ago
Thanks for the feedback, I'll try to figure something out but not sure when I'll get the time.
Feel free to submit a pull request if you're able to figure out a solution.
I am a big newbie with github (and also with docker/ansible & co). I am just somehow good at Linux.. I had a look at some YT videos about how to do a "pull request" :). I am trying to do just that but it is quite a learning curve :)
Anyway, I was breaking my head how to do this the most efficiently, without breaking too much things...
My plan would be to use :
Then restrict :
This will probably require a stop of all containers & a "mv /opt/hms-docker/apps/transmission/torrents /opt/hms-docker/media_data/_torrents" at the OS level
For Radaar & Sonarr (and maybe some other containers that I don't know about) they need access to the whole : /opt/hms-docker/media_data. This will enable them to hardlink from _torrents --> _library. They don't need 2 volumes to be mounted anymore as this one will provide access to both of the directories.
Now I think most of the heavy lifting needs to be done in the docker-compose.yml.j2 but this looks super hard for me. I may give it a try but it will probably fail or I will break a lot of things...
Well... I have tried something at least... But it fails. https://github.com/ahembree/ansible-hms-docker/compare/master...xavpitz:ansible-hms-docker:hardlink_friendly Somehow I don't understand the syntax how to build variables. The additional ones I created are "empty"...
I think I see what you're attempting to do so I'll try to think of a solution. The biggest issue is changing file paths around due to existing installations and stuff. I very much don't want my repo responsible for impacting peoples personal media libraries.
I don't personally use hardlinks or anything and don't want to break my existing setup, but I recently got a Raspberry Pi and (finally) confirmed this repo works on it, so I'll try to use that as a testbed for this
Instead of having 3+ different folder variables for the downloads, what if there were a single downloads folder (hms_docker_downloads_path: "{{ hms_docker_mount_path }}/downloads"
for example), and within this folder there was a folder for each download type (torrent, sabnzbd, nzbget).
This way, there is 1 variable that would handle parent directory and the containers will handle creating the leaf directories, such as:
/opt/hms-docker/media_data/downloads/torrent
/opt/hms-docker/media_data/downloads/sabnzbd
/opt/hms-docker/media_data/downloads/nzbget
And these could be mounted in the containers like so:
# Transmission container:
- ${hms_docker_downloads_path}/torrent:/downloads
...
# Sabnzbd container:
- ${hms_docker_downloads_path}/sabnzbd:/downloads
...
# Nzbget container:
- ${hms_docker_downloads_path}/nzbget:/downloads
If we can get away with only modifying the download directory targets, that'd be preferred.
As long as the (new) "_downloads" and (current) "_library" are both sub-directories of "media_data" the hardlinking _downloads --> _library performed by radarr & sonarr & probably readarr should work by providing the whole :
but restricting :
should also be possible.
Your idea to have one common ${hms_docker_downloads_path} instead of the 3 different variables that I tried to build is very good. You obviously better have more experience dealing with this playbook ;) .
FYI, yesterday it was the first time for me forking a github project & trying to edit a "jinja" file. I've learned a lot... Anyway something was wrong in my syntax to build variables and the "make apply" failed miserably complaining about empty variables (the torrents path one).
Ok, I managed to pass the "make check" / "make apply" on my current "hardlink_friendly" branch.
I tried not to break the plex libraries (I can confirm that plex still finds the media) by not modifying the host:container _library mount points. Not breaking the _downloads part will be more challenging because this is the folder that needs to be moved on the host layer. It will at least require a "mkdir /opt/hms-docker/media_data/_downloads" & a "mv apps/transmission/torrents media_data/_downloads"
Thing is transmission broke because of the way I built the "hms_docker_downloads_path" variable which seems empty. It tries to mount the host : /torrents (which is non existing) instead of the /opt/hms-docker/media_data/_downloads/torrents.
I'm not sure which file I need to modify : vars/default/main.yml or roles/hmsdocker/defaults/main/main.yml . Both looks the same for me. I modified both. I also built two new variables in roles/hmsdocker/defaults/main/hmsd_advanced.yml . I'm pretty sure I miss some understanding in the way to concatenate text to build variables. It seems obvious, but it is not or I am missing something obvious.
You will only ever need to modify the variables in vars/custom
due to Ansible's variable precedence. The variables in roles/hmsdocker/defaults
are just the defaults of all possible settings in the event someone doesn't declare or deletes a variable. I'm also currently working on a way to reorganize the variables so that additional hosts can be defined in the inventory with different host-level variables (basically allowing greater scalability to more systems from 1 playbook)
As for concatenating strings, here may be a helpful example:
var1: "this is"
var2: "a test"
combined_var: "{{ var1 }} {{ var2 }}"
Ansible I believe follows the Jinja templating syntax in a way, and the above combined_var
variable would turn out to equal this is a test
Thanks for the help about the variable building. It was exactly as I expected. Thing is I did a syntax mistake in the jinja file. I had a "${ xxxxxxx }" string instead of "{{ xxxxxxx }}" at one specific place.
So I managed to have something working but it is still not 100% to my liking. With hindsight, I think that artificially limiting (with the host:container volume mappings) the apps to either _library or _downloads doesn't add much to the security but annoyingly prevents hard-linking & file move across folders.
I will continue to work on this. I will probably change the root for transmission and add a symlink to maintain backward compatibility...
Of course, thanks for trying to figure out a solution!
For what it's worth, you can reference .env
file variables in the docker-compose file by using ${VAR_NAME}
. This is regular Compose syntax and is not handled by Ansible. There's a mix of Compose-style variable references and Ansible-style in the docker-compose file due to referencing some of the variables from the .env
file. This was to help keep "secret" values out of the compose file and also allow people to change some things quickly in the .env file without having to do a full playbook run again
Going back through my compose file, I can see a few places to replace the Ansible reference with a .env reference, such as hms_docker_media_path
Hello,
The "Use Hardlinks instead of Copy" advanced "Media Management" option from radarr & sonarr is not working.
The consequence is that during torrent seeding, all huge video files are using 2x the space they could use is they were hardlinked one to the other ( /opt/hms-docker/apps/transmission/torrents <--> /opt/hms-docker/media_data/_library). This drastically reduce the potential seeding time because of the disk space usage implications and/or the library capacity.
I think the hard-linking doesn't work because of the way those 2 folders are presented to the radarr & sonaar containers. /opt/hms-docker/media_data/_library --> /data/media /opt/hms-docker/apps/transmission/torrents --> /data/torrents
Within the containers, as those two folders appears as different filesystems, hardlinking is not possible. This make radaar & sonaar fallback to copying the files over from torrents --> media.
Having a common volume for /data or something along those lines will probably solve this issue.
Regards,