search

ahembree / ansible-hms-docker

Ansible playbook for automated home media server setup
GNU General Public License v3.0
399 stars 50 forks source link

Huge disk space savings / optimization possible #89

Closed xavpitz closed 3 weeks ago

xavpitz commented 1 month ago

Hello,

The "Use Hardlinks instead of Copy" advanced "Media Management" option from radarr & sonarr is not working.

The consequence is that during torrent seeding, all huge video files are using 2x the space they could use is they were hardlinked one to the other ( /opt/hms-docker/apps/transmission/torrents <--> /opt/hms-docker/media_data/_library). This drastically reduce the potential seeding time because of the disk space usage implications and/or the library capacity.

I think the hard-linking doesn't work because of the way those 2 folders are presented to the radarr & sonaar containers. /opt/hms-docker/media_data/_library --> /data/media /opt/hms-docker/apps/transmission/torrents --> /data/torrents

Within the containers, as those two folders appears as different filesystems, hardlinking is not possible. This make radaar & sonaar fallback to copying the files over from torrents --> media.

Having a common volume for /data or something along those lines will probably solve this issue.

Regards,

ahembree commented 1 month ago

Thanks for the feedback, I'll try to figure something out but not sure when I'll get the time.

Feel free to submit a pull request if you're able to figure out a solution.

xavpitz commented 1 month ago

I am a big newbie with github (and also with docker/ansible & co). I am just somehow good at Linux.. I had a look at some YT videos about how to do a "pull request" :). I am trying to do just that but it is quite a learning curve :)

Anyway, I was breaking my head how to do this the most efficiently, without breaking too much things...

My plan would be to use :

Then restrict :

This will probably require a stop of all containers & a "mv /opt/hms-docker/apps/transmission/torrents /opt/hms-docker/media_data/_torrents" at the OS level

For Radaar & Sonarr (and maybe some other containers that I don't know about) they need access to the whole : /opt/hms-docker/media_data. This will enable them to hardlink from _torrents --> _library. They don't need 2 volumes to be mounted anymore as this one will provide access to both of the directories.

Now I think most of the heavy lifting needs to be done in the docker-compose.yml.j2 but this looks super hard for me. I may give it a try but it will probably fail or I will break a lot of things...

xavpitz commented 1 month ago

Well... I have tried something at least... But it fails. https://github.com/ahembree/ansible-hms-docker/compare/master...xavpitz:ansible-hms-docker:hardlink_friendly Somehow I don't understand the syntax how to build variables. The additional ones I created are "empty"...

ahembree commented 1 month ago

I think I see what you're attempting to do so I'll try to think of a solution. The biggest issue is changing file paths around due to existing installations and stuff. I very much don't want my repo responsible for impacting peoples personal media libraries.

I don't personally use hardlinks or anything and don't want to break my existing setup, but I recently got a Raspberry Pi and (finally) confirmed this repo works on it, so I'll try to use that as a testbed for this

ahembree commented 1 month ago

Instead of having 3+ different folder variables for the downloads, what if there were a single downloads folder (hms_docker_downloads_path: "{{ hms_docker_mount_path }}/downloads" for example), and within this folder there was a folder for each download type (torrent, sabnzbd, nzbget).

This way, there is 1 variable that would handle parent directory and the containers will handle creating the leaf directories, such as:

And these could be mounted in the containers like so:

# Transmission container:
- ${hms_docker_downloads_path}/torrent:/downloads
...

# Sabnzbd container:
- ${hms_docker_downloads_path}/sabnzbd:/downloads
...

# Nzbget container:
- ${hms_docker_downloads_path}/nzbget:/downloads

If we can get away with only modifying the download directory targets, that'd be preferred.

xavpitz commented 1 month ago

As long as the (new) "_downloads" and (current) "_library" are both sub-directories of "media_data" the hardlinking _downloads --> _library performed by radarr & sonarr & probably readarr should work by providing the whole :

but restricting :

should also be possible.

Your idea to have one common ${hms_docker_downloads_path} instead of the 3 different variables that I tried to build is very good. You obviously better have more experience dealing with this playbook ;) .

FYI, yesterday it was the first time for me forking a github project & trying to edit a "jinja" file. I've learned a lot... Anyway something was wrong in my syntax to build variables and the "make apply" failed miserably complaining about empty variables (the torrents path one).

xavpitz commented 1 month ago

Ok, I managed to pass the "make check" / "make apply" on my current "hardlink_friendly" branch.

I tried not to break the plex libraries (I can confirm that plex still finds the media) by not modifying the host:container _library mount points. Not breaking the _downloads part will be more challenging because this is the folder that needs to be moved on the host layer. It will at least require a "mkdir /opt/hms-docker/media_data/_downloads" & a "mv apps/transmission/torrents media_data/_downloads"

Thing is transmission broke because of the way I built the "hms_docker_downloads_path" variable which seems empty. It tries to mount the host : /torrents (which is non existing) instead of the /opt/hms-docker/media_data/_downloads/torrents.

I'm not sure which file I need to modify : vars/default/main.yml or roles/hmsdocker/defaults/main/main.yml . Both looks the same for me. I modified both. I also built two new variables in roles/hmsdocker/defaults/main/hmsd_advanced.yml . I'm pretty sure I miss some understanding in the way to concatenate text to build variables. It seems obvious, but it is not or I am missing something obvious.

ahembree commented 1 month ago

You will only ever need to modify the variables in vars/custom due to Ansible's variable precedence. The variables in roles/hmsdocker/defaults are just the defaults of all possible settings in the event someone doesn't declare or deletes a variable. I'm also currently working on a way to reorganize the variables so that additional hosts can be defined in the inventory with different host-level variables (basically allowing greater scalability to more systems from 1 playbook)

As for concatenating strings, here may be a helpful example:

var1: "this is"
var2: "a test"
combined_var: "{{ var1 }} {{ var2 }}"

Ansible I believe follows the Jinja templating syntax in a way, and the above combined_var variable would turn out to equal this is a test

xavpitz commented 1 month ago

Thanks for the help about the variable building. It was exactly as I expected. Thing is I did a syntax mistake in the jinja file. I had a "${ xxxxxxx }" string instead of "{{ xxxxxxx }}" at one specific place.

So I managed to have something working but it is still not 100% to my liking. With hindsight, I think that artificially limiting (with the host:container volume mappings) the apps to either _library or _downloads doesn't add much to the security but annoyingly prevents hard-linking & file move across folders.

I will continue to work on this. I will probably change the root for transmission and add a symlink to maintain backward compatibility...

ahembree commented 1 month ago

Of course, thanks for trying to figure out a solution!

For what it's worth, you can reference .env file variables in the docker-compose file by using ${VAR_NAME}. This is regular Compose syntax and is not handled by Ansible. There's a mix of Compose-style variable references and Ansible-style in the docker-compose file due to referencing some of the variables from the .env file. This was to help keep "secret" values out of the compose file and also allow people to change some things quickly in the .env file without having to do a full playbook run again

Going back through my compose file, I can see a few places to replace the Ansible reference with a .env reference, such as hms_docker_media_path