[Feature] Additional host paths for library import

[scruloose]

As far as I can tell, Nefarious looks in its own Host Download Path when trying to import my existing media library, which seems like a… kind of weird assumption?

My media library lives on a NAS that the torrent box doesn't have write access to (and I intend to keep it that way). I haven't tested this, but for now I intend to try symlinking the network mount into the Nefarious download path. I have no reason to think that won't work, but I'd call it a workaround.

Personally, I would expect there to be a config option in the .env file for path to media library, which I assume would then be mounted read-only in the container. I don't know if it's possible to add paths from inside the already-running container, but having having an "Add library path" option right near the "Import TV shows" and "Import movies" buttons would be handy if it's feasible.

Ideally, it would be nice to be able to specify an arbitrary number of import paths. Personally, my media library is optimized for Kodi, so it's split into separate paths for each family member's TV shows, and then again for each family member's movies, etc. I don't know whether it would confuse Nefarious to add the parent directory, which also includes music, random video content, music videos, and so forth. I expect I'll find out soon, though. ☺

[lardbit]

I think I I'm close to understanding the request here, but I want to clarify. If your torrent box (ie. transmission?) doesn't have write access to your NAS, are you manually copying files there?

[scruloose]

Right now, yes, but the plan is to have the NAS (which is a full-fledged Debian system) do an rsync pull from the Nefarious download directory when needed. For this, I'll likely use an inotify job on the torrent box, which would trigger when files get moved to the completed directory, to connect to the NAS with a single-purpose ssh key and trigger my rsync script using the command= directive in authorized_keys, so that key won't have login access at all. That way, even if the torrent box gets totally pwned, it can't delete, corrupt, or ransomware any of the files that are already on the NAS.

By "torrent box", I mean the entire machine on which the Nefarious/Jackett/Transmission combo is installed.

Hey, I did say I'm kind of a security nerd.

Depending how well I like the file-naming and directory structure Nefarious uses, I may copy the completed downloads into the Kodi library paths as-is (with --no-clobber or equivalent), or I may drop them in a staging directory and use something like tvnamer or MediaElch to customize the filenames, arrange the directory structure the way Kodi likes it, and move 'em into the library.

The trickiest bit is probably going to be coordinating the sequence of: rsync the files over, verify that it succeeded, but leave the originals for seeding until Transmission's done with them based on some combination of ratio & time seeded, then delete them from the completed downloads directory.

But yeah, for Nefarious to have any awareness of my media library, it'll need to ingest a bunch of directories from a read-only network share. Ideally it would be nice to have an API that lets me trigger a library update from an external script too, the way I can for Kodi.

Also, some of this proposed setup is based on the assumption that Nefarious can't simply run an arbitrary command, outside the container on the host, when it has new "completed" torrents ready to harvest. If it can, that would simplify things a bit.

[lardbit]

Hey, I did say I'm kind of a security nerd.

Well, your paranoia will keep you safe!

Depending how well I like the file-naming and directory structure Nefarious uses, I may copy the completed downloads into the Kodi library paths as-is (with --no-clobber or equivalent), or I may drop them in a staging directory and use something like tvnamer or MediaElch to customize the filenames, arrange the directory structure the way Kodi likes it, and move 'em into the library.

nefarious does a mediocre job at renaming media right now. Well, it's fine for movies which just renames the file/directory as the actual title, but since TV Shows are a little more complex, it simply renames the top-level directory as the title and then organizes everything as show->season>episode. I use Plex and that seems to work well.

The trickiest bit is probably going to be coordinating the sequence of: rsync the files over, verify that it succeeded, but leave the originals for seeding until Transmission's done with them based on some combination of ratio & time seeded, then delete them from the completed downloads directory.

rsync will be great for that and I think the operations are atomic so you can be confident the transfer was successful. As for the removing transmission files after they meet some seed criteria, you'll want to use transmission's RPC (api). You can see a quick example of requesting a torrent's details. I use this python transmission library to interact with transmission.

But yeah, for Nefarious to have any awareness of my media library, it'll need to ingest a bunch of directories from a read-only network share. Ideally it would be nice to have an API that lets me trigger a library update from an external script too, the way I can for Kodi.

nefarious currently triggers the library import from the GUI, but you can certainly trigger it manually via a docker command. What's the use-case for an API though? Wouldn't you only be triggering it once when you initially install it?

Also, some of this proposed setup is based on the assumption that Nefarious can't simply run an arbitrary command, outside the container on the host, when it has new "completed" torrents ready to harvest. If it can, that would simplify things a bit.

Yeah, nefarious will need to see a directory structure to be able to import your library. It can be a read-only mount or even empty files that mirror the directory structure would do the trick.

Nefarious has a webhook built-in, but currently it's just for push notifications to know when a file is downloaded. I use Slack and get notified when something is complete. So, this could obviously be extended to do whatever.

Let me know if you have any specific ideas regarding that.

[scruloose]

rsync will be great for that and I think the operations are atomic so you can be confident the transfer was successful.

rsync is astoundingly good at what it does, it's true. I'm thinking about cases like if the NAS is down, or hits a disk full error halfway through, or the network suddenly drops… Not even rsync is quite "fire and forget". I expect the easy and robust approach will be to have the little inotify job on the torrent box hardlink all the completed files into an "outgoing" location. Then my rsync job can pull that "outgoing" directory and use --remove-source-files to delete (that second pointer to) each file after it's successfully transferred. And if Transmission is done seeding the file before rsync transfers it (eg still waiting for new hard drive to come in the mail), no problem! It can go ahead and delete (its pointer to) the file, and the backlog in "outgoing" can keep piling up. That way I don't need to parse rsync's output for files that failed to transfer, or keep a list of what's still waiting to be transferred, or try to compare Nefarious' completed directory against the (potentially renamed and reorganized) files on the NAS before deleting stuff that's finished seeding — the filesystem can do the accounting for me.

Thanks for the pointers on interacting with Transmission! I expect that will come in handy.

As for the use-case for triggering a library import via API, I was primarily thinking of keeping it up-to-date when completed downloads have been moved/reorganized/renamed etc. I suppose if Nefarious has already added them to the library and doesn't notice that they've been yoinked out of its "Host Download Path", then that's not a problem. Say I sit down and rip a bunch of DVDs, though. How will Nefarious know they've been added? If I script a "hey nefarious! update your library!" call after every out-of-band change to the collection on disk (the same way I currently do for Kodi), its index of the library will stay in sync with reality, without me having to remember to log in to the web interface and hit the import button.

And I'll get back to you about the webhook specifics. Webhooks in particular I've never used before, but if it provides an easy way to have Nefarious trigger the hardlink→ ssh → rsync sequence every time it has a completed download to report, that should be a little more straightforward and efficient than using inotify to constantly watch the completed downloads folder.

[GenesisAdam]

@lardbit I have just installed NEFARIOUS for the first time and I have the same issue as stated here by @scruloose, that's how I found this thread. My library folders are different from my download folders. I am unable to import my existing library folders. I have been using radarr and sonarr where download folders are separate from library folders.

[lardbit]

Hey @GenesisAdam,

What is the use-case for not downloading directly into your libraries? I'm trying to wrap my head around that.

[GenesisAdam]

@lardbit There is nothing wrong in doing so. But if you want to do post-processing after the files have been downloaded,it may be better doing it inside the download folders before moving the files into the media folder. Perhaps it is because the ealier movies and tv apps such as Sonarr and Radarr handled processing this way. For example you may want to rename the files or use handbrake to convert the files format before movint them into the media folder and then delete the original downloaded files in the download folders.

[lardbit]

That makes sense. nefarious technically downloads to a temporary directory and then does some basic renaming before moving it to the download path, but that won't help you with further processing. I don't have plans to implement this, since I'm aiming to satisfy the basic audience, but I am always open to contributions. A workaround, like @scruloose mentioned, is to have nefarious support multiple "download paths" and then that could satisfy your request. You could then have the 1st path as the download path, and the 2nd as the library path, and then nefarious could import your existing media while downloading elsewhere. I personally don't have use for this, so it probably won't happen too soon, just a heads up, but it is a great idea.