Storing multiple versions of the same file alongside each other

[brandonscript]

Storing multiple versions

The Sonarr team has begin investigating how different versions of a series could be stored alongside one another in the same directory. This ticket is to track the epic of work, including exploring UI/UX implementations, and required engineering changes.

To manage noise, we are limiting who can comment/reply in this thread, but we will track work to be done and communicate progress.

Goals

• Support multiple "tracks" of qualities that can be upgraded along side one another in the same directory. An example might be someone looking to store a higher quality version alongside a lower quality version to reduce the need to transcode.
• Create a user-friendly interface to make it easy to manage multiple copies, ensuring the feature is either opt-in, or easy for new or novice users to understand.
• Ensure backwards compatibility of existing setups, in that single-qualities libraries will not be impacted, with the hope that no data migration will be needed.
• Design systems that will hopefully translate to other companion platforms.

Non-Goals

• This is to support multiple qualities within the same library and in the same directory, and is not intended to manage media stored in separate directories. The recommended approach for this is still to manage multiple Sonarr instances.

TODO

(work in progress)

• UI: Handle partial completion of different qualities everywhere download or completion status is shown.
• UI: Provide easy-to-understand messaging and controls for setting up cutoffs, if changes are required.
• UI: Ensure search results that display info or recommended actions accommodate multiple qualities.
• UI: Provide informational messaging when qualities contradict or overlap each other.
• Core: Update database schema to support unique sets of files per episode instead of a single episode.
• Core: Update rules for processing when searching and manually adding.

Challenges

• Handle files/series moved to different directories
• Handle manually added qualities to an existing title, where multiple qualities may not be set up correctly (e.g., do we auto-detect the new additional quality?)
• Handle file naming conflicts where quality/media is missing from the filename (where ambiguous qualities exist)
• Explore and consider the impact to tags, particularly where they are used to obtain custom qualities, like HDR. Tags are particularly challenging because they are ranked based on priority, not mutually exclusive. It may be that we need to make tags per series and per quality going forward.

UX explorations

Working proposal

Concept: using labels input to select multiple qualities on a per-series basis

Concept: warning notice when qualities overlap in a potentially destructive, non-intuitive, or undesirable way

Rejected ideas

Nested groups/sections of qualities inside a dedicated multi-quality profile Ultimately this idea was rejected because it is confusing to build UI for, and requires a major overhaul to (and introduces some fragility into) quality profiles IA.

Ranked groups showing capped qualities, enforcing that no quality may overlap another Rejected for the same reasons as above

[moreje]

it would be really usefull!

[adityaxdiwakar]

What work is remaining here, and how can someone new contribute?

[markus101]

We've made no progress towards this in Sonarr, @Qstick has done some in Radarr, but that will only be of limited use here. This is going to take an immense amount of work and understanding of the codebase and at this stage it's not something someone new will be able to contribute much to. Even someone knowledgeable is going to spend a lot of time working on this.

[ThePragmaticArt]

My use case for this has always been having both ISOs and non-ISOs exist in parallel. It would be phenomenal if Sonarr or Radarr could have the ball moved on this.

[pawisoon]

Is there any update on the future of this feature? Its definitely missing from Sonarr for many of its users that would like to have show in 2 and more language versions.

[bakerboy448]

Is there any update on the future of this feature? Its definitely missing from Sonarr for many of its users that would like to have show in 2 and more language versions.

when there is an update there will be one. this effectively requires an entire rewrite of the frontend and backend and will not be something anytime in the near future

[brandonscript]

@bakerboy448 as the original instigator of this issue I suppose it's my responsibility to tie this off. Even if we did the work to build this, it's far simpler (and more powerful) from a usability point of view to maintain separate instances for qualities, vs. trying to manage qualities and rules in a single instance. And with pervasiveness and ease of running multiple Docker containers now, it doesn't really make sense to do a complete rewrite.

[mmarquezs]

@brandonscript

I kind of get it as is it true that it seems a simpler solution as in it doesn't require changes to the app, but I am sure I quite have a mental image of how that would work out. Like I guess they couldn't manage the same media folders right as they would fight each other? Or there would be a way? Also I guess it would require multiple download clients aswell or I guess for clients that supports tags that might be enough so it doesn't get confused with the other instance downloads.

Like the main benefit of Sonarr managing this, would be the media is all on it's respective folders following some multiversion naming convention and then like for example Emby/Jellyfin/whatever scans that. Like this naming for example: https://emby.media/support/articles/Movie-Naming.html#multi-version-movies

With the multiple instances, unless they can share the same folders that won't be very nice, not impossible to solve but yeah not great.

[brandonscript]

@mmarquezs named exactly like that, yep, then just use isolated root folders, like:

/media/TV
   /4K
   /HD

Each Sonarr instance points to the respective quality root, and either point Plex at the TV dir, or even better, just use separate libraries in Plex as well - point each to the respective quality root:

4K TV Shows → /media/TV/4K TV Shows → /media/TV/HD

[grigorescu]

Even if we did the work to build this, it's far simpler (and more powerful) from a usability point of view to maintain separate instances for qualities, vs. trying to manage qualities and rules in a single instance. And with pervasiveness and ease of running multiple Docker containers now, it doesn't really make sense to do a complete rewrite.

I understand that this is a huge lift and not a priority. That’s fine. However, while it’s simpler for the code and for development, I strongly disagree that the model that’s being proposed is simpler for users.

I have 4 instances of Sonarr (SD, 720p, 1080p, 4k) and 5 instances of Radarr (the same and one for 3D). I use Lists to sync shows and movies upstream (i.e. I’ll add something to SD and tag it with 720, 1080, 4k). Those instances pull from the lower quality instances and filter on their tag.

However, I’m now running 9 instances instead of 2. I have 9 instances I’m monitoring, 9 instances of logs I’m collecting, 9 instances I’m backing up, 9 instances I need to upgrade periodically, 9 TLS certs, etc. Everything is in Ansible, so it’s manageable, but it’s disingenuous to say that this is somehow simpler than just 2 instances.

Then there’s the waste of resources. Things are better with Prowlarr de-duplicating requests to the indexers, but I’m still sending 4x the requests I should be to TVDB, I have 4x the memory usage, 4x the CPU, etc.

That’s just from the sysadmin perspective. What about my users? It’s easy enough to give them one URL and walk them through “use this dropdown to select the qualities you want.” Instead, it’s “well, decide on the lowest quality you want, and navigate to one of these 4 URLs. Then add it, and cross reference this list of tags…”

[grigorescu]

or even better, just use separate libraries in Plex as well

Please don’t do this. Plex has really good support for multiple versions but only if in the same library. It’ll automatically choose the best source file based on what you’re watching on, your connection, etc.

[nrlcode]

I agree I would like to see this a reality. There are also instances where a movie or show only has a certain version/quality, say 1080p. So now I have multiple copies of the same 1080p file because my instances don’t know that the show is only available in 1080p.

[cscattaglia917]

I agree I would like to see this a reality. There are also instances where a movie or show only has a certain version/quality, say 1080p. So now I have multiple copies of the same 1080p file because my instances don’t know that the show is only available in 1080p.

Sad that this will probably never be a reality. I understand it is a lot of work, but it seems like this has just been ditched completely.

(Appreciate the thumbs down with no discussion lol)

[JustTooKrul]

I understand the above and realize this would be a robust change, but given the comments above, maybe it's easier to change what constitutes a series to include a quality profile? So, instead of a series having metadata for all the quality profiles needed Sonarr could just consider each series to be a <Series,Quality> tuple? This would eliminate 99% of the complexity described above. Now, you would have to think about situations where users add additional quality profiles without realizing and people asking why they have multiple versions of the same series, but this is simple enough to solve for by making it something you have to enable as an advanced feature and when adding a series simply displaying any other Qualities being monitored for that series (simply as a displayed variable and, perhaps, eliminating that quality profile from, being chosen again....). And, V1 of the feature can simply freeze in place whatever exists at the time if you remove a quality profile.

This seems like a better solution, and pushes a lot of the "niceties" of keeping things organized and tracking all the moving parts to the user vs. having to re-work all the logic for searching and storing in Sonarr.

[patrickbard]

Is there any update on the future of this feature? Its definitely missing from Sonarr for many of its users that would like to have show in 2 and more language versions.

I'm in a similar position. And since the conversation here is so heavily focused on quality, I will add my two cents.

While I accept that the most obvious use case for this feature is indeed for people who want multiple qualities available (e.g. 1080p and 4K), I think a decent amount of users (myself included) would like this feature for any kind of different version, not limited to quality. My most common use cases for it would be Director's Cut, Director's Commentary, and similar stuff, which often have all the same quality.

I just wouldn't like it if this feature got implemented and depended so much on quality that it wouldn't work for me.

[theflanman]

Adding a use-case here: Shows with separate intro tracks which are non-trivial to merge. For example, the Log Lady intros for Twin Peaks; different audio track layouts and subtitle tracks make merging without transcoding essentially impossible.

[icurnvs]

Is there any update on the future of this feature? Its definitely missing from Sonarr for many of its users that would like to have show in 2 and more language versions.

I'm in a similar position. And since the conversation here is so heavily focused on quality, I will add my two cents.

While I accept that the most obvious use case for this feature is indeed for people who want multiple qualities available (e.g. 1080p and 4K), I think a decent amount of users (myself included) would like this feature for any kind of different version, not limited to quality. My most common use cases for it would be Director's Cut, Director's Commentary, and similar stuff, which often have all the same quality.

I just wouldn't like it if this feature got implemented and depended so much on quality that it wouldn't work for me.

I have the same use case here, but just to throw some additional examples: Extended editions, the 24/48fps versions of the Hobbit movies, Star Wars despecialized, and Mad Max Fury Road Black & Chrome. Back in the late 90s/early 2000s it was popular for R rated movies to release unrated editions as a marketing ploy with added raunch. Coyote Ugly, for example...the Unrated edition imo changes the tone of the whole movie.

Personally, with modern intel iGPUs being able to transcode 4k (and soon, encode hdr/hevc), I no longer have the use case of needing to store multiple resolution versions. My Comet Lake Xeon (w1290p) igpu handles transcoding 4k stuff down to 1080p just fine for multiple streams.

I'd be curious if the technical lift is just as heavy to support these sorts of editions as it was to support multiple resolution versions.

Heck, I don't necessarily need the *arrs to support searching for and acquiring the editions for me - I'd just like the option for radarr to not delete the other editions if I manually search for and download an additional edition - maybe a checkbox on the interactive search screen or a checkbox on the edit movie screen indicating multiple editions. It would be a little bit of a pain to manage this with also having quality/score upgrades happen, but honestly, I'd be happy to have the option to make that tradeoff of having to manually delete the old/upgraded files to have even partial support for multiple editions. When following the trash naming guides, having a duplicate name isn't an issue since the edition and the other bits that make up the filename are taken into account. As such, any quality/score upgrade downloads would almost certainly have a different ending filename.

edit: also whoever reads this on the dev team, I appreciate you guys. you're doing amazing work.

[bakerboy448]

Movies have nothing to do with Sonarr, but yes any sort of multiple video file editions/types/qualities per movie or multiple video file editions/types/qualities per episode would fall under this item.

This requires in short an entire rewrite of every aspect of the app it is a herculean task that is not simple nor quick.

[icurnvs]

Ahh, drat, my apologies! I thought this thread was a radarr thread.

[icurnvs]

Movies have nothing to do with Sonarr, but yes any sort of multiple video file editions/types/qualities per movie or multiple video file editions/types/qualities per episode would fall under this item.

This requires in short an entire rewrite of every aspect of the app it is a herculean task that is not simple nor quick.

Absolutely understood - I certainly don't want to imply the dev side of this would be simple or quick. I guess from a UX perspective to us users, something may look simple in the UI - like a single check box - but behind that checkbox could be hundreds of hours of work.

Just wanted to add my particular use case for a feature like this existing.