search

pannal / Sub-Zero.bundle

Subtitles for Plex, as good you would expect them to be.
Other
1.76k stars 120 forks source link

Blocked subs on the fly + compromising Plex metadata #743

Closed VilDegard closed 3 years ago

VilDegard commented 4 years ago

My Config

My System

• Seedbox (SB) – For download and temporary storage • GDrive Unlimited – For permanent storage (cronjob offloading folder TV Shows and Movie from Seedbox to GDrive) • MergerFS – /gdrive-plex/ folder points to Plex folder in my SB and Plex folder in my Gdrive • Sub-Zero – To grab subtitles on the fly • Plex – Naming structure respecting the naming convention made by Plex {TV SHOW (YEAR) – SXXEXX – EPISODENAME [QUALITY FORMAT ACODEV VCODEC-RLSGROUP]}

Typical actions

  1. Download order from Medusa to ruTorrent.
  2. Once download done, Medusa post-process the contents.
  3. Medusa update Plex’s library, Plex displays newly downloaded contents.
  4. SubZero on top of Plex Agent (+TMDB) supposed to grab subtitles for newly downloaded contents.

Issue When content is detected in Plex, SZ is triggered to find subtitles, it stays at “Current Status: Refreshing {name show} {SXXEXX}” for a very long time (more than 10 minutes), blocking the metadata update of the remaining episodes in Plex (all displayed as Episode X instead of their original titles), ends up with just the first episode correctly analysed with right subtitles, all remaining episodes all ignored.

How can I fix SZ to grab subtitles on the fly without compromising Plex’s content detection and metadata grabbing please?

Screenshots https://i.ibb.co/KWkr86r/Screenshot-SZ-1.png https://i.ibb.co/P6by04b/Screenshot-SZ-2.png https://i.ibb.co/Lhj3B8V/Screenshot-SZ-3.png https://i.ibb.co/MDM4fpr/Screenshot-SZ-4.png https://i.ibb.co/37xG7rH/Screenshot-SZ-5.png https://i.ibb.co/bFgpD6L/Screenshot-SZ-6.png https://i.ibb.co/MntCTyM/Screenshot-SZ-7.png https://i.ibb.co/gSWSqYz/Screenshot-SZ-8.png

pannal commented 4 years ago

Please provide logs (see wiki)

VilDegard commented 4 years ago

I would have but as mentioned in the post, when I request the logs (copy/paste link) it just opens a blank page, no option to download anything. I'm running Kitana. How can I fix this please?

EDIT: now SZ is detecting only 10 missing, where actually its 1000 at least. No manual download works. I've tried a proper uninstall, deleted cache / metadata in Plex Plugin Support folder, reinstall SZ... same issue. I don't get it.

EDIT 2: And now the only downloads it make are loop downloads of the same show over and over again...

EDIT 3: Importing 1 movie into Plex triggers SZ to find subtitles... it seems to be working fine. But importing a 10 episode season show for some reasons triggers SZ to find subtitles only for the 8th episode, the rest is totally forgotten/avoided. Would that mean that by design SZ can only treat one entry at the time?

pannal commented 4 years ago

OK, first things first. You're using a remote FS and SZ relies heavily on scanning the files.

As you've already activated "low impact mode", there's little I can do.

You have activated addic7ed, which explicitly needs an anticaptcha service. This might be why SZ hangs for so long when updating, but that's hard to say without logs. Logs should be available through Kitana>Advanced - are you on the latest Kitana version? It works fine for me for retrieving logs.

First make sure you're on the latest Kitana, and try connecting to your server directly, not using the relay (if possible). Otherwise you can manually pull the Logs from the PMS instance (they're in \Library\Application Support\Plex Media Server\Logs\PMS Plugin Logs).

For your other questions:

now SZ is detecting only 10 missing, where actually its 1000 at least. No manual download works. I've tried a proper uninstall, deleted cache / metadata in Plex Plugin Support folder, reinstall SZ... same issue. I don't get it.

SZ only knows about file that have been added after SZ was enabled. This is in one of the FAQ entries and how that's solved. Basically you have to consecutively refresh the metadata of the items without subtitles until your library is all set and SZ knows about all the items and their state.

Importing 1 movie into Plex triggers SZ to find subtitles... it seems to be working fine. But importing a 10 episode season show for some reasons triggers SZ to find subtitles only for the 8th episode, the rest is totally forgotten/avoided. Would that mean that by design SZ can only treat one entry at the time?

This is a weird but known one. SZ can absolutely find subtitles in one go for multiple episodes - it just reacts to what the PMS is asking it to do. Sometimes, especially for quality upgrades, the PMS doesn't trigger agents at all.

In general: You might want to have a look at Bazarr, which incorporates most of my work on subliminal_patch and will live longer than SZ.

VilDegard commented 4 years ago

Thank you for your comprehensive answer pannal, much appreciated. First thing first, please find the logs here. (EDIT: with Plex's logs this time)

Things I've found odd:

com.plexapp.agents.none

perhaps one agent is corrumpted? should I do this? but it looks like its used for photos in Plex, weird.

ERROR (logger:38) - subliminal.providers.tvsubtitles: No show id found for (...)

DEBUG (config:350) - custom check failed for: /home/user/.apps/plex/Library/Application Support/Plex Media Server/Plug-ins/Sub-Zero.bundle/Contents/Libraries/Linux/i386/UnRAR/unrar DEBUG (config:350) - custom check failed for: unrar WARNING (config:360) - UnRAR not found

SZ on filesystem

Kitana

SZ detection of previous contents

pannal commented 4 years ago

Hey, so judging from your logs there isn't much wrong, although I can't see enough because SZ isn't set to debug logging (at the bottom of the config).

BSPlayer has throttled you, which is to be expected. You have/had Titlovi enabled without providing credentials, which isn't supported.

com.plexapp.agents.none

I wouldn't worry about that for now.

UnRAR

Not sure, it might just be missing the executable flag, or it's the wrong arch type for your setup. You can try installing unrar inside the container manually and trying again.

As I have more than 10TB of data, wouldn't be wise to empty the libraries, clean budle etc and re upload Shows and Movies gradually to avoid overloading SZ and reach API limits?

That doesn't really change much to be honest. You can do that, or you can do what I said.

All in all: I'd try Bazarr, it has way more support than SZ has (me) and is actually actively being developed on.

As for metadata sub storage: that would make the most sense, yeah.

Edit: Added additional notes

VilDegard commented 4 years ago

I've activated Log subtitle modification (debug) and Log to console (for development/debugging), tried to refresh metadata for a couple of episode, they appear as "refreshing" in SZ status but then... nothing. If I go manually to find a subtitle, I find it with a very high score (more than 300), if I select it it's then added in Plex and I can read it... I don't understand why it doesn't work automatically when Plex triggers a library scan / refresh metadata.

Here are the logs with options activated but I feel they don't record anything new

I can't try Bazarr as my actual seedb provider doesn't support it / allow it. My only option is to run it 24/24 on a raspberry pie. Plus I use Radarr but Medusa instead of Sonarr (I need Trakt support that Sonarr doesn't have).

I really just want to fix SZ to automatically grab subtitles on the fly, that's it. Do you think that's possible? Should I nuke everything and start with a fresh install?

pannal commented 4 years ago

Log to console and subtitle mod debug don't help. In the same area there's a log verbosity setting, set that to debug.

VilDegard commented 4 years ago

My bad... done. Please find the new logs here.

So to sum up, after a few tweaks and following your advices: when I request subtitles manually (Plex -- Refresh metadata), I manage to get the adequate subtitles.

However, automatically, like Radarr and Medusa send refresh libs requests... it's fine with Radarr, but not with Medusa. So I made two test: the 1st via Medusa, the 2nd without Medusa. And it seems that now my problem is not around SZ... but with Medusa...

1 st - Error – Contents from Medusa (where Medusa updates Plex lib)

1- Download a season pack on Medusa 2- Done… Medusa post-process, send library updates to Plex (perhaps too many requests?) 3- Plex refresh the season, only half the list of episodes appear, and they appear as “Episode X” without any further data 4- SZ Refresh the SXXE08, always the 8th (don’t know why) undefinitely 5- I restart SZ and then Plex show full list of episodes with real names and synopsis but no subtitles 6- I have to refresh all metadata of the season show to get subtitles

2nd - No Error – Contents moved manually into TV show Plex folder (manual updates of Plex lib from Plex)

1- Move season show into Plex TV Shows folder 2- Hit “Scan Library files” 3- Plex refresh the season, all episodes appear, and they appear as “Episode X” without any further data 4- SZ Refresh the SXXE08, always the 8th (don’t know why) 5- After a while, all episodes get their right naming, synopsis and designated subtitles from SZ. Everything works.

Perhaps the issue comes from the amount of library updates that Medusa send to Plex and then mess up with the metadata? What's your opinion?

pannal commented 4 years ago

OK, logs look fine.

I'm not really sure what Medusa is doing there, I've never used it. It might be refreshing every single episode, which usually is fine, but I can't say for sure.

Might this be a population issue with your remote FS? Might Plex, once it's getting the season refresh, not actually see all the files?

Can you set a delay in Medusa?

Also, you might try to set SZ to extract embedded subtitles, which should also work with metadata storage, to populate your libraries first (it will search for better subs regularly as extracted embeddeds are added with a score of zero), and leave off the "search immediately after extraction" option.

You can also try to move SZ further down the agent list.

Other than that, as I don't really know Medusa, I can't help very much.

VilDegard commented 4 years ago

OK, logs look fine.

Thanks to you it seems that SZ is behaving more efficiently, but I'm still having this new content automatic refresh issue. SZ don't seem to be the problem anymore.

I'm not really sure what Medusa is doing there

It's a Sickchill fork, unlike Sonarr it has a traktv watchlist update and forced manual search on season packs that is quite useful.

Can you set a delay in Medusa?

Not sure, I will take a look. I'll post an issue into their github. Perhaps it's not linked to the update of Plex libs. If you have any other idea, you're more than welcome.

Also, you might try to set SZ to extract embedded subtitles

Interesting. Why do extracted subs start with 0? Are they not supposed to me the most suitable subs for a designated video? If they suit perfectly from the beggining, why searching for better subs? (genuine curiosity).

You can also try to move SZ further down the agent list.

Good idea, I'll try that and revert asap.

Something odd but positive happened when I manually uploaded a whole season show into my Plex library. I didn't perform a library scan, but I could see that SZ detected the contents and was looking for subs when the season wasn't even displayed on Plex. After half an hour, I've hit the Scan library files button on Plex, and the season show appeared in a split second, with all metadata needed. I thought Plex would trigger SZ as agent when Plex detects new contents?

pannal commented 4 years ago

It's a Sickchill fork, unlike Sonarr it has a traktv watchlist update and forced manual search on season packs that is quite useful.

Sonarr has also the option for a season pack search, although not manual, that's true. I'm sure you could connect trakt to sonarr, I think I've seen plugins for that.

Interesting. Why do extracted subs start with 0? Are they not supposed to me the most suitable subs for a designated video? If they suit perfectly from the beggining, why searching for better subs? (genuine curiosity).

Embedded subs are often not great, and often they are, depending on who wrote them. I'll add a setting to set their score, though, good idea.

VilDegard commented 4 years ago

I've tried Sonarr, same issue... once a whole season is downloaded and sent to Plex, not all episodes are detected (perhaps the PVR triggers too many refresh lib requests, as I do a Scan Library files Plex can find the episodes left behind). I'm stubborn, I'm sure I can find the solution if I have enough help.

One thing important that I've noticed: when a new episode (e.g. S05E23) is added and metadata are searched for it, SZ is triggered to find subtitles for that episode AND for all previous episodes. So all episodes of S05, S04, S03 etc… are searched, which takes for ever.

Perhaps its because I have "Don't search for subtitles of a language if they already exist on the filesystem (metadata/filesystem)?" unticked and "Don't search for subtitles if a subtitle in any configured language exists as" in Always Search. But if I stop searching subs, it may stop looking for better subs in the future right? What do you think?

pannal commented 4 years ago

It's probably not SZ's fault that that happens. SZ doesn't refresh anything on its own, it only reacts to triggers from the PMS. The only time SZ actually refreshes stuff on its own is in the periodic tasks.

Edit: There's an update function in agents that gets called by the PMS with metadata; the PMS decides which media to include in there - can be one item or multiple.

pannal commented 4 years ago

I'll add a setting to set their score, though, good idea.

This is already there, but in advanced_settings.json's find_better_as_extracted_tv_score and find_better_as_extracted_movie_score. I've upped the score for movies to 112 from 82.

VilDegard commented 4 years ago

Thanks, I'm checking options to solve this tenacious issue. It seems that just a timeout between the libraries updates from Medusa/Sonarr to Plex should fix this. But using a remote filesystem doesn't help. I'll keep you posted. Overall I like SZ and the diligent work you've put in it, I'll surely donate at some point.

cphmichael commented 4 years ago

Plex Media Server 1.20.0.3125: https://forums.plex.tv/t/plex-media-server/30447/357

Could SZ maybe use the new API to make a partial scan of the folder where Sonarr upload a new episode - and by that way activate the "Scan Library" so Plex can find the episodes left behind?

VilDegard commented 4 years ago

Back to the main topic... @pannal timeout between libraries refresh can't be made from PVR clients at the moment. Perhaps the new API for Plex will allow partial scan...

At the moment I'm refreshing the libs manually and Subzero works fine BUT when I move loads of contents (e.g. 3 seasons of 20 episodes from different shows) suprinsingly only one of these get the matching subs, the rest Subzero mention it has found the subtitles, stored them in Plex metadata, but when I go and check... can't find them. I have to refresh metadata manually so they can appear. Weird. What do you think?

SZ is top agent in the list. Let me know if I should reproduce this issue and send you the logs. Thanks.

cphmichael commented 4 years ago

Have you tried to upgrade a series and then check in Kitana > Recently-added items > TV Shows? Does Kitana list the upgraded items? Sonarr updated a show from WEBDL to Bluray, but Kitana do not display it in Recently-added items.

pannal commented 4 years ago

So, first of all: image

As for upgrades: Upgrades have been an old issue that I can't fix. Sometimes the PMS decides not to call the agents upon a media upgrade.

cphmichael commented 4 years ago

@pannal Do you know if Bazarr is able to fetch subtitles for Movie? If Yes: I can use SZ for Series and Bazarr for Movies.

pannal commented 3 years ago

Yes it is.