SkipEmbedded does not always work on .mkv files containing subtitles

[mattmac24]

I'm a little confused as to how the SkipEmbedded setting works. The description is:

Identify how you want to handle embedded subititles if they are detected in the video file being scanned. If you set this value to 'no', you will use match embedded subtitles instead and further no further script processing will take place. If you set this to 'yes', you will ignore the fact that embedded subtitles were detected and just continue to exersice this tool to fetch some from the providers identified. NOTE: Embedded subtitles can only be detected if you are using the advanced search mode identified above. Therefore this switch has no bearing on a Basic check.

What i want is to get subtitles for all of my listed languages. But if one of these languages exist as an embedded subtitle, do not look for that language, but still look for and download the remaining languages. Is this possible? I think i want to set this to 'no' but then if an embedded english subtitle exists I will wind up with 2 english subtitles(one embedded and one external)

[caronc]

I'm a little confused that it does that myself so let me see what i can do. Please be patient though as evenings and weekends are the only days to myself and some of which i can allot to tasks like this :).

[mattmac24]

Thanks for the reply. Please let me know if you need any clarification of the issue or if you want me to test a fix. I'm now having two english subtitles show up in Plex for most videos. But I'm also getting my foreign subtitles so this is an improvement for me :).

[caronc]

I'm pretty confident i got it; who would have thought that it would come down to a silly missed true/false flag (sighs). It would have been like this for years!

Please check out the master branch when you get a chance and give it a go if you wouldn't mind! Keep me posted on your success/failure. It works for me now, but i only tested it on one video. I'll test on a few more i the meantime now as well.

Just to clarify that we're on the same page with how the option should work: If you enable it, then subtitle fetching should be skipped in the event subtitles were pre-detected as being embedded in the video file already. This detection process doesn't work on video files that have subs hard-coded into them. Also, the skip embedded feature only works on .mkv files. Recently i pushed a small fix (#30) to also skip over videos that have an .idx/.sub file paired with them (this skipping is outside of the _skipembedded check).

You may have already knew all this, but i just thought i'd put it out there just in case so we're all on the same page. :smile:

[mattmac24]

OK I've tested out the fix with a few files and I'm still having the same issue unfortunately. Also in your previous message when you say:

If you enable it, then subtitle fetching should be skipped in the event subtitles were pre-detected as being embedded in the video file already

What i think you mean is: If you disable the SkipEmbedded toggle, which enables the scan for embedded subtitles, then subtitle fetching should be skipped in the event subtitles were pre-detected as being embedded in the video file already for that particular language. Meaning that other languages not found embedded in the video file will still be fetched.

Is my interpretation correct? If so i think the nzbget toggle description should be updated to:

Identify how you want to handle embedded subititles if they are detected in the video file being scanned. If you set this value to 'no', you will attempt to match the detected embedded subtitle against a wanted language, and if a match is found no subtitle fetching will occur for that language. If you set this to 'yes', you will ignore the fact that embedded subtitles were detected and just continue to exercise this tool to fetch some from the providers identified. NOTE: Embedded subtitles can only be detected if you are using the advanced search mode identified above. Therefore this switch has no bearing on a Basic check. Embedded subtitles can only be detected in .mkv files. Hard-coded subtitles can not be detected.

Also in regards to skipping files with external subtitles, what happens in the case where there is an external english subtitle downloaded with the file, but i have listed english and spanish as my wanted languages? Perhaps there should also be an SkipExternal setting which functions in the same way as the SkipEmbedded setting?

[caronc]

I think the biggest confusion comes from it's description. Perhaps i didn't need to flip the boolean switch but rather make this easier to interpret, because even i'm confused at this point. Here is what the NZBGet Inline documentation defines:

Yet the command line output looks like this (this was what i used to test with):

  -k, --skip-embedded   If embedded subtitles were detected, choose not to use
                        them and continue to search for the subtitles hosted
                        by the identified provider(s).

So part 2 of this issue will be renaming this option and making more sense of it. Would you agree to that? I'm open to any suggestions you even have to making this option more understandable :).

I'm going to roll back my change i made last night when i get home and i'll start this task over. The second thing is maybe i should work with the same video you're testing on your end. If you're concerned about privacy, i can respect that. We can take it offline if you could email me at lead2gold@gmail.com? but i'd like to see what actual meta information is inside of it to determine what's going wrong here.

Another thing worth trying if you can spare time is running the test from the command line with the debug switch on (-D) which adds a LOT of verbosity. But it's right at the very start (before it goes off hitting web-servers) where the details on what it found (if anything) in your video file (with respect to any existing subtitles). The command line call would look something like this:

# With option
./Subliminal.py --skip-embedded -D /path/to/video

# and without
./Subliminal.py -D /path/to/video

[mattmac24]

Ok sorry for all the confusion. I think it's on my part. I agree with the wording of both the nzbget toggle and the command line option. I think the problem is that embedded subtitles are just not detected correctly.

An example file I am talking about is: Game.of.Thrones.S05E01.The.Wars.to.Come.1080p.WEB-DL.DD5.1.H.264-NTb ffprobe output: https://pastebin.com/iQVU4whX mediainfo output: https://pastebin.com/BtUNSRjf debug log with --skip-embedded flag: https://pastebin.com/kBf3rpsm Result: Spanish and English subtitles being downloaded as external .srt files debug log without --skip-embedded flag: https://pastebin.com/YHGqcJBh Result: Spanish and English subtitles being downloaded as external .srt files

So i therefore think that the fix you attempted earlier here did not help. I've undone the fix and here are the logs: debug log with --skip-embedded flag: https://pastebin.com/YUCZrPWu Result: both Spanish and english subtitles being downloaded as external .srt files debug log without --skip-embedded flag: https://pastebin.com/3F4qPtP0 Result: No subtitles downloaded

So looks like the real problem is that it doesn't recognise the embedded subtitle as english, my apologies. And then once it gets to this stage it exits the script.

Any solutions? Why is it failed to recognise that the embedded subtitle is English? Can I set the default language of unknown subtitles to English?

[caronc]

Thank you for such a detailed response! I reverted my change back in the last commit as well as adjust some of the wording. I think IgnoreEmbedded makes a bit more sense as a continue if set to Yes .

this means the output of your scripts will now be reversed again (Back how it should have been). The output here is key:

2017-07-22 14:26:33,981 - 6203 - DEBUG - Found embedded subtitle set([<Language [und]>]) with enzyme

Subliminal utilizes a third party app (written by the same brilliant guy who wrote Subliminal himself - Diaoul) called enzyme. My script just wraps all of his great work. Anyway... Enzyme is correctly being executed and scanning your .mkv vile but it hasn't seen an update in almost 4 years. It's usually very powerful and can extract all of the meta information found within the video (like your mediainfo and ffmpeg entries did). But for the one in question, it was able to find the subtitle, but it couldn't determine the language (and created it as und - short for undefined).

Here is a script you can run that exclusively uses the Enzyme plugin against a video file:

#!/usr/bin/env python
# -*- encoding: utf-8 -*-
import enzyme
import sys

if __name__ == "__main__":
    sys.argv[1]
    print(str(sys.argv[1]))
    with open(sys.argv[1], 'rb') as f:
        mkv = enzyme.MKV(f)
        for sub in mkv.subtitle_tracks:
            print(str(sub))

    exit(0)

Then you can launch it against your video like this:

# First make sure you have enzyme installed
# alternatively it's in the Subliminal Folder; you can set your PYTHONPATH to it
# if you like too:
export PYTHONPATH=/path/to/nzbget/scripts/Subliminal

# or system wide:
pip install enzyme

# scan your file like so:
python scan.py <video file>

Your output might look something like this (with respect to this video in question):

<MKV [<Info [title=None, duration=0:52:09.632000, date=2015-04-14 03:08:25]>, [<VideoTrack [1, 1912x1072, V_MPEG4/ISO/AVC, name=None, language=None]>], [<AudioTrack [2, 6 channel(s), 48000Hz, A_AC3, name=None, language=None]>], [<SubtitleTrack [3, name=None, language=None]>]]>

Long story short, it detected 'most' of the meta information, but the Subtitle language is set to None (because it couldn't be determined). I think the bug is here. I'm not sure if @Diaoul (the developer) would have input or what is involved in updating it to extract these missing details.

Otherwise the script is behaving as it should right now (it skips if an undefined track is found).

Thoughts?

[mattmac24]

So looks to be a problem with enzyme not your scripts. The output from enzyme is:

python test.py /home/matt/subliminal_test/Game.of.Thrones.S05E01.The.Wars.to.Come.1080p.WEB-DL.DD5.1.H.264-NTb.mkv /home/matt/subliminal_test/Game.of.Thrones.S05E01.The.Wars.to.Come.1080p.WEB-DL.DD5.1.H.264-NTb.mkv {'forced': False, 'name': None, 'language': None, 'default': False, 'enabled': True, 'number': 3, 'codec_name': None, 'codec_id': u'S_TEXT/UTF8', 'lacing': False, 'type': 17}

Perhaps it would be possible to set a default language when a subtitle is detected with an undefined language? I'll file a bug in enzyme regardless.

Thanks for your help with this!

[caronc]

So glad to see @aulemahal come up with a solution to enzyme as fast as he did! I would have never known the fix was that easy. My hats off to both of you for engaging and resolving this as quick as you did.

I back-ported his fix straight into the script for your feedback. :smile:

[mattmac24]

I've been testing this out today and I can confirm that everything is working as expected.

Thanks so much for all your help and support in fixing this problem :). Keep up the great work