search

caronc / nzb-subliminal

Fetches subtitles for the videos it's provided. It can be easily integrated into NZBGet and SABnzbd too.
GNU General Public License v3.0
99 stars 14 forks source link

KeyError: 'year' #40

Closed DeeMouse closed 6 years ago

DeeMouse commented 6 years ago

Hi there.

I'm getting a fatal error with one particular video file:

The Vietnam War (2017).s01e10.The Weight of Memory (March 1973-Onward).Bluray-1080p.mp4

(Just to explain the path used in the commands below, I subsequently placed this episode in a 'problematic' subfolder to enable faster testing/limited-to-relevant debug info.)

Release 0.10.0:

$ /opt/nzbget/scripts/Subliminal.py -D -t -e utf-8 -f '/media/Storage2/The Vietnam War (2017)/s01/problematic'
[INFO] [DEBUG] Script mode forced off.
2017-10-17 01:51:20,765 - 24537 - DEBUG - set(config) TIDYSUB="True"
2017-10-17 01:51:20,765 - 24537 - DEBUG - set(environment) TIDYSUB="1"
2017-10-17 01:51:20,765 - 24537 - DEBUG - set(config) FORCEENCODING="utf-8"
2017-10-17 01:51:20,765 - 24537 - DEBUG - set(environment) FORCEENCODING="utf-8"
2017-10-17 01:51:20,765 - 24537 - DEBUG - set(config) FORCE="True"
2017-10-17 01:51:20,765 - 24537 - DEBUG - set(environment) FORCE="1"
2017-10-17 01:51:20,765 - 24537 - DEBUG - get(default) SCANDIRECTORIES=None
2017-10-17 01:51:20,765 - 24537 - DEBUG - set(config) MAXAGE="24"
2017-10-17 01:51:20,765 - 24537 - DEBUG - set(environment) MAXAGE="24"
2017-10-17 01:51:20,765 - 24537 - DEBUG - set(config) MINSIZE="150"
2017-10-17 01:51:20,765 - 24537 - DEBUG - set(environment) MINSIZE="150"
2017-10-17 01:51:20,765 - 24537 - DEBUG - set(config) MINSCORE="20"
2017-10-17 01:51:20,765 - 24537 - DEBUG - set(environment) MINSCORE="20"
2017-10-17 01:51:20,765 - 24537 - DEBUG - set(config) LANGUAGES="en"
2017-10-17 01:51:20,765 - 24537 - DEBUG - set(environment) LANGUAGES="en"
2017-10-17 01:51:20,765 - 24537 - DEBUG - set(config) SYSTEMENCODING="UTF-8"
2017-10-17 01:51:20,765 - 24537 - DEBUG - set(environment) SYSTEMENCODING="UTF-8"
2017-10-17 01:51:20,765 - 24537 - DEBUG - set(config) FETCHMODE="BestScore"
2017-10-17 01:51:20,765 - 24537 - DEBUG - set(environment) FETCHMODE="BestScore"
2017-10-17 01:51:20,765 - 24537 - DEBUG - set(config) VIDEOEXTENSIONS=".mkv,.avi,.divx,.xvid,.mov,.wmv,.mp4,.mpg,.mpeg,.vob,.iso"
2017-10-17 01:51:20,765 - 24537 - DEBUG - set(environment) VIDEOEXTENSIONS=".mkv,.avi,.divx,.xvid,.mov,.wmv,.mp4,.mpg,.mpeg,.vob,.iso"
2017-10-17 01:51:20,765 - 24537 - DEBUG - set(config) SCANDIRECTORIES="/media/Storage2/The Vietnam War (2017)/s01/problematic"
2017-10-17 01:51:20,766 - 24537 - DEBUG - set(environment) SCANDIRECTORIES="/media/Storage2/The Vietnam War (2017)/s01/problematic"
2017-10-17 01:51:20,766 - 24537 - DEBUG - get(config) SCANDIRECTORIES="/media/Storage2/The Vietnam War (2017)/s01/problematic"
2017-10-17 01:51:20,766 - 24537 - DEBUG - get(config) VIDEOEXTENSIONS=".mkv,.avi,.divx,.xvid,.mov,.wmv,.mp4,.mpg,.mpeg,.vob,.iso"
2017-10-17 01:51:20,766 - 24537 - DEBUG - get(config) MAXAGE="24"
2017-10-17 01:51:20,766 - 24537 - DEBUG - get(config) MINSIZE="150"
2017-10-17 01:51:20,766 - 24537 - DEBUG - get(config) FORCE="True"
2017-10-17 01:51:20,766 - 24537 - DEBUG - get(config) SCANDIRECTORIES="/media/Storage2/The Vietnam War (2017)/s01/problematic"
2017-10-17 01:51:20,767 - 24537 - DEBUG - get(default) XREFPATHS=None
2017-10-17 01:51:20,767 - 24537 - DEBUG - get(default) SINGLE="False"
2017-10-17 01:51:20,767 - 24537 - DEBUG - get_files('/media/Storage2/The Vietnam War (2017)/s01/problematic') with 11 filter(s)
2017-10-17 01:51:20,767 - 24537 - DEBUG - Allowed The Vietnam War (2017).s01e10.The Weight of Memory (March 1973-Onward).Bluray-1080p.mp4 (suffix)
2017-10-17 01:51:20,767 - 24537 - INFO - Found 1 matched file(s).
2017-10-17 01:51:20,767 - 24537 - DEBUG - Scanning "/media/Storage2/The Vietnam War (2017)/s01/problematic/The Vietnam War (2017).s01e10.The Weight of Memory (March 1973-Onward).Bluray-1080p.mp4" (2106.86 MB)
2017-10-17 01:51:20,767 - 24537 - DEBUG - get(system) TEMPDIR="/tmp/nzbget-user"
2017-10-17 01:51:20,767 - 24537 - DEBUG - get(default) CACHEDIR="/tmp/nzbget-user"
2017-10-17 01:51:20,767 - 24537 - DEBUG - get(config) FORCEENCODING="utf-8"
2017-10-17 01:51:20,767 - 24537 - DEBUG - get(config) TIDYSUB="True"
2017-10-17 01:51:20,767 - 24537 - DEBUG - get(config) MINSCORE="20"
2017-10-17 01:51:20,767 - 24537 - DEBUG - get(default) IGNOREEMBEDDED="no"
2017-10-17 01:51:20,767 - 24537 - DEBUG - get(default) SEARCHMODE="advanced"
2017-10-17 01:51:20,767 - 24537 - INFO - Using advanced search mode
2017-10-17 01:51:20,767 - 24537 - DEBUG - get(default) MOVIEPROVIDERS=""
2017-10-17 01:51:20,767 - 24537 - DEBUG - get(default) PROVIDERS="['opensubtitles', 'tvsubtitles', 'podnapisi', 'addic7ed', 'thesubdb']"
2017-10-17 01:51:20,767 - 24537 - DEBUG - get(default) TVSHOWPROVIDERS=""
2017-10-17 01:51:20,768 - 24537 - DEBUG - get(default) PROVIDERS="['opensubtitles', 'tvsubtitles', 'podnapisi', 'addic7ed', 'thesubdb']"
2017-10-17 01:51:20,768 - 24537 - DEBUG - Using the following movie providers: podnapisi, opensubtitles, addic7ed, thesubdb, tvsubtitles
2017-10-17 01:51:20,768 - 24537 - DEBUG - Using the following tv show providers: podnapisi, opensubtitles, addic7ed, thesubdb, tvsubtitles
2017-10-17 01:51:20,768 - 24537 - DEBUG - get(default) ADDIC7EDUSER=None
2017-10-17 01:51:20,768 - 24537 - DEBUG - get(default) ADDIC7EDPASS=None
2017-10-17 01:51:20,768 - 24537 - DEBUG - get(config) LANGUAGES="en"
2017-10-17 01:51:20,768 - 24537 - DEBUG - get(config) FETCHMODE="BestScore"
2017-10-17 01:51:20,768 - 24537 - DEBUG - Fetch Mode: BestScore
2017-10-17 01:51:20,775 - 24537 - DEBUG - get(config) SYSTEMENCODING="UTF-8"
2017-10-17 01:51:20,778 - 24537 - DEBUG - get_files('/media/Storage2/The Vietnam War (2017)/s01/problematic') with 1 filter(s)
2017-10-17 01:51:20,778 - 24537 - DEBUG - Scanning [advanced] using /media/Storage2/The Vietnam War (2017)/s01/problematic/The Vietnam War (2017).s01e10.The Weight of Memory (March 1973-Onward).Bluray-1080p.mp4 lang=en
2017-10-17 01:51:20,779 - 24537 - DEBUG - get(default) TVCATEGORIES="[]"
2017-10-17 01:51:20,779 - 24537 - DEBUG - Guessing using: /media/Storage2/The Vietnam War (2017)/s01/problematic/The Vietnam War (2017).s01e10.The Weight of Memory (March 1973-Onward).Bluray-1080p.mp4
2017-10-17 01:51:20,892 - 24537 - ERROR - Fatal Exception:
  Traceback (most recent call last):
    File "/opt/nzbget/scripts/Subliminal/nzbget/ScriptBase.py", line 2935, in run
    exit_code = main_function(*args, **kwargs)
    File "/opt/nzbget/scripts/Subliminal.py", line 1936, in main
    use_nzbheaders=False,
    File "/opt/nzbget/scripts/Subliminal.py", line 1406, in subliminal_fetch
    use_nzbheaders=use_nzbheaders,
    File "/opt/nzbget/scripts/Subliminal.py", line 688, in guess_info
    guess['series'] += ' ' + str(guess['year'])
  KeyError: 'year'

2017-10-17 01:51:20,892 - 24537 - DEBUG - Exiting with return code: 94

Master (60b0ffd):

$ ./nzb-subliminal-master/Subliminal.py -D -t -e utf-8 -f '/media/Storage2/The Vietnam War (2017)/s01/problematic'
2017-10-17 01:55:52,143 - 24978 - DEBUG - Script Mode: shell
2017-10-17 01:55:52,143 - 24978 - DEBUG - set(config) TIDYSUB="True"
2017-10-17 01:55:52,143 - 24978 - DEBUG - set(environment) TIDYSUB="1"
2017-10-17 01:55:52,143 - 24978 - DEBUG - set(config) FORCEENCODING="utf-8"
2017-10-17 01:55:52,143 - 24978 - DEBUG - set(environment) FORCEENCODING="utf-8"
2017-10-17 01:55:52,143 - 24978 - DEBUG - set(config) FORCE="True"
2017-10-17 01:55:52,144 - 24978 - DEBUG - set(environment) FORCE="1"
2017-10-17 01:55:52,144 - 24978 - DEBUG - get(default) MAXAGE=None
2017-10-17 01:55:52,144 - 24978 - DEBUG - set(config) MAXAGE="24"
2017-10-17 01:55:52,144 - 24978 - DEBUG - set(environment) MAXAGE="24"
2017-10-17 01:55:52,144 - 24978 - DEBUG - get(default) MINSIZE=None
2017-10-17 01:55:52,144 - 24978 - DEBUG - set(config) MINSIZE="150"
2017-10-17 01:55:52,144 - 24978 - DEBUG - set(environment) MINSIZE="150"
2017-10-17 01:55:52,144 - 24978 - DEBUG - get(default) MINSCORE=None
2017-10-17 01:55:52,144 - 24978 - DEBUG - set(config) MINSCORE="20"
2017-10-17 01:55:52,144 - 24978 - DEBUG - set(environment) MINSCORE="20"
2017-10-17 01:55:52,144 - 24978 - DEBUG - get(default) LANGUAGES=None
2017-10-17 01:55:52,144 - 24978 - DEBUG - set(config) LANGUAGES="en"
2017-10-17 01:55:52,144 - 24978 - DEBUG - set(environment) LANGUAGES="en"
2017-10-17 01:55:52,144 - 24978 - DEBUG - get(default) SYSTEMENCODING=None
2017-10-17 01:55:52,144 - 24978 - DEBUG - set(config) SYSTEMENCODING="UTF-8"
2017-10-17 01:55:52,144 - 24978 - DEBUG - set(environment) SYSTEMENCODING="UTF-8"
2017-10-17 01:55:52,145 - 24978 - DEBUG - get(default) FETCHMODE=None
2017-10-17 01:55:52,145 - 24978 - DEBUG - set(config) FETCHMODE="BestScore"
2017-10-17 01:55:52,145 - 24978 - DEBUG - set(environment) FETCHMODE="BestScore"
2017-10-17 01:55:52,145 - 24978 - DEBUG - get(default) VIDEOEXTENSIONS=None
2017-10-17 01:55:52,145 - 24978 - DEBUG - set(config) VIDEOEXTENSIONS=".mkv,.avi,.divx,.xvid,.mov,.wmv,.mp4,.mpg,.mpeg,.vob,.iso"
2017-10-17 01:55:52,145 - 24978 - DEBUG - set(environment) VIDEOEXTENSIONS=".mkv,.avi,.divx,.xvid,.mov,.wmv,.mp4,.mpg,.mpeg,.vob,.iso"
2017-10-17 01:55:52,145 - 24978 - DEBUG - get(default) SCANDIRECTORIES=None
2017-10-17 01:55:52,145 - 24978 - DEBUG - set(config) SCANDIRECTORIES="/media/Storage2/The Vietnam War (2017)/s01/problematic"
2017-10-17 01:55:52,145 - 24978 - DEBUG - set(environment) SCANDIRECTORIES="/media/Storage2/The Vietnam War (2017)/s01/problematic"
2017-10-17 01:55:52,145 - 24978 - DEBUG - get(config) VIDEOEXTENSIONS=".mkv,.avi,.divx,.xvid,.mov,.wmv,.mp4,.mpg,.mpeg,.vob,.iso"
2017-10-17 01:55:52,145 - 24978 - DEBUG - get(config) MAXAGE="24"
2017-10-17 01:55:52,145 - 24978 - DEBUG - get(config) MINSIZE="150"
2017-10-17 01:55:52,145 - 24978 - DEBUG - get(config) FORCE="True"
2017-10-17 01:55:52,145 - 24978 - DEBUG - get(config) SCANDIRECTORIES="/media/Storage2/The Vietnam War (2017)/s01/problematic"
2017-10-17 01:55:52,146 - 24978 - DEBUG - get(default) XREFPATHS=None
2017-10-17 01:55:52,146 - 24978 - DEBUG - get(default) SINGLE="False"
2017-10-17 01:55:52,146 - 24978 - DEBUG - get_files('/media/Storage2/The Vietnam War (2017)/s01/problematic') with 11 filter(s)
2017-10-17 01:55:52,146 - 24978 - DEBUG - Allowed The Vietnam War (2017).s01e10.The Weight of Memory (March 1973-Onward).Bluray-1080p.mp4 (suffix)
2017-10-17 01:55:52,146 - 24978 - INFO - Found 1 matched file(s).
2017-10-17 01:55:52,146 - 24978 - DEBUG - Scanning "/media/Storage2/The Vietnam War (2017)/s01/problematic/The Vietnam War (2017).s01e10.The Weight of Memory (March 1973-Onward).Bluray-1080p.mp4" (2106.86 MB)
2017-10-17 01:55:52,146 - 24978 - DEBUG - get(system) TEMPDIR="/tmp/nzbget-user"
2017-10-17 01:55:52,146 - 24978 - DEBUG - get(default) CACHEDIR="/tmp/nzbget-user"
2017-10-17 01:55:52,146 - 24978 - DEBUG - get(config) FORCEENCODING="utf-8"
2017-10-17 01:55:52,147 - 24978 - DEBUG - get(config) TIDYSUB="True"
2017-10-17 01:55:52,147 - 24978 - DEBUG - get(config) MINSCORE="20"
2017-10-17 01:55:52,147 - 24978 - DEBUG - get(default) IGNOREEMBEDDED="no"
2017-10-17 01:55:52,147 - 24978 - DEBUG - get(default) SEARCHMODE="advanced"
2017-10-17 01:55:52,147 - 24978 - INFO - Using advanced search mode
2017-10-17 01:55:52,147 - 24978 - DEBUG - get(default) MOVIEPROVIDERS=""
2017-10-17 01:55:52,147 - 24978 - DEBUG - get(default) PROVIDERS="['opensubtitles', 'tvsubtitles', 'podnapisi', 'addic7ed', 'thesubdb']"
2017-10-17 01:55:52,147 - 24978 - DEBUG - get(default) TVSHOWPROVIDERS=""
2017-10-17 01:55:52,147 - 24978 - DEBUG - get(default) PROVIDERS="['opensubtitles', 'tvsubtitles', 'podnapisi', 'addic7ed', 'thesubdb']"
2017-10-17 01:55:52,147 - 24978 - DEBUG - Using the following movie providers: podnapisi, opensubtitles, addic7ed, thesubdb, tvsubtitles
2017-10-17 01:55:52,147 - 24978 - DEBUG - Using the following tv show providers: podnapisi, opensubtitles, addic7ed, thesubdb, tvsubtitles
2017-10-17 01:55:52,147 - 24978 - DEBUG - get(default) ADDIC7EDUSER=None
2017-10-17 01:55:52,147 - 24978 - DEBUG - get(default) ADDIC7EDPASS=None
2017-10-17 01:55:52,147 - 24978 - DEBUG - get(config) LANGUAGES="en"
2017-10-17 01:55:52,147 - 24978 - DEBUG - get(config) FETCHMODE="BestScore"
2017-10-17 01:55:52,147 - 24978 - DEBUG - Fetch Mode: BestScore
2017-10-17 01:55:52,154 - 24978 - DEBUG - get(config) SYSTEMENCODING="UTF-8"
2017-10-17 01:55:52,157 - 24978 - DEBUG - get_files('/media/Storage2/The Vietnam War (2017)/s01/problematic') with 1 filter(s)
2017-10-17 01:55:52,158 - 24978 - DEBUG - Scanning [advanced] using /media/Storage2/The Vietnam War (2017)/s01/problematic/The Vietnam War (2017).s01e10.The Weight of Memory (March 1973-Onward).Bluray-1080p.mp4 lang=en
2017-10-17 01:55:52,158 - 24978 - DEBUG - get(default) TVCATEGORIES="[]"
2017-10-17 01:55:52,158 - 24978 - DEBUG - Guessing using: /media/Storage2/The Vietnam War (2017)/s01/problematic/The Vietnam War (2017).s01e10.The Weight of Memory (March 1973-Onward).Bluray-1080p.mp4
2017-10-17 01:55:52,267 - 24978 - ERROR - Fatal Exception:
  Traceback (most recent call last):
    File "/home/user/nzb-subliminal-master/Subliminal/nzbget/ScriptBase.py", line 3027, in run
    exit_code = main_function(*args, **kwargs)
    File "./nzb-subliminal-master/Subliminal.py", line 1949, in main
    File "./nzb-subliminal-master/Subliminal.py", line 1428, in subliminal_fetch
    File "./nzb-subliminal-master/Subliminal.py", line 710, in guess_info
  KeyError: 'year'

2017-10-17 01:55:52,267 - 24978 - DEBUG - Exiting with return code: 1

I did find the relevant use of the key in the script, but I don't understand what's going on so I'm afraid I don't have a fix to submit. I did find that changing line 688 (in 0.10.0) from:

guess['series'] += ' ' + str(guess['year'])

to:

guess['series']

seemed to allow subtitles to download, which I think is because it prevents any year from being specified.

Related, for other episodes in this series the year is being incorrectly guessed as one of the periods stated in the title rather than the actual year of the series (2017). Example file names:

There's absolutely no urgency on this from my point of view, but I thought I should log it. Thanks!

caronc commented 6 years ago

Thank you for such a comprehensive and detailed bug report! I'm truly appreciative of all the detail you provided!

I tried to reproduce your issue without any success; here is what i did:

# Create a throw-away test directory
mkdir delme

# Create some dummy files:
touch "delme/The Vietnam War (2017).s01e07.The Veneer of Civilization (June 1968-May 1969).Bluray-1080p.mp4"
touch "delme/The Vietnam War (2017).s01e09.A Disrespectful Loyalty (May 1970-March 1973).Bluray-1080p.mp4"

# Go get our subs:
./Subliminal.py -l en -f -z 0 -s -D delme/

I ran this several times, and in all cases I didn't get the same exception you did. Can you just fill in some details for me?

Edit: Nevermind, i missed the first entry you shared: (The Vietnam War (2017).s01e10.The Weight of Memory (March 1973-Onward).Bluray-1080p.mp4). By touching this file, i can create the problem every time :). Standby for a fix (although it will be similar to yours).

DeeMouse commented 6 years ago

Many thanks for the fix. When testing master I no longer get the fatal error. :smile:

Yes, I included the last two filenames as further examples of the failure to pick up on the show year. The show year is in the filenames - I don't know if this should/can be remedied.

2017-10-18 11:46:30,078 - 30981 - DEBUG - Guessing using: /media/Storage2/TV/The Vietnam War (2017)/s01/problematic/The Vietnam War (2017).s01e10.The Weight of Memory (March 1973-Onward).Bluray-1080p.mp4
2017-10-18 11:46:30,189 - 30981 - DEBUG - Detected year as part of title.
2017-10-18 11:46:30,189 - 30981 - DEBUG - {
    [1.00] "mimetype": "video/mp4", 
    [1.00] "episodeNumber": 10, 
    [1.00] "unidentified": [
        "media", 
        "Storage2", 
        "TV", 
        "March", 
        "Onward"
    ], 
    [1.00] "container": "mp4", 
    [0.40] "title": "The Weight of Memory", 
    [0.76] "series": "The Vietnam War", 
    [1.00] "format": "BluRay", 
    [1.00] "screenSize": "1080p", 
    [1.00] "season": 1, 
    [1.00] "type": "episode"
}
caronc commented 6 years ago

Ah i see what you're saying... so you're expecting to see the year as being 2017 (with respect to this example we've been working with). That is rather odd now that you point it out.

I'll leave this ticket open and have another go at it (hopefully this weekend).

caronc commented 6 years ago

Okay, i think i fixed it. You'll need to pull out the master branch again, but i added an exclusive check for the year in the filename before we start trying to guess what it could be. It should address your concern.

As always, you know the drill: please let me know if it doesn't! :smile:

caronc commented 6 years ago

I hadn't heard back in a while, so I thought I'd follow up. Please let me know if I can close this ticket. From the few tests I did, things appear good. How is it working for you?

DeeMouse commented 6 years ago

Sorry for the late reply, and thanks for taking a look at this.

With master, I'm now getting:

2017-11-04 15:08:12,101 - 10048 - DEBUG - Guessing using: /media/Storage2/TV/The Vietnam War (2017)/s01/The Vietnam War (2017).s01e07.The Veneer of Civilization (June 1968-May 1969).Bluray-1080p.mp4
2017-11-04 15:08:12,210 - 10048 - DEBUG - Detected year (1969) updated to 2017!
2017-11-04 15:08:12,210 - 10048 - DEBUG - Detected year as part of title.
2017-11-04 15:08:12,210 - 10048 - DEBUG - {
    [1.00] "mimetype": "video/mp4", 
    [1.00] "episodeNumber": 7, 
    [1.00] "unidentified": [
        "media", 
        "Storage2", 
        "TV", 
        "June", 
        "May"
    ], 
    [1.00] "container": "mp4", 
    [0.40] "title": "The Veneer of Civilization", 
    [0.76] "series": "The Vietnam War 2017", 
    [1.00] "format": "BluRay", 
    [1.00] "season": 1, 
    [1.00] "screenSize": "1080p", 
    [1.00] "year": "2017", 
    [1.00] "type": "episode"
}

So the year is correct, but the title incorrectly includes ' 2017'. Subtitles are not downloaded for the episode. (I do have subtitles from a previous download attempt, so I believe subtitles for this episode.) I'm guessing that if the reported title were correct, subtitles would be downloaded.

Should/can numbers identified as years that are in brackets in the title part of a filename be excluded from the resultant stated series title?

caronc commented 6 years ago

This is a tough one... Having the year as part of the search is a good thing... for example. Consider the TV Series The Flash. On addic7ed.com's website, the following 2 searches yields 2 completely different responses:

The year is always appended to TV Series (not movies) as part of a bug detected back in Jan/2015 (source) where searching for The Americans yielded different results then searching for The Americans 2013.

Another example is searching for a TV series that is original and doesn't have a date in it. The following searches yield the exact same result:

I guess what I'm getting at is that: if you're not turning up results when searching for The Vietnam War 2017, but you are if you just search for The Vietnam War then the provider hosting the subtitles is denying the ambiguity it otherwise provides to everything else (they should almost be fixing it on their end). Hopefully this makes sense; your thoughts?

DeeMouse commented 6 years ago

I see - thanks for the info. It was merely a guess of mine that the year being on the end of the title in the submitted-to-subtitle-services-info was the reason for no subtitles being found.

I do wonder why I'm not getting any results now when I have previously, but I suppose that's a question for another time/issue entry. :)

Thanks again.