search

ZeroQI / Hama.bundle

Plex HTTP Anidb Metadata Agent (HAMA)
GNU General Public License v3.0
1.21k stars 113 forks source link

networking:219 errors with TVDB, then tvdb4 with all eps in one season folder metadata issue #322

Closed clowncracker closed 5 years ago

clowncracker commented 5 years ago

I am unable to get TVDB data to load. In this case I was trying to match Dragon Ball Z w/ tvdb4. I keep getting networking:219 errors.

com.plexapp.agents.hama.log

2019-06-24 20:03:39,708 (7f3e92c77700) : DEBUG (networking:166) - Requesting 'https://api.thetvdb.com/series/81472?en' 2019-06-24 20:03:39,817 (7f3e92c77700) : ERROR (networking:219) - Error opening URL 'https://api.thetvdb.com/series/81472?en' 2019-06-24 20:03:39,823 (7f3e92c77700) : DEBUG (networking:166) - Requesting 'https://api.thetvdb.com/series/81472/actors' 2019-06-24 20:03:39,906 (7f3e92c77700) : ERROR (networking:219) - Error opening URL 'https://api.thetvdb.com/series/81472/actors' 2019-06-24 20:03:39,913 (7f3e92c77700) : DEBUG (networking:166) - Requesting 'https://api.thetvdb.com/series/81472/episodes?page=1' 2019-06-24 20:03:39,989 (7f3e92c77700) : ERROR (networking:219) - Error opening URL 'https://api.thetvdb.com/series/81472/episodes?page=1' 2019-06-24 20:03:39,994 (7f3e92c77700) : DEBUG (networking:166) - Requesting 'https://api.thetvdb.com/series/81472/episodes?page=2' 2019-06-24 20:03:40,100 (7f3e92c77700) : ERROR (networking:219) - Error opening URL 'https://api.thetvdb.com/series/81472/episodes?page=2' 2019-06-24 20:03:40,107 (7f3e92c77700) : DEBUG (networking:166) - Requesting 'https://api.thetvdb.com/series/81472/episodes?page=3' 2019-06-24 20:03:40,189 (7f3e92c77700) : ERROR (networking:219) - Error opening URL 'https://api.thetvdb.com/series/81472/episodes?page=3' 2019-06-24 20:03:40,194 (7f3e92c77700) : DEBUG (networking:166) - Requesting 'https://api.thetvdb.com/series/81472/episodes?page=4' 2019-06-24 20:03:40,271 (7f3e92c77700) : ERROR (networking:219) - Error opening URL 'https://api.thetvdb.com/series/81472/episodes?page=4' 2019-06-24 20:03:40,285 (7f3e92c77700) : DEBUG (networking:166) - Requesting 'https://api.thetvdb.com/episodes/355412' 2019-06-24 20:03:40,389 (7f3e92c77700) : ERROR (networking:219) - Error opening URL 'https://api.thetvdb.com/episodes/355412' 2019-06-24 20:03:40,394 (7f3e92c77700) : DEBUG (networking:166) - Requesting 'https://api.thetvdb.com/episodes/355413' 2019-06-24 20:03:40,467 (7f3e92c77700) : ERROR (networking:219) - Error opening URL 'https://api.thetvdb.com/episodes/355413' 2019-06-24 20:03:40,472 (7f3e92c77700) : DEBUG (networking:166) - Requesting 'https://api.thetvdb.com/episodes/355414'

ZeroQI commented 5 years ago

HAMA uses log redirection so the log you attached have tons of bits missing and is not very usefull

API worked here https://api.thetvdb.com/swagger#!/Series/get_series_id so TheTVDB API OK

Common.py line 387 should handle that

Have recently been hacked on paypal with RAT+socks port for unauth transactions so:

=== TheTVDB.GetMetadata() ===================================================================================================================================
TVDBid: '81472', IMDbid: '', language_series : ['', 'en', 'ja'], language_episodes: ['', 'en', 'ja']
--- series --------------------------------------------------------------------------------------------------------------------------------------------------
common.SaveFile() - CachePath: 'C:\Users\benja\AppData\Local\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems', file: 'TheTVDB/json/81472\series_da.json'
[ ] language_rank: 3
[ ] title: 
[ ] original_title: 
[ ] IMDbid: tt0214341
[ ] zap2itId: 
[ ] content_rating: TV-PG
[ ] summary: 
[ ] originally_available_at: 1989-04-26
[ ] genres: [u'Action', u'Adventure', u'Animation', u'Anime']
[ ] studio: Fuji TV
[ ] rating: 8.7
[ ] status: Ended
[ ] duration: 1500000
[ ] banner: {u'https://thetvdb.plexapp.com/banners/graphical/81472-g8.jpg': (u'TheTVDB\\banner\\graphical/81472-g8.jpg', 1, None)}
--- actors --------------------------------------------------------------------------------------------------------------------------------------------------
common.SaveFile() - CachePath: 'C:\Users\benja\AppData\Local\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems', file: 'TheTVDB/json/81472\actors_da.json'
[ ] role: Future-Trunks                                     , name: Takeshi Kusao       , photo: https://thetvdb.plexapp.com/banners/actors/459459.jpg
[ ] role: Vegeta                                            , name: Ryou Horikawa       , photo: https://thetvdb.plexapp.com/banners/actors/459461.jpg
[ ] role: 18                                                , name: Miki Itō            , photo: https://thetvdb.plexapp.com/banners/actors/459462.jpg

Library was in "Dansk" for some reason but:

--- episodes ------------------------------------------------------------------------------------------------------------------------------------------------
common.SaveFile() - CachePath: 'C:\Users\benja\AppData\Local\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems', file: 'TheTVDB/json/81472\episodes_page1_da.json'
common.SaveFile() - CachePath: 'C:\Users\benja\AppData\Local\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems', file: 'TheTVDB/json/81472\episodes_page2_da.json'
common.SaveFile() - CachePath: 'C:\Users\benja\AppData\Local\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems', file: 'TheTVDB/json/81472\episodes_page3_da.json'
common.SaveFile() - CachePath: 'C:\Users\benja\AppData\Local\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems', file: 'TheTVDB/json/81472\episodes_page4_da.json'
[ ]    s0e1 s00e001 anidbid:         air_date: 1989-07-15

Check antivirus/security suite/antispyware/firewall, and that you use latest code

clowncracker commented 5 years ago

The issue is only with HAMA, TVDB works on my other libraries. Firewall isn't preventing anything and I am using the latest code. Episode descriptions + posters are not populating for everything after episode 40 (season 2). What other logs should I provide?

ZeroQI commented 5 years ago

i need the series update log in Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems_Logs It may be in a sub-folder with the library name

clowncracker commented 5 years ago

Dragon Ball Z.agent-search.log Dragon Ball Z.agent-update.log Dragon Ball Z.filelist.log Dragon Ball Z.scanner.log

ZeroQI commented 5 years ago

Thank you. I configured the same, but works for windows server for me, you have linux

url; 'https://api.thetvdb.com/series/81472/actors' Headers: {'Accept-Language': 'en', 'Content-type': 'application/json', 'User-agent': 'Plex/HAMA'} HEADERS_CORE = {'User-agent': 'Plex/HAMA', 'Content-type': 'application/json'}

No authentication in header, as if line 397 the tvdb test failed, make no sense... headers=UpdateDict(headers, HEADERS_TVDB if url.startswith(https://api.thetvdb.com') else HEADERS_CORE

clowncracker commented 5 years ago

@ZeroQI anything I can do?

clowncracker commented 5 years ago

@ZeroQI I created a new Plex server with a different docker and I still have TVDB issues. Here are the new logs:

com.plexapp.agents.hama.log Dragon Ball Z.agent-search.log Dragon Ball Z.agent-update.log Dragon Ball Z.filelist.log Dragon Ball Z.scanner.log

ZeroQI commented 5 years ago

updated master, please close if resolved or report if file existed it work new files it failed seemingly

clowncracker commented 5 years ago

It looks like the networking:219 issue went away, but the posters and episode descriptions still didn't update.

com.plexapp.agents.hama.log Dragon Ball Z.agent-search.log Dragon Ball Z.agent-update.log Dragon Ball Z.filelist.log Dragon Ball Z.scanner.log

clowncracker commented 5 years ago

@ZeroQI anything I can do?

ZeroQI commented 5 years ago

@clowncracker you used a season folder and all episodes in it while there is a tvdb mapping setting in the scanner tvdb mapping file... Because there is a mapping hama expect it used but it was overritten by the season folder...

You could use a local mapping tvdb4 mapping for this tvdbid but otherwise empty and this should be solved

Did you want to avail of the tvdb4 mode to put eps in season for each saga automatically or you wanted volontarilly all eps in a single season 1 to avail of thetvdb numbering and metadata for specials?

clowncracker commented 5 years ago

Maybe I'm not understanding how tvdb4 works? Ideally I would want all of the metadata to pull in for episodes 1-291 (as season 1) and also pull in for the specials (specials).

Is my folder structure wrong? Should I be making updates to the tvdb xml file?

ZeroQI commented 5 years ago

If you had not put a season folder, it would try to append mappings, since there is no local mapping file with <anime tvdbid="81472" name="Dragon Ball Z"> in it, it would have loaded the mapping from tvdb4 here: https://github.com/ZeroQI/Absolute-Series-Scanner/blob/master/tvdb4.mapping.xml

  <anime tvdbid="81472" name="Dragon Ball Z">
    01|001|039|Saiyan Saga
    02|040|074|Namek And Captian Ginyu Sagas
    03|075|107|Frieza Saga
    04|108|139|Garlic Jr., Trunks and Androids Sagas
    05|140|165|Imperfect Cell and Perfect Cell Sagas
    06|166|194|Cell Games Saga
    07|195|219|Great Saiyaman and World Tournament Sagas
    08|220|253|Babidi and Majin Buu Sagas
    09|254|291|Fusion and Kid Buu Sagas
  </anime>

and created 9 seasons and the metadata would match... Since you force a single season but there is a mapping HAMA wrongly applies it... Only happens with series with a different season mapping than the mapping file, which is rare enough...

I don't want to apply detection in the final loop and mapping seem to be line 139-140 of TheTVDBv2.py. I believe the following addition will re-test for tvdb4 only if the file mapped exist and reset to existing file otherwise. Please test and report

      elif season!='0' and metadata_source=='tvdb4':  
        ms, usl                         = Dict(mappingList, 'absolute_map', 'max_season'), Dict(mappingList, 'absolute_map', 'unknown_series_length')
        if ms and usl:  season, episode = Dict(mappingList, 'absolute_map', str(abs_number), default=(ms if usl else str(int(ms)+1), None))[0], str(abs_number)

        if metadata_source=='tvdb4' and (season not in media.seasons or episode not in media.seasons[season].episodes):
          for s in media.seasons:
          if str(abs_number) in media.seasons[s].episodes:
            season, episode = s, abs_number
            break
clowncracker commented 5 years ago

No metadata loaded at all with that fix. Attached the updated TheTVDBv2.py as well.

Dragon Ball Z.filelist.log Dragon Ball Z.scanner.log com.plexapp.agents.hama.log Dragon Ball Z.agent-search.log Dragon Ball Z.agent-update.log TheTVDBv2.zip

ZeroQI commented 5 years ago

"IndentationError: expected an indented block" Off by one or more spaces (or used tabs)

On Sun, Jul 7, 2019 at 10:18 PM clowncracker notifications@github.com wrote:

No metadata loaded at all with that fix. Attached the updated TheTVDBv2.py as well.

Dragon Ball Z.filelist.log https://github.com/ZeroQI/Hama.bundle/files/3366112/Dragon.Ball.Z.filelist.log Dragon Ball Z.scanner.log https://github.com/ZeroQI/Hama.bundle/files/3366113/Dragon.Ball.Z.scanner.log com.plexapp.agents.hama.log https://github.com/ZeroQI/Hama.bundle/files/3366114/com.plexapp.agents.hama.log Dragon Ball Z.agent-search.log https://github.com/ZeroQI/Hama.bundle/files/3366115/Dragon.Ball.Z.agent-search.log Dragon Ball Z.agent-update.log https://github.com/ZeroQI/Hama.bundle/files/3366116/Dragon.Ball.Z.agent-update.log TheTVDBv2.zip https://github.com/ZeroQI/Hama.bundle/files/3366117/TheTVDBv2.zip

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ZeroQI/Hama.bundle/issues/322?email_source=notifications&email_token=ABHMWZKW46ZJVO7CWCQAWMLP6JMSXA5CNFSM4H3MQ452YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZLTJAA#issuecomment-509031552, or mute the thread https://github.com/notifications/unsubscribe-auth/ABHMWZMWVJQ2U5UUDYSHKNDP6JMSXANCNFSM4H3MQ45Q .

clowncracker commented 5 years ago

I fixed the code, still no metadata for episodes 40+.

Dragon Ball Z.agent-update.log Dragon Ball Z.filelist.log Dragon Ball Z.scanner.log com.plexapp.agents.hama.log Dragon Ball Z.agent-search.log TheTVDBv2.zip

ZeroQI commented 5 years ago

Before new code [X] s9e38 s09e038 anidbid: air_date: 1996-01-31 abs_number: 291, title: Goku's Next Journey After new code [X] s9e38 s01e291 anidbid: air_date: 1996-01-31 abs_number: 291, title: Goku's Next Journey

Looking at TheTVDB_dict returned it's near perfect now, but the absolute ep number is not a string... Please correct by changing season, episode = s, abs_number to season, episode = s, str(abs_number)

      elif season!='0' and metadata_source=='tvdb4':  
        ms, usl                         = Dict(mappingList, 'absolute_map', 'max_season'), Dict(mappingList, 'absolute_map', 'unknown_series_length')
        if ms and usl:  season, episode = Dict(mappingList, 'absolute_map', str(abs_number), default=(ms if usl else str(int(ms)+1), None))[0], str(abs_number)

        if metadata_source=='tvdb4' and (season not in media.seasons or episode not in media.seasons[season].episodes):
          for s in media.seasons:
          if str(abs_number) in media.seasons[s].episodes:
            season, episode = s, str(abs_number)
            break
ZeroQI commented 5 years ago

@clowncracker if change works, will integrate on master code, just need "str()" arount "abs_number"

clowncracker commented 5 years ago

@ZeroQI sorry for the delay, looks like it works. Thanks for all your help!

clowncracker commented 5 years ago

@ZeroQI, the 219 error came back after the latest update.

com.plexapp.agents.hama.log Isekai Cheat Magician.agent-update.log

ZeroQI commented 5 years ago

Error opening URL 'http://api.tmdb.org/3/movie/90215? Error opening URL 'http://webservice.fanart.tv/v3/tv/360708?api_key=cfa9dc054d221b8d107f8411cd20b13f' Error opening URL 'http://api.tmdb.org/3/movie/tt10362632? http://webservice.fanart.tv/v3/tv/360502?api_key=cfa9dc054d221b8d107f8411cd20b13f

there's no tvdb error in there... fanart tv seem to genuinely have no meta for that series... is something actually not gathering metadata like before? some 219 errors are normal, before other people were impacted and had meta issue...

clowncracker commented 5 years ago

Why isn't the episode name updating? There's clearly data there from the hama log.

Line 193: [X] s1e 4 => s1e 4 air_date: 2019-07-31 language_rank: 0, title: "Crimson Pact" Line 401: [?] rank: 0, source_title: TheTVDB, title: "Episode 4"

ZeroQI commented 5 years ago 
metadata.seasons[ 1].episodes[  4]
[?] rank: 0, source_title: TheTVDB, title: "Episode 4"
[=] title                          Sources:  TheTVDB, AniDB| (TheTVDB), AniDB                             Inside: '['TheTVDB', 'AniDB']'  Value: 'Episode 4'

Your agent settings says to pick for episode title TheTVDB as source first (if language for all are at the same level) which has "Episode 4" title and it does assign TheTVDB title to the library...

Do you mean it should skip TheTVDB titles such as "Episode 4" despite TheTVDB ep titles having priority?

EndOfLine369 commented 5 years ago

@ZeroQI, we use to in the past (quite a while ago) ignore episode titles that were "Episode \d". I think we should add it into the AniDB & TVDB episode title checks and clear out the string if it matches. Sound good to you? Only potential problem would be when both sources are "Episode \d"...

ZeroQI commented 5 years ago

@EndOfLine369 I don't see that as a problem... Using open bug https://github.com/ZeroQI/Hama.bundle/issues/329 for correcting

clowncracker commented 5 years ago

That's odd, since the library has the title preference as the following: T-EM 'title' : AniDB, TheTVDB | TheTVDB, AniDB

That would indicate that the tile should be pulling from AniDB first, right? I checked DefaultPrefs.json and it has the same setting.

clowncracker commented 5 years ago

Also of note it looks like TVDB was updated with an episode title. I did the Plex dance and no episode title is still loading.

com.plexapp.agents.hama.log Isekai Cheat Magician.agent-update.log

ZeroQI commented 5 years ago

Episode's title comes from TheTVDB If your episodes gets no title your Plex database is corrupted as HAMA provides them

metadata.seasons[ 1].episodes[ 1] [?] rank: 0, source_title: TheTVDB, title: "Lost Ones from Another World" [=] title Sources: TheTVDB, AniDB| (TheTVDB), AniDB Inside: '['TheTVDB', 'AniDB']' Value: 'Lost Ones from Another World'

metadata.seasons[ 1].episodes[ 2] [?] rank: 0, source_title: TheTVDB, title: "Magic Training" [=] title Sources: TheTVDB, AniDB| (TheTVDB), AniDB Inside: '['TheTVDB', 'AniDB']' Value: 'Magic Training'

metadata.seasons[ 1].episodes[ 3] [?] rank: 0, source_title: TheTVDB, title: "Beginner Adventurers" [=] title Sources: TheTVDB, AniDB| (TheTVDB), AniDB Inside: '['TheTVDB', 'AniDB']' Value: 'Beginner Adventurers'

metadata.seasons[ 1].episodes[ 4] [?] rank: 0, source_title: TheTVDB, title: "Episode 4" [=] title Sources: TheTVDB, AniDB| (TheTVDB), AniDB Inside: '['TheTVDB', 'AniDB']' Value: 'Episode 4'

clowncracker commented 5 years ago

Why is the title reverting to TVDB when the preferences are set to AniDB?

Line 64 of hama log: 2019-08-01 19:42:40,179 (7fc0c6577700) : INFO (init:52) - Prefs[title ] = AniDB, TheTVDB | TheTVDB, AniDB (still default value)

Also - TVDB has a title for episode 4: https://www.thetvdb.com/series/isekai-cheat-magician/episodes/7254225

ZeroQI commented 5 years ago

The episode 4 that aired yesterday? Thetvdb meta was cached, file is 17 hours old seemingly: common.LoadFile() - File cached locally - url: 'https://api.thetvdb.com/series/360708/actors', Filename: 'TheTVDB/json/360708/actors_en.json', Age: '63257.5513666', Ended: None Title priority: AniDB, TheTVDB | TheTVDB, AniDB = for series title: AniDB, TheTVDB = for episodes title: TheTVDB, AniDB

TheTVDB is first for episode titles...

clowncracker commented 5 years ago

EDIT: Sorry for bugging you with this, it looks like TVDB is having API issues right now.

https://forums.thetvdb.com/viewtopic.php?f=118&t=57657