search

ZeroQI / Hama.bundle

Plex HTTP Anidb Metadata Agent (HAMA)
GNU General Public License v3.0
1.19k stars 110 forks source link

Hama not grabbing all metadata, weird title issue #411

Closed ToolB0xx closed 3 years ago

ToolB0xx commented 3 years ago 
2020-07-11 17:35:50,342 (7fbc71ffb700) :  CRITICAL (agentkit:1095) - Exception in the update function of agent named 'HamaTV', called with guid 'com.plexapp.agents.hama://tvdb-87501?lang=en' (most recent call last):
  File "/usr/lib/plexmediaserver/Resources/Plug-ins-864bbcbb7/Framework.bundle/Contents/Resources/Versions/2/Python/Framework/api/agentkit.py", line 1093, in _update
    agent.update(obj, media, lang, **kwargs)
  File "/home/plexdata/Library/Application Support/Plex Media Server/Plug-ins/Hama.bundle/Contents/Code/__init__.py", line 165, in update
    def update (self, metadata, media, lang, force ):  Update (metadata, media, lang, force,  False)
  File "/home/plexdata/Library/Application Support/Plex Media Server/Plug-ins/Hama.bundle/Contents/Code/__init__.py", line 140, in Update
    dict_AniDB, ANNid, MALid                                      =       AniDB.GetMetadata(media, movie, error_log,       source, AniDBid, TVDBid, AnimeLists.AniDBMovieSets, mappingList)
  File "/home/plexdata/Library/Application Support/Plex Media Server/Plug-ins/Hama.bundle/Contents/Code/AniDB.py", line 347, in GetMetadata
    if missing_eps:  error_log['Missing Specials' if season=='0' else 'Missing Episodes'].append("AniDBid: %s | Title: '%s' | Missing Episodes: %s" % (common.WEB_LINK % (common.ANIDB_SERIE_URL + AniDBid, AniDBid), AniDB_dict['title'], str(missing_eps)))
  File "/home/plexdata/Library/Application Support/Plex Media Server/Plug-ins/Hama.bundle/Contents/Code/common.py", line 158, in GetXml
    def GetXml          (xml,      field                ):  return xml.xpath(field)[0].text if xml.xpath(field) and xml.xpath(field)[0].text not in (None, '', 'N/A', 'null') else ''  #allow isdigit() checks
AttributeError: 'str' object has no attribute 'xpath'

2020-07-11 17:35:50,342 (7fbc71ffb700) :  DEBUG (model:229) - Serializing to /home/plexdata/Library/Application Support/Plex Media Server/Metadata/TV Shows/2/b1ffaa1304a424309bbf916dffd842c20ab761d.bundle/Contents/com.plexapp.agents.hama/Info.xml

seriesIDthingy

While trying to refresh my metadata for some of my series, Hama stopped working or correctly grabbing all the info at one point. I've had this issue of the TVDB or AniDB identifiers appearing after series, and all it took was another refresh to get rid of it and get the metadata grabbed and updated. However, now it is not grabbing anything, and even swapping metadata I previously had to this now broken format, which is strange. Adding new series shows no results either, no grabbing metadata of any kind, not even the broken title format, no art or anything. I tried going from the latest version of Hama https://github.com/ZeroQI/Hama.bundle/commit/c44961c64b96546daa157b98a3063d1bf907b6a6 (The one with some support added for AniList) and a few revisions before the initial changes for AniList, https://github.com/ZeroQI/Hama.bundle/commit/f075b75d860c28e5d28bc69c9e9653220cb33c40, to no avail. The aforementioned block of code seems to be the only thing haunting the logs, I did some quick digging, the only thing I found close to it was the dreaded AniDB anime-titles.xml issues, which I've checked and insured it is not corrupted and is the latest available one.

Not sure if this had been seen before previously, but if it was, I couldn't find one like this anywhere in the open or closed issues. If you need any more files let me know. Thanks!

com.plexapp.agents.hama.log Series-XMLs.zip

GANTZ.agent-update.log K-ON!.agent-update.log Gunslinger Girl.agent-update.log Yuusha ni Narenakatta Ore wa Shibushibu Shuushoku wo Ketsui Shimashita.agent-update.log

ZeroQI commented 3 years ago

I wouldn't trust 100% the lines mentionned by the logs, i ofter have seen the wrong lines shown and the error totally elsewhere, but the error message is valid: AttributeError: 'str' object has no attribute 'xpath'

Likely, the file return is valid but not an XML The files cached by the agent in agent data folders could help, note sure if AniDB or TheTVDB ones

Logs all stop during the AniDB section and the latest line displayed is: --- xxxx.series ------------, just before the XML loading https://github.com/ZeroQI/Hama.bundle/blob/master/Contents/Code/AniDB.py line 162

Maybe trap the xml loading by replacing line 165 with:

    try:        xml_cache  = common.LoadFileCache(filename=AniDBid+".xml", relativeDirectory=os.path.join("AniDB", "xml"))[0]
    except:  xml_cache = None
ToolB0xx commented 3 years ago

Went ahead and tried that out, no changes in behavior or log files, went ahead and added some more catches, seems like code execution completely stops at Line 167 (As in it executes up to that point, and this line is most likely responsible for the current crash) https://github.com/ZeroQI/Hama.bundle/blob/c7ebfb73a9b590a79fa1247795ba5a3432b1368a/Contents/Code/AniDB.py#L167

xml_cache does not seem to be empty as it went into the if statement, any idea how I can look and see for myself what the current xml_cache looks like at this point?

ToolB0xx commented 3 years ago

Well I seemed to have fixed it. Not sure exactly what it was or what might still be broken. What I did was changed https://github.com/ZeroQI/Hama.bundle/blob/c7ebfb73a9b590a79fa1247795ba5a3432b1368a/Contents/Code/AniDB.py#L167 to

try:
  ed = GetXml(xml_cache, 'enddate')
except:
  ed = datetime.datetime.now().strftime("%Y-%m-%d")

Now after refreshing metadata, all those series are now properly grabbing metadata and propagating.

Edit: Okay, most of them worked, then it stopped working, but this time it wasn't an exception or anything, and after looking at the update logs, I found out I was banned from AniDB. How long does a ban last? And could me being banned caused the initial issue in the first place?

AniDB_dict: {
  'Banned': True,
  'language_rank': 0,
  'original_title': 'Love Live! Sunshine!!',
  'title': 'Love Live! Sunshine!!'}
ZeroQI commented 3 years ago

@ToolB0xx Well done!

ZeroQI commented 3 years ago

Integrated your fix in the main code after re-working it slightly, thanks. that will prevent many crashes, probably more in new series with field missing in the XML, causing the crash...

Please close the ticket if the new code works, you should have been de-banned by now

EndOfLine369 commented 3 years ago

@ZeroQI, that is not the issue here. AttributeError: 'str' object has no attribute 'xpath' via GetXml(xml_cache, 'enddate') means that xml_cache got a string returned from common.LoadFileCache(). Meaning the file he had cached was not a valid xml.

The example of K-ON: http://api.anidb.net:9001/httpapi?request=anime&client=hama&clientver=1&protover=1&aid=6257 This returns a valid xml w/ enddate. <enddate>2009-06-19</enddate> which has not been touched since 2009 https://anidb.net/anime/6257/history

This all seems to just be a local file cache issue here and that try should not be added into the AniDB module. The try needs to be removed, hit his error, and look at what is in the DataItems\AniDB\xml\<anidbid>.xml that is being seen as a non-xml/string.

ZeroQI commented 3 years ago

@EndOfLine369 This message is present in the general hama log, but not the series specific log We definitely need to check that but should have gotten a similar message further if that was the case and not output anything else due to the crash...

AniDBmight have changed the shape of the banning message... We need 2 failsafe:

EndOfLine369 commented 3 years ago

Simple test for string added https://github.com/ZeroQI/Hama.bundle/blob/7d063fc9b45615669b4a92363470dd5e5fdf8bb6/Contents/Code/AniDB.py#L166

Same thing is already done later to make sure we don't get a banned or other string response from AniDB. https://github.com/ZeroQI/Hama.bundle/blob/7d063fc9b45615669b4a92363470dd5e5fdf8bb6/Contents/Code/AniDB.py#L178

ZeroQI commented 3 years ago

issue still present?

ToolB0xx commented 3 years ago

I can download the latest version of Hama and try what @EndOfLine369 suggested. I'll see what is lying in wait in that .xml file

ZeroQI commented 3 years ago

any update by any chance?

ToolB0xx commented 3 years ago

Sorry It's taken awhile to get back to you, I've since downloaded the latest version, but I haven't really had any new media to add, so I haven't seen any issues appear as of yet. So I guess you could say it's working great so far

ZeroQI commented 3 years ago

No need for new Media, if the error is gone from the series you had an issue with after HAMA update and metadata update we close the case...

ToolB0xx commented 3 years ago

Alright, sounds good. I haven't noticed any more issues since the update, and everything has a proper title, so I will go ahead and mark as closed. Thanks.