HAMA not populating metadata.

myhe01 commented 4 years ago

com.plexapp.agents.hama.log com.plexapp.system.log Plex Media Server Logs_2020-08-18_15-19-35.zip

The following logs did not exist for me: series_root_folder.filelist.log, series_root_folder.scanner.log, series_root_folder.agent-search.log, or series_root_folder.agent-update.log.

My issue is that HAMA does not want to populate any sort of metadata (titles, studio, poster, etc.) for any of the anime in my library. ASS seems to recognize the directories of the series. Fixing the match manually does not solve the issue.

HAMA and ASS do show up in the Scanner and Agent dropdowns.

I have installed and reinstalled HAMA a few times, always using the latest version. Have deleted the library, stopped Plex, updated HAMA (haven't touched ASS because it seems to work just fine), started plex, and recreated the library. Done the aforementioned several times.

The first time I installed HAMA, I had to go in and create the DataItems directories and their subdirectories.

I'm pretty sure I've been installing it into the proper directory.

I have ran chmod -R 777 '/mnt/ValleyForge/iocage/jails/plex/root/Plex Media Server/Plug-in Support/Data/com.plexapp.agents.hama/DataItems/' from the shell a few times, trying to troubleshoot. Also ran chown -R brendan:admin '/mnt/ValleyForge/iocage/jails/plex/root/Plex Media Server/Plug-in Support/Data/com.plexapp.agents.hama/'

I'm pretty sure I've done everything right, and all of the fixes I've tried don't seem to work. Reddit and Google have no more answers for me. Kinda losing my mind over this.

ZeroQI commented 4 years ago

Scanner logs were moved to Plex logs folder. Since ASS is not updated then HAMA cannot determine the root folder from the scanner logs...

myhe01 commented 4 years ago

Deleted my library, stopped Plex. Reinstalled latest versions of ASS and HAMA, updated Plex. Recreated my library, issue still persists. Tried manually fixing the match, no dice.

Please find logs below.

root.agent.log root.scanner.log root--Plex Media Server-BRENDAN-NAS-Anime.filelist.log root--Plex Media Server-BRENDAN-NAS-Anime.scanner.log com.plexapp.agents.hama.log com.plexapp.system.log Plex Media Scanner Analysis.log Plex Media Scanner Deep Analysis.log Plex Media Scanner.log Plex Media Server Logs_2020-08-18_23-38-36.zip Plex Media Server.log

ZeroQI commented 4 years ago

If the versions are the latest, why does it not recognise 'AniList' source in HAMA logs when loading settings?

Where in GitHub do you download HAMA?

10 movies are on the library root and mislabeled. Forced IDs not working on FILES...

EndOfLine369 commented 4 years ago

If the versions are the latest, why does it not recognise 'AniList' source in HAMA logs when loading settings?

As the code was not updated to know this source in validations. Now fixed.

ZeroQI commented 4 years ago

I cannot see any crash

Agent and scanner are well installed as they are both showin in plex
series show up in Plex so scanner is OK
manual scan show series "Angel Beats!" in screenshot so we should have series logs since no errors are found in system or hama generic logs.
Your series do not gather metadata. Please attach series logs such as "Angel Beats!.search.log" and "Angel Beats!.update.log" in agent data folders /_Logs/Anime. They seem awfully shorts in size. did you look at the series logs?

myhe01 commented 4 years ago

From the readme, went to the green "Code" button, clicked "Download ZIP." Extracted folders, renamed it to Hama.bundle.

Logs for the individual series seem to report that metadata was pulled? Not sure, not too experienced in this sort of thing. Please find the requested logs, for a few different series, below.

Angel Beats!.agent-search.log Angel Beats!.agent-update.log Angel Beats!.filelist.log Angel Beats!.scanner.log Black Lagoon.agent-search.log Black Lagoon.agent-update.log Black Lagoon.filelist.log Black Lagoon.scanner.log Blend S [anidb-12694].agent-search.log Demon Slayer [anidb-14107].agent-search.log Demon Slayer [anidb-14107].agent-update.log Demon Slayer [anidb-14107].filelist.log Demon Slayer [anidb-14107].scanner.log Erased [anidb-11292].agent-search.log Erased [anidb-11292].agent-update.log Erased [anidb-11292].filelist.log Erased [anidb-11292].scanner.log Lucky Star.agent-search.log Lucky Star.agent-update.log Lucky Star.filelist.log Lucky Star.scanner.log Your Lie in April [anidb-10539].agent-search.log Your Lie in April [anidb-10539].agent-update.log Your Lie in April [anidb-10539].filelist.log Your Lie in April [anidb-10539].scanner.log

ZeroQI commented 4 years ago

Now we are talking

the Search is called and logged properly per series
No error in com.plexapp.agents.hama.log nor in com.plexapp.system.log
Update ends prematurely after:
```
=== AnimeLists.GetMetadata() ================================================================================================================================
tvdb_numbering: False
```
https://github.com/ZeroQI/Hama.bundle/blob/master/Contents/Code/AnimeLists.py Line 85 is the last line displayed it calls GetAniDBTVDBMapCustom and should display something and doesn't line 54, 55 or 58
path or os.sep cause issues when GetLibraryRootPath is called?

EndOfLine369 commented 4 years ago

@ZeroQI, agreed. It should say yes/no/failed for the anime-list-custom.xml check/load. Any crash info should be shown in the _Logs/Anime/_root_.agent.log file.

@myhe01, Please attach the file with its current info.

myhe01 commented 4 years ago

Uninstalling and reinstalling the Plex plugin (and HAMA/ASS, downloaded the latest versions as of this post) didn't help, rip. Still having the exact same issue and symptoms.

Here's the file. root.agent.log

EndOfLine369 commented 4 years ago

@ZeroQI, there is no problem in GetLibraryRootPath. This is already called in Log.Open() which works fine. EX: https://github.com/ZeroQI/Hama.bundle/blob/ea5e470844b7196431edda8e637cee00445cd3b7/Contents/Code/common.py#L104-L106

=============================================================================================================================================================
common.PlexLog(file="/Plex Media Server/Plug-in Support/Data/com.plexapp.agents.hama/DataItems/_Logs/Anime/Your Lie in April [anidb-10539].agent-update.log", movie=False)
[!] file:       "//Plex Media Server/BRENDAN-NAS/Anime/Your Lie in April [anidb-10539]/[Reaktor] Your Lie in April - E20 [1080p][x265][10-bit][Dual-Audio].mkv"
[ ] library:    "Anime"
[ ] root:       "//Plex Media Server/BRENDAN-NAS/Anime"
[ ] path:       "Your Lie in April [anidb-10539]"
[ ] Plex root:  "/Plex Media Server"
[ ] Log folder: "Plug-in Support/Data/com.plexapp.agents.hama/DataItems/_Logs/Anime"
[ ] Log file:   "Your Lie in April [anidb-10539].agent-update.log"
[ ] Logger:     "0x809f48f00"
[ ] mode:       "w"

I suspect we have an infinite loop here caused by something in myhe01's setup. The only way it would stop printing is either it crashed (which we are not seeing any evidence of) or its stuck in an infinite loop

@myhe01, Add the line Log.Info("Checking: {}".format(dir)) to AnimeLists.py above line 50 shown below. Then trigger a refresh of a series and take a look at the agent-update.log for that series.

https://github.com/ZeroQI/Hama.bundle/blob/ea5e470844b7196431edda8e637cee00445cd3b7/Contents/Code/AnimeLists.py#L45-L59

EX:

def GetAniDBTVDBMapCustom(media, movie):  
  AniDBTVDBMapCustom = None
  lib, root, path = common.GetLibraryRootPath(common.GetMediaDir(media, movie))
  dir = os.path.join(root, path)
  while dir and os.path.splitdrive(dir)[1] != os.sep:
    Log.Info("Checking: {}".format(dir))      ##### HERE ######
    scudlee_filename_custom = os.path.join(dir, SCHUDLEE_CUSTOM)
    if os.path.exists( scudlee_filename_custom ):

myhe01 commented 4 years ago

@EndOfLine369

Added code to above line 50. Restarted Plex plugin. Clicked the "Refresh Metadata" option in the context menu for Darling in the FranXX, Lucky Star, and Watamote.

All of their log files are in the double-digit megabytes, lmao.

For Darling in the FranXX:

common.PlexLog(file="/Plex Media Server/Plug-in Support/Data/com.plexapp.agents.hama/DataItems/_Logs/Anime/Darling in the FranXX.agent-update.log", movie=False)
[!] file:       "//Plex Media Server/BRENDAN-NAS/Anime/Darling in the FranXX/E24 Never Let Me Go.mkv"
[ ] library:    "Anime"
[ ] root:       "//Plex Media Server/BRENDAN-NAS/Anime"
[ ] path:       "Darling in the FranXX"
[ ] Plex root:  "/Plex Media Server"
[ ] Log folder: "Plug-in Support/Data/com.plexapp.agents.hama/DataItems/_Logs/Anime"
[ ] Log file:   "Darling in the FranXX.agent-update.log"
[ ] Logger:     "0x809e43500"
[ ] mode:       "w"
=== Update() ================================================================================================================================================
id: anidb-13259, title: None, lang: en, force: True, movie: False
start: 2020-08-20 00:20:31,799761
=== AnimeLists.GetMetadata() ================================================================================================================================
tvdb_numbering: False
Checking: //Plex Media Server/BRENDAN-NAS/Anime/Darling in the FranXX
Checking: //Plex Media Server/BRENDAN-NAS/Anime
Checking: //Plex Media Server/BRENDAN-NAS
Checking: //Plex Media Server
Checking: //
Checking: //
Checking: //
Checking: //
Checking: //
Checking: //
Checking: //
Checking: //
Checking: //
Checking: //

it's just Checking: // all the way down to, hmm let's see...

line 1053525.

Checking: //
Checking: //
Checking: //
Checking: //
Checking:

Same deal for Lucky Star and Watamote.

Darling in the FranXX.agent-update.log Lucky Star.agent-update.log Watamote [anidb-9582].agent-update.log

Just realized the formatting is kinda off in this comment. Just check the logs and you'll see.

EndOfLine369 commented 4 years ago

Well, that confirms that its an infinite loop happening. You can remove that log line so it doesn't create large logs being stuck in that loop till this is fixed. I bet ASS will also get stuck on any series if you use either of the anidb2/3/4 modes.

ASS:

    if source in ["anidb2", "anidb3", "anidb4"]:
...
      dir = os.path.join(root, path)
      while dir and os.path.splitdrive(dir)[1] != os.sep:
        scudlee_filename_custom = os.path.join(dir, ANIDB_TVDB_MAPPING_LOC)
        if os.path.exists( scudlee_filename_custom ):

HAMA:

   dir = os.path.join(root, path) 
   while dir and os.path.splitdrive(dir)[1] != os.sep: 
     scudlee_filename_custom = os.path.join(dir, SCHUDLEE_CUSTOM) 
     if os.path.exists( scudlee_filename_custom ):

Issue logic is while dir and os.path.splitdrive(dir)[1] != os.sep:

ZeroQI commented 4 years ago

while dir and os.path.splitdrive(dir)[1] not in (os.sep, '//', ''):

EndOfLine369 commented 4 years ago

@ZeroQI, what is the reasoning of us going down below root? I can see no reason for this to be done. Then its a simple test of dir == root and break out of the loop.

https://github.com/ZeroQI/Absolute-Series-Scanner/commit/bbf238308aa6212941873f6a70343320677f0276#diff-edf9a02ffa5a0496c5247e048b83f667R737 https://github.com/ZeroQI/Absolute-Series-Scanner/commit/2b608f5996df476f8ccd6279b44da5a52785d075#diff-edf9a02ffa5a0496c5247e048b83f667R433 https://github.com/ZeroQI/Absolute-Series-Scanner/commit/3b32ebbc0cbe7ade467a5803afc1d68627b7a7dc#diff-edf9a02ffa5a0496c5247e048b83f667R433

HAMA was mirrored from ASS https://github.com/ZeroQI/Hama.bundle/commit/8545ed70d5421b0deedd150f393861ee91cbf516#diff-d7b7f1f1970112645f6c605242902304R43

ZeroQI commented 4 years ago

@EndOfLine369 Good point. Avoiding questions from user putting it at root level I guess...

while dir and os.path.splitdrive(dir)[1] != os.sep and dir != root:

EndOfLine369 commented 4 years ago

No, we still want to process root. I personally only put in root if I need to do anything and we don't want to break any existing setups by users. Be it in the root or a folder within it. I'll get the logic tested updated later.

EndOfLine369 commented 4 years ago

@ZeroQI, Just update the test to while dir & clear dir once we've just processed root dir = os.path.dirname(dir) if len(dir) > len(root) else ''. See any issues with this logic? This should cover any os and remote mount drive identifier.

HAMA EX:

  while dir:
    scudlee_filename_custom = os.path.join(dir, SCHUDLEE_CUSTOM)
    if os.path.exists( scudlee_filename_custom ):
      try:
        AniDBTVDBMapCustom = XML.ElementFromString(Core.storage.load(scudlee_filename_custom))
        Log.Info("Local custom mapping file loaded: {}".format(scudlee_filename_custom))
      except:  Log.Error("Failed to open: '%s', error: '%s'" % (scudlee_filename_custom, e))
      else:    break
    dir = os.path.dirname(dir) if len(dir) > len(root) else ''  # Clear variable if we've just finished processing down to (and including) root
  else:  Log.Info("Local custom mapping file not present: {}".format(SCHUDLEE_CUSTOM))

ZeroQI commented 4 years ago

@EndOfLine369 I thought you meant file system root but was obviously library root. Nothing technically wrong with your code but i like shorter code and less tests, especially if simpler...

  while  len(dir)>=len(root):
    scudlee_filename_custom = os.path.join(dir, SCHUDLEE_CUSTOM)
    if os.path.exists( scudlee_filename_custom ):
      try:
        AniDBTVDBMapCustom = XML.ElementFromString(Core.storage.load(scudlee_filename_custom))
        Log.Info("Local custom mapping file loaded: {}".format(scudlee_filename_custom))
      except:  Log.Error("Failed to open: '%s', error: '%s'" % (scudlee_filename_custom, e))
      else:    break
    dir = os.path.dirname (dir)
  else:  Log.Info("Local custom mapping file not present: {}".format(SCHUDLEE_CUSTOM))

EndOfLine369 commented 4 years ago

Hi @ZeroQI, your code might be simpler but is not the same logic. You're logic creates a an infinite loop when plex root is a drive root. We need a do/while with a '>' and not a '>=' which is what mine is to process plex root before breaking out of the loop.

EX:

>>> a1, a2
('C:\\', 'C:\\Users\\Public\\Music')
>>> test(a1, a2)
Checking: C:\Users\Public\Music
Checking: C:\Users\Public
Checking: C:\Users
Checking: C:\
Local custom mapping file not present: test.xml
>>> test2(a1, a2)
Checking: C:\Users\Public\Music
Checking: C:\Users\Public
Checking: C:\Users
Checking: C:\
Checking: C:\
...
...

>>> c1, c2
('/', '/share/Anime/Deca-Dence')
>>> test(c1, c2)
Checking: /share/Anime/Deca-Dence
Checking: /share/Anime
Checking: /share
Checking: /
Local custom mapping file not present: test.xml
>>> test2(c1, c2)
Checking: /share/Anime/Deca-Dence
Checking: /share/Anime
Checking: /share
Checking: /
Checking: /
...

EndOfLine369 commented 4 years ago

@myhe01 Please replace lines 49->57 of "AnimeLists.py": https://github.com/ZeroQI/Hama.bundle/blob/ea5e470844b7196431edda8e637cee00445cd3b7/Contents/Code/AnimeLists.py#L49-L57

With this and test please.

  while dir:
    scudlee_filename_custom = os.path.join(dir, SCHUDLEE_CUSTOM)
    if os.path.exists( scudlee_filename_custom ):
      try:
        AniDBTVDBMapCustom = XML.ElementFromString(Core.storage.load(scudlee_filename_custom))
        Log.Info("Local custom mapping file loaded: {}".format(scudlee_filename_custom))
      except:  Log.Error("Failed to open: '%s', error: '%s'" % (scudlee_filename_custom, e))
      else:    break
    dir = os.path.dirname(dir) if len(dir) > len(root) else ''  # Clear variable if we've just finished processing down to (and including) root

myhe01 commented 4 years ago

Stopped Plex. Went to the .py you pointed out.

Replaced:

Started Plex. Went to my Anime library and tried to manually fix the match and refresh metadata of a few shows.

Holy shit, it works. Thank you so much for your assistance @ZeroQI and @EndOfLine369. I'll have to go through and freshen things up, but HAMA is retrieving metadata.

I do notice that it's not getting metadata for anime movies and OVAs despite being tagged with an AniDB ID. My guess is that I just need to make seperate libraries. One with a HAMA movie agent, and one with a HAMA TV agent.

I also notice that shows are pulling posters from one source, whether it be the first or the second (or etc.) season of a show (e.g. Black Lagoon 2 only pulls posters of the first season).

EndOfLine369 commented 4 years ago

I do notice that it's not getting metadata for anime movies and OVAs despite being tagged with an AniDB ID. My guess is that I just need to make seperate libraries. One with a HAMA movie agent, and one with a HAMA TV agent.

Should not be the case. I personally only have all my stuff in a TV library. Provide 1-2 of the 'agent-update.log' files of one you say is not loading metadata.

Same request for log files of a series you think is loading the incorrect poster.

Note, in your image above you have both the series and collections being shown in the same view. You probably have "Show collections and their items" selected so you see two entries for "Bakuon!!".

myhe01 commented 4 years ago

I'm not able to see the log files of unloaded items. Deleted and recreated the library, no dice.

For example, I'm trying to find Your Name.

As an aside, my Plex library lists 72 items, whereas the folder where my media is stored lists 93 items.

ZeroQI commented 4 years ago

Seem like the scanner didn't pick them up... Are the missing ones movies in the root folder ?

EDIT: The readme for ASS states "Movies in series libraries support (names same as folder or contain " - Complete Movie")"

EndOfLine369 commented 4 years ago

It looks like your putting multiple movies just in the library root folder? As mentioned by @ZeroQI, you have neither an episode number or " - Complete Movie" string on the filename for your "Your Name" mention. Also note that putting an id tag (EX: [anidb-11829]) on a filename is never seen/used. Those are only used on series folders.

As you are shoving those all into the plex root, they will not be seen in individual log files as those are by folder. Your logs are:

..agent-update.log      (this is overridden with whatever was the last series from the root folder searched)
..agent-search.log      (this is overridden with whatever was the last series from the root folder refreshed)
_root_<root_path>.filelist.log
_root_<root_path>.scanner.log

myhe01 commented 4 years ago

@ZeroQI @EndOfLine369 Made a folder for each "series" and that seemed to fix the problem. It was kind of weird, as it recognized some individual files with AniDB IDs (like Saint Oniisan), but not others.

ZeroQI commented 4 years ago

because if there is an ep number, it is recognised... the anidbid is NOT recognised at file level, only folder level

ZeroQI / Hama.bundle

HAMA not populating metadata. #418