OP/ED misidentification issue

purposelycryptic commented 6 years ago

First of all, thank you for all your hard work on HAMA/ASS.

This is a rather odd issue I've only recently noticed, after finally figuring out how to get AniAdd to properly rename OP/EDs with ASS-compatible numbering:

For some reason, ASS will misidentify certain OP/ED files, usually identifying them as the OP/ED above or below the correct one in AniDB's internal ordering (i.e. episode C3 will be IDed as episode C2 or C4 instead).

My Files are all organized/named according to the following scheme:

Series_Name [Group_Name] [Source, Resolution]\
- Series_Name - Ep_Num [Group_Name] [Source, Resolution, Audio_Codec] [CRC_32].Ext

Where _EpNum is the AniDB Episode Number (Specials using the SP prefix, OPs and EDs the OP and ED prefix, respectively, and an [a-z] suffix when necessary, Trailer/Previews the T prefix, and non-systematic Other files the O prefix), with multi-part episodes having a ' - partX' suffix, and conjoined episodes having a '-XX' suffix, where X/XX is the respective part/episode number.

As an example, here is the file/folder layout for Owarimonogatari (2017):

Owarimonogatari (2017) [Dmon] [BD, 720p]\
- Owarimonogatari (2017) - 01 - part1 [Dmon] [BD, 720p, AAC] [A648D7B1].mkv
- Owarimonogatari (2017) - 01 - part2 [Dmon] [BD, 720p, AAC] [534EAEF0].mkv
- Owarimonogatari (2017) - 02 [Dmon] [BD, 720p, AAC] [8A34B664].mkv
- Owarimonogatari (2017) - ED1c [Dmon] [BD, 720p, AAC] [702EA02D].mkv
- Owarimonogatari (2017) - ED1d [Dmon] [BD, 720p, AAC] [930DFFFA].mkv
- Owarimonogatari (2017) - ED1e [Dmon] [BD, 720p, AAC] [66EC229F].mkv
- Owarimonogatari (2017) - OP1b [Dmon] [BD, 720p, AAC] [C7E110E0].mkv
- Owarimonogatari (2017) - OP2 [Dmon] [BD, 720p, AAC] [63B59143].mkv
- Owarimonogatari (2017) - OP3 [Dmon] [BD, 720p, AAC] [C0D815AE].mkv

And this is what the Specials look like in Plex: OwariSpecials1

Here the EDs were correctly IDed, but as for the OPs, they were instead matched like this: OwariSpecials2

OwariSpecials3

I'm not entirely sure what factor could be causing the issue, as it seems to be somewhat random - in some series, all OP/EDs are correctly matched, in some only OPs or EDs, and in some a few of both. There is obviously some common factor at work, but it eludes me.

Up until recently, all my OP/EDs were numbered according to AniDB's internal scheme (C1, C2, etc), simply because that is what AniAdd uses, and thus weren't identifiable by ASS and instead labeled as 5XX-numbered episodes. But after much script-wrangling, I finally managed to get it to name them with ASS-compatible numbering, or so I thought. If the issue is on my end, I apologize for taking up your time, and would appreciate any advice you could give to correct it.

Thanks!

ZeroQI commented 6 years ago

Please update (major scanner update done then this fix). Please report any difference

purposelycryptic commented 6 years ago

Using Owarimonogatari (2017) as my test case again, I first tried refreshing the metadata for the series, and unmatching/rematching it, both of which gave the same result as above (Even after unmatching, OP1b and OP2 were still shown in Plex as versions of the same episode, with no option to split).

So I created a new library containing only the series, which resulted in a different issue:

owarispec4

ASS identified OP1b, OP2 and OP3 as episodes 106, 107 and 108, respectively. As you can see below, the series only has four OPs on AniDB, total:

Imgur

I'm not sure what variable from AniDB you are using to match OP/EDs, but I'm guessing you're likely doing something similar to what my AniAdd renaming script does (except in reverse), that is, cycling through an array of the XMLs episode elements to match the episode number prefixes OP and ED (and their variations) to AniDB's internal episode type 3 (internal prefix C), then using the episode.title variable to match OP1/ED1 to "Opening" or "Opening 1"/"Ending" or "Ending 1" respectively, and OP/ED(n)[a-z] to "Opening (n)[a-z]"/"Ending (n)[a-z]" for the rest?

At least, that is the only way I can think of, given the limited information in AniDB's series XML files, to match them based on OP/ED number, since the internal categorization/numbering scheme doesn't distinguish between OPs and EDs at all (Off-topic, but it would be awesome if ASS could eventually identify OP/ED and "Other" files by AniDB's internal numbering, i.e. C1, C2, O1, O2, etc as well).

I don't really know what scheme you use to assign them an episode number in Plex, though (aside from the category offsets), that is, which episode becomes 101, 102, etc, since AniDB's C category (type 3) numbering frequently does not reflect the OP/ED numbering order - for Owarimonogatari (2017), for example, OP1b, OP2 and OP3 are C3, C2 and C4, respectively, and, in the latest version of ASS, were assigned to 106, 107 and 108.

If you are assigning the episode numbers by alphabetical order of the episode.title variable (in this case, OP1a, OP1b, OP2, OP3), which seems to be case, could the current situation simply be an offset issue? I.e. instead of OP1a -> 101, OP1b -> 102, etc, it is assigning them stating at 105, hence OP1b being 106, OP2 107 and OP3 108, but still associating the metadata with the correct numbering? It seems like an odd offset miss (4), but it would match the results.

Anyway, I suppose that is enough guesswork for now, hope it is in some way helpful, and, as always, thanks for all the hard work :-) I may try and take a look at the code myself, but my Python experience is essentially limited to looking at HAMA and the Shoko Metadata Agent, so I'm very much a beginner.

ZeroQI commented 6 years ago

Lines 804-814 responsible for that.

From scanner logs

No forced guid found in folder name nor id file
misc_count: {'op2': 1, 'op3': 1, 'ed1d': 1, 'ed1e': 1, 'op1b': 1, 'ed1c': 1}
"Owarimonogatari (2017)" s00e153                         "ANIDB_RX-2" "" "Owarimonogatari (2017) - ED1c [Dmon] [BD, 720p, AAC] [702EA02D].mkv"
"Owarimonogatari (2017)" s00e154                         "ANIDB_RX-2" "" "Owarimonogatari (2017) - ED1d [Dmon] [BD, 720p, AAC] [930DFFFA].mkv"
"Owarimonogatari (2017)" s00e155                         "ANIDB_RX-2" "" "Owarimonogatari (2017) - ED1e [Dmon] [BD, 720p, AAC] [66EC229F].mkv"
"Owarimonogatari (2017)" s00e106                         "ANIDB_RX-1" "" "Owarimonogatari (2017) - OP1b [Dmon] [BD, 720p, AAC] [C7E110E0].mkv"
"Owarimonogatari (2017)" s00e107                         "ANIDB_RX-1" "" "Owarimonogatari (2017) - OP2 [Dmon] [BD, 720p, AAC] [63B59143].mkv"
"Owarimonogatari (2017)" s00e108                         "ANIDB_RX-1" "" "Owarimonogatari (2017) - OP3 [Dmon] [BD, 720p, AAC] [C0D815AE].mkv"

Before they were not shifted if 1b, etc... were present before 2, 3, ... Now ending shifts it since I sorted alphabetically but put ending first need to sort per special type per ep group so sum(dict.values()) represents total offsets to date This should do the trick

misc_count: {'op2': 1, 'op3': 1, 'ed1d': 1, 'ed1e': 1, 'op1b': 1, 'ed1c': 1}
"Owarimonogatari (2017)" s00e153                         "ANIDB_RX-2" "" "Owarimonogatari (2017) - ED1c [Dmon] [BD, 720p, AAC] [702EA02D].mkv"
"Owarimonogatari (2017)" s00e154                         "ANIDB_RX-2" "" "Owarimonogatari (2017) - ED1d [Dmon] [BD, 720p, AAC] [930DFFFA].mkv"
"Owarimonogatari (2017)" s00e155                         "ANIDB_RX-2" "" "Owarimonogatari (2017) - ED1e [Dmon] [BD, 720p, AAC] [66EC229F].mkv"
"Owarimonogatari (2017)" s00e102                         "ANIDB_RX-1" "" "Owarimonogatari (2017) - OP1b [Dmon] [BD, 720p, AAC] [C7E110E0].mkv"
"Owarimonogatari (2017)" s00e103                         "ANIDB_RX-1" "" "Owarimonogatari (2017) - OP2 [Dmon] [BD, 720p, AAC] [63B59143].mkv"
"Owarimonogatari (2017)" s00e104                         "ANIDB_RX-1" "" "Owarimonogatari (2017) - OP3 [Dmon] [BD, 720p, AAC] [C0D815AE].mkv"

Please update and report and if not solved paste series specific scanner logs located in Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems_Logs If solved please close and consider donation if not already done

EndOfLine369 commented 6 years ago

Crashes. File: [Chihiro]_Amagami_SS_-_ED1_[1280x720_Blu-Ray_FLAC][970F7DCA].mkv

Error in Python: Running scanner:
Traceback (most recent call last):
  File "/share/CACHEDEV1_DATA/.qpkg/PlexMediaServer/Library/Plex Media Server/Scanners/Series/Absolute Series Scanner.py", line 810, in Scan
    offset += sum( AniDB_op[ANIDB_RX.index(rx)].values() )
KeyError: (2,)

purposelycryptic commented 6 years ago

Edit:

Just noticed from your post that your test run came up with the same results as below, i.e. 102 and 103 being set to the wrong file (or wrong entry on AniDB), likely because, like with many AniDB series, the OP entries aren't in order. /Edit

Edit 2:

The project page only just now updated to reflect the merged Pull request for me, will try the updated version. /Edit

Not solved yet, sadly, and seems to have generated other issues (Creating a new library, only half the series show up in Plex, and previously correctly IDed series misidentified).

Results this time, creating new Library with same series as example as previously (And leaving aside the other series, which were included as samples for the AniDB Series Poster Priority/Ranking Issue with HAMA), are as such:

owarispecials6

Essentially, the episode number in Plex now matches AniDB's internal C episode number and name, but is still being associated with the wrong file.

So, Imgur

...given a big Pull Request was added while I was writing this and testing things for this and the issue with HAMA, I'm not sure if any of this is still relevant or helpful, however.

I've attached the logs for the series - some of the data may be from the previous test-version of ASS, as, while I created a new Library each time, it was always named 'test1', so the logs were written to the same folder.

I've also uploaded the logs for the entire 'test1' Library (15 series) to a Google Drive folder here, in part for the other issue, but they might prove helpful here as well, at least in giving a hint as to what went wrong with the series identification.

Owarimonogatari (2017) [Dmon] [BD, 720p].agent-search.log Owarimonogatari (2017) [Dmon] [BD, 720p].agent-update.log Owarimonogatari (2017) [Dmon] [BD, 720p].filelist.log Owarimonogatari (2017) [Dmon] [BD, 720p].scanner.log

purposelycryptic commented 6 years ago

Ok, this is post-commit 306101e58727e3b542a96876fa506b0b55615cc1, using the new version of ASS:

Deleted/recreated Library, but same OP/ED issue remains, as well as the issue of half the series in the Library's folder not showing up in Plex, and one series being misidentified as its own sequel, which it had not been, using earlier revisions, although I personally started seeing similar series identification issues (mostly, series with a year and movies with 'Gekijouban' in their name no longer seeming to have the year/movie portion count as part of the name for ID weight purposes, i.e. "Shingeki no Kyojin" and Shingeki no Kyojin (2017)", a synonym, being weighted equally in this case) after commit 2ca520f1ae8241c0251e082d85911654489975ce, so that part may be nothing new.

I've uploaded the new logs for the Library, as well as the HAMA log, to a Google Drive folder here, and am attaching the logs specific to the series I have been using for testing to this post as well.

All the best!

Owarimonogatari (2017) [Dmon] [BD, 720p].filelist.log Owarimonogatari (2017) [Dmon] [BD, 720p].scanner.log

purposelycryptic commented 6 years ago

Just FYI, I rolled back to an older version of ASS (Commit be48f452bd654550031cc18dcf990e2f90b981aa), and all the missing series in my test library were detected - of course, the OP/EDs were back where we started, but it seems like something in the first of those two big recent merges caused the series detection issue (or something right before it, as I've updated ASS fairly regularly, and the first time it happened to me was after switching to that version).

Don't know if that is helpful information or not, but I'm a firm believer in there being no such thing as too much information. And now, to bed :-)

ZeroQI commented 6 years ago

@purposelycryptic That should do it. It took quite a while to spot as most times when the scanner crashes, it doesn't tell you where...

"Owarimonogatari (2017)" s00e153                         "ANIDB_RX-2" "Ending 1c" "Owarimonogatari (2017) - ED1c [Dmon] [BD, 720p, AAC] [702EA02D].mkv"
"Owarimonogatari (2017)" s00e154                         "ANIDB_RX-2" "Ending 1d" "Owarimonogatari (2017) - ED1d [Dmon] [BD, 720p, AAC] [930DFFFA].mkv"
"Owarimonogatari (2017)" s00e155                         "ANIDB_RX-2" "Ending 1e" "Owarimonogatari (2017) - ED1e [Dmon] [BD, 720p, AAC] [66EC229F].mkv"
"Owarimonogatari (2017)" s00e102                         "ANIDB_RX-1" "Opening 1b" "Owarimonogatari (2017) - OP1b [Dmon] [BD, 720p, AAC] [C7E110E0].mkv"
"Owarimonogatari (2017)" s00e103                         "ANIDB_RX-1" "Opening 2" "Owarimonogatari (2017) - OP2 [Dmon] [BD, 720p, AAC] [63B59143].mkv"
"Owarimonogatari (2017)" s00e104                         "ANIDB_RX-1" "Opening 3" "Owarimonogatari (2017) - OP3 [Dmon] [BD, 720p, AAC] [C0D815AE].mkv"

Looks better now and no longer crashes. Please test and report

purposelycryptic commented 6 years ago

It is detecting all the series in the library again, but, somehow is still not matching Owarimonogatari (2017) OP1b & OP2 correctly :-/

The log makes it look correct (ignoring the line "Forced ID in series folder: 'youtube' with id 'Dmon'", no clue as to where YouTube came from):

=============================================================================================================================================================
Library: 'Test3', root: 'Q:\Media\test2', path: 'Owarimonogatari (2017) [Dmon] [BD, 720p]', files: '9', dirs: '0', Plex scan date: 2018-08-20 14:01:19
=============================================================================================================================================================
Forced ID in series folder: 'youtube' with id 'Dmon'
misc_count: {'02': 1, '01': 2, 'op2': 1, 'ed1e': 1, 'owarimonogatari': 9, 'op3': 1, 'ed1d': 1, 'part1': 1, 'part2': 1, 'op1b': 1, 'ed1c': 1}
"Owarimonogatari (2017) [Dmon] [BD, 720p]" s01e001                         "ANIDB_RX-5" "Part1" "Owarimonogatari (2017) - 01 - part1 [Dmon] [BD, 720p, AAC] [A648D7B1].mkv"
"Owarimonogatari (2017) [Dmon] [BD, 720p]" s01e001                         "ANIDB_RX-5" "Part2" "Owarimonogatari (2017) - 01 - part2 [Dmon] [BD, 720p, AAC] [534EAEF0].mkv"
"Owarimonogatari (2017) [Dmon] [BD, 720p]" s01e002                         "Word Search" "" "Owarimonogatari (2017) - 02 [Dmon] [BD, 720p, AAC] [8A34B664].mkv"
"Owarimonogatari (2017) [Dmon] [BD, 720p]" s00e153                         "ANIDB_RX-2" "Ending 1c" "Owarimonogatari (2017) - ED1c [Dmon] [BD, 720p, AAC] [702EA02D].mkv"
"Owarimonogatari (2017) [Dmon] [BD, 720p]" s00e154                         "ANIDB_RX-2" "Ending 1d" "Owarimonogatari (2017) - ED1d [Dmon] [BD, 720p, AAC] [930DFFFA].mkv"
"Owarimonogatari (2017) [Dmon] [BD, 720p]" s00e155                         "ANIDB_RX-2" "Ending 1e" "Owarimonogatari (2017) - ED1e [Dmon] [BD, 720p, AAC] [66EC229F].mkv"
"Owarimonogatari (2017) [Dmon] [BD, 720p]" s00e102                         "ANIDB_RX-1" "Opening 1b" "Owarimonogatari (2017) - OP1b [Dmon] [BD, 720p, AAC] [C7E110E0].mkv"
"Owarimonogatari (2017) [Dmon] [BD, 720p]" s00e103                         "ANIDB_RX-1" "Opening 2" "Owarimonogatari (2017) - OP2 [Dmon] [BD, 720p, AAC] [63B59143].mkv"
"Owarimonogatari (2017) [Dmon] [BD, 720p]" s00e104                         "ANIDB_RX-1" "Opening 3" "Owarimonogatari (2017) - OP3 [Dmon] [BD, 720p, AAC] [C0D815AE].mkv"

But then in Plex, it's the same as before (The absence of episode thumbs in this screenshot is due to using a new dummy library on a different drive, rather than using symlinks as I had in previous tests):

owarispecials7

I think the issue may actually be with HAMA, or rather how ASS and HAMA interact; ASS assigns the episode number and name, but according to your ReadMe, the name is then overwritten by HAMA - at least that was my hunch, so I checked out the HAMA log, and found this:

-------------------------------------------------------------------------------------------------------------------------------------------------------------
2018-08-20 14:09:26,045 (698c) :  INFO (logkit:16) - common.PlexLog(file="", movie=False)
2018-08-20 14:09:26,048 (698c) :  INFO (logkit:16) - [!] file:       "Q:\Media\test2\Owarimonogatari (2017) [Dmon] [BD, 720p]\Owarimonogatari (2017) - OP1b [Dmon] [BD, 720p, AAC] [C7E110E0].mkv"
2018-08-20 14:09:26,048 (698c) :  INFO (logkit:16) - [!] Library access denied
2018-08-20 14:09:26,049 (698c) :  INFO (logkit:16) - [!] ASS root scanner file present: "J:\Users\C. Kloppinger\AppData\Local\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems\_Logs\_root_.scanner.log"
2018-08-20 14:09:26,049 (698c) :  INFO (logkit:16) - [!] root not found: "Q:\Media\test2\Owarimonogatari (2017) [Dmon] [BD, 720p]"
2018-08-20 14:09:26,049 (698c) :  INFO (logkit:16) - [!] root not found: "Q:\Media\test2"
2018-08-20 14:09:26,049 (698c) :  INFO (logkit:16) - [!] root not found: "Q:\Media"
2018-08-20 14:09:26,049 (698c) :  INFO (logkit:16) - [ ] library:    ""
2018-08-20 14:09:26,051 (698c) :  INFO (logkit:16) - [ ] root:       ""
2018-08-20 14:09:26,051 (698c) :  INFO (logkit:16) - [ ] path:       "_unknown_folder"
2018-08-20 14:09:26,051 (698c) :  INFO (logkit:16) - [ ] Plex root:  "J:\Users\C. Kloppinger\AppData\Local\Plex Media Server"
2018-08-20 14:09:26,052 (698c) :  INFO (logkit:16) - [ ] Log folder: "Plug-in Support\Data\com.plexapp.agents.hama\DataItems\_Logs"
2018-08-20 14:09:26,052 (698c) :  INFO (logkit:16) - [ ] Log file:   "_unknown_folder.agent-update.log"
2018-08-20 14:09:26,052 (698c) :  INFO (logkit:16) - [ ] mode:       "a"
2018-08-20 14:09:26,052 (698c) :  INFO (logkit:16) - === Update ==================================================================================================================================================
2018-08-20 14:09:26,052 (698c) :  INFO (logkit:16) - id: anidb-13033, title: None, lang: en, force: True, movie: False
2018-08-20 14:09:26,052 (698c) :  INFO (logkit:16) - -------------------------------------------------------------------------------------------------------------------------------------------------------------
2018-08-20 14:09:26,052 (698c) :  INFO (logkit:16) - AnimeLists.GetMetadata() - tvdb_numbering: False
2018-08-20 14:09:26,069 (698c) :  INFO (logkit:16) - [+] AniDBid: 13033, TVDBid: 102261, defaulttvdbseason:  0, offset:  32, name: Owarimonogatari (2017)
2018-08-20 14:09:26,069 (698c) :  INFO (logkit:16) -              -----          ------
2018-08-20 14:09:26,069 (698c) :  INFO (logkit:16) -              13033          102261
2018-08-20 14:09:26,081 (698c) :  INFO (logkit:16) - mappingList: {'TVDB': {'s0': {'13033': '32'}}, 'defaulttvdbseason': '0', 'name': 'Owarimonogatari (2017)', 'episodeoffset': '32'}
2018-08-20 14:09:26,082 (698c) :  INFO (logkit:16) - language_posters:  ['ja', 'en', 'fr', 'zh', 'sv', 'no', 'da', 'fi', 'nl', 'de', 'it', 'es', 'pl', 'hu', 'el', 'tr', 'ru', 'he', 'pt', 'cs', 'ko', 'sl', 'hr']
2018-08-20 14:09:26,082 (698c) :  INFO (logkit:16) - -------------------------------------------------------------------------------------------------------------------------------------------------------------
2018-08-20 14:09:26,082 (698c) :  INFO (logkit:16) - AniDB.GetMetadata() - AniDBid: 13033, AniDBids list: ['13033'], source: anidb
2018-08-20 14:09:26,082 (698c) :  INFO (logkit:16) - [+] 13033: []
2018-08-20 14:09:26,082 (698c) :  INFO (logkit:16) - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
2018-08-20 14:09:26,082 (698c) :  INFO (logkit:16) - AniDBid: 13033, url: http://api.anidb.net:9001/httpapi?request=anime&client=hama&clientver=1&protover=1&aid=13033
2018-08-20 14:09:26,084 (698c) :  DEBUG (logkit:13) - common.LoadFile() - file cached - CacheTime: 'Sat Aug 18 15:09:20 2018', Limit: 'Sun Aug 26 14:09:26 2018', url: 'http://api.anidb.net:9001/httpapi?request=anime&client=hama&clientver=1&protover=1&aid=13033', Filename: 'AniDB\xml\13033.xml' file_valid: 'True'
2018-08-20 14:09:26,085 (698c) :  INFO (logkit:16) - [ ] 'title': Owarimonogatari (2017), original_title: Owarimonogatari (2017), language_rank: 0
2018-08-20 14:09:26,085 (698c) :  INFO (logkit:16) - [ ] 'originally_available_at': '2017-08-12'
2018-08-20 14:09:26,085 (698c) :  INFO (logkit:16) - [ ] role: Araragi Koyomi      , name: Kamiya Hiroshi      , photo: 28404.jpg
2018-08-20 14:09:26,086 (698c) :  INFO (logkit:16) - [ ] role: Senjougahara Hitagi , name: Saitou Chiwa        , photo: 31036.jpg
2018-08-20 14:09:26,088 (698c) :  INFO (logkit:16) - [ ] role: Hachikuji Mayoi     , name: Katou Emiri         , photo: 28405.jpg
2018-08-20 14:09:26,088 (698c) :  INFO (logkit:16) - [ ] role: Oshino Ougi         , name: Mizuhashi Kaori     , photo: 199133.jpg
2018-08-20 14:09:26,088 (698c) :  INFO (logkit:16) - [ ] 'rating': '8.49'
2018-08-20 14:09:26,091 (698c) :  INFO (logkit:16) - [ ] 'genre' (2/5 above 100 weight): ['Ecchi', 'Harem']
2018-08-20 14:09:26,095 (698c) :  INFO (logkit:16) - [ ] Roles (creators tag): {'producers': ['Tou Fuyashi', 'Shinbou Akiyuki'], 'directors': ['Itamura Tomoyuki', 'Miyai Kana', 'Watanabe Akio', 'Iwasaki Taisuke', 'Shinbou Akiyuki'], 'writers': ['Nishio Ishin']}
2018-08-20 14:09:26,095 (698c) :  INFO (logkit:16) - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
2018-08-20 14:09:26,095 (698c) :  INFO (logkit:16) - [ ] s0e201 => s0e201 air_date: 
2018-08-20 14:09:26,095 (698c) :  INFO (logkit:16) - [ ] s1e  2 => s1e  2 air_date: 2017-08-13
2018-08-20 14:09:26,095 (698c) :  INFO (logkit:16) - [ ] s1e  1 => s1e  1 air_date: 2017-08-12
2018-08-20 14:09:26,096 (698c) :  INFO (logkit:16) - [ ] s0e151 => s0e151 air_date: 2017-08-12
2018-08-20 14:09:26,098 (698c) :  INFO (logkit:16) - [X] s0e102 => s0e102 air_date: 2017-08-13 language_rank: 1, title: "Opening 2"
2018-08-20 14:09:26,098 (698c) :  INFO (logkit:16) - [ ] s0e101 => s0e101 air_date: 2017-08-12
2018-08-20 14:09:26,098 (698c) :  INFO (logkit:16) - [ ] s0e401 => s0e401 air_date: 2017-08-12
2018-08-20 14:09:26,098 (698c) :  INFO (logkit:16) - [ ] s0e404 => s0e404 air_date: 2017-08-12
2018-08-20 14:09:26,099 (698c) :  INFO (logkit:16) - [ ] s0e405 => s0e405 air_date: 2017-11-29
2018-08-20 14:09:26,099 (698c) :  INFO (logkit:16) - [ ] s0e406 => s0e406 air_date: 2017-11-29
2018-08-20 14:09:26,101 (698c) :  INFO (logkit:16) - [ ] s0e402 => s0e402 air_date: 2017-10-25
2018-08-20 14:09:26,101 (698c) :  INFO (logkit:16) - [ ] s0e403 => s0e403 air_date: 2017-10-25
2018-08-20 14:09:26,101 (698c) :  INFO (logkit:16) - [ ] s0e407 => s0e407 air_date: 2017-12-27
2018-08-20 14:09:26,101 (698c) :  INFO (logkit:16) - [ ] s0e408 => s0e408 air_date: 2017-12-27
2018-08-20 14:09:26,101 (698c) :  INFO (logkit:16) - [ ] s0e409 => s0e409 air_date: 2017-12-27
2018-08-20 14:09:26,102 (698c) :  INFO (logkit:16) - [ ] s0e104 => s0e104 air_date: 2017-11-29
2018-08-20 14:09:26,102 (698c) :  INFO (logkit:16) - [ ] s0e155 => s0e155 air_date: 2017-12-27
2018-08-20 14:09:26,104 (698c) :  INFO (logkit:16) - [ ] s0e154 => s0e154 air_date: 2017-11-29
2018-08-20 14:09:26,104 (698c) :  INFO (logkit:16) - [ ] s0e103 => s0e103 air_date: 2017-10-25
2018-08-20 14:09:26,105 (698c) :  INFO (logkit:16) - [ ] s0e156 => s0e156 air_date: 2017-12-27
2018-08-20 14:09:26,105 (698c) :  INFO (logkit:16) - [ ] s0e153 => s0e153 air_date: 2017-10-25
2018-08-20 14:09:26,105 (698c) :  INFO (logkit:16) - [ ] s0e152 => s0e152 air_date: 2017-08-13
2018-08-20 14:09:26,105 (698c) :  INFO (logkit:16) - Season: 0 Episodes: [] not on disk
2018-08-20 14:09:26,107 (698c) :  INFO (logkit:16) - Season: 1 Episodes: ['1', '2'] not on disk
2018-08-20 14:09:26,107 (698c) :  INFO (logkit:16) - ANNid: '19397', MALid: '35247', xml loaded: 'True'
2018-08-20 14:09:26,107 (698c) :  INFO (logkit:16) - -------------------------------------------------------------------------------------------------------------------------------------------------------------

Despite being for the file "Owarimonogatari (2017) - OP1b [Dmon] [BD, 720p, AAC] [C7E110E0].mkv", this line shows HAMA associating s0e102 with "Opening 2":

2018-08-20 14:09:26,098 (698c) :  INFO (logkit:16) - [X] s0e102 => s0e102 air_date: 2017-08-13 language_rank: 1, title: "Opening 2"

So, that would explain why changing things in ASS hasn't had the expected effect - there appears to be a disconnect between HAMA and ASS. Which side is responsible for the resulting issue, I'm not sure, since HAMA depends on ASS for the episode number, but HAMA discards the episode name and assigns it metadata, and I'm not sure how HAMA associates the Specials episode numbers with actual metadata, given those episode numbers are unique to ASS/HAMA (reverse-translating them to AniDB episode numbers? In which case, changes to how ASS numbers them would require changes to HAMA as well...)

But that seems to solve part of the mystery, at least - still getting the same series-level mismatches, seems to be due to precise name matches being given the same matching score as ones with a year/period/apostrophe at the end, which then appear above them on the list (see images below for examples from sample library), but that is a separate issue.

oreimomismatch

snkmismatch

ZeroQI commented 6 years ago

I fixed the obvious, the specials numbering in ASS, which now look grand' BUT relies on files present so if missing 1a, 1b, 1c, OP2 is 102... I need to download anidb xml from ASS if the anidb id is present which i wanted to avoid... Need to match logic on hama side now

purposelycryptic commented 6 years ago

I'm actually seriously impressed ASS has done such a good job so far in almost every case without referencing a series' AniDB XML - it's essentially like working blind with a 99% success rate.

But, given a the majority of releases don't contain a full set of OP/EDs (or, rather, the OP/ED list on AniDB tries to cover every variation from every format the series was released in, so different media-type groups simply don't have them to release), using the XML is probably the only way to achieve full accuracy. I don't suppose it's as simple as using the same AniDB XML storage folder for both ASS and HAMA, so that, for either one, if a sufficiently recent version is present locally, it can use that vs. downloading a fresh copy... Or rather, even if it is, the larger part is then integrating the use of the XML data into ASS.

Knowing that definitely explains a lot about the code and how it is structured, though - I spent a couple hours yesterday reading over it (HAMA/ASS have been my first major exposure to Python outside of minor script editing), and somehow completely missed the fact that the XML is never fetched/referenced.

Up until a few months ago, I had been using Emby/Mediabrowser for almost a decade, and Emby Server uses it's own scanner, with plugins only able to serve as agents, so I never got a real look at the scanner side; for what it's worth, HAMA was essentially the reason I switched to Plex, despite preferring Emby over Plex in almost every other respect.

ZeroQI commented 6 years ago

AniDB XML example: http://api.anidb.net:9001/httpapi?request=anime&client=hama&clientver=1&protover=1&aid=13033

epno type="3" = C1, title = Opening 1a
epno type="3" = C2, title = Opening 2
epno type="3" = C3, title = Opening 1b
epno type="3" = C4, title = Opening 3
epno type="3" = C5, title = Ending 1a
epno type="3" = C6, title = Ending 1b
epno type="3" = C7, title = Ending 1c
epno type="3" = C8, title = Ending 1d
epno type="3" = C9, title = Ending 1e
epno type="3" = C10, title = Ending 1f

Solutions:

leave as is as works for most files (Hama still needs updating)
ask to rename files from OP1b to s00e102 for example
anidb id in series name + anidb xml downloaded from here (otherwise new files mapped wrong on the first scan as not downloaded if using hama cache) with numbering with a letter not sequential (listing op1a op2 op1b order for example)

Still researching how to solve efficiently

I like my solution without AniDB XML as it covers most issues
XML assisted numbering will need the anidb id forced which wasn't present in this case

Detective Conan currently has 41 openings and 49 endings.

Edit: I cannot achieve 100% with no anidbid forced and numerous OP/ED with lettered versions with normal methods as i cannot determine if ep 1b is ep 2 or 3 in plex without, especially since they are not in sequence in the xml ...

I could take the biggest steps (a-h=8?) and spread every episode away by this step but op1a = 1 and op2 = 9 in this case

i could make openings 1-50 and 1b 51, 2b 52, 1c 101, etc... as a variant...

both ideas would work needing no xml building the list, the second option looks more natural though but bad numbering for second versions and after of the op/ed... It could load the xml and no need for episode ranges anymore...

i need to load the xml it seems but you have to force the anidb id when opening/Endings are wrong...

OR i leave as is and adjust in hama...

ZeroQI commented 6 years ago

@purposelycryptic re-did the op/ed part.

Issue was, if op1a and op1b not present, op2 is ep 2 (+100/150)... Also, in your case, op1b and 2 are swapped and not in order in xml so can't use the Cxx number... In regards to the mapping, I feel the numbering is right but hama should swap the meta If the numbering is off compared to what it should, force the anidb id.

Please test and report, but this is for the scanner part only. Once all is confirmed ok will change the agent part if needed.

https://github.com/ZeroQI/Absolute-Series-Scanner/blob/master/Scanners/Series/Absolute%20Series%20Scanner.py

New house with mortgage starting any day so time is scarce. I am glad the agent is good enough for you, it is way more complex than other anime agents but tried to make it the best and glad it is used.

EndOfLine369 commented 6 years ago

@ZeroQI, doesn't this new dl of anidb info urlopen(ANIDB_HTTP_API_URL+id, context=SSL_CONTEXT).read() look like a prime candidate of an easy AniDB ban? EX: I have 110 series with op/ed episodes and I list all my folders with an anidb id

ZeroQI commented 6 years ago

@EndOfLine369 it only download if the anidb id is present and there are openings and wait 6s afterwards.

EndOfLine369 commented 6 years ago

ah, just rechecked. found the time.sleep(6) this time

EndOfLine369 commented 6 years ago

Did a test. Fixed test to remove: FutureWarning: The behavior of this method will change in future versions. Use specific 'len(elem)' or 'elem is not None' test instead.

And corrected the check to pull when 'anidb_xml' is None and NOT when it 'is not None':


FROM:
if source.startswith('anidb') and id and not anidb_xml and rx in ANIDB_RX[1:3]:  #2nd and 3rd rx
TO:
if source.startswith('anidb') and id and anidb_xml is None and rx in ANIDB_RX[1:3]:  #2nd and 3rd rx

ZeroQI commented 6 years ago

What is afterwards that test populate anidb-xml so it is downloaded only once in the file loop if needed Your test is True if var was None and so was mine but you removed the warning and clearer, so better...

EndOfLine369 commented 6 years ago

Looks like there are still issues in the test I have of openings. https://anidb.net/perl-bin/animedb.pl?show=anime&aid=7506 There is a 'OP1d' in AniDB (that I don't have) that is not being reserved as e104.

AniDB URL: http://api.anidb.net:9001/httpapi?request=anime&client=hama&clientver=1&protover=1&aid=7506, length: 65620, AniDB_op: {1: {1: 3, 2: 4}}
"Amagami SS -lj -sU -e25 -a7506 [anidb-7506] -n11474" s00e151                         "ANIDB_RX-2" "Ending 1" "[Chihiro]_Amagami_SS_-_ED1_[1280x720_Blu-Ray_FLAC][970F7DCA].mkv"
"Amagami SS -lj -sU -e25 -a7506 [anidb-7506] -n11474" s00e152                         "ANIDB_RX-2" "Ending 2" "[Chihiro]_Amagami_SS_-_ED2_[1280x720_Blu-Ray_FLAC][438325D1].mkv"
"Amagami SS -lj -sU -e25 -a7506 [anidb-7506] -n11474" s00e153                         "ANIDB_RX-2" "Ending 3" "[Chihiro]_Amagami_SS_-_ED3_[1280x720_Blu-Ray_FLAC][6A1BF043].mkv"
"Amagami SS -lj -sU -e25 -a7506 [anidb-7506] -n11474" s00e154                         "ANIDB_RX-2" "Ending 4" "[Chihiro]_Amagami_SS_-_ED4_[1280x720_Blu-Ray_FLAC][4F0C2A86].mkv"
"Amagami SS -lj -sU -e25 -a7506 [anidb-7506] -n11474" s00e155                         "ANIDB_RX-2" "Ending 5" "[Chihiro]_Amagami_SS_-_ED5_[1280x720_Blu-Ray_FLAC][A71D8850].mkv"
"Amagami SS -lj -sU -e25 -a7506 [anidb-7506] -n11474" s00e156                         "ANIDB_RX-2" "Ending 6" "[Chihiro]_Amagami_SS_-_ED6_[1280x720_Blu-Ray_FLAC][89893E4D].mkv"
"Amagami SS -lj -sU -e25 -a7506 [anidb-7506] -n11474" s00e157                         "ANIDB_RX-2" "Ending 7" "[Chihiro]_Amagami_SS_-_ED7_[1280x720_Blu-Ray_FLAC][56FF6FAB].mkv"
"Amagami SS -lj -sU -e25 -a7506 [anidb-7506] -n11474" s00e158                         "ANIDB_RX-2" "Ending 8" "[Chihiro]_Amagami_SS_-_ED8_[1280x720_Blu-Ray_FLAC][CD2A011D].mkv"
"Amagami SS -lj -sU -e25 -a7506 [anidb-7506] -n11474" s00e101                         "ANIDB_RX-1" "Opening 1a" "[Chihiro]_Amagami_SS_-_OP1a_[1280x720_Blu-Ray_FLAC][90B8637C].mkv"
"Amagami SS -lj -sU -e25 -a7506 [anidb-7506] -n11474" s00e102                         "ANIDB_RX-1" "Opening 1b" "[Chihiro]_Amagami_SS_-_OP1b_[1280x720_Blu-Ray_FLAC][1CB96475].mkv"
"Amagami SS -lj -sU -e25 -a7506 [anidb-7506] -n11474" s00e103                         "ANIDB_RX-1" "Opening 1c" "[Chihiro]_Amagami_SS_-_OP1c_[1280x720_Blu-Ray_FLAC][AD82C54E].mkv"
"Amagami SS -lj -sU -e25 -a7506 [anidb-7506] -n11474" s00e104                         "ANIDB_RX-1" "Opening 2a" "[Chihiro]_Amagami_SS_-_OP2a_[1280x720_Blu-Ray_FLAC][4B969037].mkv"
"Amagami SS -lj -sU -e25 -a7506 [anidb-7506] -n11474" s00e105                         "ANIDB_RX-1" "Opening 2b" "[Chihiro]_Amagami_SS_-_OP2b_[1280x720_Blu-Ray_FLAC][DBDEB2E4].mkv"
"Amagami SS -lj -sU -e25 -a7506 [anidb-7506] -n11474" s00e106                         "ANIDB_RX-1" "Opening 2c" "[Chihiro]_Amagami_SS_-_OP2c_[1280x720_Blu-Ray_FLAC][5CE1AD00].mkv"
"Amagami SS -lj -sU -e25 -a7506 [anidb-7506] -n11474" s00e107                         "ANIDB_RX-1" "Opening 2d" "[Chihiro]_Amagami_SS_-_OP2d_[1280x720_Blu-Ray_FLAC][13BEC602].mkv"
"Amagami SS -lj -sU -e25 -a7506 [anidb-7506] -n11474" s00e108                         "ANIDB_RX-1" "Opening 2e" "[Chihiro]_Amagami_SS_-_OP2e_[1280x720_Blu-Ray_FLAC][6866FEAB].mkv"

ZeroQI commented 6 years ago

Look like it is in manual mode and doesn't use the mappings First 1: is openings and 1:3 is ep 1 +3 (1d last op mapped) so cumulative offset should be 3 not 2 For op2x and the mapping were overwritten or the calculation array is wrong...

EndOfLine369 commented 6 years ago

ep type offset: 100, ep: 1, offset: 0, cumulative_offset: 0, final ep number: 101
ep type offset: 100, ep: 1, offset: 1, cumulative_offset: 0, final ep number: 102
ep type offset: 100, ep: 1, offset: 2, cumulative_offset: 0, final ep number: 103
ep type offset: 100, ep: 2, offset: 0, cumulative_offset: 2, final ep number: 104
ep type offset: 100, ep: 2, offset: 1, cumulative_offset: 2, final ep number: 105
ep type offset: 100, ep: 2, offset: 2, cumulative_offset: 2, final ep number: 106
ep type offset: 100, ep: 2, offset: 3, cumulative_offset: 2, final ep number: 107
ep type offset: 100, ep: 2, offset: 4, cumulative_offset: 2, final ep number: 108

Looks like an issue in setting 'cumulative_offset' as should = 3 for ep2 openings

EndOfLine369 commented 6 years ago

Think I have narrowed it down to this piece in issue.

          if anidb_xml is not None:
            if ANIDB_RX.index(rx) in AniDB_op:  AniDB_op [ ANIDB_RX.index(rx) ]   [ ep ] = offset # {101: 0 for op1a / 152: for ed2b} and the distance between a and the version we have hereep, offset                         = str( int( ep[:-1] ) ), offset + sum( AniDB_op.values() )                             # "if xxx isdigit() else 1" implied since OP1a for example... # get the offset (100, 150, 200, 300, 400) + the sum of all the mini offset caused by letter version (1b, 2b, 3c = 4 mini offset)
            else:                               AniDB_op [ ANIDB_RX.index(rx) ] = { ep:    offset }
          Log.info("AniDB_op: {}".format(AniDB_op))

LOG:

AniDB URL: http://api.anidb.net:9001/httpapi?request=anime&client=hama&clientver=1&protover=1&aid=7506, length: 65620, AniDB_op: {1: {1: 3, 2: 4}}
AniDB_op: {1: {1: 3, 2: 4}, 2: {'1': 0}}
ep type offset: 150, ep: 1, offset: 0, cumulative_offset: 0, final ep number: 151
AniDB_op: {1: {1: 3, 2: 4}, 2: {'1': 0, '2': 0}}
ep type offset: 150, ep: 2, offset: 0, cumulative_offset: 0, final ep number: 152
AniDB_op: {1: {1: 3, 2: 4}, 2: {'1': 0, '3': 0, '2': 0}}
ep type offset: 150, ep: 3, offset: 0, cumulative_offset: 0, final ep number: 153
AniDB_op: {1: {1: 3, 2: 4}, 2: {'1': 0, '3': 0, '2': 0, '4': 0}}
ep type offset: 150, ep: 4, offset: 0, cumulative_offset: 0, final ep number: 154
AniDB_op: {1: {1: 3, 2: 4}, 2: {'1': 0, '3': 0, '2': 0, '5': 0, '4': 0}}
ep type offset: 150, ep: 5, offset: 0, cumulative_offset: 0, final ep number: 155
AniDB_op: {1: {1: 3, 2: 4}, 2: {'1': 0, '3': 0, '2': 0, '5': 0, '4': 0, '6': 0}}
ep type offset: 150, ep: 6, offset: 0, cumulative_offset: 0, final ep number: 156
AniDB_op: {1: {1: 3, 2: 4}, 2: {'1': 0, '3': 0, '2': 0, '5': 0, '4': 0, '7': 0, '6': 0}}
ep type offset: 150, ep: 7, offset: 0, cumulative_offset: 0, final ep number: 157
AniDB_op: {1: {1: 3, 2: 4}, 2: {'1': 0, '3': 0, '2': 0, '5': 0, '4': 0, '7': 0, '6': 0, '8': 0}}
ep type offset: 150, ep: 8, offset: 0, cumulative_offset: 0, final ep number: 158
AniDB_op: {1: {1: 0, 2: 4}, 2: {'1': 0, '3': 0, '2': 0, '5': 0, '4': 0, '7': 0, '6': 0, '8': 0}}
ep type offset: 100, ep: 1, offset: 0, cumulative_offset: 0, final ep number: 101
AniDB_op: {1: {1: 1, 2: 4}, 2: {'1': 0, '3': 0, '2': 0, '5': 0, '4': 0, '7': 0, '6': 0, '8': 0}}
ep type offset: 100, ep: 1, offset: 1, cumulative_offset: 0, final ep number: 102
AniDB_op: {1: {1: 2, 2: 4}, 2: {'1': 0, '3': 0, '2': 0, '5': 0, '4': 0, '7': 0, '6': 0, '8': 0}}
ep type offset: 100, ep: 1, offset: 2, cumulative_offset: 0, final ep number: 103
AniDB_op: {1: {1: 2, 2: 0}, 2: {'1': 0, '3': 0, '2': 0, '5': 0, '4': 0, '7': 0, '6': 0, '8': 0}}
ep type offset: 100, ep: 2, offset: 0, cumulative_offset: 2, final ep number: 104
AniDB_op: {1: {1: 2, 2: 1}, 2: {'1': 0, '3': 0, '2': 0, '5': 0, '4': 0, '7': 0, '6': 0, '8': 0}}
ep type offset: 100, ep: 2, offset: 1, cumulative_offset: 2, final ep number: 105
AniDB_op: {1: {1: 2, 2: 2}, 2: {'1': 0, '3': 0, '2': 0, '5': 0, '4': 0, '7': 0, '6': 0, '8': 0}}
ep type offset: 100, ep: 2, offset: 2, cumulative_offset: 2, final ep number: 106
AniDB_op: {1: {1: 2, 2: 3}, 2: {'1': 0, '3': 0, '2': 0, '5': 0, '4': 0, '7': 0, '6': 0, '8': 0}}
ep type offset: 100, ep: 2, offset: 3, cumulative_offset: 2, final ep number: 107
AniDB_op: {1: {1: 2, 2: 4}, 2: {'1': 0, '3': 0, '2': 0, '5': 0, '4': 0, '7': 0, '6': 0, '8': 0}}
ep type offset: 100, ep: 2, offset: 4, cumulative_offset: 2, final ep number: 108

ZeroQI commented 6 years ago

@EndOfLine369 Two behaviours:

There is an AniDB id and opening and the xml is not yet loaded, load it then pause 6s then build AniDB_op once and tested with "anidb_xml is None".
Build manually from natural sorted titles if "anidb_xml is None". This cannot be applied to already build AniDB_op as there is no safety to avoid downgrade back offset value for an episode, if 1d is not present, the numering will not be shifted as it should.

i then add the offset for all prior episodes (1a=0, ab=1, 1c=2...) then the offset for that episode then the epis

I believe you are better than me at python, but your pull request changed dramatically the behaviour: " - - Good: "not anidb_xml" replaced with "anidb_xml is None"

Bad: "not anidb_xml"replaced with "anidb_xml is not None:". That is wrong. i didn't wrote it that way, and make the manual list over-write existing one and not update it for manual estimation when anidb id not present...

EndOfLine369 commented 6 years ago

Episodes need at least a default offset entry of '0' otherwise a KeyError will be hit.
-------------------------
    cumulative_offset = sum( [ AniDB_op [ ANIDB_RX.index(rx) ][x] for x in Dict(AniDB_op, ANIDB_RX.index(rx), default={0:0}) if x<ep ] )
KeyError: (2,)
-------------------------
If a regex other than an op/en is matched, the 'ep_' variable is not known so it need to be updating the 'ep' variable instead of creating a new one.
-------------------------
    add_episode_into_plex(media, file, root, path, show, int(season), int(ep_), title, year, int(ep2) if ep2 and ep2.isdigit() else int(ep_), rx, tvdb_mapping, unknown_series_length, offset_season, offset_episode, mappingList)
UnboundLocalError: local variable 'ep_' referenced before assignment
-------------------------

EndOfLine369 commented 6 years ago

I see you undid the typo. Fixed crashes from that code as noted above. Strange that GitHub closed this on a commit. Never seen that happen.

dgw commented 6 years ago

@EndOfLine369 "fixes" from the end of the top (summary) line combined with the issue number at the beginning of the description, which is unintuitive behavior at best.

Closing an issue referenced by number with "fixes", "closes", "resolves" etc. before it is documented GitHub behavior, but it probably shouldn't happen in this special case. I'd say it's worth writing to GitHub support.

EndOfLine369 commented 6 years ago

Thanks @dgw for the explanation on the unexpected issue closure

dgw commented 6 years ago

Glad to help! I wrote GitHub support about it, not that I expect anything to change.

EndOfLine369 commented 6 years ago

FYI, @ZeroQI, Another crashe found

    cumulative_offset = sum( [ AniDB_op [ ANIDB_RX.index(rx) ][x] for x in Dict(AniDB_op, ANIDB_RX.index(rx), default={0:0}) if x<ep ] )
KeyError: (0,)

Due to if rx in ANIDB_RX[:-2]: being used but only ANIDB_RX[1:3] being used for AniDB_op population. Looks like the fix I put in earlier for the KeyError: (2,) crash needs a diff fix to handle any non-entry in AniDB_op for any of the ANIDB_RX[:-2] indexes. Checking to see what is needed to fix.

EndOfLine369 commented 6 years ago

Updated code to confirm 'AniDB_op's contents on the 'cumulative_offset=' line. seems to have fixed the issue.

ZeroQI commented 6 years ago

@purposelycryptic latest version is stable for you ?

ZeroQI commented 6 years ago

@purposelycryptic latest version is stable for you ? Will close without reply

ZeroQI / Absolute-Series-Scanner

OP/ED misidentification issue #172

Edit:

Edit 2: