Gaps in Absolute Numbering

sven-7 commented 6 years ago

Hi @ZeroQI - Using the latest scanner -- related to #184 -- I have found a lot more similar cases now with the latest scanner. I'm having issues where there are gaps between absolute numbers, the scanner seems to be assuming that I'm missing the SXEX equivalent.

Scanner seems to be handling something different when absolute numbering isn't continuous.

i.e. -- with .hack, if Season 1 ends at episode 26 and absolute 27-32 are specials, and I have season 2 starting at absolute 33, it assumes I'm missing S2E1-E6. I am finding several series where this is the case now.

Below, Absolute should 33 = S2E1:

"hack SIGN [tvdb2-79099]" s01e025                         "Word Search" "Catastrophe" "25 - Catastrophe.mkv"
"hack SIGN [tvdb2-79099]" s01e026                         "Word Search" "Return" "26 - Return.mkv"
"hack SIGN [tvdb2-79099]" s02e007     (Orig: s01e033    ) "Word Search" "A Legendary Hero" "33 - A Legendary Hero.mkv"
"hack SIGN [tvdb2-79099]" s02e008     (Orig: s01e034    ) "Word Search" "Kite'S Bracelet" "34 - Kite's Bracelet.mkv"
"hack SIGN [tvdb2-79099]" s02e009     (Orig: s01e035    ) "Word Search" "Wings Phoenix" "35 - Wings of Phoenix.mkv"

A few of the series I've found this on are:

.hack (says I'm missing S2E1-E6)
The World God Only Knows (says I'm missing S1E2-E12)

Another case appears to be where there are recaps that are marked as specials, but maintain absolute numbering. Ai Techchi Muyo has this occurring for me, among others.

Logs.zip

ZeroQI commented 6 years ago

Should tvdb2 mode disregard specials and only the EPs in seasons in sequence unconditionally? Is there cases where we need to keep this behaviour?

sven-7 commented 6 years ago

I would think so. For Ai Tenchi Muyo, as an example, whether I have the specials labeled as absolute or S0xEX, it showing up incorrect. I would typically have my EPs in seasons be absolute, while specials are S0E01 (or s0xe01) etc. as specials often do not have absolute numbering. It seems when they do, that's causing some issues?

ai tenchi all absolute.zip

sven-7 commented 6 years ago

Note - the "The World God Only Knows" issue seems to have been title related (contained episode names with '1.0' and '2.0' in the titles themselves. I removed the title and it re-populated.

EndOfLine369 commented 6 years ago

Looks like the 'absoluteNumber' is not being pulled from the json entry. Its just being set manually below with absolute_number variable as well as ignoring season 0.

          # Loop through sorted episodes list
          absolute_number, tvdb_mapping = 0, {}
          for index in sorted_episodes_index_list:
            if Dict(sorted_episodes_json[index], 'airedSeason')>0: #continue
              absolute_number = absolute_number + 1
              tvdb_mapping[int(absolute_number)] = (Dict(sorted_episodes_json[index], 'airedSeason'), Dict(sorted_episodes_json[index], 'airedEpisodeNumber') if source =='tvdb2' else int(absolute_number))

Was changed in how to find the abs number here https://github.com/ZeroQI/Absolute-Series-Scanner/commit/565f3b4ce804f4bbcc2708f394668415c4edfd7a in conversion to TVDB APIv2 back in May.

sven-7 commented 6 years ago

Is that something the ASS can use without changing anything drastically or does the new TVDB APIv2 limit it?

EndOfLine369 commented 6 years ago

@sven-7, looks like its just a change in the logic. Not a limitation. @ZeroQI would have to advise why it was changed.

I have not tested myself but just pulled the old code and updated it to fit in with the json setup currently used. You can try to make the update in your scanner of the below and advise if it works again for you as previously.

FROM:
          # Loop through sorted episodes list
          absolute_number, tvdb_mapping = 0, {}
          for index in sorted_episodes_index_list:
            if Dict(sorted_episodes_json[index], 'airedSeason')>0: #continue
              absolute_number = absolute_number + 1
              tvdb_mapping[int(absolute_number)] = (Dict(sorted_episodes_json[index], 'airedSeason'), Dict(sorted_episodes_json[index], 'airedEpisodeNumber') if source =='tvdb2' else int(absolute_number))
TO:
          tvdb_mapping, ep_count, abs_manual_placement_info, number_set = {}, 0, [], False
          for index in sorted_episodes_index_list:
            if Dict(sorted_episodes_json[index], 'airedSeason')>0: #continue
              ep_count = ep_count + 1
              if not Dict(sorted_episodes_json[index], 'absoluteNumber'):
                Dict(sorted_episodes_json[index], 'absoluteNumber'), number_set = str(ep_count), True
                abs_manual_placement_info.append("s%se%s = abs %s" % (Dict(sorted_episodes_json[index], 'airedSeason'), Dict(sorted_episodes_json[index], 'airedEpisodeNumber'), Dict(sorted_episodes_json[index], 'absoluteNumber')))
              elif not number_set:  ep_count = int(Dict(sorted_episodes_json[index], 'absoluteNumber'))
              else:  Log.error("different abs number found on ep (s%se%s) after starting to manually place our own abs numbers. all episodes will be added as season 1." % (Dict(sorted_episodes_json[index], 'airedSeason'), Dict(sorted_episodes_json[index], 'airedEpisodeNumber')) );  break
          else:
            Log.debug("abs_manual_placement_worked, abs_manual_placement_info: '%s'" % (str(abs_manual_placement_info)))
            for index in sorted_episodes_index_list:
              if Dict(sorted_episodes_json[index], 'absoluteNumber'):  tvdb_mapping[int(Dict(sorted_episodes_json[index], 'absoluteNumber'))] = (int(Dict(sorted_episodes_json[index], 'airedSeason')), int(Dict(sorted_episodes_json[index], 'airedEpisodeNumber')) if source=='tvdb2' else int(Dict(sorted_episodes_json[index], 'absoluteNumber')))

ZeroQI commented 6 years ago

we cannot have movies disrupting the absolute count or even specials so will have to do manually without reading the absolute index, so there are no longer scanner numbering issues when an absolute index is missing as well... There were specials that had to be inserted back at the end of seasons that justified it initially but we have to revert to the old code. absolute tvdb numbering would then become back all season episodes in sequence.

sven-7 commented 6 years ago

@EndOfLine369 - something went wrong, but admittedly I could have messed up.

Oct 13, 2018 10:15:11.626 [3384] DEBUG - Performing a scan with 'Absolute Series Scanner EOLtest' (language: en virtual: 0).
Oct 13, 2018 10:15:11.626 [3384] DEBUG -   * Scanning H:\Test2
Oct 13, 2018 10:15:11.627 [3384] DEBUG -     * Scanning directory H:\Test2 (parent: no)
Oct 13, 2018 10:15:11.627 [3384] DEBUG - Adding subdirectory for scanner: "H:\Test2\Ai Tenchi Muyo!"
Oct 13, 2018 10:15:11.630 [3384] ERROR - No module in VideoFiles
Oct 13, 2018 10:15:11.630 [3384] ERROR - Error scanning directory .
Oct 13, 2018 10:15:11.644 [3384] ERROR - No module in Absolute Series Scanner EOLtest
Oct 13, 2018 10:15:11.644 [3384] ERROR - We got an error scanning in H:\Test2
Oct 13, 2018 10:15:11.644 [3384] DEBUG -     * Scanning directory H:\Test2\Ai Tenchi Muyo! (parent: yes)

ZeroQI commented 6 years ago

The joy of coding a scanner... Had plenty of that, had to check line per line from good version when it happens using notepad++ compare.

Car stolen this morning or yesterday after 8pm (Toyota Mr-S 2003) and building a new house so will be busy but will try to edit the source asap.

@sven-7 edit:

hack SIGN season 1 of 26 eps last one "return"
hack SIGN season 2 of 12 eps first one "A Legendary Hero"

How did you end up with "A Legendary Hero" as episode 33? it's absolute number 27.

I do not think following thetvdb absolute index is wise since it includes movies.
Specials and movies should stay in specials and the way it is now doesn't need tvdb absolut index that caused many issues in the past.
working as designed for me. if you disagree please argument what episodes you put from 27 to 32...

"hack SIGN [tvdb2-79099]" s01e026                         "Word Search" "Return" "26 - Return.mkv"
"hack SIGN [tvdb2-79099]" s02e001     (Orig: s01e027    ) "Word Search" "A Legendary Hero" "27 - A Legendary Hero.mkv"

sven-7 commented 6 years ago

@ZeroQI - sorry to hear about your car! That sucks.

Where are you seeing it's absolute 27? Per TVDB, S2E1 ("A Legendary Hero") is Absolute episode 33.

Absolute 27 for .hack is S0E1 - Intermezzo..

Another Example I have is Sword Art Online. S3E1 (Absolute 51) is showing up as the second episode in season 3. This is because Ordinal Scale, the movie, is Absolute 50.

ZeroQI commented 6 years ago

@sven-7 I live across from work, it's not too bad... It one one of the cheapest in the lot ffs...

i removed tvdb absolute numbering from the absolute index variable for few reasons:

broken for long running series as soon as 1 episode miss an absolute index, making tvdb2 mode 1 season back
movies and specials ending between seasons makes it un-natural, different time length, making 26 eps seasons have more eps in absolute numbering. if somebody have all eps in sequence, it will be right straight away while a new special insterted between seasons even released after the last season would shift everything making it wrong. That is not desired nor keep it stupid simple

The absolute numbering I support in TheTVDB is the all season episodes (except season 0) in sequence, that's it because it makes no sense to have a movie in the middle of a series absolute numbering... From that point of view, you follow strict TheTVDB but that's now how i wrote it for the reason above.

sven-7 commented 6 years ago

Okay. I guess I'll try to work out a work-around to make things work again. It looks like about 8 series are broken as a result, so I will sort something else out.

ZeroQI commented 6 years ago

Please renumber the episodes. Nobody follow the tvdb absolute index numbering in torrent/usenet releases.

sven-7 commented 6 years ago

I'm just going to move those specific series to regular 'tvdb' S1E1, etc. numbering. I don't want to have to manually keep updating newer shows like SAO as they come out and get mixed up with what is what since the Absolute numbering is off going forward.

ZeroQI commented 6 years ago

TheTVDB absolute index help time-wise to position specials but is not correct for absolute numbering.

Hack SIGN season 1 is composed of 26 eps. the first episode of season 2 is episode 27 in absolute numbering. I am sorry but I really don't see what is confusing you nor how can anybody say logically the episode after 26 is 33, or expect a movie in the middle of an absolute numbering.

Numbering down is the best way and will keep working even if new seasons and specials come out (26+12=season 3 starts at 39, etc...) which wasn't the case with the old way, as a movie could come and insert before in the middle or after a season... That is no longer a liability.

sven-7 commented 6 years ago

It’s not that I’m confused, I just disagree with that processes. I understand your logic and don’t disagree with you. It doesn’t make sense to have movies as absolute numbers, but that’s their policy if it’s canonical. What I had envisioned, in tvdb2 mode since it restarts the numbering at the beginning of each season, would for regardless if there is a gap in absolute numbering, Absolute 33 would map to S2E1 because that’s where it’s listed. The issues you listed above make sense and I understand if it causes more issues to change it.

For the argument of hack sign... With hack sign, the logical specials were OVA episodes that occur between absolute 27-32, thus that’s why they got out there. OVA often have episodes release after seasons.

For SAO, the author has said that Ordinal Scale happens between seasons 2-3. I don’t think it should get absolute numbering as a movie, but that’s what they said it would be. S3E1 is absolute 51 instead of 50.

I fixed the few series with gaps to standard tvdb mode.

ZeroQI commented 6 years ago

So in pure tvdb mode all is in sequence and make sense

I don't want people to have numbering done on eps then needing to change it because an in-between season special/movie came out...
anidb2 will translate to thetvdb season and ep number so not using thetvdb absolute numbering for that or anything else.
They should have named it viewing order and have separate absolute index like plex allowing to view chronologically...

sven-7 commented 6 years ago

Yep. In pure TVDB mode with episode numbering done via SxxExx format, it’s good.

It’s unfortunate they do it that way, but I have everything fixed up now with pure tvdb mode.

sven-7 commented 6 years ago

Hey @ZeroQI - was playing around with this and just wanted to leave an FYI.

Toriko has a weird special at the end of season 2, which is absolute 99. Having on tvdb3 caused the last episode to get pushed into a fake season 4. I did a local tvdb4 mapping file for Toriko that looks like this:

1|001|049|Season 1
2|050|098|Season 2
3|100|147|Season 3

That got the intended result and correct metadata for each episode.

ZeroQI commented 6 years ago

For me it is working as designed as the absolute numbering in tvdb2/3/4 modes is composed of season episodes without specials, just the sum of season episodes put in an absolute numbering, while tvdb absolute index could insert specials or movie and thus the absolute index becomes meaningless. Only tvdb 5 mode stricly follows the absolute index

Here is correct tvdb2/3 numbering: s00e04 toriko dbz crossover [ep 99]

1|001|049|Season 1
2|050|098|Season 2
3|099|146|Season 3

sven-7 commented 6 years ago

Ahhh, sweet. Thanks for the info. That’s helpful for checking into other series

ZeroQI commented 6 years ago

I tried to document in the readme, will make it clearer but these modes exist only in scanner and agent. The tvdb mode aupport seem to have a small following...

sven-7 commented 6 years ago

Thanks!

ZeroQI / Absolute-Series-Scanner

Gaps in Absolute Numbering #185