One Piece tvdb4 S00 Movies and Specials

DarthKnight12 commented 5 years ago

Hi,

I don't know what I am doing wrong, everytime I add a file to my specials folder of One Piece, Plex always matches everything correctly (including Metadata and all) and after a while Specials will no longer be listed and they just disappear.

Heres a screenshot of the folder structure and naming Im using

example

everything else usually works but this section is really starting to be difficult for me

For Clarification: For the entire series (all episodes) im using the absolute numbering scheme Exxx

Update 1:

Log

One Piece [tvdb4-81797].scanner.log

Update 2:

Filelist Log

One Piece [tvdb4-81797].filelist.log

ZeroQI commented 5 years ago

Need scanner logs as i am unsure

DarthKnight12 commented 5 years ago

where can i find that ?

ZeroQI commented 5 years ago

Scanner readme should have location. Currently in Hama agent data folders.

DarthKnight12 commented 5 years ago

would you need filelist.log aswell ?

ZeroQI commented 5 years ago

Please

DarthKnight12 commented 5 years ago

added filelist log

DarthKnight12 commented 5 years ago

Dont know why this is happening especially since it works (same naming and everything) with other shows such as dragon ball super

DarthKnight12 commented 5 years ago

No indication yet as to why this keeps happening only with one piece ?

ZeroQI commented 5 years ago

If it scans right, then dissappear, then the scanner worked at least once, seem like plex database corruption No S00 in logs (agent and filelist)

DarthKnight12 commented 5 years ago

how can i fix it then ? plex dance ?

ZeroQI commented 5 years ago

Either scanner issue (use legacy scanner but since scan properly first time doubt it, you could scan manually twice to prove the point) or Plex corruption (Plex dance)

DarthKnight12 commented 5 years ago

It happened again, checked an s00 has disappeared

ZeroQI commented 5 years ago

Check Plex scanner log. After Plex dance and it scans and gets added (capture first copy of filelist and my scanner logs), does it dissapear on the next scan? (capture plex scanner logs+ my scanner logs)

DarthKnight12 commented 5 years ago

ill do that later, initially i thought it has to do with the naming im using but the scanner does pick everything up

DarthKnight12 commented 5 years ago

so this is just after adding it again (i removed it and readded it, scanned it, and now its back)

One Piece [tvdb4-81797].filelist.log One Piece [tvdb4-81797].scanner.log

EndOfLine369 commented 5 years ago

@ZeroQI, First log has 'Manual scan'. This log has 'Plex scan'. And the last log has multiple rescans probably as plex setting to scan on changes seen instead of schedule so it would only call a scan on a specific folder and not at a library level.

@DarthKnight12, I assume your using the latest version of the scanner?

If you run a requested scan on the library post your dance addition, I'm assuming it will disappear at that time? If that is the case, can you try with this older version of the scanner Absolute Series Scanner.py from before the last commit. And repeat to see if it again disappears at requested scan run?

DarthKnight12 commented 5 years ago

@EndOfLine369 yes i am using the latest version, but it happened on the previous version aswell and i thought the latest one fixed it but after two days or so s00 disappeared again

DarthKnight12 commented 5 years ago

and it is gone again, here are the logs

One Piece [tvdb4-81797].filelist.log One Piece [tvdb4-81797].scanner.log

DarthKnight12 commented 5 years ago

i think i found the error, according to http://thexem.de/xem/show/4468 s00 only has one entry e25 all the other ones are not mapped, is there any work around to include them ?

EndOfLine369 commented 5 years ago

It looks like the root of the issue is your bad folder structure. And is a larger issue then just missing S0 entries. S0 is good at a good path One Piece [tvdb4-81797]/S00 - Specials/<files>

But in this last log, your 1 too many folders deep in all those listed. One Piece [tvdb4-81797]/S05 - Drum Island/[5.1] Arc Drum Island/<files> [5.1] Arc Drum Island/ should not be there. As it is, its changing to seeing it as a grouping folder.

That is why it is seeing the plex tv series as "Arc Drum Island" for this log snip.

=============================================================================================================================================================
Library: 'no valid X-Plex-Token.id', root: '/share/CACHEDEV5_DATA/TV_Series_Anime', path: 'One Piece [tvdb4-81797]/S05 - Drum Island/[5.1] Arc Drum Island', files: '14', dirs: '0'
Manual scan start: 2019-03-09 08:05:14,999278
=============================================================================================================================================================
No forced id found in series folder name nor id file
-------------------------------------------------------------------------------------------------------------------------------------------------------------
misc_count: {"wapol's": 2, 'resolve!': 1, 'corps': 1, 'dream!': 1, 'lives!': 1, 'tyrant': 1, 'e090': 1, 'e091': 1, 'faith': 1, 'reindeer!': 1, 'cherry': 2, 'to': 1, 'going': 1, 'plate': 1, "outcast's": 1, 'e089': 1, 'e088': 1, 'e083': 1, 'e082': 1, 'e081': 1, 'e080': 1, 'e087': 1, 'e086': 1, 'e085': 1, 'e084': 1, 'miracle': 1, 'where': 1, 'stars!': 1, "chopper's": 2, 'witch!': 1, 'are': 1, 'beyond': 1, 'out': 1, 'blue': 1, 'secret!': 1, 'forever!': 1, 'island!': 2, 'falling': 1, 'power': 1, 'wapol!': 1, 'sea!': 1, 'hiriluk': 1, 'on': 2, 'drum': 3, 'carried': 1, 'of': 2, "kingdom's": 1, "hiriluk's": 2, 'climb': 1, 'flies': 1, 'zoan': 1, 'seven': 1, 'one': 14, 'blossoms!': 1, 'devil': 1, 'fruit!': 2, 'rockies!': 2, 'fight': 1, 'blossoms': 1, 'type': 1, 'happy': 1, 'form': 1, 'that': 1, 'on!': 1, "dalton's": 1, 'land!': 1, 'flag': 1, 'doctors': 1, "i'm": 1, 'raid!': 1, 'crew!': 1, 'will': 1, 'transformation!': 1, 'piece': 14, 'called': 1, 'and': 2, 'e078': 1, 'e079': 1, 'an': 2, "nami's": 1, 'in': 2, 'doctor': 1, 'when': 1, 'snow': 2, 'tin': 2, 'sick': 1, 'you': 1, 'gets': 1, 'goodbye': 1, 'quack!': 1, 'island': 2, 'munch': 2, 'adventure': 1, 'nameless': 1, 'a': 2, 'land': 1, 'rule': 1, 'nosed': 1, 'without': 1, 'ends!': 1, 'the': 14}
misc_words: ['piece', 'one', 'the']
-------------------------------------------------------------------------------------------------------------------------------------------------------------
"Arc Drum Island" s05e078                         "Word Search" "Nami'S Sick Beyond Snow Falling On The Stars!" "One Piece - E078 - Nami`s Sick Beyond the Snow Falling on the Stars!.mkv"
"Arc Drum Island" s05e079                         "Word Search" "A Raid! Tin Tyrant And Tin Plate Wapol!" "One Piece - E079 - A Raid! The Tin Tyrant and Tin Plate Wapol!.mkv"

DarthKnight12 commented 5 years ago

Don´t Know why but im not having any issues with the folder structure for the entire series (besides s00 that is). Everything else is matching properly straight out of the gate

ZeroQI commented 5 years ago

"[5.1] Arc Drum Island" should be seen as "Transparent folder" as it matches cic(r'^(Saga|(Story )?Ar[kc])')] since square brackets are removed and not trigger grouping folders

"No forced id found in series folder name nor id file" is not consistent with path: 'One Piece [tvdb4-81797]/S05 - Drum Island/[5.1] Arc Drum Island'

We need to display when folders (season/grouping) are removed to follow the scanner logic

DarthKnight12 commented 5 years ago

@zeroqi please explain to me what you are saying because I don't understand when you say "...we need to display when folders are removed.."

ZeroQI commented 5 years ago

@DarthKnight12 apologies, this part was more addressed to @EndOfLine369 I am saying it should work with your naming as you use "arc xxx" so shouldn't be seen as grouping folder, or we can make it work (maybe the brackets make it not match "transparent folders" regex) EndOfLine369 or me have to improve logging as:

no id forced reported whereas tvdb4 is used
doesn't indicate season and transparent folders as they are processed

DarthKnight12 commented 5 years ago

Ok now that I understand :) back to the xem link, could the sudden disappearance of s00 be due to the fact that not all of s00 EPS are mapped (only EP 25 is seen when you look at it)

EndOfLine369 commented 5 years ago

"[5.1] Arc Drum Island" should be seen as "Transparent folder" as it matches cic(r'^(Saga|(Story )?Ar[kc])')] since square brackets are removed and not trigger grouping folders

"No forced id found in series folder name nor id file" is not consistent with path: 'One Piece [tvdb4-81797]/S05 - Drum Island/[5.1] Arc Drum Island'

We need to display when folders (season/grouping) are removed to follow the scanner logic

@ZeroQI, I don't think so. As there can't be >1 season folders. So /S05 - Drum Island/[5.1] Arc Drum Island/ and only the first one is being filtered as a season folder. So logic being used is <grouping_folder>/<season_folder>/<series_folder> as you had enhanced it to be able to have a season folder first in a grouping folder setup.

Additional logging would def help make it clearer, but these facts I believe prove it:

## Scanner log
Folder:
One Piece [tvdb4-81797]/S05 - Drum Island/[5.1] Arc Drum Island
File:
One Piece - E078 - Nami`s Sick Beyond the Snow Falling on the Stars!.mkv

Plex entry:
"Arc Drum Island" s05e078                         "Word Search" "Nami'S Sick Beyond Snow Falling On The Stars!" "One Piece - E078 - Nami`s Sick Beyond the Snow Falling on the Stars!.mkv"

## Facts I see:
Grouping folder logic used == <grouping_folder>/<season_folder>/<series_folder>/<file>

Series Found == Arc Drum Island (from <series_folder>)
Season Found == s05 (from <season_folder>)
Episode Found == e078 (from <file>)

EndOfLine369 commented 5 years ago

Ok now that I understand :) back to the xem link, could the sudden disappearance of s00 be due to the fact that not all of s00 EPS are mapped (only EP 25 is seen when you look at it)

@DarthKnight12, This is all from a bad folder structure being used as it is trying to use both grouping folder (albeit inadvertently) and series folder (for specials). As I noted earlier (https://github.com/ZeroQI/Absolute-Series-Scanner/issues/205#issuecomment-470874914) you add in a special and Plex does a direct scan on One Piece [tvdb4-81797] and as a standard scan sees One Piece [tvdb4-81797]/<season>/<file> and ignores all the other folders as ignored grouping setup. You then get a scheduled scan at library level that again scans starting at the root folder, sees One Piece [tvdb4-81797] as a grouping folder, removes it known as a standard series folder (where your specials now get deleted), then scans One Piece [tvdb4-81797] as a grouping folder with format <grouping_folder>/<season_folder>/<series_folder>/<file>.

DarthKnight12 commented 5 years ago

Ok, I'm not going to argue but please explain to me why does it work perfectly fine if I exclude s00 and it deletes s00 (after initially matching it) after two days, while everything else Satya and is matched correctly ?

DarthKnight12 commented 5 years ago

Ah sorry I didn't see your last post, I'll try something maybe it will work that way maybe it won't but I'll try to add [0.1] movies and [0.2] specials to s00

EndOfLine369 commented 5 years ago

Please fix your folder structure (remove the "[5.1] Arc Drum Island" folder levels so should just be One Piece [tvdb4-81797]/S05 - Drum Island/<file>) and your issue should disappear.

DarthKnight12 commented 5 years ago

I dont get it why [5.1] Arc Drum Island messes with s00 - Specials, whilst like i said everything else matches correctly

EndOfLine369 commented 5 years ago

Everything under One Piece [tvdb4-81797] must either be a standard series folder:

<series>/<file>
<series>/<season>/<file> <-- your specials

OR a grouping folder:

<grouping_folder>/<series_folder>/<file>
<grouping_folder>/<series_folder>/<season_folder>/<file>
<grouping_folder>/<season_folder>/<series_folder>/<file> <-- your s1+ files

As you have both in it, the grouping folder wins and removes the series folder setup files.

DarthKnight12 commented 5 years ago

what i did now is i changed the structure of s00 to

One Piece [tvdb4-81797]/S00 - Specials/[0.1] Arc Movies/files

One Piece [tvdb4-81797]/S00 - Specials/[0.2] Arc Specials/files

EndOfLine369 commented 5 years ago

Going the other way to make the series folder to grouping folder structure should also work. As long as all files are just one setup design.

DarthKnight12 commented 5 years ago

thank you for the help! really appreciate you guys and the level of support you're showing ! thank you !

DarthKnight12 commented 5 years ago

Alright well changing s00 to the arc folder structure didn't work, its gone once again :/

ZeroQI commented 5 years ago

@EndOfLine369 My philosophy is to remove complexity first, which is why Season folders and ark folders (which i call transparent folders, last ones in the SEASON_RX list hence the [:-1] and [-1] used for regex) are removed from the structure and are not meant to trigger the grouping folder logic. Now the regex might be badly written and [5.1] disrupting it. We definitely need logging to know every instance of removed folders, so we confirm the situation. I still think i am right :)

SEASON_RX       = [                                                                                                                                                              ### Seasons Folders
                    cic(r'^Specials'),                                                                                                                                           # Specials (season 0)
                    cic(r'^(Season|Series|Book|Saison|Livre|S)[ _\-]*(?P<season>\d{1,2})'),                                                                                      # Season / Series / Book / Saison / Livre / S
                    cic(r'^(?P<show>.*?)[\._\- ]+S(?P<season>\d{2})$'),                                                                                                          # (title) S01
                    cic(r'^(?P<season>\d{1,2})'),                                                                                                                                # ##
                    cic(r'^(Saga|(Story )?Ar[kc])')]

EndOfLine369 commented 5 years ago

Objection, assumption of rightness is not based on facts in evidence. 😁

Seen in code:

There is no clean_string() call on folder from for folder in reverse_path[:-1] so the [5.1] in folder names are indeed interfering with it.

There is a difference seen where one has reverse_path.remove(folder) run before and one run after the test. Seems like it should be after in both.

...
### Extract season folder to reduce complexity and use folder as serie name ###
folder_season, season_folder_first = None, False
for folder in reverse_path[:-1]:                  # remove root folder from test, [:-1] Doesn't thow errors but gives an empty list if items don't exist, might not be what you want in other cases
for rx in SEASON_RX:                            # in anime, more specials folders than season folders, so doing it first
  match = rx.search(folder)                     #
  if match:                                     # get season number but Skip last entry in seasons (skipped folders)
    if rx!=SEASON_RX[-1]: 
      folder_season = int( match.group('season')) if match.groupdict().has_key('season') and match.group('season') else 0 #break
      if len(reverse_path)>=2 and folder==reverse_path[-2]:  season_folder_first = True
    reverse_path.remove(folder)                 # Since iterating slice [:] or [:-1] doesn't hinder iteration. All ways to remove: reverse_path.pop(-1), reverse_path.remove(thing|array[0])
    break
if not kwargs and len(reverse_path)>1 and path.count(os.sep):  return       #if not grouping folder scan, skip grouping folder
...
...
    ### Extract season and transparent folder to reduce complexity and use folder as serie name ###
    reverse_path, season_folder_first = list(reversed(Utils.SplitPath(path))), False
    for folder in reverse_path[:-1]:                 # remove root folder from test, [:-1] Doesn't thow errors but gives an empty list if items don't exist, might not be what you want in other cases
      for rx in SEASON_RX :                          # in anime, more specials folders than season folders, so doing it first
        if rx.search(folder):                        # get season number but Skip last entry in seasons (skipped folders)
          reverse_path.remove(folder)                # Since iterating slice [:] or [:-1] doesn't hinder iteration. All ways to remove: reverse_path.pop(-1), reverse_path.remove(thing|array[0])
          if rx!=SEASON_RX[-1] and len(reverse_path)>=2 and folder==reverse_path[-2]:  season_folder_first = True
          break
...

EndOfLine369 commented 5 years ago

would explain why your expectation of ALL season folders to be cleansed are not happening. And thus being seen as a grouping folder in format <grouping_folder>/<season_folder>/<series_folder>/<file> as shown from the log breakdown above https://github.com/ZeroQI/Absolute-Series-Scanner/issues/205#issuecomment-471212687
could explain the differences in scan behavior seen for this being seen as grouping folder on Plex scan and not a grouping folder on manual scan (or vise-versa) and episodes to disappear...

ZeroQI commented 5 years ago

Well spotted, completelly missed that...

Format: ////

you have 1 season_rx per reverse_path entry max, the break doesn't cascade
transparent folder can be before or after season folder

Seem like the fix is to run "clean_string" on all reverse_path entries or:

match = rx.search(clean_string(folder))
rx.search(clean_string(folder))

"No forced id found in series folder name nor id file" bugs me since "[tvdb4-...]" is present but hopefully fixed with that

DarthKnight12 commented 5 years ago

so whats the solution then :) will i have to add anything ? Oh also, S00 magically reappeared again without me having to change anything, i have no idea why thou

ZeroQI commented 5 years ago

@DarthKnight12 You need to wait for a new release of the scanner fixing this (or at least improving logging and attempting fixing this)

EndOfLine369 commented 5 years ago

@ZeroQI, reverse_path is used in later code folder_show = reverse_path[0] if reverse_path else "" as a single example. So we can't clean it as would remove the forced ids at a minimum.

EndOfLine369 commented 5 years ago

There is a difference seen where one has reverse_path.remove(folder) run before and one run after the test. Seems like it should be after in both.

@ZeroQI, For this, I am having a hard time determining which one is correct to action reverse_path.remove(folder) before or after the season_folder_first test.

It really depends on when season_folder_first should be True. I'm having a hard time figuring that out. Need you to please advise.

season_folder_first:

<series>/<file>  <== False
<series>/<season_rx_folder>{1,inf}/<file>  <== True?
<grouping_folder>/<series_folder>/<file>   <== False
<grouping_folder>/<series_folder>/<season_rx_folder>{1,inf}/<file>  <== True?
<grouping_folder>/<season_rx_folder>{1,inf}/<series_folder>/<file>  <== True?

ZeroQI commented 5 years ago

season_folder_first was added 2018-04-21 here https://github.com/ZeroQI/Absolute-Series-Scanner/commit/b25d73a84294ecd2e17d76426be1844cceab7cd0#diff-edf9a02ffa5a0496c5247e048b83f667

season_folder_first is used to see if series > season > xxx > yyy > zzz so i do trigger grouping folders by depth only if season folder isn't second because that means it's not a grouping folder since series folder would be first... Transparent folders - seasons_rx[-1] - are taken out of reverse_path beforehand

<series>/<file>  <== False
<series>/<season_rx_folder>{1,inf}/<file>  <== True
<grouping_folder>/<series_folder>/<file>   <== False
<grouping_folder>/<series_folder>/<season_rx_folder>{1,inf}/<file> False
<grouping_folder>/<season_rx_folder>{1,inf}/<series_folder>/<file>  <== True but season folders go into series, this is unsupported and bad

EndOfLine369 commented 5 years ago

Well, i'm going to leave this intriguing find alone. As could be correct. Don't know as brain💥 is not working on processing this information at this time. Main problem on clean not running on season_rx run is now fixed. Considering this closed unless @DarthKnight12 comes back with issues still after clean fix.

DarthKnight12 commented 5 years ago

Everything is working like a charm, once again thank you guys very much for the help and support on this! Truly heroes of this amazing scanner and agent !

_{Sent with GitHawk}

ZeroQI commented 5 years ago

@EndOfLine369 Thanks for the fix. Good team work :D

ZeroQI / Absolute-Series-Scanner

One Piece tvdb4 S00 Movies and Specials #205