search

ZeroQI / Absolute-Series-Scanner

Seasons, absolute mode, Subfolders...
1.01k stars 155 forks source link

Fallback [from behavior changing modes; anidb3/4] to anidb when mapping fails #243

Closed avluis closed 4 years ago

avluis commented 4 years ago

Is your feature request related to a problem? Please describe. Whenever anidb3/4 doesn't have a matching tvdb/tvdb6 entry, the result is tvdb-/tvdb6-.

For example: tvdb- tvdb6-

anime-list-master.xml:

<anime anidbid="2899" tvdbid="movie" defaulttvdbseason="1" episodeoffset="" tmdbid="" imdbid="tt0093399">
  <name>Mori no Densetsu</name>
</anime>

Mori no Densetsu (1988) [anidb3-2899].agent-search.log:

=============================================================================================================================================================
common.PlexLog(file="/config/Library/Application Support/Plex Media Server/Plug-in Support/Data/com.plexapp.agents.hama/DataItems/_Logs/Anime/Mori no Densetsu (1988) [anidb3-2899].agent-search.log", movie=False)
[!] file:       "/media/anime/Mori no Densetsu (1988) [anidb3-2899]/01 - Complete Movie [Anonymoose][Blu-ray](3d47a23d).mkv"
[ ] library:    "Anime"
[ ] root:       "/media/anime"
[ ] path:       "Mori no Densetsu (1988) [anidb3-2899]"
[ ] Plex root:  "/config/Library/Application Support/Plex Media Server"
[ ] Log folder: "Plug-in Support/Data/com.plexapp.agents.hama/DataItems/_Logs/Anime"
[ ] Log file:   "Mori no Densetsu (1988) [anidb3-2899].agent-search.log"
[ ] Logger:     "0x14d0fb0cb700"
[ ] mode:       "w"
=== Search() ================================================================================================================================================
title: 'Mori no Densetsu (1988) [tvdb-]', name: 'Complete Movie', filename: '%2Fmedia%2Fanime%2FMori%20no%20Densetsu%20(1988)%20[anidb3-2899]%2F01%20-%20Complete%20Movie%20[Anonymoose][Blu-ray](3d47a23d)%2Emkv', manual: 'True', year: 'None'
start: 2019-11-12 15:39:44,200696
=============================================================================================================================================================
--- force id ------------------------------------------------------------------------------------------------------------------------------------------------
Forced ID - source: tvdb, id: , title: 'Mori no Densetsu (1988)'
=============================================================================================================================================================
end: 2019-11-12 15:39:44,201752

Mori no Densetsu (1988) [anidb3-2899].agent-update.log:

=============================================================================================================================================================
common.PlexLog(file="/config/Library/Application Support/Plex Media Server/Plug-in Support/Data/com.plexapp.agents.hama/DataItems/_Logs/Anime/Mori no Densetsu (1988) [anidb3-2899].agent-update.log", movie=False)
[!] file:       "/media/anime/Mori no Densetsu (1988) [anidb3-2899]/01 - Complete Movie [Anonymoose][Blu-ray](3d47a23d).mkv"
[ ] library:    "Anime"
[ ] root:       "/media/anime"
[ ] path:       "Mori no Densetsu (1988) [anidb3-2899]"
[ ] Plex root:  "/config/Library/Application Support/Plex Media Server"
[ ] Log folder: "Plug-in Support/Data/com.plexapp.agents.hama/DataItems/_Logs/Anime"
[ ] Log file:   "Mori no Densetsu (1988) [anidb3-2899].agent-update.log"
[ ] Logger:     "0x14d0fb0cb700"
[ ] mode:       "w"
=== Update() ================================================================================================================================================
id: tvdb-, title: None, lang: en, force: True, movie: False
start: 2019-11-12 15:57:02,431647
=== AnimeLists.GetMetadata() ================================================================================================================================
tvdb_numbering: False
Local custom mapping file not present: anime-list-custom.xml
--- AniDBTVDBMap --------------------------------------------------------------------------------------------------------------------------------------------
ERROR: Could not find tvdb: 
             -----          ------

[ ] collection: TVDBid '' is not part of any collection
[ ] studio: 
--- return --------------------------------------------------------------------------------------------------------------------------------------------------
AniDB_id: '', AniDB_id2: '', AniDBid: '', TVDB_id: '', TVDB_id2: '', TVDBid: ''
mappingList: {
  's1e1_mapped': False}
AnimeLists_dict: {}
=== common.GetMetadata() ====================================================================================================================================
not tvdb4 mode
=== TheTVDB.GetMetadata() ===================================================================================================================================
TVDBid: '', IMDbid: '', language_series : ['en', 'ja'], language_episodes: ['en', 'ja']
TVDBid non-digit
=== AniDB.GetMetadata() =====================================================================================================================================
Source: tvdb, AniDBid: , Full AniDBids list: [], Active AniDBids list: []
language_posters: ['en']
--- return --------------------------------------------------------------------------------------------------------------------------------------------------
relations_map: {}
AniDB_dict: {}
=== TheMovieDb.GetMetadata() ================================================================================================================================
TVDBid: '', TMDbid: '', IMDbid: ''
=== FanartTv.GetMetadata() ==================================================================================================================================
movie:'False', TVDBid: '', tmdbid: '', imdbid:'', season: '0', num: '100'
=== Plex.GetMetadata() ======================================================================================================================================
Prefs['themes']: 'TVTunes, Plex', TVDBid: ''
--- themes --------------------------------------------------------------------------------------------------------------------------------------------------
Not pulling meta - 'Plex' in Prefs['themes']: 'True', TVDBid: ''
--- return --------------------------------------------------------------------------------------------------------------------------------------------------
Plex_dict: {}
=== TVTunes.GetMetadata() ===================================================================================================================================
Prefs['themes']: 'TVTunes, Plex', title: '', title2: ''
--- themes --------------------------------------------------------------------------------------------------------------------------------------------------
--- return --------------------------------------------------------------------------------------------------------------------------------------------------
TVTunes_dict: {}
=== OMDb.GetMetadata() ======================================================================================================================================
IMDbid: ''
--- return --------------------------------------------------------------------------------------------------------------------------------------------------
OMDb_dict: {}
=== Local.GetMetadata() =====================================================================================================================================
dir:     /media/anime/Mori no Densetsu (1988) [anidb3-2899]
library: Anime
root:    /media/anime
path:    Mori no Densetsu (1988) [anidb3-2899]
series_root_folder:  /media/anime/Mori no Densetsu (1988) [anidb3-2899]
Grouping folder:     Mori no Densetsu (1988) [anidb3-2899]
subfolder_count:     0
reverse_path:        ['Mori no Densetsu (1988) [anidb3-2899]']
season_folder_first: False
Grouping folder not found
--- return --------------------------------------------------------------------------------------------------------------------------------------------------
Local_dict: {}
=== common.AdjustMapping() ==================================================================================================================================
adjusting mapping for 'anidb3/tvdb' & 'anidb4/tvdb6' usage
season_map: {}
relations_map: {}
--- tvdb mapping adjustments --------------------------------------------------------------------------------------------------------------------------------
TVDB Before: {}
TVDB After : 
--- tvdb meta episode adjustments ---------------------------------------------------------------------------------------------------------------------------
adjustments: {}
--- return --------------------------------------------------------------------------------------------------------------------------------------------------
is_modified: False
=== Update() ================================================================================================================================================
AniDBid: '', TVDBid: '', TMDbid: '', IMDbid: '', ANNid:'', MALid: ''
=== common.write_logs() =====================================================================================================================================
anime-list AniDBid missing : []
anime-list studio logos    : []
AniDB summaries missing    : []
Plex themes missing        : []
TVDB posters missing       : []
Missing Specials           : []
Missing Episodes           : []
AniDB posters missing      : []
anime-list TVDBid missing  : []
TVDB season posters missing: []
Missing Special Summaries  : []
Missing Episode Summaries  : []
=== common.UpdateMeta() =====================================================================================================================================
Fields in Metadata Sources per movie/serie, season, episodes
-------------------------------------------------------------------------------------------------------------------------------------------------------------
Metadata Fields (items #), type, source provider, value
[!] field: "title" not in meta_root, source: "None"
[!] 'MyAnimeList' source not in MetaSources dict, please Check case and spelling
[!] 'MyAnimeList' source not in MetaSources dict, please Check case and spelling
[!] 'MyAnimeList' source not in MetaSources dict, please Check case and spelling
metadata.seasons[ 1]--------------------------------------------------------------------------------------------------------------------------------------
metadata.seasons[ 1].episodes[  1]
[!] 'MyAnimeList' source not in MetaSources dict, please Check case and spelling
-------------------------------------------------------------------------------------------------------------------------------------------------------------
end: 2019-11-12 15:57:02,797757

Describe the solution you'd like Fallback to anidb/2 instead of current behavior.

Describe alternatives you've considered Manually mapping entries in Plex to anidb with Fix Match can override it properly, but this only works for a single series/folder at a time with the current Plex version (otherwise Plex will merge all entries together; this requires performing The Plex Dance® to fix). I'm currently importing my entire library one folder at a time to catch when anidb3 doesn't match a tvdb entry.

Additional context If this behavior is undesirable; can I get an idea of where I need to modify the scanner to implement this?

ZeroQI commented 4 years ago

Can't you group them in a collection ? Putting them in a grouping folder would create the collection for you if using the right collection source ("Local")... That is generally the easiest way. you can order in grouping folder by prepending "[01]" in front of the series name to order them or use that in sorting title so they are sorted in the web interface...

This is a MOVIE linked to another MOVIE...

There is no reason to use anidb3 on every member of your library. Use tvdb numbering, anidb numbering, or even anidb numbering using anidb2 mode IF a tvdb exists (Series, no movies).

Why do you think it is logical to use anidb3 mode on a series on two movies ???

avluis commented 4 years ago

My apologies for not clarifying but the images attached above is the same entry -- the difference is that I swapped between anidb3 and anidb4 to take those screenshots.

There is no TheTVDB id, TheTVDB doesn't store movies and even ScudLee list "movie" in thetvdb id...

I'm not sure how the above is relevant but a quick look at my library I can see two movies that are on TheTVBD:

Magical Girl Lyrical Nanoha The Movie 1st (2010) [anidb3-6075] & Magical Girl Lyrical Nanoha The Movie 2nd A's (2012) [anidb3-8821] = [tvdb-81115].

My reason for wanting to approach this way is due to my library size. I make use of scripts and helper apps to automatically hash the files and store into the appropriate directory(ies).

anidb3 fits my usage case for the library overall but as mentioned there are cases where there is no tvdb entry and that is okay - all I'm asking is if the scanner can fallback to anidb mode instead of eating the id and outputting tvdb-.

So, for example with the series mentioned above; if TheTVDB didn't have an entry for it: Magical Girl Lyrical Nanoha The Movie 1st (2010) [anidb3-6075] --> no tvid = Magical Girl Lyrical Nanoha The Movie 1st (2010) [anidb-6075] and Magical Girl Lyrical Nanoha The Movie 2nd A's (2012) [anidb3-8821] --> no tvid = Magical Girl Lyrical Nanoha The Movie 2nd A's (2012) [anidb-8821].

Another reason to look at it is this way; If it were to end up with a tvdb entry in the future (as it can be with currently airing series) then the scanner should be able to revert back to the anidb3 behavior and attach the tvdb id instead.

ZeroQI commented 4 years ago

These movies are considered special in a TV series. The movies you used as exemple have no series linked to them, therefore are not in TheTVDB.

Reverting to AniDB makes sense to me, to anidb2 does not.

On Tue, Nov 12, 2019, 10:35 PM Luis E Alvarado notifications@github.com wrote:

My apologies for not clarifying but the images attached above is the same entry -- the difference is that I swapped between anidb3 and anidb4 to take those screenshots.

There is no TheTVDB id, TheTVDB doesn't store movies and even ScudLee list "movie" in thetvdb id...

I'm not sure how the above is relevant but a quick look at my library I can see two movies that are on TheTVBD: Magical Girl Lyrical Nanoha The Movie 1st (2010) [anidb3-6075] & Magical Girl Lyrical Nanoha The Movie 2nd As (2012) [anidb3-8821]=[tvdb-81115]`

My reason for wanting to approach this way is due to my library size. I make use of scripts and helper apps to automatically hash the files store into the appropriate directory.

anidb3 fits my usage case for the library overall but as mentioned there are cases where there is not tvdb entry; and that is okay.

All I'm asking is if the scanner can fallback to anidb mode instead of eating the id and outputting tvdb-.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ZeroQI/Absolute-Series-Scanner/issues/243?email_source=notifications&email_token=ABHMWZNKQCO4WCOQUGSM5XTQTMVUPA5CNFSM4JMJEHR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOED4GENA#issuecomment-553148980, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABHMWZPI2QSOK746FU3QW5DQTMVUPANCNFSM4JMJEHRQ .

avluis commented 4 years ago

I agree with reverting to anidb as well; after taking a look at the code behind anidb2; anidb makes the most sense given the issue presents itself only when there is no tvdb entry to match to -- making anidb2 pointless.

--

Is there a part of the code you can point me to where you would consider this could be implemented?

ZeroQI commented 4 years ago

Now we could test for anidb 2/3/4 and tvdb empty and replace with anidb mode, to avoid going through all that code un-necessarilly, line 665 seem very adequate if source in ["anidb2", "anidb3", "anidb4"]: but we need to load the tvdb id then to figure out so not that easy...

We need to locate the re-writing of the series folder first. located Line 775 i believe (and also 692) folder_show = clean_string(folder_show) + " [tvdb%s-%s]" % ('' if source=="anidb3" else "6", a3_tvdbid)

Simplest seem to modify it like such: if a3_tvdbid: folder_show = clean_string(folder_show) + " [tvdb%s-%s]" % ('' if source=="anidb3" else "6", a3_tvdbid) else: folder_show = clean_string(folder_show) + " [anidb-%s]" % (id)

Please test and report (and submit change if you can, i will approve it).

avluis commented 4 years ago

Got it; I'll get on this as soon as I'm home as I'm not in front of a good terminal at the moment (editing is painful).

avluis commented 4 years ago

Working on anidb3 at the moment and the following logic works if line 775 is replaced with:

- folder_show = clean_string(folder_show) + " [tvdb%s-%s]" % ('' if source=="anidb3" else "6", a3_tvdbid)
+ if a3_tvdbid: folder_show = clean_string(folder_show) + " [anidb-%s]" % (id)
+ else: folder_show = clean_string(folder_show) + " [tvdb%s-%s]" % ('' if source=="anidb3" else "6", a3_tvdbid)
  except Exception as e:  Log.error("Error parsing content, Exception: '%s'" % e)

~~Only issue left is to figure out how to map the file/folder path itself, meaning, for example: Yumekuri [anidb3-9392] --> no tvdb --> Yumekuri [anidb-9392] (path Yumekuri [anidb3-9392])~~

Is this something that would need to be addressed in HAMA or can it be resolved in the scanner directly? I am wondering if the effort is worth the results if it has to be mapped in the agent instead of the scanner.

Had to redownload my copy of the scanner as it seems it had some issues before I started tweaking (probably from trying to modify it remotely). It seems to be working as intended with the following change (as you suggested):

- folder_show = clean_string(folder_show) + " [tvdb%s-%s]" % ('' if source=="anidb3" else "6", a3_tvdbid)
+ if a3_tvdbid: folder_show = clean_string(folder_show) + " [tvdb%s-%s]" % ('' if source=="anidb3" else "6", a3_tvdbid)
+ else: folder_show = clean_string(folder_show) + " [anidb-%s]" % (id)
  except Exception as e:  Log.error("Error parsing content, Exception: '%s'" % e)

anidb3 --> anidb when no tvdb match: anidb3_as_anidb

Sakura Nagashi [anidb3-12423].filelist.log:

=============================================================================================================================================================
Library: 'Anime', root: '/media/anime', path: 'Sakura Nagashi [anidb3-12423]', files: '1', dirs: '0'
Plex scan start: 2019-11-13 15:55:08,314233
plexignore_files: '[]', plexignore_dirs: '[]'
=============================================================================================================================================================
[file] Sakura Nagashi [anidb3-12423]/01 - Music Video [GanbaRaw][www](48c3263a).mkv
=============================================================================================================================================================
Plex scan end: 2019-11-13 15:55:08,314750

Sakura Nagashi [anidb3-12423].scanner.log:

Library: 'Anime', root: '/media/anime', path: 'Sakura Nagashi [anidb3-12423]', files: '1', dirs: '0'
Plex scan start: 2019-11-13 15:55:08,315798
=============================================================================================================================================================
Forced ID (series folder) - source: "anidb3", id: "12423"
-------------------------------------------------------------------------------------------------------------------------------------------------------------
AniDB mode (anidb3) enabled, loading mapping xml file (Local->ASS mod->ScudLee master)
URL: 'https://rawgit.com/ZeroQI/Absolute-Series-Scanner/master/anime-list-corrections.xml', Using cached file: '/tmp/ASS-anime-list-corrections.xml'
-- No valid tvbdbid found for anidbid '12423'
URL: 'https://rawgit.com/ScudLee/anime-lists/master/anime-list-master.xml', Using cached file: '/tmp/ASS-anime-list-master.xml'
-- No valid tvbdbid found for anidbid '12423'
-------------------------------------------------------------------------------------------------------------------------------------------------------------
AniDB mode (anidb3) enabled, loading season and relation mapping for all associated tvdbid entries
URL: 'https://rawgit.com/ScudLee/anime-lists/master/anime-list-master.xml', Using cached file: '/tmp/ASS-anime-list-master.xml'
URL: 'https://rawgit.com/ZeroQI/Absolute-Series-Scanner/master/anime-list-corrections.xml', Using cached file: '/tmp/ASS-anime-list-corrections.xml'
anidbid: '12423', tvdbid: '', max_season: '0', season_map: {}, no override set so using unmodified mapping
-------------------------------------------------------------------------------------------------------------------------------------------------------------
misc_count: {'Video': 1, '01': 1, 'Music': 1}
misc_words: []
-------------------------------------------------------------------------------------------------------------------------------------------------------------
"Sakura Nagashi [anidb-12423]" s01e001                         "Word Search" "Music Video" "01 - Music Video [GanbaRaw][www](48c3263a).mkv"
=============================================================================================================================================================
Plex scan end: 2019-11-13 15:55:08,456100

The same code above should work[s] for anidb4 as well. I'll try and test for any side-effects later today and create a PR if its a pass.

In summary; After making the changes above, instead of having to import my collection one by one; I can now just dump everything at once and the scanner with sort it out for me 🥇

avluis commented 4 years ago

@sven-7 @EndOfLine369 @ZeroQI Quick question for you in regards to [anidb2] mode; In my efforts to learn how this mode works and how the mapping files work, I'm attempting to test out a local custom mapping file. Here is that file:

<?xml version="1.0" encoding="utf-8"?>
<anime-list>
  <anime anidbid="3468" tvdbid="79895" defaulttvdbseason="1" episodeoffset="" tmdbid="" imdbid="">
    <name>Gintama</name>
    <mapping-list>
      <mapping anidbseason="0" tvdbseason="0">;3-0;4-3;</mapping>
      <mapping anidbseason="1" tvdbseason="2" start="50" end="99" offset="-49"/>
      <mapping anidbseason="1" tvdbseason="3" start="100" end="150" offset="-99"/>
      <mapping anidbseason="1" tvdbseason="4" start="151" end="201" offset="-150"/>
    </mapping-list>
    <supplemental-info>
      <studio>Sunrise</studio>
    </supplemental-info>
  </anime>
</anime-list>

The above still ends up with what I posted on the Plex forums so having a bit of trouble figuring out what needs to change. For example; Season One (50 episodes) starts with a special and then episode 1 - 49 which match with AniDB S1 and 1 - 49. Season Two (50 additional episodes) correspond with AniDB episode 50 - 99. Season Three (51 more) -- AniDB 100 - 150. Season Four (1 - 26, 2nd special, 27 - 51) -- AniDB 151 - 176, S2, 177 - 201. https://www.thetvdb.com/series/gintama/allseasons/official https://anidb.net/anime/3468

I've re-read the instructions (https://github.com/ScudLee/anime-lists/#format) but I'm having a hard time mapping that out; got any clues for me?

ZeroQI commented 4 years ago

defaulttvdbseason="1" not 'a' so not using absolute numbering AniDb untouched so tvdb s01e1-49 AniDb 50-99 to tvdb s02e01-50 AniDb 100-150 so tvdb s03e01-51 AniDb 151-201 so tvdb s04e01-50 S1s2 untouched so AniDB Special 4 mapped to tvdb S00e03

If you want tvdb absolute numbering you use tvdb2 and it will show it using seasons if you wish...

avluis commented 4 years ago

@ZeroQI Thank you -- I hope to learn from this so I can start contributing to ScudLee's lists.