search

ZeroQI / Hama.bundle

Plex HTTP Anidb Metadata Agent (HAMA)
GNU General Public License v3.0
1.19k stars 110 forks source link

Any way to force "thetvdb movie id"? If not, any way to force TheMovieDB ID? Documentation says deprecated. #447

Closed RainOne closed 3 years ago

RainOne commented 3 years ago

Hi!

I've been using ASS+HAMA for a while, very successfully to index my anime collection. Since it allows for TVs and Movies to be indexed together, I thought about using it to index my Tokusatsu collection - not present on AniDB, but presented on TheTVDB and The Movie Database.

This is entirely unsupported, and I'm aware - Tokusatsu indexing is terrible on TheTVDB (everything is under a 40+ season for Super Sentai, for instance, with some shows having their unique database entry), so manually forcing matches with "tvdb" on the folders seem to be the most sure way of properly matching everything.

The thing is that TheTVDB now has a "movies" entry - for instance, on https://thetvdb.com/movies/shin-kamen-rider-prologue Reading ASS documentation, it seems I can no longer force TheMovieDatabase IDs with the "tmdb-" tag. Not only that, I can't find a way to force tag a "THETVDB.COM MOVIE ID" as it's listed on the website - just putting that id finds the TV ID instead.

I know about the "complete movie" workaround, but manually forcing the ID works well for me on the Anime collection and I was thinking I could do something like that on the Tokusatsu library, specially since most of the items i need to match are present on TheTVDB. Am I missing something or is there no current way to force TheTVDB.com movie ids? Is TheMovie Database forced matching also no longer possible?

Thank you in advance.

ZeroQI commented 3 years ago

Notes

Are you aware you can manually select another agent (TheTVDB) in a library for matching, so i don't have to code a workaround just for that...

RainOne commented 3 years ago

Oh, I'm entirely aware.

I mostly wanted to go to ASS/HAMA to support movies and series in the same library - Not through AniDB but for a merge between TheTVDB and TheMovieDB.

Honestly, I didn't know TheTVDB supported Movies now - I'm not sure how the TVDB agent on plex supports it, or the regular scanner does or how to manually point to entries using it - I'm rearranging my collection (and decided that everything I did in the past 3 hours was a waste of time).

I couldn't get TheMovieDB and TheTVDB manual assignment to work with ASS+HAMA, which is why I made this message. Now that I know it works (I just wasn't getting a hit, for some reason?), I'll look into it more once I get back to it.

Thank you for the support and i'll come back with my findings, but now I need to work a little :P

RainOne commented 3 years ago

Ok, so, it doesn't seem to be working?

I have multiple unique movie folders assigned with the TMDB tag (much like I do with anidb on my anime collection), plex itself seems to automatically "attach" the movie to that tag, but no metadata is downloaded - neither name, description or poster.

I don't see anything outstanding in the logs either, here's an example:

match_content.txt

This log is when I click "match content" on a movie that has a folder named "Choushinsei Flashman Movie (1986) [tmdb-440442]". The "match content" interface comes up with that as a perfect match, but no metadata is downloaded.

the TVDB forced matches are working (mostly), one of them I uploaded a Poster yesterday and added an episode list, it only downloaded the episode list and not the poster yet - maybe something to do with plex's cache system.

Any other logs or a better way to provide you these logs?

ZeroQI commented 3 years ago

HAMA series logs in agent data folders (.search.log and .update.log) missing

RainOne commented 3 years ago

Choushinsei Flashman Movie (1986) [tmdb-440442].agent-update.log

This is the tmdb page for the movie:

https://www.themoviedb.org/movie/440442-choushinsei-flashman-the-movie

Don't really know what I am doing wrong.

Edit: I tried matching the exact movie in your example (as I also have it) and it found it, but didn't download any assets:

Kamen Rider Shin (1992) [tmdb-194777].agent-update.log Kamen Rider Shin (1992) [tmdb-194777].agent-search.log

ZeroQI commented 3 years ago 
=== TheMovieDb.GetMetadata() ================================================================================================================================
TVDBid: '', TMDbid: '', IMDbid: ''

Need to look at code, why no metadata is loaded

RainOne commented 3 years ago

Yeah, I noticed that.

I also noticed that if I put the entire url of the movie as the tmdb-info (tmdb-440442-choushinsei-flashman-the-movie) it does match it, but still doesn't load any metadata.

Choushinsei Flashman Movie (1986) [tmdb-440442-choushinsei-flashman-the-movie].agent-update.log Choushinsei Flashman Movie (1986) [tmdb-440442-choushinsei-flashman-the-movie].filelist.log Choushinsei Flashman Movie (1986) [tmdb-440442-choushinsei-flashman-the-movie].scanner.log Choushinsei Flashman Movie (1986) [tmdb-440442-choushinsei-flashman-the-movie].agent-search.log

If it's deprecated, it's fine, but it would be nice if it parsed the code only, as the codes are unique, as far as I know.

But if nobody uses, eh, I'll figure something out.

RainOne commented 3 years ago

Another thing I just noticed: It's not downloading posters from TheTVDB.

On another series, I added a poster to the thetvdb database, in multiple languages:

https://www.thetvdb.com/series/tokumei-sentai-go-busters

However, it doesn't find a poster: Tokumei Sentai Go-Busters (2012) [tvdb-348803].agent-update.log Tokumei Sentai Go-Busters (2012) [tvdb-348803].filelist.log Tokumei Sentai Go-Busters (2012) [tvdb-348803].scanner.log Tokumei Sentai Go-Busters (2012) [tvdb-348803].agent-search.log

It does download tv show, description, episode names (all were not available on the database before i added them). Not the poster though: the cached information points to the wrong URL for the poster:

"poster":"series/348803/posters/601e3d47ddd3e.jpg" when it should be "https://thetvdb.com/banners/series/348803/posters/601e3d47ddd3e.jpg"

series_en.json.txt

On another series, it also fails to grab TheTVDB Posters, but it matches the series to tmdb database instead (which is not working for actual movies) and downloads that poster instead:

Tokusatsu GaGaGa (2019) [tvdb-358297].agent-update.log Tokusatsu GaGaGa (2019) [tvdb-358297].filelist.log Tokusatsu GaGaGa (2019) [tvdb-358297].scanner.log Tokusatsu GaGaGa (2019) [tvdb-358297].agent-search.log

ZeroQI commented 3 years ago

Tokumei Sentai Go-Busters (2012)

Tokusatsu GaGaGa (2019) [tvdb-358297]: you forced the tvdb id and it used it for TMDB meta on top of theTVDB meta...

Doesn't fail to get theTVDB poster (rank 17) but TheMovieDb has highter priority since it's first (rank 1), since you changed the poster Metadata Sourcesto be: TheMovieDb, TheTVDB, AniDB, AniList, FanartTV, tvdb4

[x] posters ( 1) Sources: (TheMovieDb), TheTVDB, AniDB, AniList, FanartTV, tvdb4 Inside: '['TheTVDB', 'TheMovieDb']' Value: '{'https://image.tmdb.org/t/p/original/ekrEZFXoKuDJ6YGtV4mbXWcL4GN.jpg': ('TheMovieDb/poster/ekrEZFXoKuDJ6YGtV4mbXWcL4GN.jpg', 1, None)}' [x] posters ( 1) Sources: TheMovieDb, (TheTVDB), AniDB, AniList, FanartTV, tvdb4 Inside: '['TheTVDB', 'TheMovieDb']' Value: '{'https://thetvdb.plexapp.com/banners/posters/5cd605f93e703.jpg': ('TheTVDB/posters/5cd605f93e703.jpg', 17, 'https://thetvdb.plexapp.com/banners/posters/5cd605f93e703_t.jpg')}' "it [...] fails to grab TheTVDB Posters"

Metadata is an agent issue so this ticket needs closing. This is the path to functionnality, this should allow IMDB and TMDb forced ID to Work

RainOne commented 3 years ago

You are correct, I made some mistakes. However, I did the thing you told me (if I understand correctly, was to edit AnimeLists.py with your modifications). That didn't work at all.

I did all you asked and the scans of TMDB ids are still not working and a couple of TVDB entries are not grabbing the posters still. For instance:

TVDB posters missing : ["TVDBid: 317847 | Title: 'Dengeki Sentai Changeman'"]

Dengeki Sentai Changeman PTBR (1985) [tvdb-317847].agent-update.log Dengeki Sentai Changeman PTBR (1985) [tvdb-317847].filelist.log Dengeki Sentai Changeman PTBR (1985) [tvdb-317847].scanner.log Dengeki Sentai Changeman PTBR (1985) [tvdb-317847].agent-search.log

But if you go to that link, you'll see that there is a poster (two, as I even uploaded another one), but they just don't show up (and are being reported as missing on the logs).

This should be closed as this seems to be a Hama issue and not a ASS issue? Do I just close this entry and open another one on HAMA? There's a lot of issues that I still can't figure out.

I don't really want to take your time for my very specific issue (Force TVDB and TMDB on a Agent that allows for both TV and Movie entries to live together in the same library). Specially since TMDB seems to be deprecated - I could just make a new "Tokusatsu (TV)" and "Tokusatsu (Movies)" libraries and live off the same category names from plex internal scrapers (for when I want series and movies to live together), but I do find it strange that some TVDB entries simply doesn't find the posters (I think it might be a thing between V2 and V3 apis/entries and when they were created).

Thank you so much for your time and feel free to say "sorry i don't plan on supporting any of this"

ZeroQI commented 3 years ago

Let's try to summarize current situation

The action plan at this point is:

RainOne commented 3 years ago

Well, I did what you asked (i commented the lines you asked to delete): image image

and i created a new database and made a new scan. The new scan only brought in the data from the TVDB forced entries. Looking at the logs, it seems that the TMDB is only being partially assigned:

Kamen Rider Shin (1992) [tmdb-194777].agent-update.log Kamen Rider Shin (1992) [tmdb-194777].filelist.log Kamen Rider Shin (1992) [tmdb-194777].scanner.log

the id is being assigned, but the tmdb.getmetadata function is still not getting it.

What do you mean about "not being able to get the json poster"? Is that in relation to the TVDB entires that are not downloading posters?

ZeroQI commented 3 years ago

You are right, no TMDbid detected, I messed up

=== TheMovieDb.GetMetadata() ================================================================================================================================
TVDBid: '', TMDbid: '', IMDbid: ''

Found where, put back the commented/deleted lines, use this line to replace (forgot an undercore for 2 variables), now it should fallback to the forced TMDBid and use the metadata return AnimeLists_dict, AniDB_winner, TVDB_winner if TVDB_winner.isdigit() else "", Dict(mappingList, 'tmdbid') or TMDB_id, Dict(mappingList, 'imdbid') or IMDB_id, mappingList

RainOne commented 3 years ago

So that worked, the logs now show that the TMDB is being assigned and OMDb (i already set up the API key) is also identifying the IMDB instance of the movie.

Kamen Rider Shin (1992) [tmdb-194777].agent-search.log Kamen Rider Shin (1992) [tmdb-194777].agent-update.log

However, only the "genre" is being pulled from TMDB, and I imagine that's because TheMovieDB and OMDb are on the list of "T--M 'genres'" on HAMA configuration. I don't really understand how the service tag configurationn works (you said before that posters were not being pulled because of priorities, but as I understand, priorities only work when the request works - if it doesn't, it shouldn't prioritize "no answer"). It seems that addimg TheMovieDB to those different tags didn't work - unless it requires a full plex server reboot/cache clean.

I also noticed that the title is not being pulled on either (TMDB or OMDb), as per logs. Everything else is (description, posters, background images), but nothing is being applied on my library. OMDB (and therefore IMDB) posters are being detected, but not being added to the pool of "available posters" for me to select. While the TMDB poster does appear there, it doesn't get automatically set and replace the "no available poster".

I wonder if these are something related to the plex cache? I always have issues with Plex cached items and I don't know how to forcibly clean them other than deleting everything - and I don't really want to remove all the cached information from my other libraries. I do notice that using other agents (mainly the ones that are plex's default), every time I "force" a metadata change, it will update - Maybe HAMA is not forcing a cache flush, I wonder if there's a way to code that on your side.

Seems we're going the right way.

ZeroQI commented 3 years ago

If no title is set, no metadata is saved

Metadata Fields (items #), type, source provider, value
[!] field: "title" not in meta_root, source: "None"
[#] originally_available_at        Sources: AniDB,TheTVDB                                                 Inside: ['OMDb']  Values: 
[!] 'MyAnimeList' source not in MetaSources dict, please Check case and spelling
[=] duration                       Sources: TheTVDB, AniDB, MyAnimeList, (TheMovieDb), OMDb               Inside: '['TheMovieDb', 'OMDb']'  Value: '88'
[!] 'MyAnimeList' source not in MetaSources dict, please Check case and spelling
[=] rating                         Sources: AniDB, TheTVDB, MyAnimeList, (TheMovieDb), OMDb | TheTVDB, AniDB  Inside: '['TheMovieDb', 'OMDb']'  Value: '4.4'
[!] 'MyAnimeList' source not in MetaSources dict, please Check case and spelling
[=] genres                   ( 3)  Sources: TheTVDB, AniDB, MyAnimeList, (TheMovieDb), OMDb               Inside: '['TheMovieDb', 'OMDb']'  Value: '['Action', 'Adventure', 'Science Fiction']'
[!] 'MyAnimeList' source not in MetaSources dict, please Check case and spelling
[#] summary                        Sources: TheTVDB, AniDB                                                Inside: ['TheMovieDb', 'OMDb']  Values: 
[#] countries                      Sources: TheMovieDb                                                    Inside: ['OMDb']  Values: 
[=] studio                         Sources: AnimeLists, AniDB, (TheMovieDb)                               Inside: '['TheMovieDb']'  Value: 'Toei Company'
[#] countries                      Sources: TheMovieDb                                                    Inside: ['OMDb']  Values: 
[?] meta_new: {
      'https://image.tmdb.org/t/p/original/2FRSGUsD571KxR7Rne60PKZ99EW.jpg': ('TheMovieDb/poster/2FRSGUsD571KxR7Rne60PKZ99EW.jpg', 17, None)}
    meta_old: [
      ]
[x] posters                  ( 1)  Sources: tvdb4, (TheMovieDb), TheTVDB, AniList, FanartTV, AniDB        Inside: '['TheMovieDb', 'OMDb']'  Value: '{'https://image.tmdb.org/t/p/original/2FRSGUsD571KxR7Rne60PKZ99EW.jpg': ('TheMovieDb/poster/2FRSGUsD571KxR7Rne60PKZ99EW.jpg', 17, None)}'
[x] art                      ( 1)  Sources: TheTVDB, (TheMovieDb), FanartTV                               Inside: '['TheMovieDb']'  Value: '{'https://image.tmdb.org/t/p/original/178rrQRHM4RnswUMXjC1JB6ilqU.jpg': ('TheMovieDb/artwork/178rrQRHM4RnswUMXjC1JB6ilqU.jpg', 34, 'https://image.tmdb.org/t/p/w300/178rrQRHM4RnswUMXjC1JB6ilqU.jpg')}'
RainOne commented 3 years ago

I added as you requested (I imagine it's the T-EM 'title"?) image

Still no Title found on the logs: Kamen Rider Shin (1992) [tmdb-194777].agent-search.log Kamen Rider Shin (1992) [tmdb-194777].agent-update.log

I also tried adding to the title_sort and original_title, but they also didn't provide a title in the logs.

ZeroQI commented 3 years ago

It is moving forward, metadata in update log looks great apart from the title which exists in 'ja' only but not found: https://api.tmdb.org/3/movie/194777?api_key=7f4a0bd0bd3315bb832e17feda70b5cd&append_to_response=releases,credits,trailers,external_ids&language=en

language_series : ['en', 'ja'] is set so it needs code amended to get the 'ja' title from TMDB.

RainOne commented 3 years ago

I didn't really understand the problem: The api link you pasted does have both the "original title" and the "title", it's just way down below (after languages):

"runtime":88,"spoken_languages":[{"english_name":"Japanese","iso_639_1":"ja","name":"日本語"}],"status":"Released","tagline":"","title":"Shin Kamen Rider: Prologue","video":false,"vote_average":4.4,"vote_count":5,"releases":{"countries":[{"certification":"G","iso_3166_1":"JP","primary":false,"release_date":"1992-02-20"},{"certification":"","iso_3166_1":"JP","primary":false,"release_date":"2005-12-24"}]},"credits":{"cast":[{"adult":false,"gender":2,"id":1175053,"known_for_department":"Acting","name":"Shin Ishikawa","original_name":"Shin Ishikawa","popularity":0.608,"profile_path":null,

I don't know if you mean "title from ja" as in the original title in japanese (which wouldn't be useful) or if i'm just misunderstanding you.

Probably the latter.

ZeroQI commented 3 years ago

i missed the "title":"Shin Kamen Rider: Prologue" if a movie it's called 'title', if a series it's called 'name', and here it's a movie in a series library...

https://github.com/ZeroQI/Hama.bundle/blob/master/Contents/Code/TheMovieDb.py line 43 change Log.Info("[ ] title: {}" .format(SaveDict( Dict(json, 'title' if mode=="movie" else 'name'), TheMovieDb_dict, 'title' ))) into Log.Info("[ ] title: {}" .format(SaveDict( Dict(json, 'title') or Dict(json, 'name'), TheMovieDb_dict, 'title' )))

Submitted 2 fixed to include in master code Please attach just "Kamen Rider Shin (1992) [tmdb-194777].agent-update.log" if there are still issues

RainOne commented 3 years ago

yeah, that worked.

A couple more things:

OMDb.py is looking for "title" on line 29, it should be "Title", as the OMDb api request calls it (with a capital T): {"Title":"Shin Kamen Rider","Year":"1992","Rated":"N/A","Released":"20 Feb 1992","Runtime":"90 min","Genre":"Action, Adventure, Drama, Horror, Sci-Fi, Thriller","Director":"Makoto Ishikawa, Makoto Tsuji","Writer":"Shotaro Ishinomori, Hayato Miyashita, Joe Onodera","Actors":"Kohisa Ishikawa, Yumi Nomura, Joe Onodera, Akira Ishihama","Plot":"In Shotaro Ishinomori's own deconstructionist entry in his famous \"Masked Rider\" series, young Shin Kazamatsuri is transformed into a powerful grasshopper mutant by a scientific institute known only as the \"Foundation,\" which targets him.","Language":"Japanese, English","Country":"Japan","Awards":"N/A","Poster":"https://m.media-amazon.com/images/M/MV5BOTZhNWM5YTEtYzNlNi00YzdlLWJkYmMtOGUzMzBhYjA5NWUyXkEyXkFqcGdeQXVyNjUwMTE5MjI@._V1_SX300.jpg","Ratings":[{"Source":"Internet Movie Database","Value":"5.7/10"}],"Metascore":"N/A","imdbRating":"5.7","imdbVotes":"113","imdbID":"tt0157028","Type":"movie","DVD":"N/A","BoxOffice":"N/A","Production":"N/A","Website":"N/A","Response":"True"}

TMDb.py doesn't grab "originally_available_at" on line 49 because it's looking for "first_air_date", while TMDb offers that value as "release_date":"1992-02-20" (for movies, I imagine?). I added an "or dict(json, 'release_date')", on line 49 of TheMovieDb.py and that is also scrapped now.

I was gonna say that Summary and Year were not being added to the plex interface, but I noticed that TheMovieDB and OMDb were not listed on Hama as sources for that information, so I added them and now they show up.

I'll keep looking if there's any more interesting things to add - how should I suggest them? Just paste them in here?

ZeroQI commented 3 years ago

You can do a pull request or add them here Can't find in ASS or HAMA readme where it says deprecated... Will close that in the meantime to have the impression the number of incidents goes down...