Utilize AniDb Scrapping to DB matching

salomj commented 3 weeks ago

Description

Summary

Utilize an AniDB Scrapper library to automatically pull AniDB matching, rather than have to enter everything manually.

Details

Added some support for TMDb/tv
Stopped using "private" methods outside of convert.py
Switched X_to_anidb and anidb_to_X to consistently be 1 to Many.

Related Changes

Type of Change

[x] New feature (non-breaking change which adds functionality)

Checklist

[x] My code was submitted to the nightly branch of the repository.

salomj commented 3 weeks ago

Coding part of these changes are done, in process of testing.

meisnate12 commented 3 weeks ago

the linkage to that github json should just be put into https://github.com/Kometa-Team/Anime-IDs

meisnate12 commented 3 weeks ago

Also do have more information on where this data is coming from? Did you just find this repo or are you involved with it?

https://github.com/notseteve/AnimeAggregations

meisnate12 commented 3 weeks ago

Kometa also uses TVDb IDs as the main source for show ids not TMDb anytime i get TMDb Show IDs theyre just converted to TVDb IDs

salomj commented 3 weeks ago

the linkage to that github json should just be put into https://github.com/Kometa-Team/Anime-IDs This doesn't work for many many anime, for two different reasons. 1) This json file does not include TMDb/tv nor a way to distinguish between TMDb/movie and TMDb/tv. So I am unsure how you'd like me to modify this file. Or are you saying you'd like me to make changes to regularly import the JSON to that library?

Also do have more information on where this data is coming from? Did you just find this repo or are you involved with it? https://github.com/notseteve/AnimeAggregations 1) Data is straight from AniDB, it gets aggregated every 15 days.

Kometa also uses TVDb IDs as the main source for show ids not TMDb anytime I get TMDb Show IDs theyre just converted to TVDb IDs 1) Yes, and that's a problem a lot of AniDb doesn't have TVDb for one reason or another, but nearly all of them have TMDb.

salomj commented 3 weeks ago

I have ran this against my local database, and I went from a little over 1200 unmatched items to 83.

Arial-Z commented 3 weeks ago

Not sure why switching from the most trusted source for animes mappings (anime-lists) used by all animes agent (HAMA, anidb.net, MAL agent ...) to this repo is interesting. A simple search on the list (https://raw.githubusercontent.com/notseteve/AnimeAggregations/main/aggregate/AnimeToExternal.json) return 1056 tmdb tv entry, animes-lists got over 6600 tvdb id entry (including seasons and special mappings), and if looking only for season 1 (what you're doing here i think), it's got 3800 entry. I found 476 tmdb movie entry, animes-lists got 1600 entry for movies with an imdbid. Also what happen to seasons and episodes offsets, kometa currently don't use them, but they are needed to work on the seasons (that all other agent use and maybe kometa one day)

salomj commented 3 weeks ago

Not sure why switching from the most trusted source for animes mappings (anime-lists) used by all animes agent (HAMA, anidb.net, MAL agent ...) to this repo is interesting. A simple search on the list (https://raw.githubusercontent.com/notseteve/AnimeAggregations/main/aggregate/AnimeToExternal.json) return 1056 tmdb tv entry, animes-lists got over 6600 tvdb id entry (including seasons and special mappings), and if looking only for season 1 (what you're doing here i think), it's got 3800 entry. I found 476 tmdb movie entry, animes-lists got 1600 entry for movies with an imdbid. Also what happen to seasons and episodes offsets, kometa currently don't use them, but they are needed to work on the seasons (that all other agent use and maybe kometa one day)

Values are taken straight from AniDB (Values come from: https://anidb.net/anime/17906/resource and are compiled into a single file for easier matching.)
Change also allows matching for tmdb/tv, which the hand-crafted list does not handle tmdb/tv
If you check the code, the hand-craft list takes priority over the automated list. So it is not a list "switch", it is a list augmentation.
The only thing that uses episode and season offset seems to be TVDb, and AniDb does not directly support TVDb.
Priority for shows is still TVDb, IMDb, and now TMDb/tv

Arial-Z commented 3 weeks ago

As Sohjiro said i think the best would be to add them to kometa animes-id repo, with key like tmdb_tv and tmdb_movie so they could be used as fallback if tvdb or imdb fail.

salomj commented 3 weeks ago

As Sohjiro said i think the best would be to add them to kometa animes-id repo, with key like tmdb_tv and tmdb_movie so they could be used as fallback if tvdb or imdb fail.

That should be easy enough.

salomj commented 3 weeks ago

As Sohjiro said i think the best would be to add them to kometa animes-id repo, with key like tmdb_tv and tmdb_movie so they could be used as fallback if tvdb or imdb fail.

https://github.com/Kometa-Team/Anime-IDs/pull/1

salomj commented 2 weeks ago

@meisnate12 Why was this closed?

meisnate12 commented 1 week ago

its been added to Kometa in a different way

Kometa-Team / Kometa