search

trakt / Plex-Trakt-Scrobbler

Add what you are watching on Plex to trakt.tv
1.45k stars 167 forks source link

OpenEntityMap - Github rate limit #511

Open RickDB opened 5 years ago

RickDB commented 5 years ago

OpenEntityMap (anidb / tvdb lists) no longer lists releases on their repo according to the below error but Trakt still tries to retrieve it from there on startup and periodically resulting in this error:

2018-11-08 00:43:05,292 - requests.packages.urllib3.connectionpool (2f468) : DEBUG (requests.packages.urllib3.connectionpool:401) - https://api.github.com:443 "GET /repos/OpenEntityMap/oem-database-anidb-tvdb/releases/latest HTTP/1.1" 403 None

This means Trakt is no longer able match any Anime as it has no reference points for Anidb that way.

RickDB commented 5 years ago

When debugging in Firefox I noticed Github's API is rate limiting my IP:

"API rate limit exceeded for x.x.x.x. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)"

Any chance you can add authentication options to Trakt for the Github syncing part (OATH token)?

Xalaxis commented 5 years ago

@RickDB Whilst the releases of OEM are currently somewhat out of date (which I've mentioned to @fuzeman elsewhere) you shouldn't be getting a 403 for that request given there are some at https://github.com/OpenEntityMap/oem-database-anidb-tvdb/releases. You should be getting a 200 like I got a couple of days ago:

2018-11-04 21:00:48,669 - requests.packages.urllib3.connectionpool (7fee0fcbf700) : DEBUG (requests.packages.urllib3.connectionpool:401) - https://api.github.com:443 "GET /repos/OpenEntityMap/oem-database-anidb-tvdb/releases/latest HTTP/1.1" 200 None 2018-11-04 21:00:48,674 - oem_client_provider_release.incremental (7fee0fcbf700) : INFO (oem_client_provider_release.incremental:124) - [anidb -> tvdb] Collection is up to date (v1.18.10)

Have you figured this out and think the rate limiting is the reason for the 403s? Good investigation 🙂 .

RickDB commented 5 years ago

Tested it with VPN server and returns proper 200 status messages that way for those OEM API requests, do have some other projects that use the Github API so it might be the combined requests are locking me out (hard to pinpoint atm). Does work that way at least and now seeing the anime shows on Trakt collections as well as scrobble events, not an ideal fix but routing api.github.com to a VPN server for the time being.

About it being out of date just noticed that as well with recently aired shows like SSSS.Gridman while that one is listed on the Fuzeman list, is this because the OEM lists are manually packed and put in repo releases? If so understand why though considering the sheer amount of separate anidb xml files in the OEM repo which Github will handle poorly if at all.

fuzeman commented 5 years ago

I'm working on a better solution for the distribution of mappings (for another project), but will likely port it over to here once it's ready.

Any chance you can add authentication options to Trakt for the Github syncing part (OATH token)?

This would be a bit complicated to implement as we can't easily support the usual OAuth flow, and in the future the GitHub API won't be required at all.

About it being out of date just noticed that as well with recently aired shows like SSSS.Gridman while that one is listed on the Fuzeman list, is this because the OEM lists are manually packed and put in repo releases?

I've published a new update that includes all the recent changes from ScudLee/anime-lists, the build server has also been switched to automatically publish updates every Sunday ~2PM GMT.

RickDB commented 5 years ago

Good to hear 👍 Want me to leave the issue open as a reminder or should I close it?