knightcrawler-stremio / knightcrawler

A selfhosted Stremio addon
Apache License 2.0
238 stars 39 forks source link

Create service to populate Imdb data collection within mongo #99

Closed iPromKnight closed 4 months ago

iPromKnight commented 4 months ago

We can use this collection as an alternative source to lookup imdb ids, which would be executed before name_to_imdb is called in the consumer.

As discussed in Discord, there is currently an issue where rate limiting may cause an IMDB id lookup to fail - this PR will address that by having a service that periodically updates and caches all possible IMDB ids from imdb's datasets.

iPromKnight commented 4 months ago

Total import size:

CleanShot 2024-02-27 at 22 49 00

iPromKnight commented 4 months ago

import in batches of 50k takes roughly 15 mins

Gabisonfire commented 4 months ago

awesome, very good idea