Upgrade the crawler to use the "games" API resource, which can return 100 games at a time.
The API reports that there are 49,361 games currently on Giant Bomb, so we would be able to crawl the entire database in 494 requests. At a casual 1 request every 5 minutes this would take two days, which is a fantastic improvement over the current speed of database growth.
Clidus
commented
8 years ago
Upgrade the crawler to use the "games" API resource, which can return 100 games at a time.
The API reports that there are 49,361 games currently on Giant Bomb, so we would be able to crawl the entire database in 494 requests. At a casual 1 request every 5 minutes this would take two days, which is a fantastic improvement over the current speed of database growth.