sselph / scraper

A scraper for EmulationStation written in Go using hashing
MIT License
449 stars 88 forks source link

Update to support TheGamesDB v2 api #252

Closed J-Swift closed 5 years ago

J-Swift commented 5 years ago

This should address #230, #237, #244, and #245

TheGamesDB updated to a new api a while back and v2 is pretty different:

This is the initial conversion of the existing code to support the new version. It relies on a "temporary" swagger client I generated here, but there is a decent amount of hand cleanup to get to that point. See this issue I opened for more info. Once that is cleaned up, addressing future API updates should be fairly straightforward.

The new API supports batching which is great, but not currently implemented/supported in the codebase. I might tackle that next, as the API limits are not very generous (1500 req/month) and so batching is the only way a large initial scrape is going to work. I currently am making 4 + 2 * {number of games} API requests per run of the scraper.

Note I had to learn golang for this and so let me know if there are any issues with how I've coded it up.

sselph commented 5 years ago

Thanks for the hard work. I added some comments and questions.

J-Swift commented 5 years ago

Updated the quick ones, commented on the others. I can still make those style changes you mentioned and add an API key if you provide one, otherwise I defer to you on merging this in.

sselph commented 5 years ago

Thanks!