search

torch2424 / vaporBoy

Gameboy / Gameboy Color Emulator PWA built with Preact. ⚛️ Powered by wasmBoy. 🎮Themed with VaporWave. 🌴🐬
https://vaporboy.net/
GNU Affero General Public License v3.0
252 stars 21 forks source link

Build a ROM Scraper #2

Closed torch2424 closed 6 years ago

torch2424 commented 6 years ago

Would be nice for box art and things: https://github.com/RetroPie/RetroPie-Setup/wiki/scraper

torch2424 commented 6 years ago

So looking at schelphs scraper it uses a 4MB csv file as it's DB, thus this would be impractical on a client. And it would require reverse engineering hw they are hashing what part of the ROM, since it does not seem to be a standard MD5 or SHA1 of the file.

I think Our best bet for now is just to pop up a modal, that allows the user to manually add info, or search using a simple search box, and then we can just save the id. Giant Bomb seems to have the best/most reasonable API with 2,000 requests per hour. Though the games db is unlimited, and idb doesn't support CORs for free, or unlimited requests.

torch2424 commented 6 years ago

Apparently games DB is all messed up right now, and can't find an api key:

https://www.reddit.com/r/RetroPie/comments/8wpgfp/thegamesdb_new_api/

torch2424 commented 6 years ago

Found how to get API Keys, but it is 1000 requests per month :p

https://forums.thegamesdb.net/viewtopic.php?f=10&t=111

torch2424 commented 6 years ago

Probably have to go with GIant Bomb: https://www.giantbomb.com/api/

This modern app also did: https://github.com/irwintsay/VGDB

torch2424 commented 6 years ago

Gonna do IGDB. And just set up keys in the Caddy to do rewrites on the GET request to get the proper responses.

torch2424 commented 6 years ago

IGDB Platform Ids:

GB: 33, GBC: 22, GBA: 24

Example IGDB request: https://vaporboy.net/scrape/igdb/games/?search=Links%20Awakening&fields=name,cover,platforms,first_release_date&filter[release_dates.platform][any]=22,33&order=popularity:desc

Need to add user_key in headers and stuff though, and will be proxied through

torch2424 commented 6 years ago

IGDB Doesn't support search and both filtering by platform 😢

So going with Giant Bomb.

Giant Bomb Platform Ids:

GB: 3, GBC: 57

Example Request: https://www.giantbomb.com/api/games/?api_key=[API_KEY_HERE]&format=json&filter=name:zelda,platforms:57|3&field_list=name,image,original_release_date,platforms

torch2424 commented 6 years ago

Made endpoint in Caddyfile for this:

https://vaporboy.net/scrape/giantbomb?format=json&filter=name:zelda,platforms:57|3&field_list=name,image,original_release_date,platforms

https://github.com/torch2424/dotFiles/blob/master/.files_caddy/aaronTheDev-apps/Caddyfile#L23

Simply need to set up in app!