Add more scrapers

[parasiteoflife]

Is your feature request related to a problem? Please describe. Currently only IGDB is supported, in order to get an API key from there the account must have 2FA enabled which needs a phone number. For several reasons, including privacy, giving our phone number to services is a big no-no, even more so when there are alternatives that don't require that.

Describe the solution you'd like Support for additional scrapers.

Describe alternatives you've considered For a good list you can read here the great summary done by @gantoine which I'll quote:

Alright I've gone through the more prominent DBs/API out there and collected some notes. The ones I think are worth exploring are marked with a ⭐ . Please let me know if I missed any and I'll add them to the list.

IGBD: ~250K games, ~190 platforms ⭐

Pros

* Free (owned and run by Twitch)

* Extensive list of API endpoints that can get pretty specific

* Great rate limits (4 requests/second limit with no global max)

* Protobuf support

Cons

* Greater focus on modern games

* Low number of art assets

TheGamesDB: ~76K games, ~150 platforms

It seems the preferred way to interact with it is to mass-download the database and use the API async to keep your local copy up-to-date.

Pros

* Focused on retro games

Cons

* API key has to be requested and approved by staff

* Local copy of DB must be synced

* Sparse metadata and related assets

MobyGames: ~330K games, ~320 platforms ⭐

Pros

* Free for personal use

* Sizeable collection of games and platforms

* Decent collection of crowdsourced assets

Cons

* Low API request limits (1 requests/second, max 360 requests/hour)

* API only supports game and platform queries (incl. art assets)

As per the API TOS: "MobyGames must be credited wherever data from the API is used. Typically, a link to the MobyGames page for the game will suffice. "

RAWG: ~350K games, ~50 platforms ⭐

Pros

* Free for personal use

* Rich metadata on par with IGDB

* Support for DLCs, achievements, trailers, and more

Cons

* Unclear API limits

* Some endpoints locked behind paid plans

* Poor collection of art assets

As per the TOS: "Free for personal use as long as you attribute RAWG as the source of the data and/or images and add an active hyperlink from every page where the data of RAWG is used."

GiantBomb: ~80K games, ~190 platforms

Pros

* Decent rate limits (450 requests/15 minutes)

* Extensive API endpoints

* Wiki-style entries for most popular games

Cons

* **Paid only (~5$/month)**

* Mediocre collection of art assets

ScreenScraper.fr: ~300K games, ~105 platforms ⭐

Pros

* **Extensive** collection of art assets

* Free for personal use and open-source projects

* Decent API footprint

Cons

* Low API rates (20K requests/24 hours for registered users, 3K otherwise)

As per the TOS: "L'API ScreenScraper ne peut être intégré que dans les applications entièrement gratuites et distribuées, ou, dans le cas contraire, avec l'autorisation préalable et les conditions dictées par l'équipe de ScreenScraper."

SteamGridDB: ? games, ? platforms

Seems to be mostly focused on art assets based around their concept of "grids", collections of assets uploaded by users.

Pros

* Prebuilt python library

* Asset collection split by logos, hero images and icons

* ~290K Grids, ~75K heroes, ~75K logos, ~35K icons

Cons

* Unclear API limits

* No game metadata provided

* More focused on modern games

GameTDB: ? games, 7 platforms

Only the following platforms are currently supported: Wii, WiiU, DS, 3DS, Switch, PS3, Gamecube

Pros

* Solid collection of box art assets (for supported platforms)

Cons

* Limited platforms support

* No API/undocumented API

[michael-j-green]

Oh wow... I was unaware of most of those. Thankyou for that.

I'll have to see how these fall in line with my plans, as API sign up has been a pain point I've wanted to erase for users since the day I started this project.

I have another project that I'll be getting back into now that auth in Gaseous is nearly done - it's called Hasheous. The idea of this project is to proxy and cache metadata lookup to IGDB so the end user doesn't have to get their own API key. It would also store hash to IGDB game mappings (editable by the community - hence the auth part) so that end users wouldn't need to store DAT's locally either.

Some of these sources do worry me in regards to the Hasheous project when they say "for personal use only", because while Gaseous will always be free, I have toyed with the idea of having a donation system for Hasheous (this would NEVER be a requirement for pulling data - I want it to be free forever) to cover hosting costs. I don't know if this could then be interpreted as commercial use.

Hope this makes sense! :)