[Feature] Support additional scrapers (ScreenScraper, TGDB, SteamGridDB)

XargonWan commented 1 year ago

Please add ScreenScraper as additional scraper as it seems to be more complete of IGDB on retro gaming.

Describe the solution you'd like There could be a list in the options (similarly to Plex agents) to select which scarapers the user wants, and the config (for an ulimited/ish scraping ScreenScraper needs an user account). Romm could scrape with the first scraper in the list and then fallback on the other ones if no match (or match but no media) is found.

Screenscraper is providing more than just cover, but wheel, descriptions, logo and more.

Describe alternatives you've considered I don't know any other scraper services, but maybe someone does.

zurdi15 commented 1 year ago

Yeah seems like a good idea to integrate more than just IGDB and let the user decide. I need to work a little bit more in the code base to allow more database integrations easily. When I reach a better quality code ScreenScraper will be integrated.

victor10100111001 commented 1 year ago

I think this might be a good choice. Especially since it seems like Zurdi can request an api key that everyone running this project can use. From their website (translated):

Who can use the API?
The ScreenScraper API can only be integrated into fully free and distributed applications, or, if not, with the prior authorisation and conditions dictated by the ScreenScraper team. Failure to comply with this rule may be the subject of a breakdown of account, see any legal proceedings.
If you are a developer and want to integrate our API, contact us via the forum to present your software and obtain your usernames and passwords to provide to the API to validate your rights to use it

This would remove some barrier of entry

binarygeek119 commented 1 year ago

theres also https://thegamesdb.net/ and https://gamesdb.launchbox-app.com/ (althought in not sure is launchbox lets you use their db openly)

gantoine commented 1 year ago

Alright I've gone through the more prominent DBs/API out there and collected some notes. The ones I think are worth exploring are marked with a ⭐ . Please let me know if I missed any and I'll add them to the list.

IGBD: ~250K games, ~190 platforms ⭐

Pros

Free (owned and run by Twitch)
Extensive list of API endpoints that can get pretty specific
Great rate limits (4 requests/second limit with no global max)
Protobuf support

Cons

Greater focus on modern games
Low number of art assets

TheGamesDB: ~76K games, ~150 platforms ⭐

It seems the preferred way to interact with it is to mass-download the database and use the API async to keep your local copy up-to-date.

Pros

Focused on retro games
JSON dump can be downloaded from CDN

Cons

Local copy of DB must be synced
Sparse metadata and related assets

MobyGames: ~330K games, ~320 platforms ⭐

Pros

Free for personal use
Sizeable collection of games and platforms
Decent collection of crowdsourced assets

Cons

Low API request limits (1 requests/second, max 360 requests/hour)
API only supports game and platform queries (incl. art assets)

As per the API TOS: "MobyGames must be credited wherever data from the API is used. Typically, a link to the MobyGames page for the game will suffice. "

RAWG: ~350K games, ~50 platforms

Pros

Free for personal use
Rich metadata on par with IGDB
Support for DLCs, achievements, trailers, and more

Cons

20K API calls/month
Some endpoints locked behind paid plans
Poor collection of art assets

As per the TOS: "Free for personal use as long as you attribute RAWG as the source of the data and/or images and add an active hyperlink from every page where the data of RAWG is used."

GiantBomb: ~80K games, ~190 platforms

Pros

Decent rate limits (450 requests/15 minutes)
Extensive API endpoints
Wiki-style entries for most popular games

Cons

Paid only (~5$/month)
Mediocre collection of art assets

ScreenScraper.fr: ~300K games, ~105 platforms ⭐

Pros

Extensive collection of art assets
Free for personal use and open-source projects
Decent API footprint

Cons

Low API rates (20K requests/24 hours for registered users, 3K otherwise)

As per the TOS: "L'API ScreenScraper ne peut être intégré que dans les applications entièrement gratuites et distribuées, ou, dans le cas contraire, avec l'autorisation préalable et les conditions dictées par l'équipe de ScreenScraper."

SteamGridDB: ? games, ? platforms

Seems to be mostly focused on art assets based around their concept of "grids", collections of assets uploaded by users.

Pros

Prebuilt python library
Asset collection split by logos, hero images and icons
~290K Grids, ~75K heroes, ~75K logos, ~35K icons

Cons

Unclear API limits
No game metadata provided
More focused on modern games

GameTDB: ? games, 7 platforms

Only the following platforms are currently supported: Wii, WiiU, DS, 3DS, Switch, PS3, Gamecube

Pros

Solid collection of box art assets (for supported platforms)

Cons

Limited platforms support
No API/undocumented API

Casuallynoted commented 1 year ago

Possibly the ability to import Launchbox's metadata xml files? Not sure how practical that is though

https://forums.launchbox-app.com/topic/52709-launchbox-db-api/

They definitely seem to have their stuff together when it comes to resources, I can say that. The submission process with IGDB has been a time haha

parasiteoflife commented 1 year ago

Yes, please, implement other scrapers, giving my phone number to twitch is something I wont ever do and unfortunately for us is the only way to enable 2FA on their site.

Emporea commented 9 months ago

@zurdi15 Ok thanks for pointing out this existing issue. I searched for IGDB / alternative, but this issue didn't pop up. Can you estimate how long it will take to implement other sources? What needs to be achieved first in your opinion so this could happen?

Thanks for all your work on this project.

maxexcloo commented 8 months ago

Screenscraper is also nice as most frontends (such as ES-DE) use it also so it would be easier to potentially match/share data.

LeeTeng2001 commented 2 weeks ago

bump

rommapp / romm