search

Clidus / gwl

Video game collection, wishlist and backlog tracker.
https://www.gamingwithlemons.com/
MIT License
18 stars 6 forks source link

Investigate alternate datasources #75

Open Clidus opened 8 years ago

Clidus commented 8 years ago

I've had the concern since the beginning that relying on a single API might not be a good idea. As much as I love Giant Bomb, they are a commercial entity owned by CBSi and they haven't updated the API in years. They also recently introduced a bunch of rate limits and IP blocks which we haven't fallen foul of yet, but introduces risks to the long term investment of the project.

It would be a good idea to investigate alternative datasources, not only to have a fall back, but also to acquire more data. I would love to have barcode information for example.

Delicious Library uses Amazon data apparently. I imagine this will be better for more recent games, than old games. Probably a good place to start.

Steam's API would be another good source of information, and would allow a collection import function.

The big challenge here is identifying the same game across multiple datasources and not listing it multiple times.

crunchprank commented 8 years ago

I'm sure you've heard of TheGamesDB but just in case you hadn't, I thought I'd throw it out there. From all your issue responses today, looks like Giant Bomb is really starting to enforce some strict API rules which is a bummer.

Clidus commented 8 years ago

I've seen that before, but forgot what it was called. Thanks! I'll check out what it offers.

It's understandable, Giant Bomb has been hit with a lot of abuse (sites stealing content and reposting it as their own) and attacks, and we might be able to live quite happily under the rate limits (I don't believe the site's traffic is anywhere near being a problem yet). Even if we can't, it seems like exceptions can be made for people using the API as it was intended.

I would like the project to stand on its own two feats a little however, and having alternative sources of information seems like a safe bet incase one of those sources goes away. It will also allow us to fill in missing information that GB doesn't provide, such as barcodes (which would make a scanning app possible!).

Clidus commented 8 years ago

Seems like the site is currently being blocked by Giant Bomb, proving my concerns...

Currently trying to get in contact with someone to get us unblocked.

Clidus commented 8 years ago

I got API access restored, but I'm fairly unhappy that the block was put in place. Im going to advance by plans to decouple the site from Giant Bomb and move to alternate data sources.

Also, this restriction on the API seems utterly insane?

capture

Non-businesses do occasionally use EC2...

crunchprank commented 8 years ago

That reasoning makes no sense...

iedame commented 8 years ago

Wow, that's really worrisome.

+1 for alternate datasources. the ones mentioned here are a good start.