search

GwentAPI / gwentapi

The code for GwentAPI
https://gwentapi.com
BSD 3-Clause "New" or "Revised" License
47 stars 1 forks source link

Question: Using a JSON of the database #29

Closed Soreine closed 6 years ago

Soreine commented 7 years ago

TL;DR:

What do you advise to retrieve a dump of the all the cards data ?

Context:

We are developing a Chrome extension to find and preview cards on any web page. https://github.com/Soreine/hyper-gwent/

For now, we are generating a JSON of the card list, taken from GwentAPI. We would like to avoid overloading GwentAPI, and rather have this data packaged with the extension.

Recently, our script that fetches all the cards data from the API, before packaging the extension, has been encountering errors. It happens on random cards: request to https://api.gwentapi.com/v0/cards/Uh25y0wnUhGokjVR5Jnu7A failed, reason: read ECONNRESET. Either this is an issue on your end, or just some rate limiting, since adding a delay between each request works. It would be much simpler for us to use your data source directly. Can we rely on this ? https://github.com/GwentAPI/data/blob/master/input/latest.jsonl

zhouzi commented 7 years ago

May be worth noting that we use the gwent-api-client to load all the cards with a cards.map(load). Which means there are ~305 requests flying at the same time. I can add a method to the gwent-api-client to bulk load resources from the API and limit to ~10 requests at a time.

It's not a fix for this particular issue but would let the API breath a bit.

P.S.: I'm working with Soreine on hyper-gwent.

Tri125 commented 7 years ago

I don't recommend using the data repo (or latest.jsonl for that matter). I actually forgot to mark it as deprecated. I used to use a web crawler to generate latest.jsonl which I was then using to seed the database. As of July 14th this is no longer the case. The crawler is no longer being maintained and as a consequence, the dump will no longer be part of GwentAPI releases.

I would advice that you spam the API to get what you need. The alternative would be that you use the crawler but you will just end up spamming gwentify instead of my service anyways.

10 requests at a time is awful. If you can find the exact limit, reproduce the issue, and send me logs (with timestamp) that would help be figure out what exactly is going on. I didn't use any rate limiter on the software or the machine that I'm operating on. However, I'm using Cloudflare which could think that it was spam and decided to block you. It could also be a problem with your software.

Tri125 commented 6 years ago

Closing this as there wasn't any comment for a certain time now.