search

SpaceTradersAPI / api-docs

The API documentation for the SpaceTraders API
143 stars 35 forks source link

Rate limit response is sometimes different #49

Closed dbeckwith closed 1 year ago

dbeckwith commented 1 year ago

I was trying to write a simple loop to fetch every page of systems using the List Systems endpoint, which takes a while. I get normal rate limit 429 responses for a while, and have my code wait for the time provided in the Retry-After header. But after a while (almost 150 pages), I get a different kind of rate limit response. It doesn't have a Retry-After header, and the response body isn't JSON, but what looks like HTML:

<!doctype html><meta charset="utf-8"><meta name=viewport content="width=device-width, initial-scale=1"><title>429</title>429 Too Many Requests

It almost looks like something else is rate limiting my requests, like a proxy or something.

Also, a bit off-topic, but is there a better way to fetch the full list of systems? Seems like the documented page size limit is only 20, but according to the docs there are 5,000 systems, which would be 250 pages, which takes quite a while with the rate-limiting. Maybe I shouldn't be fetching every single system? But then how can I do things like discover nearby systems?

dbeckwith commented 1 year ago

39 would actually be a great solution for my last question about discovering nearby systems

tyrope commented 1 year ago

Unrelated to the issue, but for your purpose, you're looking for /v2/systems.json (though this is rumoured to go away at some point!)

space-admiral commented 1 year ago

Usually that will happen if you are consistently going past the API rate limit, we have an added layer for DDoS protection, and we can't control the headers. If you stay within the defined limit of 2 requests a second, the headers should be visible.

Let me know though if it still doesn't work. Thanks for reporting this! We also added a section in the docs for this rate limiter.