search

esi / esi-issues

Issue tracking and feature requests for ESI
https://esi.evetech.net/
208 stars 23 forks source link

Error 500 "Undefined 429 response. Original message: Too many errors.You have been temporarily throttled." #1227

Open JimmyRoth opened 4 years ago

JimmyRoth commented 4 years ago

Bug

When requesting a character ID, or Alliance ID endpoint, I receive an unexpected 500 error, stating I have been throttled. However, X-Esi-Error-Limit-Remain claims I have 99 errors left. More request samples available upon request.

Request

GET /latest/alliances/822344869/?datasource=tranquility&language=en-us

Response

Status Code

500

Headers

'Date': 'Fri, 14 Aug 2020 11:26:05 GMT',
'X-Esi-Error-Limit-Remain': '77',
'X-Esi-Error-Limit-Reset': '55',
'X-Esi-Request-Id': '98a1c20b-c8e9-44d3-8a57-ea2ae2604bd2'

Body

Please provide the response body, feel free to scrub any opsec details you wish. E.g.:

{
    "error": "Undefined 429 response. Original message: Too many errors.You have been temporarily throttled"
}

Expected

200

Checklist

Check all boxes that apply to this issue:

cvweiss commented 4 years ago

@JimmyRoth I've solved this issue on my server. I had several threads making calls to the ESI server and each of them would "sleep" to slow down the number of requests. As it turns out this was the problem.

The requests were finishing, but my code wasn't completing, so the requests were bottling up on the server, and it appears that too many calls were happening at once. I solved the issue, rather than sleep for a 1/10th of a second, by sleeping for 1/1000th a second 100 times and allowing the requests to "tick" between those sleeps(I use PHP's Guzzler). With many calls happening you could inadvertently have too many requests in flight, causing the 429 throttling error.

If you're not using a language that can async/await properly, this may be similar to your issue. Something to look into.

jamesrobb commented 3 years ago

@JimmyRoth Can you confirm that you aren't experiencing the situation outlined by @cvweiss?

mateuszkrasucki commented 3 years ago
  1. The reason is you're hitting another, internal error limiter that protects game server itself. It's quite stringent, it allows you to cause only 10 errors affecting game server of those per minute.
  2. To be perfectly honest, I suspect that it's quite possible that part of the reason you are hitting those because of other errors on our side. Please report them, we promise to find bandwidth to fix them. Nevertheless, When you encounter those you should enhance your calm and backoff requests a bit.
  3. It's definitely a bug that we're returning 500 with cryptic message indicating 429 which is not defined in Swagger spec. I'm opening internal issue (TTC-3809) to handle this and consider other changes to this internal error rate limiter.
cvweiss commented 3 years ago

To be perfectly honest, I suspect that it's quite possible that part of the reason you are hitting those because of other errors on our side. Please report them, we promise to find bandwidth to fix them.

@mateuszkrasucki We aren't given a good message to report back, should we report the timestamp, url, etc?

Is there a limit to the number of requests that can reach the game server per minute? Before I drastically reduced the number of calls my code makes, I've noticed that the error occurs between the :35 and :45 second marks within a minute (without errors) extremely consistently, so I must be hitting some sort of rate limiting to the number of calls per minute. In the next minute, at :00 second mark, I stop receiving the 500 errors.

mateuszkrasucki commented 3 years ago

@cvweiss As pointed out, there's no limit to number of calls you're making. There's a limit to number of errors you're causing at the game server level, and it's 10 (not 100 as with ESI's error rate limiter that headers refer to). ^ I believe that this weird :35 - :45 second mark for 500s to start to occur is just because of you hitting that error level. This can be confirmed if you check and confirm that 500s that you're getting are the ones with "429 (...)" message.

warlof commented 3 years ago

This is also valid for the following endpoints :