Excessive requests

[Novynn]

Hello,

It's come to our attention that users of this tool are sending excessive requests to pathofexile.com resulting in their accounts being restricted. Please implement some sort of incremental back-off on failed requests (4xx's) with an automatic stop after a number of failures.

Additionally this tool should send an identifiable user-agent when making requests so that we can report these issues to you quicker without having to investigate individual cases.

Thanks, Rory.

Grinding Gear Games.

[thisismydesign]

Hey Rory,

Thanks for letting us know. Adding a user agent shouldn't be a problem.

We are limiting the requests based on your headers as explained here. We apply this both to WS connections and HTTP requests. We don't retry HTTP failed requests. We will attempt to reconnect to WS but always respecting the rate limit.

We've had a few issues with this:

• it seems WS connections should also be part of the rate limit, even though we couldn't find any specification for this
• bursting requests don't seem to be allowed, even within the bounds of the rate limit - we implemented an arbitrary 1 query / sec limit
• in January the limit was changed back and forth and for a certain period (as far we could tell) was not respected by the server. This caused some issues for the users but seemed to have resolved after some time: https://github.com/5k-mirrors/poe-live-search-manager/issues/70, https://github.com/5k-mirrors/poe-live-search-manager/issues/68

We didn't receive any such reports recently. One hypothetical issue could be if the user uses both the browser and the tool extensively, in which case it might run into a rate limit (we'd have to verify this case though). That said, I'm not sure what can lead to an account restriction, it sounds like a drastic measure, that hopefully isn't applied by hitting the limit one time by accident.

Could you provide more details? I've tried to reach out earlier with no success. If there's a channel for devs to ask questions (e.g. about WS limitations) or get notified about API or limitation changes it would be great to get access: git.thisismydesign@gmail.com.

[Novynn]

If you're sure of your rate-limiting implementation then these issues are likely caused by players using older releases. This is one reason why setting a user agent is vitally important.

This tool is using endpoints we don't explicitly allow or encourage so we're unable to provide support for interacting with them. Rate-limits can and will change at any time depending on our requirements of providing a stable experience to our playerbase.

[Novynn]

We're still getting requests from accounts that are hitting the rate-limit and not stopping. These requests are exceeding 1 per second and are coming from the 1.13.1 version.

We recommend implementing an incremental back-off with a failure state that requires manual intervention when a connection can't be re-established.

[thisismydesign]

Thanks for the update. Could you show an example with request timestamps? How many users are affected?

Because of network delay, requests could overlap within 1 second. As a quick improvement, I will increase that. And we will look into the incremental back-off.

[Novynn]

Sure, here's a snippet from one user:

[12/Oct/2020:08:36:10 +1300] "GET /api/trade/live/Standard/v0XeQaJuE HTTP/1.1" 429 "PoE Live Search Manager/1.13.1"
[12/Oct/2020:08:36:11 +1300] "GET /api/trade/live/Standard/ylYQz8ghR HTTP/1.1" 429 "PoE Live Search Manager/1.13.1"
[12/Oct/2020:08:36:11 +1300] "GET /api/trade/live/Standard/Wq6vDr0hm HTTP/1.1" 429 "PoE Live Search Manager/1.13.1"
[12/Oct/2020:08:36:12 +1300] "GET /api/trade/live/Standard/k7yQ2XQc5 HTTP/1.1" 429 "PoE Live Search Manager/1.13.1"
[12/Oct/2020:08:36:12 +1300] "GET /api/trade/live/Standard/BBgLeOKu8 HTTP/1.1" 429 "PoE Live Search Manager/1.13.1"
[12/Oct/2020:08:36:13 +1300] "GET /api/trade/live/Standard/Q2LaoaeCw HTTP/1.1" 429 "PoE Live Search Manager/1.13.1"
[12/Oct/2020:08:36:14 +1300] "GET /api/trade/live/Standard/zabP5YvT4 HTTP/1.1" 429 "PoE Live Search Manager/1.13.1"
[12/Oct/2020:08:36:14 +1300] "GET /api/trade/live/Standard/Dm6W8jXf5 HTTP/1.1" 429 "PoE Live Search Manager/1.13.1"
[12/Oct/2020:08:36:15 +1300] "GET /api/trade/live/Standard/7jLnJ86H5 HTTP/1.1" 429 "PoE Live Search Manager/1.13.1"
[12/Oct/2020:08:36:16 +1300] "GET /api/trade/live/Standard/Dm6M0W2t5 HTTP/1.1" 429 "PoE Live Search Manager/1.13.1"
[12/Oct/2020:08:36:16 +1300] "GET /api/trade/live/Standard/JaZZ2RJSl HTTP/1.1" 429 "PoE Live Search Manager/1.13.1"
[12/Oct/2020:08:36:17 +1300] "GET /api/trade/live/Standard/r2PQvMMTQ HTTP/1.1" 429 "PoE Live Search Manager/1.13.1"
[12/Oct/2020:08:36:18 +1300] "GET /api/trade/live/Standard/95zK2a8hK HTTP/1.1" 429 "PoE Live Search Manager/1.13.1"
[12/Oct/2020:08:36:18 +1300] "GET /api/trade/live/Standard/ylYQz8ghR HTTP/1.1" 429 "PoE Live Search Manager/1.13.1"
[12/Oct/2020:08:36:19 +1300] "GET /api/trade/live/Standard/Q2MLa47Cw HTTP/1.1" 429 "PoE Live Search Manager/1.13.1"
[12/Oct/2020:08:36:19 +1300] "GET /api/trade/live/Standard/k7yQ2XQc5 HTTP/1.1" 429 "PoE Live Search Manager/1.13.1"
[12/Oct/2020:08:36:21 +1300] "GET /api/trade/live/Standard/ver6MvJSE HTTP/1.1" 429 "PoE Live Search Manager/1.13.1"
[12/Oct/2020:08:36:21 +1300] "GET /api/trade/live/Standard/Q2LaoaeCw HTTP/1.1" 429 "PoE Live Search Manager/1.13.1"
[12/Oct/2020:08:36:22 +1300] "GET /api/trade/live/Standard/r2PMJ27CQ HTTP/1.1" 429 "PoE Live Search Manager/1.13.1"
[12/Oct/2020:08:36:22 +1300] "GET /api/trade/live/Standard/Dm6W8jXf5 HTTP/1.1" 429 "PoE Live Search Manager/1.13.1"
[12/Oct/2020:08:36:23 +1300] "GET /api/trade/live/Standard/Dm6M0W2t5 HTTP/1.1" 429 "PoE Live Search Manager/1.13.1"
[12/Oct/2020:08:36:23 +1300] "GET /api/trade/live/Standard/JaVokyWul HTTP/1.1" 429 "PoE Live Search Manager/1.13.1"
[12/Oct/2020:08:36:24 +1300] "GET /api/trade/live/Standard/r2PQvMMTQ HTTP/1.1" 429 "PoE Live Search Manager/1.13.1"
[12/Oct/2020:08:36:25 +1300] "GET /api/trade/live/Standard/BBgLy4XH8 HTTP/1.1" 429 "PoE Live Search Manager/1.13.1"
[12/Oct/2020:08:36:26 +1300] "GET /api/trade/live/Standard/ylYQz8ghR HTTP/1.1" 429 "PoE Live Search Manager/1.13.1"
[12/Oct/2020:08:36:26 +1300] "GET /api/trade/live/Standard/68Zazm0UG HTTP/1.1" 429 "PoE Live Search Manager/1.13.1"
[12/Oct/2020:08:36:27 +1300] "GET /api/trade/live/Standard/k7yQ2XQc5 HTTP/1.1" 429 "PoE Live Search Manager/1.13.1"
[12/Oct/2020:08:36:27 +1300] "GET /api/trade/live/Standard/v0XeQaJuE HTTP/1.1" 429 "PoE Live Search Manager/1.13.1"
[12/Oct/2020:08:36:28 +1300] "GET /api/trade/live/Standard/Q2LaoaeCw HTTP/1.1" 429 "PoE Live Search Manager/1.13.1"
[12/Oct/2020:08:36:29 +1300] "GET /api/trade/live/Standard/Wq6vDr0hm HTTP/1.1" 429 "PoE Live Search Manager/1.13.1"

Not too many users are stuck in this state but enough to cause notable spikes in our analytics.

[thisismydesign]

Thanks. Hopefully v1.13.4 already helps with this, let me know. The app will auto-update upon restart so users won't get stuck on the old version for long.

[Novynn]

Looks like people are still able to get into a state where they endlessly spam requests in response to a 401 error. The tool really shouldn't attempt to make the same request again if it encounters this response code.

[thisismydesign]

It is important for us to automatically reconnect in case of e.g. a network outage. Didn't make a distinction between error codes so far. Provided the rate limit is respected the connection should eventually recover.

When does 401 happen? Is it an invalid session id? - In that case, you're right, the connection wouldn't recover automatically.

[thisismydesign]

@Novynn Since yesterday we're not getting items through the websocket even though the socket is connected. Can you give any information about this?

[Novynn]

No changes have been made on our end, and data is still being streamed.

[Novynn]

This tool is still generating excessive 401 and 404 responses. If this issue isn't resolved we'll have to block it's access to our servers.

[thisismydesign]

@Novynn Hey Rory, we've made some improvements to the app:

• Upon receiving 429 we stop requests. Every request from within the app is throttled but this fallback should help in case users hit the rate limit elsehwere.
• We fetch items in one go instead of 1-by-1, therefore making fewer requests.
• We're making the app open-source so the community can be more involved in the development.

With that we would like to request that you enable the client from v2.0.0 and above.

[Novynn]

That's great to hear! Do 401's and 404's also prompt the requests to stop? These are the worst offenders as mentioned:

This tool is still generating excessive 401 and 404 responses. If this issue isn't resolved we'll have to block it's access to our servers.

[thisismydesign]

@Novynn Thanks for the quick reply! We'll include those as well, no problem.

[thisismydesign]

@Novynn Hey Rory, we'll release the improved app before 3.19. It would be a web page so that it's easy to use and there's no need to install anything. In order to connect your websockets, we would need our origin whitelisted on your side, is that feasible? Thank you!

[Novynn]

The endpoint in use isn't part of our third-party offerings. I'm afraid that we aren't able to offer additional support for it at this time.

[thisismydesign]

Resolved in v2.0.0