Request rate limit

[atis]

Since yesterday update - after couple of minutes active actions (panning map, zooming) server starts to respond with:

403 Forbidden Rate limit exceeded

Sometimes when big battles are going on, this is quite critical drawback, as operator is essentially locked out of helping team.

1) Actual allowed rate limits should be tested 2) Some request limiting should be implemented to avoid this from happening.

Simple panning could use more local cache, not request them again. Some settings to fine tweak this would be nice.

[jonatkins]

a big culprit of this is the chat window. as we filter by different types, and the backend only returns the two tabs used by the native frontend, we often have to request much more to fill the window you have in view. Keeping your chat tab on faction or full is often the best, compact and public are not so good.

[atis]

@jonatkins This happens with Faction open (for communications), however it's necessary to refresh events for player tracker - so the habit to pan often which leads subsequentive calls to getThinnedEntities

However your comment made me thing of alternate solution - plugin to manually refresh full log, so the operator could just press the button and watch changes. That plugin could also have rate limiting.

[jonatkins]

IITC will refresh itself, unless left idle for an extended period of time. This is currently configured at 30 seconds. Are you saying this is too slow for you?

How does the standard intel site behave with similar user actions? Does it hit the request rate limit error easily?

I've not yet checked - does the standard intel site have a faster refresh for chat messages?

Perhaps certain user actions should trigger an immediate refresh, at least for chat. e.g. starting to enter text into the chat input box (you're about to say something - so seeing the latest said by others makes sense). Looking at recent message rates is another option (when things are happening quickly, refresh faster to keep current)

Care needs to be taken to avoid hammering the ingress servers excessively, but still keeping things up to date.

[atis]

@jonatkins yes, when operation on battlefield, 30 seconds is too slow, that's why we're doing panning to refresh.

I just tested with stock map, and when doing panning, it requests just getPaginatedPlextsV2 After panning stops, it takes a second or two for getThinnedEntitiesV2 to appear (perhaps they are not parallel).

So, just panning fast on stock map will use half of the requests IITC does, and the "Rate limit" doesn't appear.

[jonatkins]

I'll have to have a close look at the stock intel site - see if we can match it's behaviour better.

I might try other things too - e.g. click into the text input box causes an immediate refresh of the chat window

[jonatkins]

Had a look at the IITC code in detail. Making some improvements:

• there was a short 0.5 second delay between the completion of a move/zoom and the refresh starting. I've increased this to 0.8 seconds
• there was no restriction on these repeated refreshes. I'm now preventing a second refresh from starting within 5 seconds of the previous one
• the default idle refresh was 30 seconds, plus a factor based on zoom level - I'm changing this to 60 seconds + zoom factor

With these changes I now cannot cause requests to fail however much I zoom/move around the map. I will commit shortly.

I will look into making chat refresh on a separate faster cycle and/or refresh on focus of the text entry field - that'll be a separate change though. Feel free to open a new issue on this as I won't be starting it immediately.

[jonatkins]

you may want to try out the test build - at http://iitc.jonatkins.com/?page=test - to see how it is now

[Teleken]

This bug has not been fixed. Still the same problems as before.

Happens particularly often when changing zoom levels, but if I'm zoomed in low enough to see all portals (for example) I'll get between 6-10 "drags" before failures start to occur. Very easy to reproduce. I'm on the latest version (May 2).

[atis]

@Teleken @jonatkins it's beed siginficantly improved, however still happening sometimes. At least it's now possible to use draw tools with zooming in/out

[Teleken]

Ok - can anything be done to alleviate the problem, or any further diagnosis to help? Like I mentioned it's pretty easy to reproduce. I'm wondering if launching 40 concurrent requests all the time may be contributing to this.

[jonatkins]

@Teleken I made further improvements very recently. If you could try the current test version and see if it's any better?

[Teleken]

I tried the update but I didn't notice a difference, but then I tried to reinstall the original and I'm not sure that the update actually took.

I had one idea though, since it almost always happens when either zooming or panning multiple times how about a small delay? Currently as soon as the drag completes it fires 40 requests to the server. If I drag again then it sends another 40, which can trigger the rate limit. A 2-second delay between last drag/zoom action and loading the results could avoid this problem.

[jonatkins]

@Teleken That's exactly what IITC does

• after a move/zoom, the refresh is delayed for 1.25 seconds (current test build - was 0.8 or 0.5 seconds)
• refreshes will never be closer than 5 seconds apart, so if you move/zoom immediately after a refresh starts, the delay is longer

Also, although it shows '40 requests', they don't all hit the server at once. The browser will limit them, only allowing 4 or so at a time, sending another each time one completes.

40 requests does seem particularly high though - I can't get anywhere near that number. Are you using your browser's 'zoom' feature to see more detail over a larger area? What size display are you using and is your browser fullscreen?

[Teleken]

Ahh good call!

Umm, I have a 2560x1440 screen running at 75% zoom in order to show portals sooner, so effective resolution is about 3400x1920.

However in terms of the delay - it takes at least a couple of seconds from one pan to the next, so perhaps a value a bit closer to 2 would be better?

How can I try your test build? I'll give it a whirl to see if it fixes my problem, but it seems my resolution is to blame (didn't realize number of concurrent calls was proportional to the resolution).

Thanks for the fast responses and help on this BTW.

[jonatkins]

@Teleken The test build is available on the website - http://iitc.jonatkins.com/?page=test - install as you would the regular build.

And yes, resolution has a big impact on the number of requests made. The back-end data is tile-based, much like the map images themselves, so the higher the effective resolution, the more requests made. I get 14 requests per refresh on a window sized approx 1280x1000

[jonatkins]

@Teleken (and others) I'd been keeping this issue in mind, despite it being closed for a while and various tweaking of timeouts helping but not fixing things, when I had a sudden flash of inspiration on what IITC was doing differently compared to the stock intel site.

It was a simple change, and I made it just before making the new 0.12.0 release. Now the number of requests does NOT depend on screen resolution - it should not go above 6 (4 for the map data, 2 for chat). This should now be fine, working more like the stock site does, and as IITC did before we had to change the code due to the protocol changes.

[Teleken]

I noticed that today! Mine automatically updated and it doesn't go above six now, and I can't reproduce the rate limiting problem. It feels a little bit slower, but I can deal with that. Good job and thanks again for all the work!