Aggressive rate limiting prevets acquisition from loading all tabs

[aiolos01]

It seems something has changed in 3.21 regarding rate limiting. Acquisition used to load 60 tabs before throttling. Now it loads 31 and the next few times the number of loaded tabs is constantly reduced until it doesn't load anything any more.

It seems retrying too often causes this.

I contacted support but (as expected) didn't get any help regarding any changes to their APIs, just a URL that basically says their policies can change at any time without notice. https://www.pathofexile.com/developer/docs#ratelimits

[came-here-to-learn]

Good "future proof" solution for this would be to allow to choose number of tabs X refreshed per Y seconds.

Where user could choose X and Y.

For example 1 tab every 120 seconds.

[testpushpleaseignore]

Copying this comment from elsewhere in this repo to this problem

So it looks like they are setting limits for each type of request being made. Here are the limits that I've run into so far:

"backend-character-request-limit"

X-Rate-Limit-Policy: "backend-character-request-limit"
X-Rate-Limit-Rules "Account,Ip"
X-Rate-Limit-Account "10:60:60,50:1800:600"
X-Rate-Limit-Account-State "11:60:60,12:1800:0"
X-Rate-Limit-Ip "30:60:120,120:1800:600"
X-Rate-Limit-Ip-State "12:60:0,12:1800:0"

"backend-item-request-limit"

X-Rate-Limit-Policy: "backend-item-request-limit"
X-Rate-Limit-Rules "Account,Ip"
X-Rate-Limit-Account "30:60:60,100:1800:600"
X-Rate-Limit-Account-State "31:60:60,85:1800:0"
X-Rate-Limit-Ip "45:60:120,180:1800:600"
X-Rate-Limit-Ip-State "31:60:0,85:1800:0"

So for characters, we can make 10 requests every 60 seconds, and 50 every 1800 seconds. And for item tabs, we can make 30 requests every 60 seconds, and 100 every 1800 seconds.

The original people who wrote this code just assumed that there was one global request limit, that never lasted for more than 60 seconds. So yeah worst case scenario, having 1 request per minute should prevent us from ever hitting any throttling, but I'd imagine that a number of people would get upset if I don't have this program operate as fast as possible without hitting the true limits. Might take a while to figure out how to queue these requests up to have their own limits, but I currently wouldn't have much time right now to figure this out.

Since GGG has implemented a sort of dynamic throttling, I'm wondering if I can ask their servers for the current limits, so that every time a batch of transactions are about to be sent it out, it can check to see what the current limits are? Or perhaps all requests are showing the current X-Rate-Limit-Account-State? If so, then I would have to see if this header is in a good transaction after making the first transaction to their servers.

[testpushpleaseignore]

I wonder what the rate limits are for https://api.pathofexile.com? If they're a lot higher than when what we use currently at https://www.pathofexile.com/character-window/, then I bet that this is their way of trying to get us to switch over. Though that would mean a complete overhaul of how we are allowed to log in with this app, since we would have to then switch to using OAuth 2.0, rather than what this app allows currently.

[aiolos01]

That is a distinct possibility since the PoB ssid issue some months ago. I believe Oath 2 for non Web based apps is still very difficult to implement and GGG promised some changes that haven't been done yet. At this point I's be happy With 1 tab/minute just to be able to load all my tabs. That should be easy to do I suppose.

[came-here-to-learn]

I think a stop gap would have to be 1 char / tab per minute and a break for 60 minutes after 29 chars/tabs.

Would take ages to load my 500 tabs, but this is still better than nothing.

[csherman03]

i also have this problem and cannot refresh my ~300 tabs, is there a solution for this yet for acquisition i would greatly appreciate it i basically cannot make any builds in standard right now and i messaged support 4-5 times and theres nothing they can do

[aiolos01]

Looty works for me. I cleared all tabs and reloaded everything. I takes longer but works.

For anyone interested here's a workaround. You let acquisition load as many tabs as it can and then block it with your firewall for some time (e.g.20 min) after you unblock it throttling will be disabled and it will load some tabs. Then block it again and repeat as needed until it loads everything. Annoying yes but it's the best we've got at the moment.

As a side note I've already started pricing all my new items with the built-in system (premium tabs). Xyzz abandoned acquisition and testpush (while very responsive in the past) doesn't seem to have time for this any more. So we might as well stop relying on this as much as possible as it's almost certain than soon it won't work any more.

[CrappyUnoptimizedCode]

For anyone interested here's a workaround. You let acquisition load as many tabs as it can and then block it with your firewall for some time (e.g.20 min) after you unblock it throttling will be disabled and it will load some tabs. Then block it again and repeat as needed until it loads everything. Annoying yes but it's the best we've got at the moment.

I've done something similar, I ran Acquisition on a old laptop and used PowerShell to enable/disable wifi. I enable wifi for 3,5 minutes and then disable wifi for 33 minutes, every cycle I'm able to fetch about 70 tabs. I posted the details in a reddit topic for those who would like to try this: https://dd.reddit.com/r/pathofexile/comments/13mhuxd/acquisition_finally_synced_my_stash/

[testpushpleaseignore]

I won't have time to develop a fix for this, but if someone wants to update the "itemsmanagerworker.cpp" file to change how the program determines when to automatically throttle itself (without going over the 4 different timeouts), then I could test that for you once you submit a Pull Request. If you manage to get the code pretty close to working, then I could tidy it up and create a release that fixes these timeout issues.

I totally understand that this is frustrating, but yeah I wish I had the free time to put more extensive work into this fix.

[aiolos01]

I understand you don't have time to fix this properly but I have a suggestions that you can implement which will mostly solve the problem for us.

Just remove the auto refresh on startup. This way we can simply refresh the tabs we use for selling and avoid refreshing the hundreds of remove only tabs that never change anyway. Auto refresh just stalls and we can't refresh the tabs we really want to without resorting to tricks like I described above.

[came-here-to-learn]

Any chance to add a pop-up window or setting somehwere:

1) dont refresh on start

2) time between refreshing each tab: default X (default 2 seconds)

3) number of tabs to refresh before "long sleep" Y (default 10)

4) number of time to wait after refreshing Y tabs -> what would be Z minutes (default 31 minutes)

When we can change Z, Y and Z on our own, we can slowly download the data

Sadly I dont know how to make a popup window

[gerwaric]

So... I'm working on a fix for this. It won't be ready for 3.22, but I'm able to dynamically parse the rate limit policies and states from reply headers as they are received. The goal is to use those headers to proactively throttle requests without hard-coding any limits at all.

For now, I'm just checking replies for the Retry-After header to detect a rate limit violation, then I'm pausing for the specified time before restarting http requests.

As much as possible, I'm trying to keep changes to ItemsManagerWorker to a minimum, so I'm adding a new class with it's own source and header files. It basically wraps calls to QNetworkAcessManager::get(). Once I get that working with the dynamic limit detection, throttling should be almost transparent to ItemsManagerWorker and the rest of the application.