Acquisition 0.9.7 is blocked by GGG for rate limit violations

[gerwaric]

For anyone who's been trying to use acquisition recently, GGG finally blocked version 0.9.7 at the server level for rate limit violations.

I tried to fix this before, but my approach broke if you were using a non-English computer. Now I'm back to working this problem. I've updated the version string, and my test builds are not being blocked, so I can make progress. I also have someone from the EU helping me. Between the two of us, we are testing two different timezones and languages. Once it's working for both of us, I'd like to get a few more volunteers before making an official release. I want to make sure any update does not break and cause acquisition to be banned again.

Hopefully we'll be able to get an update that works again, but I'm not sure how long it will take.

[gerwaric]

For people wondering why rate limiting is not easy to solve, please read all the related documentation if you haven't already:

https://www.pathofexile.com/developer/docs#ratelimits

The main issue is that there are multiple rate limit policies, and they are subject to change without notice. Each policy has to be tracked separately and each policy can apply to multiple api endpoints. However, there is no canonical list of policies or endpoints. Additionally, each policy can have multiple rules to keep track of. Typically there is an account-based limitation as well as an ip-based limitation, but GGG could add others at any time. Furthermore, each rule has multiple components.

This means that acquisition needs to track everything dynamically.

The more a solution is hard-coded, the greater the risk that acquisition will start breaking rules when GGG changes something. That would put acquisition at risk of being banned again. For example, this might happen if GGG decides to temporarily tighten rates limits around the launch of a new league or the launch of POE2 due to a surge in bot or trade assistant activity.

Right now the solution I'm working on hard-codes the policy names and api endpoints, but reads the rules and individual components dynamically. Once that works, it should be possible to make even the policy names and endpoints fully dynamic. That's just more complexity, so I'm tackling it one step at a time.

[Marconium2]

Hey thats great to hear, if i can help with anything let me know :)

[Jozen2023]

Someone on the other thread mentioned a possible aproach to avoid rate limit hammering: after the initial fetching disable the autoupdate at all and let only the manual updating available for now, with 10 or 20 tabs at max per minute or so. Dunno if something like this can work, but it seem logical.

[eugenikus8]

@tomholz Sounds good. Is it possible to get your compiled version for testing?

[aiolos01]

Fixing the rate limit issue would be ideal but it may take some time and might stop working any time GGG decides to mess with the rules again.

I've already suggested multiple times that simply disabling auto update solves the issue. This is a standard-exclusive issue. We all have hundreds or remove-only tabs and 20-40 or so normal tabs. There's no need to update 500 tabs every time. We just need to update the 40 normal tabs that actually change content. And we can do this manually when necessary.

In my case the already existing acquisition (0.97 I believe) stalls permanently after updating 100+ tabs. This is more than enough for the normal (not remove-only) tabs. So even without ANY solution to the rate limiting issue we can all update our normal tabs if the damned auto-update is removed.

I still don't understand why none of the people working on this have replied to my suggestion. It seems like an easy fix. WAY easier than dealing with all the rate limiting mess. Is there some flaw in my reasoning? Is the auto-update so difficult to remove? I really don't know...

[gerwaric]

@Marconium2 @eugenikus8 I'll let you know when I have a stable version for testing. Thank you for offering to help test.

[gerwaric]

@Jozen2023 that's a good idea, although even 10 tabs a minute will break the current account-based rule in the backend-item-request-limit policy within fifteen minutes. I believe the fastest you can sustainably request tabs under the current policies and remain "safe" is 3 tabs per minute right now.

[CrappyUnoptimizedCode]

@tomholz If you would like me to join in testing versions, let me know. I have about 700 tabs in Standard.

[gerwaric]

@aiolos01 I can take a look at it, but not a software developer, so I'd ask for some consideration while I'm learning. I have zero experience with QT, I haven't written code in c++ since college twenty years ago, I have a day job, and I do have some friends I like to spend time with every now and then :-)

The good news is that this is an open source project. You can clone the project yourself, download QT community edition, install OpenSSL, tinker around with your idea, and make a pull request once you've got it working. That's what I did when I started, and I barely knew what I was doing. (That's also why I can empathize a bit.. I was in your shoes a few months ago when I saw that rate-limiting was breaking third-party tools like acquisition).

[gerwaric]

BTW, thank you to everyone for taking interest and sharing your thoughts, and thank you for your patience. I'm new to github and open source, so please bear with me. I am as eager to see this fixed, but I'm still learning the dev tools and the codebase.

[aiolos01]

I appreciate your efforts in this but I don't understand your answer to be honest. You're trying to solve a difficult problem and that takes time and software development expertise. What I'm suggesting is to leave everything as it is and just disable auto-update. This should be much easier and faster. I'm trying to help you, not make your effort more difficult.

P.S. You said it yourself 10 tabsX15 min = 150 tabs (for me it's around 105). That's way more than we need to actually refresh.

P.S.2. I haven't set this up on a debugger or anything and I'm no C++ expert either but maybe line 64 of application.cpp is a good place to start looking. Just a suggestion.

[gerwaric]

@aiolos01 I agree with you that it's a good idea to disable automatic updates, and I will look into it.

P.S. It looks like application.cpp:64 is the initial update that happens when acquisition first loads. I'm turning that off for my own sanity, but that won't stop the automatic refreshes. The code for those appears to be auto-generated, so I'll have to figure out the QT UI form editor.

[aiolos01]

There's no need to stop the automatic refreshes. There's a checkbox for that in the UI and the user can disable them. It's only the initial update that needs to be turned off.

[gerwaric]

Unfortunately, I've discovered that rate limiting is needed until the ItemsManagerWorker class gets a significant rewrite. This is because acquisition ends up making a network API request for nearly every tab in your stash regardless of how many tabs you have checked or selected due to a server change GGG made sometime after 2016.

Under the hood, acquisition uses a customized QNetworkDiskCache object. This allowed acquisition to redirect requests for unchecked or unselected tabs to the cache instead of making API calls. Based on comments in the code, it looks like this approach was developed in 2016 when GGG was running their own nginx web server. However, GGG now uses Cloudflare and tab caching no longer works as reliably as it presumably did in 2016.

I've tried updating the custom QNetworkDiskCache, but it's still not working consistently. As a result, you cannot just select a handful of active tabs to be refreshed periodically, because every refresh can make API requests for more tabs than you selected or checked. If you have hundreds of tabs, it could take hours for a refresh to complete under current rate limit policies, even if you only had a single tab checked.

This rabbit hole is going deeper than I had hoped. However, with some luck I hope to have rate-limiting ready for testing in the next week or two. Watch for a post on the pathofexiledev subreddit. Then if I'm not burned out I'll look at reworking ItemsManagerWorker to be less dependent on GGG's server configuration.

[came-here-to-learn]

What exactly do you mean when you refer to "tab caching"?

Do you mean saving them on the local computer so the tabs dont need to be refreshed again after turning the program off and then on? That functionality never worked for me - acquistion always had to download all the information from scratch.

It;'s like it didnt save anything at all.

[gerwaric]

@came-here-to-learn I'm seeing some (but not all) tab caches persist between runs. It's also happening to me between refreshes within a single run.

[aiolos01]

It's strange to hear that. I've never had issues with caching (except at one time due to a bug). I've been using this for years and have made many refreshes of 5-10 tabs. I never encountered throttling which means it probably didn't read more tabs than it should have. Maybe it's user dependent?

I hope this is solved at some point. I don't really use acquisition for selling any more but it's still great for searching your stash. Can't really believe there are so many great and complex tools out there but only this and looty for searching your on stash.

[leet0rz]

It's strange to hear that. I've never had issues with caching (except at one time due to a bug). I've been using this for years and have made many refreshes of 5-10 tabs. I never encountered throttling which means it probably didn't read more tabs than it should have. Maybe it's user dependent?

I hope this is solved at some point. I don't really use acquisition for selling any more but it's still great for searching your stash. Can't really believe there are so many great and complex tools out there but only this and looty for searching your on stash.

It will happen if you have a lot of tabs, it will go on cooldown. Any fix on this yet or are people still working on one?

[aiolos01]

It's strange to hear that. I've never had issues with caching (except at one time due to a bug). I've been using this for years and have made many refreshes of 5-10 tabs. I never encountered throttling which means it probably didn't read more tabs than it should have. Maybe it's user dependent? I hope this is solved at some point. I don't really use acquisition for selling any more but it's still great for searching your stash. Can't really believe there are so many great and complex tools out there but only this and looty for searching your on stash.

It will happen if you have a lot of tabs, it will go on cooldown. Any fix on this yet or are people still working on one?

I guess you missed the part where I say "refreshes of 5-10 tabs". That doesn't mean I have 10 tabs. Couldn't play the game for years with 10 tabs...

[leet0rz]

It's strange to hear that. I've never had issues with caching (except at one time due to a bug). I've been using this for years and have made many refreshes of 5-10 tabs. I never encountered throttling which means it probably didn't read more tabs than it should have. Maybe it's user dependent? I hope this is solved at some point. I don't really use acquisition for selling any more but it's still great for searching your stash. Can't really believe there are so many great and complex tools out there but only this and looty for searching your on stash.

It will happen if you have a lot of tabs, it will go on cooldown. Any fix on this yet or are people still working on one?

I guess you missed the part where I say "refreshes of 5-10 tabs". That doesn't mean I have 10 tabs. Couldn't play the game for years with 10 tabs...

That was not the point, users who refresh 30+ tabs will get a limit and it will continue with the tabs after a minute or so until its done. For users with less tabs they'll be fine as they will not hit that limit.

@tomholz Are you aware of a fix yet or is this still being worked on? Thanks.

[gerwaric]

@leet0rz are you on discord? I have a build I've been testing. It's not ready for public release, but I'd love to see if it works for you.

[aiolos01]

It's strange to hear that. I've never had issues with caching (except at one time due to a bug). I've been using this for years and have made many refreshes of 5-10 tabs. I never encountered throttling which means it probably didn't read more tabs than it should have. Maybe it's user dependent? I hope this is solved at some point. I don't really use acquisition for selling any more but it's still great for searching your stash. Can't really believe there are so many great and complex tools out there but only this and looty for searching your on stash.

It will happen if you have a lot of tabs, it will go on cooldown. Any fix on this yet or are people still working on one?

I guess you missed the part where I say "refreshes of 5-10 tabs". That doesn't mean I have 10 tabs. Couldn't play the game for years with 10 tabs...

That was not the point, users who refresh 30+ tabs will get a limit and it will continue with the tabs after a minute or so until its done. For users with less tabs they'll be fine as they will not hit that limit.

@tomholz Are you aware of a fix yet or is this still being worked on? Thanks.

I have 200+ tabs. Learn how to read.

[leet0rz]

@leet0rz are you on discord? I have a build I've been testing. It's not ready for public release, but I'd love to see if it works for you.

Appreciate it but I think I'll wait for a public release, please tag me when there is one and I'll help test and report issues if I find any. I use Acqu extensively.

[leet0rz]

@aiolos01 I am not talking about your specific case, I am talking about in general. If users have a lot of tabs that it needs to look through it will reach a limiter. Especially on league starts when it needs to configure again it will have to go through all the tabs on the first run. Maybe you are able to understand now.

[gerwaric]

I've muddled my way through creating an installer and tagging a release with dynamic rate limiting: https://github.com/gerwaric/acquisition/releases/tag/v0.9.9-beta.2 (updated to beta 2)

Feedback on what works or doesn't would be helpful, especially things that changed from 0.9.7. This build is time-limited and should stop working in about 2 weeks. I'm being cautious because until we have more confidence that 0.9.9 works and the new rate limiting is rock solid.

@aiolos01 @leet0rz @CrappyUnoptimizedCode

[CrappyUnoptimizedCode]

Hi @tomholz , I tried refreshing all my standard tabs and I think you nailed it! I had a look at the rate limiting info panel (great addition) and noticed it adjusted the request frequency a few times, so this all seems to work quite well. Is there any particular info you would like to know regarding my settings/stash etc?

[gerwaric]

@CrappyUnoptimizedCode mainly just watch for anything that's changed from 0.9.7 that seems like a bug. You can open issues on my repo if you find something.

[gerwaric]

I'm closing this issue since it's been resolved with newer releases, available here: https://github.com/gerwaric/acquisition/releases/