gerwaric / acquisition

Stash and forum shop management for Path of Exile
GNU General Public License v3.0
18 stars 3 forks source link

Loading Standard Tabs Causes 100+ Errors and a 24 Hour Ban #56

Closed livejamie closed 3 months ago

livejamie commented 3 months ago

This happened to me last week when we were troubleshooting #48

image

It happens only when I log on to Standard.

Today I logged onto my league account and it worked as expected, but I tried to log onto my Standard account and it showed 120+ errors, something about being unable to parse tabs and the program crashed.

I'm now banned for 24 hours again.

I can't provide any logs because I can't use my SESSID or OpenID to log on to the Path of Exile website.

I'll ask Novynn if they can provide any additional information to help troubleshoot. They said they would pass it along to you.

gerwaric commented 3 months ago

EDIT: I just saw the email from GGG. Crap. I'm working on this now. I will leave the rest of this comment because it still covers my current understanding of the problem. I've also deleted affected installers.

You're the second person I'm aware of who was blocked by Cloudflare for this issue. (See https://github.com/gerwaric/acquisition/issues/50). I'm almost certain this was caused by a change I introduced in v0.11.1-alpha.12, which made it into the v0.11.1 release, and was not fixed until v0.11.1.1 (EDIT: the latest version is still bugged). See below for details. If I'm right about this, this bug was also the cause of https://github.com/gerwaric/acquisition/issues/51. I have removed the installers and added warnings to all the affected version so that nobody else can accidentally install these versions.

First, I'm sorry.

This bug, like you experienced first-hand, not only impacts acquisition, but it blocks people from using other third-party tool, and it blocks them from accessing the official website. I introduced this bug while trying to troubleshoot a mysterious crash that turned out to be unrelated, https://github.com/gerwaric/acquisition/issues/49.

If you want to know what was happening under the hood, here's a summary:

At startup, acquisition doesn't know the status of the API rate limit policies, so it makes special HEAD requests to check the status before making any API calls. These HEAD requests don't count against the rate limit policy, so they are "safe" in one sense. I've tried to be very cautious about things like this, because Acquisition was black-listed at the application level by GGG last year for violating rate limit policies. However, that caution backfired in this case.

The problem was two-fold.

First, acquisition was incorrectly rejecting replies to some of these HEAD requests. In alpha.12, I started strictly checking response codes for 200 OK, but some of them were coming back with 204, which acquisition should have accepted because 200-299 are all considered "success". This was my first mistake.

Second, once a HEAD reply was rejected, acquistion would immediately continue onto the next stash request. This triggered another HEAD request because acquistion was still trying to check the status of the rate limits. However, the reply to this second HEAD request would be incorrectly rejected just like the first. I should have blocked accquisition from sending requests after experiencing a failure. This was my second mistake.

With as many stash tabs as you and other have in Standard, this was causing enough requests to trigger Cloudflare's own rate limit policies. These are different from GGG's rate limit policies, and the Cloudflare policies are undocumented as far as I know, but several hundred or thousand HEAD requests in the span of a few seconds is enough to violate them.

I'm cautiously hopeful that this issue has been resolved with v0.11.1.1...

Honestly, this bug was due to my being a rookie c++ web software developer. When acquisition was black-listed for rate limit violations last year, I decided to teach myself enough to solve the problem. Since then, I've been learning more with each issue. My apologies again.

gerwaric commented 3 months ago

Since this is still an issue with v0.11.1.1, I'm pulling the installer for that version, too, until I can figure out what's going on. I've also posted in the related issues so other current users will hopefully see what's going on.

gerwaric commented 3 months ago

I found the problem. In addition to the two mistakes described above, there was a third mistake in how requests were batched at startup. Fixing the first two was necessary, but not enough to prevent this third one from causing the flood of HEAD requests.

I missed this in testing because I only have about 250 tabs in Standard--not enough to trigger Cloudflare. Now that I know what to look for, I can see it happening on my account, too.

livejamie commented 3 months ago

Great troubleshooting, and you don't need to be so apologetic! Things happen and I know you're trying your best. :)

gerwaric commented 3 months ago

Thank you. I think I've fixed the problem, but would love confirmation from someone brave enough to try:

https://github.com/gerwaric/acquisition/releases/tag/v0.11.2-rc.1

livejamie commented 3 months ago

My IP has been whitelisted so somebody else with a bunch of tabs would have to try. :)

gerwaric commented 3 months ago

Back when I was more panicked this morning, I almost bought a couple hundred tabs just to test it myself!

livejamie commented 3 months ago

Aw man I hope I didn't cause any unneeded stress on you, I wasn't upset or anything I promise!

gerwaric commented 3 months ago

No worries at all. Your issue report was quite useful in helping me figure out what was going on, so I'm happy you filed it!

What worried me was the idea that everyone who used acquisition suddently finding themselves blocked from using acquisiton, other tools like awakened trade, and even the trade site. What made it worse is that I had no idea this morning what was going on, because up until your report I was sure I had fixed the issue.

Shadetorrent commented 3 months ago

Just dropping by to thank you for your work gerwaric!

livejamie commented 3 months ago

It appears to be working as expected for me, but I'd have somebody with a different IP test before closing this out.

image

gerwaric commented 3 months ago

@Shadetorrent thank you! Hopefully the 0.11.2-rc.1 release is stable for a while, so I can get back to my league starter :-)

@livejamie good to hear. I'll close this ticket and push out 0.11.2 as a final release in a couple days if nothing else shows up.

A bunch of the recent issues were due to rate limiting and OAuth. Now that those are done, fingers crossed, I'm planning to work on a couple minor features, then maybe tackle some bigger stuff like pseudo-mods and internal database rework towards the end of the league.

livejamie commented 3 months ago

It got to ~300 tabs and then errored out. I don't know if this would have resulted in a rate limit for a regular IP. You can ask Novynn.

Errors and warnings will be printed here
ERROR 2024-08-02T07:09:10.761 Cannot parse the stash tab "$$$ (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.761 Cannot parse the stash tab "Card (Remove-only)" colour: "bf00" 
ERROR 2024-08-02T07:09:10.761 Cannot parse the stash tab "$$$ (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.761 Cannot parse the stash tab "Card (Remove-only)" colour: "bf00" 
ERROR 2024-08-02T07:09:10.761 Cannot parse the stash tab "$$$ (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.761 Cannot parse the stash tab "Card (Remove-only)" colour: "bf00" 
ERROR 2024-08-02T07:09:10.761 Cannot parse the stash tab "Card (Remove-only)" colour: "bf00" 
ERROR 2024-08-02T07:09:10.761 Cannot parse the stash tab "$$$ (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.761 Cannot parse the stash tab "$$$ (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.761 Cannot parse the stash tab "$$$ (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.761 Cannot parse the stash tab "~price 1.5 divine (Remove-only)" colour: "80" 
ERROR 2024-08-02T07:09:10.761 Cannot parse the stash tab "~price 2.5 divine (Remove-only)" colour: "80" 
ERROR 2024-08-02T07:09:10.761 Cannot parse the stash tab "~price 4.9999 divine (Remove-only)" colour: "ff" 
ERROR 2024-08-02T07:09:10.761 Cannot parse the stash tab "$$$" colour: "4d00" 
ERROR 2024-08-02T07:09:10.761 Cannot parse the stash tab "~price 2.99999 divine (Remove-only)" colour: "80" 
ERROR 2024-08-02T07:09:10.761 Cannot parse the stash tab "~price 3.9999 divine (Remove-only)" colour: "80" 
ERROR 2024-08-02T07:09:10.761 Cannot parse the stash tab "~price 4.9999 divine (Remove-only)" colour: "ff" 
ERROR 2024-08-02T07:09:10.761 Cannot parse the stash tab "Card (Remove-only)" colour: "bf00" 
ERROR 2024-08-02T07:09:10.761 Cannot parse the stash tab "Card (Remove-only)" colour: "bf00" 
ERROR 2024-08-02T07:09:10.763 Cannot parse the stash tab "~price 3 divine (Remove-only)" colour: "80" 
ERROR 2024-08-02T07:09:10.763 Cannot parse the stash tab "~price 4 divine (Remove-only)" colour: "80" 
ERROR 2024-08-02T07:09:10.763 Cannot parse the stash tab "~price 5 divine (Remove-only)" colour: "ff" 
ERROR 2024-08-02T07:09:10.763 Cannot parse the stash tab "$$ (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.763 Cannot parse the stash tab "MA (Remove-only)" colour: "bf00" 
ERROR 2024-08-02T07:09:10.763 Cannot parse the stash tab "Card" colour: "bf00" 
ERROR 2024-08-02T07:09:10.763 Cannot parse the stash tab "~price 1.5 chaos (Remove-only)" colour: "ff" 
ERROR 2024-08-02T07:09:10.763 Cannot parse the stash tab "~price 3 divine (Remove-only)" colour: "80" 
ERROR 2024-08-02T07:09:10.763 Cannot parse the stash tab "~price 5 divine (Remove-only)" colour: "ff" 
ERROR 2024-08-02T07:09:10.763 Cannot parse the stash tab "Essence (Remove-only)" colour: "80" 
ERROR 2024-08-02T07:09:10.763 Cannot parse the stash tab "CA (Remove-only)" colour: "ff" 
ERROR 2024-08-02T07:09:10.763 Cannot parse the stash tab "FR (Remove-only)" colour: "80" 
ERROR 2024-08-02T07:09:10.764 Cannot parse the stash tab "Breach (Remove-only)" colour: "80" 
ERROR 2024-08-02T07:09:10.764 Cannot parse the stash tab "~price 3 exalted (Remove-only)" colour: "80" 
ERROR 2024-08-02T07:09:10.764 Cannot parse the stash tab "~price 4 exalted (Remove-only)" colour: "80" 
ERROR 2024-08-02T07:09:10.764 Cannot parse the stash tab "~price 5 exalted (Remove-only)" colour: "ff" 
ERROR 2024-08-02T07:09:10.764 Cannot parse the stash tab "~price 20 chaos (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.764 Cannot parse the stash tab "~price 25 chaos (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.764 Cannot parse the stash tab "~price 30 chaos (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.764 Cannot parse the stash tab "~price 35 chaos (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.764 Cannot parse the stash tab "~price 40 chaos (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.764 Cannot parse the stash tab "~price 50 chaos (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.764 Cannot parse the stash tab "CA (Remove-only)" colour: "ff" 
ERROR 2024-08-02T07:09:10.764 Cannot parse the stash tab "~price 75 chaos (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.764 Cannot parse the stash tab "~price 1.5 exalted (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.764 Cannot parse the stash tab "FR (Remove-only)" colour: "80" 
ERROR 2024-08-02T07:09:10.764 Cannot parse the stash tab "~price 3 exalted (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.764 Cannot parse the stash tab "~price 4 exalted (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.764 Cannot parse the stash tab "~price 5 exalted (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.764 Cannot parse the stash tab "meat shields (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.764 Cannot parse the stash tab "Smite (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.764 Cannot parse the stash tab "EA (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.764 Cannot parse the stash tab "DC (Remove-only)" colour: "ff" 
ERROR 2024-08-02T07:09:10.764 Cannot parse the stash tab "FR (Remove-only)" colour: "80" 
ERROR 2024-08-02T07:09:10.765 Cannot parse the stash tab "Cluster (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.765 Cannot parse the stash tab "CR (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.765 Cannot parse the stash tab "$$ (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.765 Cannot parse the stash tab "D1 (Remove-only)" colour: "bf00" 
ERROR 2024-08-02T07:09:10.765 Cannot parse the stash tab "D2 (Remove-only)" colour: "bf00" 
ERROR 2024-08-02T07:09:10.765 Cannot parse the stash tab "FO (Remove-only)" colour: "ff" 
ERROR 2024-08-02T07:09:10.765 Cannot parse the stash tab "ME (Remove-only)" colour: "80" 
ERROR 2024-08-02T07:09:10.765 Cannot parse the stash tab "D3 (Remove-only)" colour: "bf00" 
ERROR 2024-08-02T07:09:10.766 Cannot parse the stash tab "S (Remove-only)" colour: "ff" 
ERROR 2024-08-02T07:09:10.766 Cannot parse the stash tab "U (Remove-only)" colour: "80" 
ERROR 2024-08-02T07:09:10.766 Cannot parse the stash tab "C (Remove-only)" colour: "ff" 
ERROR 2024-08-02T07:09:10.766 Cannot parse the stash tab "C (Remove-only)" colour: "80" 
ERROR 2024-08-02T07:09:10.766 Cannot parse the stash tab "BL (Remove-only)" colour: "bf00" 
ERROR 2024-08-02T07:09:10.766 Cannot parse the stash tab "F (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.766 Cannot parse the stash tab "E (Remove-only)" colour: "80" 
ERROR 2024-08-02T07:09:10.766 Cannot parse the stash tab "BS (Remove-only)" colour: "bf00" 
ERROR 2024-08-02T07:09:10.766 Cannot parse the stash tab "E (Remove-only)" colour: "80" 
ERROR 2024-08-02T07:09:10.766 Cannot parse the stash tab "F (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.766 Cannot parse the stash tab "~price 3 alch (Remove-only)" colour: "bf00" 
ERROR 2024-08-02T07:09:10.766 Cannot parse the stash tab "Brains (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.766 Cannot parse the stash tab "~price 2 chaos (Remove-only)" colour: "bf00" 
ERROR 2024-08-02T07:09:10.766 Cannot parse the stash tab "Eyes (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.766 Cannot parse the stash tab "CT1 (Remove-only)" colour: "bf00" 
ERROR 2024-08-02T07:09:10.766 Cannot parse the stash tab "~price 3 chaos (Remove-only)" colour: "bf00" 
ERROR 2024-08-02T07:09:10.766 Cannot parse the stash tab "~price 3 chaos (Remove-only)" colour: "bf00" 
ERROR 2024-08-02T07:09:10.766 Cannot parse the stash tab "CT2 (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.766 Cannot parse the stash tab "Hearts (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.766 Cannot parse the stash tab "F (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.766 Cannot parse the stash tab "DO (Remove-only)" colour: "bf00" 
ERROR 2024-08-02T07:09:10.766 Cannot parse the stash tab "Livers (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.766 Cannot parse the stash tab "~price 1 alch (Remove-only)" colour: "bf00" 
ERROR 2024-08-02T07:09:10.766 Cannot parse the stash tab "I1 (Remove-only)" colour: "bf00" 
ERROR 2024-08-02T07:09:10.767 Cannot parse the stash tab "~price 1 chaos (Remove-only)" colour: "bf00" 
ERROR 2024-08-02T07:09:10.767 Cannot parse the stash tab "Dump (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.767 Cannot parse the stash tab "~price 2 chaos (Remove-only)" colour: "bf00" 
ERROR 2024-08-02T07:09:10.767 Cannot parse the stash tab "~price 3 chaos (Remove-only)" colour: "bf00" 
ERROR 2024-08-02T07:09:10.767 Cannot parse the stash tab "~price 5 chaos (Remove-only)" colour: "bf00" 
ERROR 2024-08-02T07:09:10.767 Cannot parse the stash tab "I5 (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.767 Cannot parse the stash tab "~price 7 chaos (Remove-only)" colour: "bf00" 
ERROR 2024-08-02T07:09:10.767 Cannot parse the stash tab "Flasks (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.767 Cannot parse the stash tab "Soulrend (Remove-only)" colour: "bf00" 
ERROR 2024-08-02T07:09:10.767 Cannot parse the stash tab "PR (Remove-only)" colour: "bf00" 
ERROR 2024-08-02T07:09:10.767 Cannot parse the stash tab "~price 1 chaos (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.767 Cannot parse the stash tab "~price 5 chaos (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.767 Cannot parse the stash tab "Bug (Remove-only)" colour: "80" 
ERROR 2024-08-02T07:09:10.768 Cannot parse the stash tab "DE (Remove-only)" colour: "bf00" 
ERROR 2024-08-02T07:09:10.768 Cannot parse the stash tab "M (Remove-only)" colour: "ff" 
ERROR 2024-08-02T07:09:10.768 Cannot parse the stash tab "Crafting (Remove-only)" colour: "bf00" 
ERROR 2024-08-02T07:09:10.768 Cannot parse the stash tab "Private/Leveling (Remove-only)" colour: "80" 
ERROR 2024-08-02T07:09:10.768 Cannot parse the stash tab "~price 1 alch (Remove-only)" colour: "80" 
ERROR 2024-08-02T07:09:10.768 Cannot parse the stash tab "~price 1 alch (Remove-only)" colour: "80" 
ERROR 2024-08-02T07:09:10.768 Cannot parse the stash tab "Loreweave (Remove-only)" colour: "80" 
ERROR 2024-08-02T07:09:10.768 Cannot parse the stash tab "~price 1 gcp (Remove-only)" colour: "ff" 
ERROR 2024-08-02T07:09:10.768 Cannot parse the stash tab "ES Gear (Remove-only)" colour: "ff" 
ERROR 2024-08-02T07:09:10.768 Cannot parse the stash tab "Private (Remove-only)" colour: "80" 
ERROR 2024-08-02T07:09:10.768 Cannot parse the stash tab "~price 3 chisel (Remove-only)" colour: "ff" 
ERROR 2024-08-02T07:09:10.769 Cannot parse the stash tab "Ice Golem (Remove-only)" colour: "ff" 
ERROR 2024-08-02T07:09:10.769 Cannot parse the stash tab "Necro (Remove-only)" colour: "80" 
ERROR 2024-08-02T07:09:10.769 Cannot parse the stash tab "Talismans (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.769 Cannot parse the stash tab "Private (Remove-only)" colour: "80" 
ERROR 2024-08-02T07:09:10.769 Cannot parse the stash tab "~b/o 1 chaos (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.769 Cannot parse the stash tab "~b/o 1 chaos (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.769 Cannot parse the stash tab "~b/o 1 chaos (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.769 Cannot parse the stash tab "~b/o 1 chaos (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.769 Cannot parse the stash tab "~b/o 1 chaos (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.769 Cannot parse the stash tab "~b/o 1 chaos (Remove-only)" colour: "4d00" 
ERROR 2024-08-02T07:09:10.769 Cannot parse the stash tab "Graveyard (Remove-only)" colour: "80" 
ERROR 2024-08-02T08:00:45.400 Received a reply for "stash-request-limit" without rate limit headers. 
gerwaric commented 3 months ago

@livejamie I don't think those errors would have results in a rate-limit violation, but there's something strange about that last one, so it's possible. If you have time, could you copy and paste the 10-20 lines from your log file leading up to that error message:

ERROR 2024-08-02T08:00:45.400 Received a reply for "stash-request-limit" without rate limit headers

The other messages about parsing the color should be warnings at best. That's because the stash tab colour field is defined as having 6 digits in the current documentation, but on some legacy tabs it's 2 or 4 digits, so acquisition ignores the colors of those tabs. I'll change them in the next RC.

livejamie commented 3 months ago

I gave it a shot tonight and it appears to be working as expected.

image

gerwaric commented 3 months ago

Glad to hear it!

I'm planning to add the ability to update the list of stash tabs and characters without updating their contents, since just updating the lists will be almost instant.

However, I'm waiting until I can get more of the bugs people are reporting now fixed before adding features.