Improve request-backoff throttling

I've noticed (on stable v3.1.1.1) if you happen to get blocked by Steam/AkamaiGHost for some reason (they can be very sensitive), ASF repeatedly keeps retrying requests, leading to a longer and longer IP block.

In condition of connecting when blocked:

2018-03-16 12:19:01|dotnet-14890|INFO|STM|Connect() Connecting...
2018-03-16 12:19:02|dotnet-14890|INFO|STM|OnConnected() Connected to Steam!
2018-03-16 12:19:02|dotnet-14890|INFO|STM|OnConnected() Logging in...
2018-03-16 12:19:04|dotnet-14890|INFO|STM|OnLoggedOn() Successfully logged on!
2018-03-16 12:19:04|dotnet-14890|INFO|STM|Init() Logging in to ISteamUserAuth...
2018-03-16 12:19:05|dotnet-14890|INFO|STM|Init() Success!
2018-03-16 12:19:05|dotnet-14890|INFO|STM|IsAnythingToFarm() Checking first badge page...
2018-03-16 12:19:05|dotnet-14890|WARN|STM|UrlGetToStringRetry() Request failed after 5 attempts!
2018-03-16 12:19:05|dotnet-14890|DEBUG|STM|UrlGetToStringRetry() Request failing: https://steamcommunity.com/my/badges?l=english&p=1
2018-03-16 12:19:05|dotnet-14890|WARN|STM|IsAnythingToFarm() Could not get badges' information, we will try again later!
2018-03-16 12:19:05|dotnet-14890|WARN|STM|UrlGetToStringRetry() Request failed after 5 attempts!
2018-03-16 12:19:05|dotnet-14890|DEBUG|STM|UrlGetToStringRetry() Request failing: https://steamcommunity.com/dev/apikey?l=english
2018-03-16 12:19:06|dotnet-14890|WARN|STM|UrlPostRetry() Request failed after 5 attempts!
2018-03-16 12:19:06|dotnet-14890|DEBUG|STM|UrlPostRetry() Request failing: https://steamcommunity.com/gid/103582791440160998
2018-03-16 12:19:06|dotnet-14890|WARN|STM|UrlGetToStringRetry() Request failed after 5 attempts!
2018-03-16 12:19:06|dotnet-14890|DEBUG|STM|UrlGetToStringRetry() Request failing: https://steamcommunity.com/dev/apikey?l=english
2018-03-16 12:19:06|dotnet-14890|WARN|STM|UrlGetToStringRetry() Request failed after 5 attempts!
2018-03-16 12:19:06|dotnet-14890|DEBUG|STM|UrlGetToStringRetry() Request failing: https://steamcommunity.com/dev/apikey?l=english

ASF should have seen that my IP got blocked and backed off after the first badges request, but instead if kept hitting tons of times, making the problem worse.

Yes, "just don't get banned by AkamaiGHost, Steam problems aren't ASF's problem", but maybe this will convince you: if you send a ton of tradeoffers to an ASF user all at once, this victim's IP can end up blocked. To make matters worse, when a new offer comes in after reaching this block condition, ASF retries all these pending offers at the same time (it really doesn't care about ConfirmationsLimiterDelay for example), further making a huge problem.

2018-03-16 11:38:26|dotnet-3918|INFO|STM|ParseTrade() Accepting trade: 3084339671
2018-03-16 11:38:26|dotnet-3918|INFO|STM|ParseTrade() Accepting trade: 3084307886
2018-03-16 11:38:26|dotnet-3918|INFO|STM|ParseTrade() Accepting trade: 3084284886
2018-03-16 11:38:26|dotnet-3918|INFO|STM|ParseTrade() Accepting trade: 3084279018
2018-03-16 11:38:26|dotnet-3918|INFO|STM|ParseTrade() Accepting trade: 3084273375
2018-03-16 11:38:26|dotnet-3918|INFO|STM|ParseTrade() Accepting trade: 3084267242
2018-03-16 11:38:26|dotnet-3918|INFO|STM|ParseTrade() Accepting trade: 3084243124
2018-03-16 11:38:26|dotnet-3918|INFO|STM|ParseTrade() Accepting trade: 3084220145
2018-03-16 11:38:26|dotnet-3918|INFO|STM|ParseTrade() Accepting trade: 3084214422
2018-03-16 11:38:26|dotnet-3918|INFO|STM|ParseTrade() Accepting trade: 3084205099
2018-03-16 11:38:26|dotnet-3918|INFO|STM|ParseTrade() Accepting trade: 3084199135
2018-03-16 11:38:26|dotnet-3918|INFO|STM|ParseTrade() Accepting trade: 3084178315
2018-03-16 11:38:26|dotnet-3918|INFO|STM|ParseTrade() Accepting trade: 3084172488
2018-03-16 11:38:26|dotnet-3918|INFO|STM|ParseTrade() Accepting trade: 3084161075
2018-03-16 11:38:28|dotnet-3918|WARN|STM|UrlPostRetry() Request failed after 5 attempts!
2018-03-16 11:38:28|dotnet-3918|DEBUG|STM|UrlPostRetry() Request failing: https://steamcommunity.com/tradeoffer/3084284886/accept
2018-03-16 11:38:28|dotnet-3918|WARN|STM|UrlPostRetry() Request failed after 5 attempts!
2018-03-16 11:38:28|dotnet-3918|DEBUG|STM|UrlPostRetry() Request failing: https://steamcommunity.com/tradeoffer/3084339671/accept
2018-03-16 11:38:28|dotnet-3918|WARN|STM|UrlPostRetry() Request failed after 5 attempts!
2018-03-16 11:38:28|dotnet-3918|DEBUG|STM|UrlPostRetry() Request failing: https://steamcommunity.com/tradeoffer/3084214422/accept
2018-03-16 11:38:29|dotnet-3918|WARN|STM|UrlPostRetry() Request failed after 5 attempts!
2018-03-16 11:38:29|dotnet-3918|DEBUG|STM|UrlPostRetry() Request failing: https://steamcommunity.com/tradeoffer/3084273375/accept
2018-03-16 11:38:29|dotnet-3918|WARN|STM|UrlPostRetry() Request failed after 5 attempts!
2018-03-16 11:38:29|dotnet-3918|DEBUG|STM|UrlPostRetry() Request failing: https://steamcommunity.com/tradeoffer/3084267242/accept
2018-03-16 11:38:29|dotnet-3918|WARN|STM|UrlPostRetry() Request failed after 5 attempts!
2018-03-16 11:38:29|dotnet-3918|DEBUG|STM|UrlPostRetry() Request failing: https://steamcommunity.com/tradeoffer/3084243124/accept
2018-03-16 11:38:29|dotnet-3918|WARN|STM|UrlPostRetry() Request failed after 5 attempts!
2018-03-16 11:38:29|dotnet-3918|DEBUG|STM|UrlPostRetry() Request failing: https://steamcommunity.com/tradeoffer/3084205099/accept
2018-03-16 11:38:29|dotnet-3918|WARN|STM|UrlPostRetry() Request failed after 5 attempts!
2018-03-16 11:38:29|dotnet-3918|DEBUG|STM|UrlPostRetry() Request failing: https://steamcommunity.com/tradeoffer/3084199135/accept
2018-03-16 11:38:29|dotnet-3918|WARN|STM|UrlPostRetry() Request failed after 5 attempts!
2018-03-16 11:38:29|dotnet-3918|DEBUG|STM|UrlPostRetry() Request failing: https://steamcommunity.com/tradeoffer/3084178315/accept
2018-03-16 11:38:29|dotnet-3918|WARN|STM|UrlPostRetry() Request failed after 5 attempts!
2018-03-16 11:38:29|dotnet-3918|DEBUG|STM|UrlPostRetry() Request failing: https://steamcommunity.com/tradeoffer/3084172488/accept
2018-03-16 11:38:29|dotnet-3918|WARN|STM|UrlPostRetry() Request failed after 5 attempts!
2018-03-16 11:38:29|dotnet-3918|DEBUG|STM|UrlPostRetry() Request failing: https://steamcommunity.com/tradeoffer/3084161075/accept
2018-03-16 11:38:29|dotnet-3918|WARN|STM|UrlPostRetry() Request failed after 5 attempts!
2018-03-16 11:38:29|dotnet-3918|DEBUG|STM|UrlPostRetry() Request failing: https://steamcommunity.com/tradeoffer/3084279018/accept
2018-03-16 11:38:29|dotnet-3918|WARN|STM|UrlPostRetry() Request failed after 5 attempts!
2018-03-16 11:38:29|dotnet-3918|DEBUG|STM|UrlPostRetry() Request failing: https://steamcommunity.com/tradeoffer/3084220145/accept
2018-03-16 11:38:29|dotnet-3918|WARN|STM|UrlPostRetry() Request failed after 5 attempts!
2018-03-16 11:38:29|dotnet-3918|DEBUG|STM|UrlPostRetry() Request failing: https://steamcommunity.com/tradeoffer/3084307886/accept

All in all, it's my opinion that this would be a desired change for ASF: if ASF sees that you're blocked (the response usually includes You don't have permission to access "http://steamcommunity.com/........." on this server.), it should globally back off from all external requests -- no further bots should continue to request steamcommunity until it's possible again.

This will probably be quite difficult (all the different threadings, how it should cooperate with the different *LimiterDelay config options, etc), and I can understand if you're not interested in this simply due of that, but please consider the benefits of it.

Hope this issue was clear, and thank you for your time!

You didn't post any core details which would make this issue possible to evaluate.

When exactly is the block triggered, by which actions, globally, locally, is it the amount of requests, amount of open connections, concurrent connections, specific resource or maybe some other matter considered?
After what time the block disappears, is it fixed time since last request? When should ASF retry after hitting first 403?
What resources are blocked? Does the block affect only steamcommunity part, or maybe steam store, steam API, steam network too?

Without answer to those 3 questions I can't even consider this issue in the first place since I don't know what ASF is supposed to do - saying "it should not send requests" does not specify any details. Do you want to globally stop all bots once first 403 is received? Because I guess not, and if not, then I must know the details in order to find the best solution.

The issue itself is OK and I guess we could further enhance ASF's logic in this case, but once you provide all details that are crucial to code it in the first place. At worst, time of the block and scope (steamcommunity, steam store, maybe both? If not both, are they independent and logic should be applied independently for one and another, or maybe just steamcommunity suffers from this?).

I understand your questions, but I'm not an expert on it either -- and don't think anyone outside of Valve HQ could really say for certain, because so much of Steam is a black-box. Will try to connect my personal experiences:

When exactly is the block triggered

After too many requests per minute to Steam's web servers. There are a few different blocks I know of:

Too many requests to "normal json-rpc" endpoints that are used in Steam's built-in functions (eg https://steamcommunity.com/inventory/76561198006963719/753/1?l=english&count=75). As these endpoints are expected to get lots of hits by legitimate client activity, it doesn't do a full block -- just returns null.
Too many requests to normal pages, or endpoints that aren't intended for intensive use.

Run this once or twice, and then you will get a nice AkamaiGHost IP block from all SteamCommunity links: for i in {1..20}; do curl -sS "https://steamcommunity.com/market/listings/578080/MILITIA%20CRATE" -o/dev/null & done;

(This case is the same as with my example in first post, when you're accepting too many tradeoffers at once)

by which actions, globally, locally, is it the amount of requests, amount of open connections, concurrent connections, specific resource or maybe some other matter considered?

Block seems to be 100% IP-based (and turning on a VPN/changing IP gets you instantly past it). Don't think it has any relation to open connections;; ASF's WebSocket login connections remain stable to the Steam network even when blocked by SteamCommunity.

After what time the block disappears, is it fixed time since last request? When should ASF retry after hitting first 403?

I don't know((( But I think it's some kind of bucket: if you get blocked, you're blocked for ~10 minutes, but each further request adds a small amount of time to the bucket. In the current case of ASF, it's hammering the bucket, so it takes a longer time until reset, rather than just backing off for a short while and retrying once.

What resources are blocked? Does the block affect only steamcommunity part, or maybe steam store, steam API, steam network too?

Very smart question. store.steampowered and steamcommunity have similar blocking mechanisms (requesting too many pages per minute get you an AkamaiGHost block), but the blocks are separate from each other. So you can be blocked from Community, but still able to use Store, or vice-versa. API, bit harder for me to say. I think it's in its own block as well, but a block gives you some kind of null response, not an AkamaiGHost block. Steam Network has its own functionality for blocking, so even when blocked from Store or Community, you can still log in and such -- you just won't be able to use most ASF functionality because it depends on those other domains.

Do you want to globally stop all bots once first 403 is received?

I want it to identify when there's some problem, and hold back on any new HTTP requests to the domains you're blocked on. An incomplete solution that would still help a lot could even just be seeing that the response from steamcommunity is from AkamaiGHost saying "You don't have permission to access.*", or combined with header when blocked:

< HTTP/1.1 403 Forbidden
< Server: AkamaiGHost

vs normal Apache header when not blocked. (Though I think it's better to see that it's AkamaiGHost, in case Valve decides to switch to nginx or something :))

HTH, let me know if I missed something big here, or some other input I can give, thanks!

Run this once or twice, and then you will get a nice AkamaiGHost IP block from all SteamCommunity links: for i in {1..20}; do curl -sS "https://steamcommunity.com/market/listings/578080/MILITIA%20CRATE" -o/dev/null & done;

I tried to reproduce your akamai block and didn't succeed. Instead, steamcommunity replied to me with 429:

                        <h1>Sorry!</h1>
                        <p class="sectionText">
                                An error was encountered while processing your request:<br><br>
                        </p>
                        <h3>You've made too many requests recently. Please wait and try your request again later.</h3><br><br>

Unless you can come up with reproducable ASF pattern that brings the block you're talking about, I'll consider it out of scope since I don't see how ASF could run into your problem by its own actions. ArchiBoT once parsed like 50 trades at once and didn't get any block, I've also never experienced what you're claiming, and I tried to reproduce it not only by following your own method but by following mine own ones as well.

OK, took me some serious DoS but I reached what you claim:

root@debian:~# curl -I "https://steamcommunity.com/market/listings/578080/MILITIA%20CRATE"
HTTP/1.1 403 Forbidden
Server: AkamaiGHost
Mime-Version: 1.0
Content-Type: text/html
Content-Length: 317
Expires: Fri, 13 Apr 2018 03:52:34 GMT
Date: Fri, 13 Apr 2018 03:52:34 GMT
Connection: close

I still don't see how ASF can reach THAT big amount of requests, considering there is global limit of 10 open connections at any given time, I'll need to check if the same hammering while maintaining 10 outstanding jobs at a time leads to that.

After deeper analysis:

Block happens based on number of requests sent across given time period, which isn't connected with number of active connections.
I managed to send over 500 requests with 200 ms delay between each without triggering the block, while clearly getting one after ~50 requests with 100 ms of delay.

Based on the above, ASF received an enhancement that ensures better compatibility in regards of above rate-limiting. The code is good enough in order to:

Limit 3 services (API, store and community) independently, since as you found out yourself and I confirmed, they're calculated (and therefore blocked) within a local scope. This means that they can be all accessed (and counted) in parallel.
Queue requests and never allow to spawn more than one per 200 ms.
If we reach maximum number of connections to given ServicePoint (which is currently 10), wait for open connection before queueing the next request. Thanks to that we won't cause a backlog of ongoing requests, therefore hammering Steam in case of it being unresponsive (as we could create 50 requests across 10 seconds and execute them all at once because Steam wasn't responsive during those 10 seconds and weren't able to catch up with us).

Thanks to that ASF alone is not able to trigger rate-limiting that you described above, which is how it should be done instead if trying to workaround the issue that should not happen in the first place. If you have other services accessing the same limited resources then you should double WebLimiterDelay and use similar method for all other tools, so e.g. using ASF alone with default of 200 ms, or using ASF and another tool of your choice, both having a delay of 400 ms.

JustArchiNET / ArchiSteamFarm

Improve request-backoff throttling #764