Closes #3017

[JustArchi]

So, this started really badly by me finding out that in "clean" issue proposal I have over 1000 games to check, out of which not a single one has any card drops remaining. Majority of those were F2P games (where indeed, we can craft booster pack but don't have booster pack eligibility), on top of possibilities of some removed games and what else.

Therefore, this PR comes with a lot of enhancements on top of original proposal to minimize the risk of getting Steam banned and other DoS measures by doing as much 200 IQ moves as possible:

• Firstly, we can enable the additional logic by setting EnableRiskyCardsDiscovery bot config property to true. Without that, nothing changes in regards to before.
• Once enabled, we use additional fallback logic that triggers in two situations:
  • We failed loading first badge page, or
  • We failed loading at least one other badge page and successes from remaining ones resulted in zero games to farm
• Fallback logic fetches appIDs that we can craft boosters for, and appIDs that we're eligible for getting boosters from. Since first one means we own the game, and second one means we already farmed everything possible (exceptions possible), the subtraction gives us games that we can potentially idle for more cards.
• Just because we found a game like that, doesn't mean anything due to exceptions mentioned above. Since every single game check means firing one request, we absolutely can't afford checking e.g. 1000 games at once, as that'd guarantee a Steam ban and DoS, violating our contributing guidelines on top of being stupidly dangerous.
• This is why we're sorting this list further:
  • We apply our appID rules right away to filter out entries that make no sense to be checked, i.e. sales blacklist, global blacklist, bot blacklist, package-data temporary failures, priority queue only rules
  • Then we begin sorting remaining results. We start with cache hits (explained below), and move on to remaining rules
  • Next we order results by priority queue right away, we want to give user a way to prioritize not only final sorting but also the checking, we don't do other sorting (i.e farming orders)
  • Finally we apply randomization on top, that is, we firstly check cache hits (randomly), then prioritized appIDs (randomly) and then everything else (randomly), to ensure we have as little misses in a row as possible regardless of user settings
• Each miss we cache for up to 7 days in bot database, which is not perfect (tomorrow you can spend money in cs:go and have cards available), but it's required to not check the same 1k entries every 3 hours
• We also cache each hit for as long as we determine it's not yet fully farmed. This allows us to be smart about checking games after farming restart, since if in the previous run we already spent significant time to find those 3 out of 900 entries we were missing, we should start from checking them first.
• Since this approach is greedy and risky, we stop game checking as soon as we find, either:
  • 1 game with card drops and >= 3 hours (configurable HoursUntilCardDrops)
  • or 32 games with card drops total.
• This is not perfect but we can't afford to check 1k entries purely to allow user to sort them in FarmingOrders, we need to find a balance between performance and greedyness, this allows us to be as fast as possible in regards to performance, and it's sufficient for our needs.
• When we run out of games to farm (earlier than in normal mode, due to stopping at 32 games at the most), we do everything from beginning, starting from trying to use badge pages (again).

This overly complex approach guarantees us the following:

• As soon as user is able to use badge pages, we use them, since it's important to note that the whole logic in this PR is risky and unwanted in long-term, and we should do our best to minimize using it. This is why it's disabled by default and even if user enables it, we don't use it if possible.
• Once we determine that we need to use it, we try to send as little requests and overhead as possible. This is why we filter, sort, cache, prioritize and skip a lot of stuff, to ensure that people won't get banned by using this option. The worst possibility is, indeed, sending 1k requests or so to finding out what we need, but since this logic is expected to fire only for people that have excessive number of card drops, and they should have those games rather easily found when using this approach, I don't expect from this approach to put a lot of load on the Steam servers.
• At the same time we're still prioritizing ASF functionality and performance. We do our best to allow people to idle in Simple and Complex modes, we also respect their settings (such as idle priority queue and farming orders), even if some of them are applied only on a subset of all the results (that we had to limit due to overhead reasons).

All things considered, I'm very proud of this 200 IQ approach to solving original #3017 issue, I believe this should not cause as much harm as I initially expected when I found out that the most simple approach results in 1k requests each 3 hours, which would be dramatic in outcome.

Reviews welcome. @MavericksNightmare I'd appreciate if you could test if this addresses your original issue since I obviously don't have a test account with such situation, so a bit of this code remains untested. You can do so by picking top build from here and grabbing artifact matching your ASF setup from the bottom. You need to manually put "EnableRiskyCardsDiscovery": true in your bot config. Let me know your thoughts, whether it works or doesn't :+1:

Closes #3017

[github-actions[bot]]

Qodana for .NET

It seems all right 👌

No new problems were found according to the checks applied

💡 Qodana analysis was run in the pull request mode: only the changed files were checked ☁️ View the detailed Qodana report

Contact Qodana team

Contact us at [qodana-support@jetbrains.com](mailto:qodana-support@jetbrains.com) - Or via our issue tracker: https://jb.gg/qodana-issue - Or share your feedback: https://jb.gg/qodana-discussions

[MavericksNightmare]

Wow, that's quite a solution you came up with here ... and in such a short time!

I ran it on my account with the following result: After unsuccessfully trying to load badge overview pages for 5 times in a row, ASF started loading booster creator and eligibility pages and according to the logfile 7 of my badge pages. From these 7 badge pages, it identified 5 games with a total of 19 cards left to farm and an estimated fariming time of 12 hours 30 mins. Farming started immediately as usual with no issues. After farming of the first game was finished, I checked my Booster Eligibility list and found the game in there. This should rule out this game as candidate to farm at the next iteration and narrow search to those games still requiring farming.

Looks to me like it works as intended. Thanks a lot for it! I will definitely continue to use it for farming cards on my account!

The 5 games found in the initial run is actually only a minor fraction of games I have left to farm as I have not been able to farm cards for a long time and must have accumulated easily 100+ games with cards since then. I assume the number of 5 games is due to the limitations metioned above to avoid excessive querying of badge pages. I fully agree that limitations like these are necessary in order to rule out high server load / violations of Steam terms and have no issue with that. I am confident that repeating farming under the approach you suggested, including randomization, will eventually allow me to farm at least the vast majority of games over time.

As you pointed out, the majority of games not showing up on Booster Eligibility list will usually be F2P games. I do not really think F2P games need to be included into the "RiskyCardsDiscovery" solution. Each card drop requires to spend 10 USD on ingame items which many owners of F2P games will never do. So the vast majority of F2P games is irrelevant for farming upfront. The minority of F2P game owners who actually do spend 10 USD for ingame items will usually do this because they actually want to play the game. So they will most likely receive card drops anyway by playing and will usually not need a solution like ASF to farm the first card. In addition, the "RiskyCardsDiscovery" solution cannot help with farming second, third card drops etc. anyway as Booster Eligibility is awarded along with first card drop. This should add the F2P game to Booster Eligibility list such that it henceforth will be ruled out completely by the check against that list. So the only kind of benefit of including F2P games into "RiskyCardsDiscovery" solution is to provide assistance to a few people having spent 10 USD in the shop and lost interest in playing before having received first card drop. This very limited benefit comes at the high cost of adding a large number of irrelevant games to the checks. I therefore think F2P games should be ruled out completely from this solution if possible. The small benefit is just not worth the big increase in effort. Maybe ruling them out is not so easy though as this would require data about which appid is F2P and which one is not.

One more thing: According to Steam rules, keeping up booster eligibility requires to login at least once every 7 days. I have no idea what would happen to Booster Eligibility list if this 7-days-requirement was violated. Will it come up empty although a substantial number of games actually has been farmed? ASF logins count as valid logins with respect to this 7-days-requirement. So maybe the login preceding the farming process itself is already enough to guarantee that the list never be empty, even if 7 days after the previous login have passed. But maybe Booster Eligibility list is not refresed in real time so it might still come up empty. You may want to check whether unexpected behavior like this could maybe cause issues and require special handling.

[JustArchi]

Great to hear it @MavericksNightmare, will merge in a second then!

After farming of the first game was finished, I checked my Booster Eligibility list and found the game in there. This should rule out this game as candidate to farm at the next iteration and narrow search to those games still requiring farming.

Yes, and even if it didn't, we actually do record the game as ignored (cache miss) for 7 days once it's finished farming, so whether Steam adds it to the list immediately or not should not make a difference for us. Worst case we check it manually with a request, find out no card drops, then cache a miss.

I am confident that repeating farming under the approach you suggested, including randomization, will eventually allow me to farm at least the vast majority of games over time.

That's the point, what ASF will state (you have X games, Y card drops left) is only a part of the bigger picture, due to the fact that calculating bigger picture is overly costly in regards to this method. You will get whole picture the moment badge pages start to work - for this algorithm, it's sufficient for ASF to have the minimum subset of drops that allows it to operate with optimal speed.

I therefore think F2P games should be ruled out completely from this solution if possible. The small benefit is just not worth the big increase in effort. Maybe ruling them out is not so easy though as this would require data about which appid is F2P and which one is not.

The last part. Finding out which games are F2P and are possible to be skipped is even more requests, and considering the fact that you can't be sure we don't have any drops in them, it's better to leave things as they are and count on cache misses to do their job. Checking each F2P game once per week should not cause big overhead, and together with randomization, should not cause us overflow of bad requests in a row.

You may want to check whether unexpected behavior like this could maybe cause issues and require special handling.

Using ASF preserves boosters eligibility, that one we already checked. Considering the fact that you're logged in with Steam network and everything, this is such an edge case that I don't believe is worth doing anything for - worst case we spend more time finding games since we'll be evaluating whole list rather than a subset of it, but with our approach of randomization and everything, it should still finish sooner rather than later, and this is extreme for user to not be able to use badge pages, not logging in to Steam for prolonged amount of time, and the boosters eligibility not updating in time after us logging in. It's fine imho.

Thanks for testing! I'll get down to merging it and writing docs then, you can use this test version for now, it'll automatically update to next stable/prerelease (according to your update channel), or you can also update manually the moment I release next version if you'd like to.