Limit requests to SteamGifts

[at46]

Is your enhancement request related to a problem? Please describe. Starting yesterday some people (including myself) are getting 429 error from steamgifts for too many requests while using esgst see here or here.

Describe the solution you'd like. Limit the amount of requests ESGST sends to SG

Additional Context Affected features seems to be bookmarks, syncing1, syncing2 and decrypted giveaways.

[rafaelgomesxyz]

On it.

[marlop352]

CG posted the current limits https://www.steamgifts.com/discussion/yLOhv/rate-limiting-limits

[rafaelgomesxyz]

@marlop352 Thanks for letting me know!

This is gonna be a bit tough, specially considering that so many features make a ton of requests. Giveaway Extractor, for example, would easily reach the hourly limit in large trains, although I don't think those are super common.

I think at first I'll just limit all requests to SteamGifts, because that's a simple solution, and then I can slowly deal with each problematic feature.

[at46]

Does this also affect your server e.g. for fetching all the bundled games from sg? And could you provide a list of problematic features so that we can turn them off for now?

[rafaelgomesxyz]

It does, but it shouldn't be an issue, because there are 653 bundle pages, so I can adapt it to make 120 per minute and it should be good.

Unfortunately there are so many features that making a list would take a while. I'm thinking of adding some stats to the settings menu that show you how many requests have been made to SteamGifts and to what pages. This should help identify which features are making those.

[rafaelgomesxyz]

I've limited SteamGifts requests to 1 per second. This ensures that, if continuous requests are being made, a maximum of 60 requests will be made per minute, which is way below the 120 limit and leaves 60 requests for browsing SG.

If this turns out to be too slow, I can think about lowering it.

[Eiion]

Please speed it up quite a bit more or remove it completely from endless scrolling. Endless scrolling is now a pain to use even on the group page that's only about 8 pages long for me (assuming this is what causes the issue).

[at46]

@Eiion have you checked what requests you see in the SteamGifts Request Log (ESGST drop-down menu between Donations and Current Version). Also are you talking about https://www.steamgifts.com/account/steam/groups or https://www.steamgifts.com/giveaways/search?type=group when you say group page? Endless scrolling works fine for me so it is probably some other feature making a lot of requests which slows down everything.

[Eiion]

@at46 I was talking about https://www.steamgifts.com/giveaways/search?type=group I checked that log just now - I hadn't known of it before. Going to the won page (that shows 50 GAs at once) it takes like forever. And then the page got stuck with the last three entries and the log showing this:

https://www.steamgifts.com/giveaway/xVbZM/men-of-war-assault-squad-2-gold-edition/winners/search?page=1 (1)
https://www.steamgifts.com/giveaways/won/search?page=3 (1)
https://www.steamgifts.com/giveaway/L6TJy/super-squash/winners/search?page=40 (1)
https://www.steamgifts.com/giveaway/L6TJy/super-squash/winners/search?page=39 (1)
https://www.steamgifts.com/giveaway/L6TJy/super-squash/winners/search?page=38 (1)
https://www.steamgifts.com/giveaway/L6TJy/super-squash/winners/search?page=37 (1)
https://www.steamgifts.com/giveaway/CBUps/tragedy-of-prince-rupert/winners/search?page=1 (1)
https://www.steamgifts.com/giveaway/vHgKg/die-with-glory-point-and-click-adventure-game (1)
https://www.steamgifts.com/giveaway/L6TJy/super-squash/winners/search?page=36 (1)
https://www.steamgifts.com/giveaway/L6TJy/super-squash/winners/search?page=35 (1)
https://www.steamgifts.com/giveaway/L6TJy/super-squash/winners/search?page=34 (1)
https://www.steamgifts.com/giveaway/L6TJy/super-squash/winners/search?page=33 (1)
https://www.steamgifts.com/giveaway/L6TJy/super-squash/winners/search?page=32 (1)
https://www.steamgifts.com/giveaway/L6TJy/super-squash/winners/search?page=31 (1)
https://www.steamgifts.com/giveaway/L6TJy/super-squash/winners/search?page=30 (1)
https://www.steamgifts.com/giveaway/L6TJy/super-squash/winners/search?page=29 (1)
https://www.steamgifts.com/giveaway/L6TJy/super-squash/winners/search?page=28 (1)
https://www.steamgifts.com/giveaway/L6TJy/super-squash/winners/search?page=27 (1)
https://www.steamgifts.com/giveaway/L6TJy/super-squash/winners/search?page=26 (1)
https://www.steamgifts.com/giveaway/L6TJy/super-squash/winners/search?page=25 (1)
https://www.steamgifts.com/giveaway/L6TJy/super-squash/winners/search?page=24 (1)
https://www.steamgifts.com/giveaway/L6TJy/super-squash/winners/search?page=23 (1)
https://www.steamgifts.com/giveaway/L6TJy/super-squash/winners/search?page=22 (1)
https://www.steamgifts.com/giveaway/L6TJy/super-squash/winners/search?page=21 (1)
https://www.steamgifts.com/giveaway/L6TJy/super-squash/winners/search?page=20 (1)
https://www.steamgifts.com/giveaway/L6TJy/super-squash/winners/search?page=19 (1)
https://www.steamgifts.com/giveaway/L6TJy/super-squash/winners/search?page=18 (1)
https://www.steamgifts.com/giveaway/L6TJy/super-squash/winners/search?page=17 (1)
https://www.steamgifts.com/giveaway/L6TJy/super-squash/winners/search?page=16 (1)
https://www.steamgifts.com/giveaway/L6TJy/super-squash/winners/search?page=15 (1)
https://www.steamgifts.com/giveaway/L6TJy/super-squash/winners/search?page=14 (1)
https://www.steamgifts.com/giveaway/L6TJy/super-squash/winners/search?page=13 (1)
https://www.steamgifts.com/giveaway/L6TJy/super-squash/winners/search?page=12 (1)
https://www.steamgifts.com/giveaway/L6TJy/super-squash/winners/search?page=11 (1)
https://www.steamgifts.com/giveaway/L6TJy/super-squash/winners/search?page=10 (1)
https://www.steamgifts.com/giveaway/L6TJy/super-squash/winners/search?page=9 (1)
https://www.steamgifts.com/giveaway/L6TJy/super-squash/winners/search?page=8 (1)
https://www.steamgifts.com/giveaway/L6TJy/super-squash/winners/search?page=7 (1)
https://www.steamgifts.com/giveaway/L6TJy/super-squash/winners/search?page=6 (1)
https://www.steamgifts.com/giveaway/L6TJy/super-squash/winners/search?page=5 (1)
https://www.steamgifts.com/giveaway/L6TJy/super-squash/winners/search?page=4 (1)
https://www.steamgifts.com/giveaway/L6TJy/super-squash/winners/search?page=3 (1)
https://www.steamgifts.com/giveaway/L6TJy/super-squash/winners/search?page=2 (1)
https://www.steamgifts.com/giveaway/L6TJy/super-squash/winners/search?page=1 (1)
https://www.steamgifts.com/giveaway/H0tSv/ctu-counter-terrorism-unit/winners/search?page=1 (1)
https://www.steamgifts.com/giveaway/XyHFw/the-walking-dead-season-two/winners/search?page=1 (1)

No idea what that's all about... why ESGST is checking for the winners of that giveaway when I was the winner. That giveaway has 1000 winners - but how's that relevant for my Won page.

Other than that it seems like just simple giveaway links - nothing else. But again, with 50 GAs on the won page that's at least 50 seconds for just that one page. But it even takes much longer than that for endless scrolling to move on to the second page... restarting the broswer and the PC didn't do anything at all. What took seconds or 1-2 minutes before to get done now takes 15 minutes.

[at46]

@Eiion ESGST checking all winner of that giveaway seems to be a bug and should be reported in #1660 (Edit: Nevermind, I just saw now that you already did that). Besides that as far as I've tested it giveaway groups and for the won page also Created/Entered/Won Giveaway Details are slowing down the page loading.