Open Eiion opened 3 years ago
I have to correct that - it's all pages that take forever to load with endless loading. And I'm indeed talking about minutes per page.
After further roaming around Steamgifts I have to come to the conclusion that after the update to 8.8.5 everything takes minutes with endless scrolling enabled. Disabling endless scrolling gets it partially back up to normal speed. Entering a giveaway still takes very long
EDIT: Even with endless scrolling disabled e.g. getting the group icons for groups we are members of shown is not working properly. While for about half the giveaways on the page everything is shown the rest isn't. (https://i.imgur.com/IlBkIuT.jpg)
EDIT 2: Just going to the entered page to see the first page with filters applied makes is a trial of patience to get everything shown what before the update only took maybe 5-10 seconds per page. Loading all the entered pages until the first giveaway that has already ended (which should only be a hand full) to sort them all by time entered (to find the last one that I've entered to test all the issues with ESGST) - impossible. Getting the second page in alone already takes minutes.
EDIT 3: Tired of waiting for several minute to get data on my screen for just one page with endless scrolling I started going through manually... still with endless scrolling enabled. Surprisingly page 4 got loaded after a long while with all filters applied while data for page 3 still is completely absent. I'm stopping here... this is bad. Really bad. To me it feels like 8.8.5 is completely broken.
EDIT 4: I didn't stop - I started again from page 1 of the entered pages and again, on page 3 filters don't get applied, still it only shows 7 giveaways while there's certainly more since there's a page 4.
EDIT 5: Even disabling all the ESGST filters via the slider on top doesn't help. e.g. now I'm also not getting the chance of winning percentage or game details displayed for page 2.
Ok, so apparently this is all due to "Rate limiting" by CG. 120 requests per minute with basically everything being a request. Considering before there was practically no limit this seems like overkill to me. Either way, limiting the request to only 1 per second also seems a bit overkill. Grez has mentioned on his comment in the SG discussions how extremely slow the groups page is - basically my issue. Though mine seems to be even worse if you move on to the entered page where it's 50 requests instead of 20 (or 25?). Also, considering how now one basically has to disable endless scrolling and go through page by page which now takes many minutes every single time instead of going to SG maybe 5 times a day, check the pages if there's e.g. a wishlist giveaway and leave again - which didn't take longer than a minute or two in the past... man, that's a huge blow to usability.
EDIT: I've noticed that "Entered Giveaways Stats" only shows N/A anymore. No idead if that's related to this issue as well or something else, but I assume it's all the same.
Not a bug, it's intended due to the rate limiting on SG. 1 per second is not overkill, since 2 per second could lead to 120, which is the minute limit. I'll be slowly working on improvements to individual features in the near future, but unfortunately there's not a lot else that I can do. You'll just have to be patient for now. As for things not loading properly, it could just be that you're making too many requests and there's a long queue, have you tried with a single tab open and doing a single task at a time?
I am only using one tab and I'm doing no task at all besides waiting for pages to load. Only then I start entering or leaving a giveaway or checking it's description (or whatever else you can do like bookmarking, hiding,...)
You could make the limit user adjustable. At least I wouldn't mind having 90 or 100 requests a minute instead of 60 - that would probably speed things up by a lot. And even if I hit the limit for one minute, I can still resume the next minute. In the end I still would be done in three to five minutes the most rather than 6-10.
EDIT: Actually you could still leave it at 120 with no delay at all. Let people have there results as quick as possible. That would be e.g. 2 pages of giveaways with 20 to enter/leave/bookmark/... - and then a timer makes them wait until the minute is over to make another 120 reuqests. Or lets them continue right away if not all 120 requests have been used. That seems in most cases much quicker than being slowed down with only 1 request per second when often 120 requests could be enough for everything one has to do in a visit. Or maybe let pages fully load in the background (which they currently don't do - as in open a tab with steamgifts, go watch a short video on Youtube before getting back to the SG tab) to have them ready when turning to them a few minutes later to then enter/leave/whatever. That again would be much faster than having to sit in front of the screen having to wait for a second for each GA data to appear for minutes to have e.g. all groups pages loaded in (again, that's only about 8 - 5 times a day, that's not like forcing the server to the ground. Even having SG in it's own window - when another window is in the front (again, e.g. Youtube) SG/ESGST doesn't continue to load GA data. Either way, that current given limit is - in my opinion - way to restrictive and (as with other cases where quick and dirty solutions have been implied on Steamgifts instead of better good ones) I only hope that CG adjusts the numbers. So I can get my things done on the page in a few minutes instead of making me wait for maybe up to an hour a day in total (I don't know, just rough estimating and maybe slightly overestimating - it certainly was an hour today).
Apparently the current update is out for 2 days already - when I only got it today and I still didn't have any of the 429 errors for the past two days. So apparently my load on the server was still within the limits with the previous version of ESGST.
An API could solve so many issues on that page and given how big it is and all around it that would be the best solution for everyone involved. But that surely won't ever happen.
You're forgetting that on top of the minute limit, there's also an hourly and a daily limit, so if too many requests are made you could be blocked from using SG for a whole day. And the main problem isn't actually that you could be blocked, but that this high number of requests in a short time is causing a stress on SG, which is exactly why cg implemented the rate limiting system.
Pages should already load in the background, though. If they're not doing that, it could be your browser not running scripts while a tab is inactive.
And you have to understand that your use case isn't everybody's use case. Several ESGST users have complained about getting 429 errors.
Like I said, for now, you'll just have to be patient. I'll be working on optimizing features. I just wanted to get an update out as soon as possible to deal with all of the users getting 429 errors. But now I can start reviewing features and setting different limits on a case-by-case basis. For example, I'm removing the limit from Enter/Leave Giveaway Button in the next version, since that feature shouldn't cause such a heavy load.
I didn't forget that - it just doesn't matter to me (and presumably even less to the average user) when I do what I do on SG only a few times a day. In total I wouldn't reach the limits. I understand that the stress is why he implemented the limits. But had he gone to twice the limits I imagine the impact would already have been massive, which is why I called these overkill before, considering there were no such limits before.
I'm just using basic Chrome broswer - no fancy different variant of it. I think last time it worked was when I was still using the script instead of the extension. But for sure the past year there's no loading of pages in the background. This is what should happen: I open SG in a new tab and the first page gets loaded as well as data of giveaways to which filters are applied. Because of the filters the list is shrinking to a point where endless scrolling loads in the next page - rinse and repeat until all pages (or at least some in most cases) are loaded. What actually happens: I open SG in a new tab and as soon as I leave the tab put the window in the background then no new pages are loaded in, no filters on frehsly shown GAs are applied and even loading GA data for already displayed games (like chance of winning, trading cards,...) stops.
Sure my case isn't everybody's use case. But the whole purpose of the limitation was to limit "power-users". So there should be a threshold. I actually consider myself quite more of a heavy user of the page then the average Joe who checks in maybe twice a day or less. And as said before, I still didn't get the errors in the past two days even without the limitations of the new extension version. So in my opinion the threshold should be at a point where it's still working as before for most users and only the extreme power-users who're putting the most stress on the servers, those because of whom there is now a limitation, should be those who should still run into the even strikter limitations by ESGST. In other words: It shouldn't be the average users who just need the script a few times a days to visit SG who are now running into issues when before they didn't (again, even though limitations existed for a few days before the ESGST update). I know you're still trying to figure things out, I'm just saying that for me personally - believing that I'm already using the script more heavy then most other users I personally know using it - now ESGST is causing issues that were non-issues from SG's point of view before the update.
Yes patience... I know. And even if we could fix issues within an hour - there's still days for Google to update the store. So... yeah. That's going to be a tough time.
Or maybe just let users select the request limits themselves? Add an option that allows them to put in any limit they want and if SG puts them on hold because they overdo it, it's within their own hands to adjust the limit. This would avoid several version updates until there's a fixed middle ground setting which would only work for some and not for others, most or not - who knows. This could also help finding out what works for most if you request feedback on which limitation settings works best for users of the script.
Pages load in the background just fine for me on Chrome, so I don't know.
For now, I'll just focus on optimizing features. I'd rather not allow users to set any limits they want, because if cg has set this threshold, ESGST should comply with it. I could definitely allow users to set a limit in the 0-100 range though, I'll implement that for the next version.
Anyway, your main problems appear to be coming from Created/Entered/Won Giveaway Details and Endless Scrolling. For CEWGD, the problem is definitely all those winner pages being loaded, which will be fixed in the next version. And for ES, I'm thinking about removing the limit from this feature, since it shouldn't really cause that much of a heavy load (a user has to scroll down to load the next page anyway, which might already take ~1 second), except maybe for the continuous load option.
Have you activated Chrome to be able to run in the background even when it's closed? I have that deactivated - but it shouldn't be the issue since I'm not closing chrome in these cases, it's just in the background.
I've googled for a Chrome setting/flag to let extension run in the background but found nothing. Is it possible that there's an issue with the background.js according to how it has to be set up or is there even one implemented in the extension? I didn't check as I'm currently on my phone?
It could be a "secret" setting as in one that could only be found via search, unlisted otherwise? Possibly even locked at a later point with another update. That would give users the chance to adjust it to their needs without being slowed down (as long as still within limits) without having to wait for probably multiple updates with adjusted settings in regards to limits. But seriously, if he sets limits and the adjustable range is equal to that range, maybe even one less e.g. 1-119, then that absolutely does comply.
It's also the All page - I think, I'll see when I'm on PC. Also, I'm not sure if the winner list checking is the issue as for e.g. the "die-with-glory-point-and-click-adventure-game (1)" entry there's no wonder list check. Though, that's only a part of the log and I don't know if there was a second request for that giveaway for the winner page. I'll check later to see if that's only for multi key giveaways or all of them.
That continuous load option? That's the one that pre-loads following pages without the user already scrolling, right? I've noticed that setting for first time yesterday and I'm not using it.
I've just checked the log for This Minute on the Won page - and it's not checking the winner for every giveaway - but for many:
This Minute (27 / Max: 120 / Left: 93)
https://www.steamgifts.com/giveaway/79sZm/the-little-acre (1)
https://www.steamgifts.com/giveaway/HZBz6/the-office-quest (1)
https://www.steamgifts.com/giveaway/RKQXW/monstrum/winners/search?page=1 (1)
https://www.steamgifts.com/giveaway/ABaGA/forgotten-in-hell (1)
https://www.steamgifts.com/giveaway/upFPD/randals-monday (1)
https://www.steamgifts.com/giveaway/o6JiY/gangsta-woman/winners/search?page=1 (1)
https://www.steamgifts.com/giveaway/I9sXL/the-king-of-fighters-2002-unlimited-match/winners/search?page=1 (1)
https://www.steamgifts.com/giveaway/bx623/butcher (1)
https://www.steamgifts.com/giveaway/oojG4/call-of-duty-2 (1)
https://www.steamgifts.com/giveaway/dui3K/deep-blue-3d-maze-in-space (1)
https://www.steamgifts.com/giveaway/sunlq/munin (1)
https://www.steamgifts.com/giveaway/P7xXq/rpg-tycoon/winners/search?page=1 (1)
https://www.steamgifts.com/giveaway/FViKh/pew-paw/winners/search?page=1 (1)
https://www.steamgifts.com/giveaway/Rncpa/lawnmower-game-4-the-final-cut/winners/search?page=1 (1)
https://www.steamgifts.com/giveaway/hf04U/tales-from-candlekeep-tomb-of-annihilation/winners/search?page=1 (1)
https://www.steamgifts.com/giveaway/o0LhE/amazeing-adventures/winners/search?page=1 (1)
https://www.steamgifts.com/giveaway/7T4gD/albino-lullaby-episode-1/winners/search?page=1 (1)
https://www.steamgifts.com/giveaway/qJjsA/psichodelya (1)
https://www.steamgifts.com/giveaway/crlFg/gods-trigger (1)
https://www.steamgifts.com/giveaway/VWrtD/tales-of-symphonia (1)
https://www.steamgifts.com/giveaway/IMNTj/renzo-racer (1)
https://www.steamgifts.com/giveaway/LXPQb/bendy-and-the-ink-machine (1)
https://www.steamgifts.com/giveaway/kAx4O/space-raiders-rpg/winners/search?page=1 (1)
https://www.steamgifts.com/giveaway/2S7Sa/fortresscraft-evolved (1)
https://www.steamgifts.com/giveaway/oJMWG/metal-slug-3 (1)
https://www.steamgifts.com/giveaway/dAqAI/wings-of-vi (1)
https://www.steamgifts.com/giveaway/S2bTI/run (1)
Loading the All pages doesn't seem to have the same issue for multi key giveaways controray to what I've assumed but oddly this just showed up in the log:
This Minute (1 / Max: 120 / Left: 119)
https://www.steamgifts.com/giveaways/search?type=wishlist (1)
Why would the wishlist page be requested when loading the second page with endless scrolling? (I'm adding that to the other issue about odd requests.) Also, this following request shows up every now and then - also on other pages if I recall correctly:
https://www.steamgifts.com/ajax.php
Yesterday evening and today things have sped back up at bit more. I'm assuming cached data was an issue as well (though I certainly hadn't removed any - I wouldn't know how). Just as a heads up. Still, looking forward to the proposed changes.
As for running in the background: I opened a tab with Steamgifts on the All page and lets it run. When pages 1-3 where loaded I went to other pages, Windows Explorer, Windows Settings - for about 15 Minutes I did all kind of things. When I returned to the Steamgifts tab it was still no further than when I left it. Only then when I had it back in foreground it started to load in more pages. So there's certainly something not working right with background activity. Is there anything I can do to find out or help finding out what the issue is?
I've also noticed a few considerably long stalls (several minutes with exactly zero requests happening), but the affected tabs eventually started doing their stuff again and ultimately completed it. This was particularly pronounced when I was doing a full sync, which only made about 200 requests to SG but took about 15 minutes to complete, with no or almost no other requests being made to SG by other tabs during the process. I'm not certain there's any relation to whether or not the offending tab is on the background, though, and in my experience those stalls were the exception, not the norm, but I figured they're worth mentioning anyway.
Today everything takes very extra long again... so I thought I'll have a look at the request numbers and although I consider myself a somewhat heavy user, seeing these numbers I guess I'm far from it. So that doesn't explain why everything is soo slow:
This Minute (0 / Max: 120 / Left: 120) This Hour (5 / Max: 2400 / Left: 2395) This Day (405 / Max: 14400 / Left: 13995) Last Minute (1) Last Hour (29) Last Day (601) <- which includes an automatic sync
Seeing this I'm very much within my limits. I have to admit that there might be days where I do twice as much as I did yesterday but even then this would still be well within any limits. And I can see no reason to limit the frequency of my requests - instead I'm sitting here waiting 10 minutes for all group pages to load, having to start over again because after several minutes the second or whatever other page isn't loading. As a result I fell like this makes the request issue worse than getting the data right away without possibly running into time outs causing users to reload everyting again.
While yesterday most of the day everything went fine, last night things started to basically stop working. Nothing past the first page would be loaded with endless scrolling enabled. Today, still the same as last night when I gave up trying to spend my points... I'm having the Group page open in the foreground and while the first page was there rather quick I'm now waiting for over 5 minutes for the second page to load - but nothing happens. I've tried before some hour ago, I even reconnected my internet connection to get a new IP and deleted all SG cookies - to no effect. It feels just like over 20 years ago, with 56kbit dail-up modems, where you had to wait for minutes for a picture to load in on the internet. Only back then you could watch the progress of it getting more and more completed until you eventually saw the full picture while this issues seems to be like an actual complete stop of anything.
These are the errors I'm getting with the last two repeating whenever ESGST is trying to load a page I guess:
Checking backup...
chrome-extension://ibedmjbicclcdfmghnkfldnplocgihna/esgst.js:1 1605897455743 NaN
3chrome-extension://ibedmjbicclcdfmghnkfldnplocgihna/esgst.js:88 Uncaught (in promise) TypeError: Cannot read property 'setAttribute' of undefined
at Ro.cfh_saveReply (chrome-extension://ibedmjbicclcdfmghnkfldnplocgihna/esgst.js:88)
at Ro.cfh_setReply (chrome-extension://ibedmjbicclcdfmghnkfldnplocgihna/esgst.js:88)
at Object.setPopout (chrome-extension://ibedmjbicclcdfmghnkfldnplocgihna/esgst.js:88)
at Ro.init (chrome-extension://ibedmjbicclcdfmghnkfldnplocgihna/esgst.js:88)
at async R.loadFeatures (chrome-extension://ibedmjbicclcdfmghnkfldnplocgihna/esgst.js:1)
at async t (chrome-extension://ibedmjbicclcdfmghnkfldnplocgihna/esgst.js:88)
search:161 USP is not accessible
checkIfUspIsReady @ search:161
chrome-extension://ibedmjbicclcdfmghnkfldnplocgihna/esgst.js:1 [ESGST] No result. Error: No result.
at qt.gc_getCategories (chrome-extension://ibedmjbicclcdfmghnkfldnplocgihna/esgst.js:88:233721)
at async qt.gc_getGames (chrome-extension://ibedmjbicclcdfmghnkfldnplocgihna/esgst.js:88:228013)
warning @ chrome-extension://ibedmjbicclcdfmghnkfldnplocgihna/esgst.js:1
chrome-extension://ibedmjbicclcdfmghnkfldnplocgihna/esgst.js:1 [ESGST]
warning @ chrome-extension://ibedmjbicclcdfmghnkfldnplocgihna/esgst.js:1
chrome-extension://ibedmjbicclcdfmghnkfldnplocgihna/esgst.js:1 [ESGST] No result. Error: No result.
at qt.gc_getCategories (chrome-extension://ibedmjbicclcdfmghnkfldnplocgihna/esgst.js:88:233721)
at async qt.gc_getGames (chrome-extension://ibedmjbicclcdfmghnkfldnplocgihna/esgst.js:88:228013)
warning @ chrome-extension://ibedmjbicclcdfmghnkfldnplocgihna/esgst.js:1
search:161 USP is not accessible
checkIfUspIsReady @ search:161
16chrome-extension://ibedmjbicclcdfmghnkfldnplocgihna/esgst.js:1 [ESGST] No result. Error: No result.
at qt.gc_getCategories (chrome-extension://ibedmjbicclcdfmghnkfldnplocgihna/esgst.js:88:233721)
at async qt.gc_getGames (chrome-extension://ibedmjbicclcdfmghnkfldnplocgihna/esgst.js:88:228013)
warning @ chrome-extension://ibedmjbicclcdfmghnkfldnplocgihna/esgst.js:1
gc_getCategories @ chrome-extension://ibedmjbicclcdfmghnkfldnplocgihna/esgst.js:88
DevTools failed to load SourceMap: Could not load content for chrome-extension://ibedmjbicclcdfmghnkfldnplocgihna/lib/browser-polyfill.min.js.map: HTTP error: status code 404, net::ERR_UNKNOWN_URL_SCHEME
esgst.js:1 [ESGST] No result. Error: No result.
at qt.gc_getCategories (chrome-extension://ibedmjbicclcdfmghnkfldnplocgihna/esgst.js:88:233721)
at async qt.gc_getGames (chrome-extension://ibedmjbicclcdfmghnkfldnplocgihna/esgst.js:88:228013)
This morning and afternoon everything was fine - not it's like life in slowmotion again... everything takes minutes to process. Two CPU cores run on 75-100 percent with SG open and the page gets time outs every other minute (those that give you the option to either wait or close the tab). Looking at the console I'm getting A LOT of errors and problem reported with ESGST again - just like last time:
DevTools failed to load SourceMap: Could not load content for chrome-extension://ibedmjbicclcdfmghnkfldnplocgihna/lib/browser-polyfill.min.js.map: HTTP error: status code 404, net::ERR_UNKNOWN_URL_SCHEME
(anonymous) @ cmp_vm.min.js:1
(anonymous) @ cmp_vm.min.js:1
content.js:63021 [Report Only] Refused to create a worker from 'blob:https://www.steamgifts.com/10882e87-be5b-49c5-ba52-bd0ca77ce410' because it violates the following Content Security Policy directive: "worker-src 'none'".
(anonymous) @ content.js:63021
__webpack_require__ @ content.js:20
(anonymous) @ content.js:23495
__webpack_require__ @ content.js:20
(anonymous) @ content.js:23466
__webpack_require__ @ content.js:20
(anonymous) @ content.js:23454
__webpack_require__ @ content.js:20
(anonymous) @ content.js:84
(anonymous) @ content.js:87
(index):1 [Report Only] Refused to create a worker from 'blob:https://www.steamgifts.com/10882e87-be5b-49c5-ba52-bd0ca77ce410' because it violates the following Content Security Policy directive: "worker-src 'none'".
esgst.js:1 Checking backup...
esgst.js:1 1611946560621 NaN
about:blank:1 POST about:blank net::ERR_UNKNOWN_URL_SCHEME
cmp_vm.min.js:1 USP is not accessible
(anonymous) @ cmp_vm.min.js:1
setInterval (async)
(anonymous) @ cmp_vm.min.js:1
(anonymous) @ cmp_vm.min.js:1
cmp_vm.min.js:1
That's from the All page. On Group it was about 5 times as many errors/prolems.
Here's some more errors from the same page after changing a filter (watched a 25 minute video in the meantime while waiting for the page to sort itself out):
chrome-extension://ibedmjbicclcdfmghnkfldnplocgihna/esgst.js:1 Cannot set property 'cache' of undefined
chrome-extension://ibedmjbicclcdfmghnkfldnplocgihna/esgst.js:1 Cannot set property 'cache' of undefined
chrome-extension://ibedmjbicclcdfmghnkfldnplocgihna/esgst.js:1 Cannot set property 'cache' of undefined
chrome-extension://ibedmjbicclcdfmghnkfldnplocgihna/esgst.js:1 Cannot set property 'cache' of undefined
chrome-extension://ibedmjbicclcdfmghnkfldnplocgihna/esgst.js:88 Uncaught (in promise) SyntaxError: Invalid regular expression: /^(.|(*)?)$/: Nothing to repeat
at new RegExp (<anonymous>)
at tc.elgb_openPopup (chrome-extension://ibedmjbicclcdfmghnkfldnplocgihna/esgst.js:88)
chrome-extension://ibedmjbicclcdfmghnkfldnplocgihna/esgst.js:88 Uncaught (in promise) SyntaxError: Invalid regular expression: /^(.|(*)?)$/: Nothing to repeat
at new RegExp (<anonymous>)
at tc.elgb_openPopup (chrome-extension://ibedmjbicclcdfmghnkfldnplocgihna/esgst.js:88)
chrome-extension://ibedmjbicclcdfmghnkfldnplocgihna/esgst.js:88 Uncaught (in promise) SyntaxError: Invalid regular expression: /^(.|(*)?)$/: Nothing to repeat
at new RegExp (<anonymous>)
at tc.elgb_openPopup (chrome-extension://ibedmjbicclcdfmghnkfldnplocgihna/esgst.js:88)
chrome-extension://ibedmjbicclcdfmghnkfldnplocgihna/esgst.js:1 Cannot set property 'cache' of undefined
DevTools failed to load SourceMap: Could not load content for chrome-extension://ibedmjbicclcdfmghnkfldnplocgihna/lib/browser-polyfill.min.js.map: HTTP error: status code 404, net::ERR_UNKNOWN_URL_SCHEME
esgst.js:1
By the way, this currently is for Chrome Version 88.0.4324.104 (Official Build) (64-Bit)
So... since there also were errors for Steam Web Integration I played around with settings a bit, turning off extensions back and forth and so on. Turns out: If I deavtivate ESGST then Steamgifts runs perfectly normal with Steam Web Integration turned on. If I activate ESGST and Steam Web Integration off then Steamgifts runs perfectly normal as well. But, if both ESGST and Steam Web Integration are turned on at the same time, that is when Steamgifts goes down the gutter and is basically useless unless you don't mind waiting for half an hour to get 6 pages loaded in.
The last version of SWI is 39 days old though so this makes me believe that this likely isn't the main source of the recent issues... I would guess that a Chrome update caused the incompatibilities of the two extensions since both work fine on their own and the only thing updated really is Chrome itself. I don't know though - I'm just guessing.
Changing Steam Web Integration from Observe to Check webpage every 1.5 seconds almost completely or even completely solves the recent issue. So maybe it's neither caused by Chrome nor SWI but possibly Steam...
Is everything here sorted or still a problem?
Regarding the "fix" I've told about in my last comment? Yes, without that it's still a massive issue. After about 5 minutes I'm still only at page 10 of the all page.
Description On the won page endless loading takes minutes to load a following page.
Steps to Reproduce
Expected Behavior Following pages get loaded without much waiting
Console Errors
System (please complete the following information):