simonk83
commented
3 years ago Just seems to be those two stores, so I've disabled them for now. The rest are behaving
Closed simonk83 closed 3 years ago
simonk83
commented
3 years ago Just seems to be those two stores, so I've disabled them for now. The rest are behaving
simonk83
commented
3 years ago Hmm now it’s doing it for Umart as well
TDCroPower
commented
3 years ago there all is well, you are probably just too slow and the purchase bots faster ;)
jef
commented
3 years ago there all is well, you are probably just too slow and the purchase bots faster ;)
Perhaps, but also the selectors may be off as well 😀 Did streetmerchant happen to get any screenshots for you? That'll help in debugging.
simonk83
commented
3 years ago Pretty sure it's not bots, as the stock situation here in Aus is pretty dire and they're not coming in that quickly, plus it's firing off way too many times to be real stock. I did attach some screenshots to the issue above, but here's everything I have from running just now.
If you need me to do any debugging or anything just yell out :)
simonk83
commented
3 years ago I had a look at the code for Mwave:
outOfStock: {
container: '.stockAndDelivery > li:nth-child(1) > dl > dd',
text: ['Currently No Stock']
}
},
I've tried to mess around with this but as my html is non-existent I'm just clutching at straws really. One strange thing though is that this page produces a false positive:
https://www.mwave.com.au/product/asus-geforce-rtx-3080-tuf-gaming-oc-10gb-video-card-ac38205
But this page doesn't:
https://www.mwave.com.au/product/evga-geforce-rtx-3080-ftw3-ultra-gaming-10gb-video-card-ac38322
I can't see a difference in the elements personally, but again I don't really know what I'm doing :)
jef
commented
3 years ago I see the problem. The user agent that it's using was a mobile user agent and the selectors are different for user agents. This needs to get updated.
Thanks!
simonk83
commented
3 years ago Ah great, happy to do it and push it up if you like as I think it needs to happen for a few of the Aussie ones. I'll see if I can find the right spot, otherwise I'll ask back here ;)
EDIT: Ok maybe it's not as simple as just changing it per store, I see you're using a dynamic user agent switcher. I'll leave it alone (unless it's simple enough, in which case point me in the right direction and I'll see what I can do. I have no html knowledge but I can scrape by with other stuff).
StanZha
commented
3 years ago as mentioned above pages are rendered for mobile device. Selectors are different
ui-mobile used
no stock found twice
this thing works properly for a page opened in desktop browser: .stockAndDelivery > li:nth-child(1) > dl > dd
simonk83
commented
3 years ago So streetmerchant always uses a mobile user agent? Or it cycles through different agents and sometimes it uses a mobile one? Presumably the latter as that would explain why sometimes it detects properly. If that’s the case it’s probably beyond me to try and fix it to account for both mobile and non mobile, I’ll leave it to the experts 😄
kin3tik
commented
3 years ago When the store model was submitted the USER_AGENT config item had just been deprecated in favour of a package that pulls in a random user-agent string - is that right @jef ? As per #1335
Am I correct in saying there are two options to fix this?
USER_AGENT (is this still possible?)simonk83
commented
3 years ago For now I'm using a non-mobile USER_AGENT (so yep, still possible) which has done the trick for mwave, but bpctech is really stubborn and constantly brings up in stock messages regardless, so I assume something else is going on there. They're extra expensive anyway so I've just disabled that store for now.
Also, and this likely can't be helped, but looks like Centrecom are smart and are detecting the activity and blocking IP's pretty quickly.
StanZha
commented
3 years ago I can't tell how useful it is to randomize user agents. In below example it uses weird mix of mobile browsers and desktop browsers across different OS. One of them is chrome 45 on android 4.2, that is very old. I removed all calls to getRandomUserAgent() and got all pages opened properly in desktop mode, at least for test 5 min run against Mwave. When user agent not set explicitly via page.setUserAgent correct user agent for corresponding chromium version will be used. On screenshot every time when user agent for mobile device was used, page was opened in mobile mode and produced false positive.
It seems second argument for winston logger should be some type of collection. When string passed it prints nothing in console output. I added square brackets and it started printing properly
StanZha
commented
3 years ago The random-useragent has property deviceType for user agents. If necessary this condition ua.browserName === 'Chrome' && ua.browserVersion > '20' can be modified to exclude mobile and tablet user agents. I don't see single deviceType defined as desktop, so I would assume when deviceType is empty it means desktop: https://github.com/skratchdot/random-useragent/blob/master/useragent-data.json
jef
commented
3 years ago Hey @StanZha, I've updated the user agent library and it's pulling from the top 50 UAs (not mobile). Can you see if this is still an issue for you?
Thanks!
When the store model was submitted the
USER_AGENTconfig item had just been deprecated in favour of a package that pulls in a random user-agent string - is that right @jef ? As per #1335Am I correct in saying there are two options to fix this?
- Add inStock and outOfStock selectors for mobile
- Explicitly specify a non-mobile
USER_AGENT(is this still possible?)
Yes, that use to be the case, but no longer. Try out the latest and see if you run into any issues.
StanZha
commented
3 years ago Thank you @jef, I tried to run recent code from main for 10 minutes and got all appropriate results - no pages opened in mobile or tablet mode for 3080/mwave store, no false positives observed.
I was just troubleshooting this problem, issue was opened by different user. @simonk83 can you confirm that changes are working as expected?
simonk83
commented
3 years ago Yep looking good! Thanks Jef
jef
commented
3 years ago Awesome! Thank you for the confirmation 😁
Expected Behavior
These seem to be false positives, so they should be seen as Out of Stock
Current Behavior
Bot marks them as in stock and opens the browser page
Steps to Reproduce
Here's a snippet of the output
Environment
OS: Mac OS Catalina dotenv file:
Logs