search

Prince25 / StockAlertBot

Faced with the scenario of scalpers using bots to hog up all the inventory of popular holiday toys and sell them at ridiculously high markup prices, I decided to put up a fight so we can get our hands on things we need to survive the #Coronavirus quarantine(s).
https://github.com/PrinceS25/StockAlertBot/wiki/Beating-Scalpers
MIT License
231 stars 44 forks source link

2 different errors when executing #7

Closed jpall12 closed 3 years ago

jpall12 commented 3 years ago

(node:2180) Warning: Setting the NODE_TLS_REJECT_UNAUTHORIZED environment variable to '0' makes TLS connections and HTTPS requests insecure by disabling certificate verification. (Use node --trace-warnings ... to show where the warning was created)

and

Unhandled error for Newegg. Written to logNewegg.log

jpall12 commented 3 years ago

This maybe my error but I cant seem to find why its happening.

Any help is appreciated

Prince25 commented 3 years ago

The first error warning is normal and is not a concern. For the second, can you open logNewegg.log and tell me what it says?

jpall12 commented 3 years ago

So I can't seem to find the logNewegg.log

Where would it normally be?

jpall12 commented 3 years ago

Also I'd like to thank you for your quick response.

jpall12 commented 3 years ago

lol I'm an idiot, found the log here is what it says.

TypeError: Cannot read property 'innerHTML' of undefined at Timeout.newegg [as _onTimeout] (file:///D:/julia/Frenzy%20Flips/Monitors/StockAlert/StockAlertBot/stores/newegg.js:39:71) at processTicksAndRejections (internal/process/task_queues.js:93:5)

Prince25 commented 3 years ago

How often does this error occur? If it doesn't often, refer to #5

Prince25 commented 3 years ago

@jpall12 Has this been resolved?

jpall12 commented 3 years ago

Hi sorry for the slow response. It does seem to happen relatively often. check attached ss. image

jpall12 commented 3 years ago

also I did a test with an item that was in stock and it didn't post to my discord webhook is this expected?

Jeisonx commented 3 years ago

also I did a test with an item that was in stock and it didn't post to my discord webhook is this expected?

I can potentially try to take a look at your issue, would you mind providing the URLs you are searching on Newegg, also when you attempted to send an in stock notification to discord did the console log reproduce an error message to why it was not able to send the notification to discord?

jpall12 commented 3 years ago

"https://www.newegg.com/p/N82E16868110293?", "https://www.newegg.com/p/N82E16868110291?", "https://www.newegg.com/Product/ComboDealDetails?ItemList=Combo.4205033", "https://www.newegg.com/p/N82E16868105273", "https://www.newegg.com/p/N82E16868105274", "https://www.newegg.com/asus-geforce-rtx-3080-tuf-rtx3080-10g-gaming/p/N82E16814126453?Description=3080&cm_re=3080-_-14-126-453-_-Product", "https://www.newegg.com/pny-geforce-rtx-3080-vcg308010tfxmpb/p/N82E16814133810?", "https://www.newegg.com/Product/ComboDealDetails?ItemList=Combo.4207523", "https://www.newegg.com/Product/ComboDealDetails?ItemList=Combo.4207521", "https://www.newegg.com/Product/ComboDealDetails?ItemList=Combo.4208292", "https://www.newegg.com/evga-geforce-rtx-3090-24g-p5-3987-kr/p/N82E16814487526?Description=3090&cm_re=3090-_-14-487-526-_-Product", "https://www.newegg.com/evga-geforce-rtx-3090-24g-p5-3987-kr/p/N82E16814487526?Item=N82E16814487526&quicklink=true", "https://www.newegg.com/evga-geforce-rtx-3080-10g-p5-3885-kr/p/N82E16814487520?Item=N82E16814487520", "https://www.newegg.com/msi-geforce-rtx-3080-rtx-3080-gaming-x-trio-10g/p/N82E16814137597?Item=N82E16814137597", "https://www.newegg.com/asus-geforce-rtx-3080-tuf-rtx3080-o10g-gaming/p/N82E16814126452?Item=N82E16814126452", "https://www.newegg.com/msi-geforce-rtx-3080-rtx-3080-ventus-3x-10g-oc/p/N82E16814137598?Item=N82E16814137598", "https://www.newegg.com/asus-geforce-rtx-3080-rog-strix-rtx3080-o10g-gaming/p/N82E16814126457?Item=N82E16814126457", "https://www.newegg.com/asus-geforce-rtx-3090-tuf-rtx3090-24g-gaming/p/N82E16814126455?Item=N82E16814126455", "https://www.newegg.com/asus-geforce-rtx-3090-tuf-rtx3090-o24g-gaming/p/N82E16814126454?Item=N82E16814126454", "https://www.newegg.com/asus-geforce-rtx-3090-rog-strix-rtx3090-o24g-gaming/p/N82E16814126456?Item=N82E16814126456", "https://www.newegg.com/evga-geforce-rtx-3080-10g-p5-3881-kr/p/N82E16814487522?Item=N82E16814487522", "https://www.newegg.com/evga-geforce-rtx-3090-24g-p5-3975-kr/p/N82E16814487524?Item=N82E16814487524", "https://www.newegg.com/evga-geforce-rtx-3090-24g-p5-3973-kr/p/N82E16814487523?Item=N82E16814487523", "https://www.newegg.com/evga-geforce-rtx-3090-24g-p5-3985-kr/p/N82E16814487525?Item=N82E16814487525", "https://www.newegg.com/evga-geforce-rtx-3080-10g-p5-3883-kr/p/N82E16814487521?Item=N82E16814487521", "https://www.newegg.com/msi-geforce-rtx-3090-rtx-3090-gaming-x-trio-24g/p/N82E16814137595?Item=N82E16814137595", "https://www.newegg.com/gigabyte-geforce-rtx-3090-gv-n3090gaming-oc-24gd/p/N82E16814932327?Item=N82E16814932327", "https://www.newegg.com/gigabyte-geforce-rtx-3090-gv-n3090eagle-oc-24gd/p/N82E16814932328?Item=N82E16814932328", "https://www.newegg.com/GEFORCE-RTX-30-SERIES/EventSaleStore/ID-10492", "https://www.newegg.com/zotac-geforce-rtx-3090-zt-a30900d-10p/p/N82E16814500503?Item=N82E16814500503", "https://www.newegg.com/gigabyte-geforce-rtx-3080-gv-n3080eagle-oc-10gd/p/N82E16814932330?Item=N82E16814932330", "https://www.newegg.com/gigabyte-geforce-rtx-3080-gv-n3080gaming-oc-10gd/p/N82E16814932329?Item=N82E16814932329", "https://www.newegg.com/zotac-geforce-rtx-3060-ti-zt-a30610h-10m/p/N82E16814500507?", "https://www.newegg.com/Product/ComboDealDetails?ItemList=Combo.4208270", https://www.newegg.com/evga-geforce-rtx-3060-ti-08g-p5-3667-kr/p/N82E16814487537?Description=RTX%203060%20Ti&cm_re=RTX_3060%20Ti-_-14-487-537-_-Product&quicklink=true https://www.newegg.com/asus-geforce-rtx-3060-ti-rog-strix-rtx3060ti-o8g-gaming/p/N82E16814126470?Description=RTX%203060%20Ti&cm_re=RTX_3060%20Ti-_-14-126-470-_-Product&quicklink=true https://www.newegg.com/asus-geforce-rtx-3060-ti-ko-rtx3060ti-o8g-gaming/p/N82E16814126474?Description=RTX%203060%20Ti&cm_re=RTX_3060%20Ti-_-14-126-474-_-Product&quicklink=true https://www.newegg.com/zotac-geforce-rtx-3060-ti-zt-a30610h-10m/p/N82E16814500507?Description=RTX%203060%20Ti&cm_re=RTX_3060%20Ti-_-14-500-507-_-Product https://www.newegg.com/Product/ComboDealDetails?ItemList=Combo.4208270 https://www.newegg.com/Product/ComboDealDetails?ItemList=Combo.4207457 https://www.newegg.com/Product/ComboDealDetails?ItemList=Combo.4207461 https://www.newegg.com/Product/ComboDealDetails?ItemList=Combo.4207455 https://www.newegg.com/Product/ComboDealDetails?ItemList=Combo.4207453 https://www.newegg.com/Product/ComboDealDetails?ItemList=Combo.4208144 https://www.newegg.com/Product/ComboDealDetails?ItemList=Combo.4208318 https://www.newegg.com/Product/ComboDealDetails?ItemList=Combo.4211037 https://www.newegg.com/Product/ComboDealDetails?ItemList=Combo.4211039 https://www.newegg.com/msi-geforce-rtx-3060-ti-rtx-3060-ti-gaming-x-trio/p/N82E16814137611?Description=RTX%203060%20Ti&cm_re=RTX_3060%20Ti-_-14-137-611-_-Product https://www.newegg.com/asus-geforce-rtx-3060-ti-dual-rtx3060ti-o8g/p/N82E16814126468?Description=RTX%203060%20Ti&cm_re=RTX_3060%20Ti-_-14-126-468-_-Product https://www.newegg.com/msi-geforce-rtx-3060-ti-rtx-3060-ti-ventus-2x-oc/p/N82E16814137612?Description=RTX%203060%20Ti&cm_re=RTX_3060%20Ti-_-14-137-612-_-Product https://www.newegg.com/asus-geforce-rtx-3060-ti-tuf-rtx3060ti-o8g-gaming/p/N82E16814126471?Description=RTX%203060%20Ti&cm_re=RTX_3060%20Ti-_-14-126-471-_-Product https://www.newegg.com/evga-geforce-rtx-3060-ti-08g-p5-3663-kr/p/N82E16814487535?Description=RTX%203060%20Ti&cm_re=RTX_3060%20Ti-_-14-487-535-_-Product https://www.newegg.com/gigabyte-geforce-rtx-3060-ti-gv-n306tgaming-oc-8gd/p/N82E16814932377?Description=RTX%203060%20Ti&cm_re=RTX_3060%20Ti-_-14-932-377-_-Product https://www.newegg.com/gigabyte-geforce-rtx-3060-ti-gv-n306tgamingoc-pro-8gd/p/N82E16814932376?Description=RTX%203060%20Ti&cm_re=RTX_3060%20Ti-_-14-932-376-_-Product https://www.newegg.com/gigabyte-geforce-rtx-3060-ti-gv-n306teagle-8gd/p/N82E16814932379?Description=RTX%203060%20Ti&cm_re=RTX_3060%20Ti-_-14-932-379-_-Product https://www.newegg.com/gigabyte-geforce-rtx-3060-ti-gv-n306taorus-m-8gd/p/N82E16814932375?Description=RTX%203060%20Ti&cm_re=RTX_3060%20Ti-_-14-932-375-_-Product https://www.newegg.com/gigabyte-geforce-rtx-3060-ti-gv-n306teagle-oc-8gd/p/N82E16814932378?Description=RTX%203060%20Ti&cm_re=RTX_3060%20Ti-_-14-932-378-_-Product https://www.newegg.com/zotac-geforce-rtx-3060-ti-zt-a30610e-10m/p/N82E16814500506?Description=RTX%203060%20Ti&cm_re=RTX_3060%20Ti-_-14-500-506-_-Product

jpall12 commented 3 years ago

oh looks horrible.

and for the webhook issue i'll run another test I didn't check if it made an error and i don't see a log file for the webhook.

Jeisonx commented 3 years ago

oh looks horrible.

and for the webhook issue i'll run another test I didn't check if it made an error and i don't see a log file for the webhook.

Will take a look when i get home from work, if an error is reproduced from the webhook it will not create a file it will just log it on your terminal console

Prince25 commented 3 years ago

`TypeError: Cannot read property 'innerHTML' of undefined at Timeout.newegg [as _onTimeout]

After looking more at your error, I believe this error is on Newegg's end. They are timing out your later requests because the program is sending too many requests in a short amount of time, hence the "as _onTimeout". This might be fixed if I spaced out the requests similar to what I did with Amazon but it's not guaranteed and certainly not a permanent fix.

Furthermore, there is no easy fix to this problem because sending too many requests in a short time to any website will usually result the website denying access. The only free fix here is to reduce the amount of links you have and space out requests. Otherwise, you will need to switch between VPN or proxies every so often.

Perhaps you can try running multiple copies of the program each with 5-10 links. Space out the execution of these programs by 10-15 seconds.

Jeisonx commented 3 years ago

Well i reviewed the URLs posted, most of them are not formatted correctly missing ( "", ) also found combo urls 90% of the combo are no longer available, and the ones that are available the website html is different were some classname wouldn't be found. Also found multiple duplicated and or expired URLs.

Here's a more clean version of the URLs you've provided I've been running them for 10+ mins without an issue.

FixedURLs.txt

jpall12 commented 3 years ago

@Jeisonx3 Thanks for that info about the expired URLs etc. But i'm not too sure why you said my URLs are not formatted correctly. I do have the ("", ) for all of my URLs.

Am I missing something?

jpall12 commented 3 years ago

`TypeError: Cannot read property 'innerHTML' of undefined at Timeout.newegg [as _onTimeout]

After looking more at your error, I believe this error is on Newegg's end. They are timing out your later requests because the program is sending too many requests in a short amount of time, hence the "as _onTimeout". This might be fixed if I spaced out the requests similar to what I did with Amazon but it's not guaranteed and certainly not a permanent fix.

Furthermore, there is no easy fix to this problem because sending too many requests in a short time to any website will usually result the website denying access. The only free fix here is to reduce the amount of links you have and space out requests. Otherwise, you will need to switch between VPN or proxies every so often.

Perhaps you can try running multiple copies of the program each with 5-10 links. Space out the execution of these programs by 10-15 seconds.

Ahh ok, ill adjust the interval to prevent ip based bans, and based off of what @Jeisonx3 said would the HTML being different another part of my issue?

jpall12 commented 3 years ago

oh looks horrible. and for the webhook issue i'll run another test I didn't check if it made an error and i don't see a log file for the webhook.

Will take a look when i get home from work, if an error is reproduced from the webhook it will not create a file it will just log it on your terminal console

Oh so it will log in the terminal bc I didn't see an error or any indication that it attempted to send a message to my discord webhook.

Jeisonx commented 3 years ago

Ahh ok, ill adjust the interval to prevent ip based bans, and based off of what @Jeisonx3 said would the HTML being different another part of my issue?

I believe that's the main part of your issue as you're getting timeout because most URLs are expired and or not available. try using the FixedURLs.txt that i provided and see if your experience the same issue.

Also I've fixed your first issue in this pull request regarding to the warning message of NODE_TLS_REJECT_UNAUTHORIZED, but i would let @PrinceS25 take a look at it as I'm not 100% sure of how the code i removed is being used on Tesco.js https://github.com/PrinceS25/StockAlertBot/pull/10/commits/80547926c2eed0efb996b463de4b604756b3e55b

@Jeisonx3 Thanks for that info about the expired URLs etc. But i'm not too sure why you said my URLs are not formatted correctly. I do have the ("", ) for all of my URLs.

Am I missing something?

It could've potentially been the way it was copy and pasted, but if you take a look at your reply with the URLs majority of them are supposed to be inside of (" ") and ended with a ( , ) to start a new line for example "https://newegg.com/test", "https://newegg.com/test2"

most of them are just https://newegg.com/test https://newegg.com/test

Prince25 commented 3 years ago

@Jeisonx3 is correct to say that only normal product links work at the time. However either I, or @Jeisonx3 if he's/she's up to it, can work on adding the combo deals in the near future. At the moment, I'm working on adding Gamestop as well as an interface so you won't have to edit the file manually, you'll be able to change settings through a browser page.

Jeisonx commented 3 years ago

@Jeisonx3 is correct to say that only normal product links work at the time. However either I, or @Jeisonx3 if he's/she's up to it, can work on adding the combo deals in the near future. At the moment, I'm working on adding Gamestop as well as an interface so you won't have to edit the file manually, you'll be able to change settings through a browser page.

I can take a look at adding combos, the code would be a bit messy as I'm in my learning stages. I saw that you're working with electron to potentially add the interface that would be fun. Also I've manage to kind of add Gamestop and Walmart. Gamestop seems to know that you're looking at their website from axios requests and doing too many resulted in an ip ban so I stopped. I was doing some research into potentially seeing if I was able to add proxies for each request. Walmart to me was similar to Amazon where you will get error response status = 412 but changing the interval automatically seems to work. I will submit a pull request to Walmart later in the date to let you take a look at it.

Prince25 commented 3 years ago

Yeah. I can take a look at it later. Don't worry about Gamestop. I have working code for it already, but haven't yet pushed. Also, thank you so much for the support, @Jeisonx !! I really appreciate it. If I get any more donations, I don't mind splitting it with you for your share of the work! Twitter seems like the best way to raise awareness of this project.

Also, do you have a Discord?

Jeisonx commented 3 years ago

Yeah. I can take a look at it later. Don't worry about Gamestop. I have working code for it already, but haven't yet pushed. Also, thank you so much for the support, @Jeisonx !! I really appreciate it. If I get any more donations, I don't mind splitting it with you for your share of the work! Twitter seems like the best way to raise awareness of this project.

Also, do you have a Discord?

I appreciate the gesture to share donations but I won't accept them you have earned every penny that's donated to you, you have put in so many hours of work into this project I doesn't sound right to me, I'm completely satisfied with helping and learning from this project. Twitter sounds like a great idea and will be the best place to bring awarenesses, also creating a Discord server to support user will be helpful. I've started to use Discord this is my user:Jeisonx#4900

Prince25 commented 3 years ago

@jpall12 Newegg combo links should now work thanks to @Jeisonx. Let us know if everything is good for you now so we can close this thread.

jpall12 commented 3 years ago

Yup that's all. Thank you @Jeisonx and @PrinceS25 for all of your help!