search

jef / streetmerchant

๐Ÿค– The world's easiest, most powerful stock checker
https://jef.buzz/streetmerchant
MIT License
4.96k stars 1.31k forks source link

Connection timeout on some stores with new version #677

Closed Tomson124 closed 3 years ago

Tomson124 commented 3 years ago

Description

Really weird issue. I am getting connection time outs for media markt, notebooksbilliger and some other stores when I run the new docker container or the the cloned version on my vps. If I run the same version and config on my local machine it works fine. But if I run the older pre 1.6. version as a pm2 instance with just the updated store lists and the same config it works just fine. As far as I understand it is not the IP of the vps being blacklisted as the pm2 instance should be timing out as well then, right? Really confused right now on whats going on.

Possible solution

No solution as of right now, not sure if it even is a problem with the software or not.

Tomson124 commented 3 years ago

Should probably also add that pinging the stores that are making problemsis working fine. And there are also some navigation timeouts without a store attached to it. Just tried it again as a simple npm run instead of docker but the same outcome with the newer version: grafik

jef commented 3 years ago

Whoops, didn't mean to close.

Do you mind sharing your .env? I'll run on my end and test.

Tomson124 commented 3 years ago

The .env of the pm2 instance which runs fine.

The .env : ``` # ** All configuration variables are optional ** # Read https://github.com/jef/nvidia-snatcher#customization for help on customizing this file ############################################################################################# BROWSER_TRUSTED=true COUNTRY=germany DESKTOP_NOTIFICATIONS= DISCORD_NOTIFY_GROUP= DISCORD_WEB_HOOK=https://discordapp.com/api/webhooks/... EMAIL_PASSWORD= EMAIL_TO= EMAIL_USERNAME= HEADLESS= IN_STOCK_WAIT_TIME= LOG_LEVEL= LOW_BANDWIDTH= MAX_PRICE_SERIES_3070= MAX_PRICE_SERIES_3080= MAX_PRICE_SERIES_3090= MICROCENTER_LOCATION= MQTT_BROKER_ADDRESS= MQTT_BROKER_PORT= MQTT_CLIENT_ID= MQTT_PASSWORD= MQTT_QOS= MQTT_TOPIC= MQTT_USERNAME= NVIDIA_ADD_TO_CART_ATTEMPTS= NVIDIA_SESSION_TTL= OPEN_BROWSER=false PAGE_BACKOFF_MIN= PAGE_BACKOFF_MAX= PAGE_SLEEP_MIN= PAGE_SLEEP_MAX= PAGE_TIMEOUT= PAGERDUTY_INTEGRATION_KEY= PAGERDUTY_SEVERITY= PHONE_CARRIER= PHONE_NUMBER= PLAY_SOUND= PROXY_ADDRESS= PROXY_PORT= PUSHBULLET= PUSHOVER_TOKEN= PUSHOVER_USER= PUSHOVER_PRIORITY= SCREENSHOT= SHOW_ONLY_BRANDS= SHOW_ONLY_MODELS= SHOW_ONLY_SERIES=3070 SLACK_CHANNEL= SLACK_TOKEN= SMTP_ADDRESS= SMTP_PORT= STORES=nvidia,amazon-de,alternate,mediamarkt,proshop-de,notebooksbilliger,caseking,mindfactory,nvidia-api TELEGRAM_ACCESS_TOKEN= TELEGRAM_CHAT_ID= TWILIO_ACCOUNT_SID= TWILIO_AUTH_TOKEN= TWILIO_FROM_NUMBER= TWILIO_TO_NUMBER= TWITCH_ACCESS_TOKEN= TWITCH_CHANNEL= TWITCH_CLIENT_ID= TWITCH_CLIENT_SECRET= TWITCH_REFRESH_TOKEN= TWITTER_ACCESS_TOKEN_KEY= TWITTER_ACCESS_TOKEN_SECRET= TWITTER_CONSUMER_KEY= TWITTER_CONSUMER_SECRET= TWITTER_TWEET_TAGS= USER_AGENT= WEB_PORT= ```

But if I change some the stores and add 3080 model it also doesnt work anymore in the pm2 instance(pre 1.6.x), could not yet test if it is adding the model or removing amazon-de cause I have to sleep some now. But really weird issue.

The changed .env : ``` # ** All configuration variables are optional ** # Read https://github.com/jef/nvidia-snatcher#customization for help on customizing this file ############################################################################################# BROWSER_TRUSTED=true COUNTRY=germany DESKTOP_NOTIFICATIONS= DISCORD_NOTIFY_GROUP= DISCORD_WEB_HOOK=https://discordapp.com/api/webhooks/... EMAIL_PASSWORD= EMAIL_TO= EMAIL_USERNAME= HEADLESS= IN_STOCK_WAIT_TIME= LOG_LEVEL= LOW_BANDWIDTH= MAX_PRICE_SERIES_3070= MAX_PRICE_SERIES_3080= MAX_PRICE_SERIES_3090= MICROCENTER_LOCATION= MQTT_BROKER_ADDRESS= MQTT_BROKER_PORT= MQTT_CLIENT_ID= MQTT_PASSWORD= MQTT_QOS= MQTT_TOPIC= MQTT_USERNAME= NVIDIA_ADD_TO_CART_ATTEMPTS= NVIDIA_SESSION_TTL= OPEN_BROWSER=false PAGE_BACKOFF_MIN= PAGE_BACKOFF_MAX= PAGE_SLEEP_MIN= PAGE_SLEEP_MAX= PAGE_TIMEOUT= PAGERDUTY_INTEGRATION_KEY= PAGERDUTY_SEVERITY= PHONE_CARRIER= PHONE_NUMBER= PLAY_SOUND= PROXY_ADDRESS= PROXY_PORT= PUSHBULLET= PUSHOVER_TOKEN= PUSHOVER_USER= PUSHOVER_PRIORITY= SCREENSHOT= SHOW_ONLY_BRANDS= SHOW_ONLY_MODELS= SHOW_ONLY_SERIES=3070,3080 SLACK_CHANNEL= SLACK_TOKEN= SMTP_ADDRESS= SMTP_PORT= STORES=nvidia,alternate,mediamarkt,proshop-de,notebooksbilliger,caseking,mindfactory,nvidia-api TELEGRAM_ACCESS_TOKEN= TELEGRAM_CHAT_ID= TWILIO_ACCOUNT_SID= TWILIO_AUTH_TOKEN= TWILIO_FROM_NUMBER= TWILIO_TO_NUMBER= TWITCH_ACCESS_TOKEN= TWITCH_CHANNEL= TWITCH_CLIENT_ID= TWITCH_CLIENT_SECRET= TWITCH_REFRESH_TOKEN= TWITTER_ACCESS_TOKEN_KEY= TWITTER_ACCESS_TOKEN_SECRET= TWITTER_CONSUMER_KEY= TWITTER_CONSUMER_SECRET= TWITTER_TWEET_TAGS= USER_AGENT= WEB_PORT= ```
jef commented 3 years ago

Models and pricing were kind of in a weird state with a couple commits that change how models/max price works. I would make sure that you run the latest.

jef commented 3 years ago

That is a weird issue though.

jef commented 3 years ago

Perhaps I also misread.

To confirm:

Correct?

AelithBlanchett commented 3 years ago

Hi! I'm running 1.7.0.

I was listening to 3070 series only, I added the 3080 and I also get those timeouts now. I'm listening to bestbuy-ca, amazon-ca and newegg-ca.

[11:23:58 PM] info :: โ„น selected stores: amazon-ca, bestbuy-ca, newegg-ca
[11:23:58 PM] info :: โ„น selected series: 3070, 3080
[11:23:59 PM] warn :: โœ– couldn't find sound player
[11:24:15 PM] error :: Failed to parse card: PNY GeForce RTX 3070 8GB Dual Fan Ampere Graphics Card
[11:24:39 PM] error :: Navigation timeout of 30000 ms exceeded
[11:24:47 PM] error :: Navigation timeout of 30000 ms exceeded
[11:24:57 PM] error :: โœ– [newegg-ca] asus 3080 tuf - Navigation timeout of 30000 ms exceeded
[11:25:09 PM] error :: โœ– [amazon-ca] msi 3080 gaming x trio - Navigation timeout of 30000 ms exceeded
[11:25:18 PM] error :: Navigation timeout of 30000 ms exceeded
[11:25:23 PM] error :: Navigation timeout of 30000 ms exceeded
[11:25:28 PM] info :: โœ– [bestbuy-ca] [zotac (3080)] trinity :: OUT OF STOCK
[11:25:42 PM] error :: โœ– [amazon-ca] msi 3080 ventus 3x oc - Navigation timeout of 30000 ms exceeded
[11:25:50 PM] error :: Navigation timeout of 30000 ms exceeded
[11:25:55 PM] error :: โœ– [newegg-ca] evga 3080 xc3 black - Navigation timeout of 30000 ms exceeded
[11:26:04 PM] info :: โœ– [bestbuy-ca] [msi (3080)] ventus 3x :: OUT OF STOCK

After only listening to the 3080 GPUs, I can confirm they are definitely the ones messing up.

jef commented 3 years ago

Thanks for the comment @AelithBlanchett. Unfortunately, I can't duplicate the issue :/

                    $$\       $$\ $$\                                                  $$\               $$\
                    \__|      $$ |\__|                                                 $$ |              $$ |
$$$$$$$\ $$\    $$\ $$\  $$$$$$$ |$$\  $$$$$$\           $$$$$$$\ $$$$$$$\   $$$$$$\ $$$$$$\    $$$$$$$\ $$$$$$$\   $$$$$$\   $$$$$$\
$$  __$$\\$$\  $$  |$$ |$$  __$$ |$$ | \____$$\ $$$$$$\ $$  _____|$$  __$$\  \____$$\\_$$  _|  $$  _____|$$  __$$\ $$  __$$\ $$  __$$\
$$ |  $$ |\$$\$$  / $$ |$$ /  $$ |$$ | $$$$$$$ |\______|\$$$$$$\  $$ |  $$ | $$$$$$$ | $$ |    $$ /      $$ |  $$ |$$$$$$$$ |$$ |  \__|
$$ |  $$ | \$$$  /  $$ |$$ |  $$ |$$ |$$  __$$ |         \____$$\ $$ |  $$ |$$  __$$ | $$ |$$\ $$ |      $$ |  $$ |$$   ____|$$ |
$$ |  $$ |  \$  /   $$ |\$$$$$$$ |$$ |\$$$$$$$ |        $$$$$$$  |$$ |  $$ |\$$$$$$$ | \$$$$  |\$$$$$$$\ $$ |  $$ |\$$$$$$$\ $$ |
\__|  \__|   \_/    \__| \_______|\__| \_______|        \_______/ \__|  \__| \_______|  \____/  \_______|\__|  \__| \_______|\__|
1.7.0

[7:35:52 PM] info :: โ„น selected stores: bestbuy-ca, amazon-ca, newegg-ca
[7:35:52 PM] info :: โ„น selected series: 3070, 3080
[7:35:57 PM] error :: Failed to parse card: PNY GeForce RTX 3070 8GB Dual Fan Ampere Graphics Card
[7:36:05 PM] info :: โœ– [bestbuy-ca] [zotac (3080)] trinity :: OUT OF STOCK
[7:36:05 PM] info :: โœ– [amazon-ca] [msi (3080)] gaming x trio :: OUT OF STOCK
[7:36:06 PM] info :: โœ– [newegg-ca] [asus (3080)] tuf :: OUT OF STOCK
[7:36:06 PM] info :: โœ– [amazon-ca] [msi (3080)] ventus 3x oc :: OUT OF STOCK
[7:36:07 PM] info :: โœ– [amazon-ca] [evga (3080)] ftw3 :: OUT OF STOCK
[7:36:07 PM] info :: โœ– [bestbuy-ca] [msi (3080)] ventus 3x :: OUT OF STOCK
[7:36:07 PM] info :: โœ– [amazon-ca] [evga (3080)] ftw3 ultra :: OUT OF STOCK
[7:36:08 PM] info :: โœ– [amazon-ca] [evga (3080)] xc3 ultra :: OUT OF STOCK
[7:36:09 PM] info :: โœ– [bestbuy-ca] [evga (3080)] xc3 ultra :: OUT OF STOCK
[7:36:09 PM] info :: โœ– [newegg-ca] [evga (3080)] ftw3 gaming :: OUT OF STOCK
[7:36:09 PM] info :: โœ– [amazon-ca] [evga (3080)] xc3 :: OUT OF STOCK
[7:36:10 PM] info :: โœ– [amazon-ca] [evga (3080)] xc3 black :: OUT OF STOCK
[7:36:11 PM] info :: โœ– [amazon-ca] [gigabyte (3080)] gaming oc :: OUT OF STOCK
[7:36:11 PM] info :: โœ– [bestbuy-ca] [asus (3080)] tuf :: OUT OF STOCK
[7:36:11 PM] info :: โœ– [amazon-ca] [gigabyte (3080)] eagle oc :: OUT OF STOCK
[7:36:12 PM] info :: โœ– [newegg-ca] [evga (3080)] ftw3 ultra :: OUT OF STOCK
[7:36:12 PM] info :: โœ– [amazon-ca] [asus (3080)] tuf :: OUT OF STOCK
^C

Are you using USER_AGENT? I'd also suggest running HEADLESS=false to see what's happening.

Tomson124 commented 3 years ago

To confirm:

It's not working on latest via Docker It works on pre 1.6.0 via Docker Correct?

Sorry was very late yesterday. Yes it is not working in the latest (nightly) docker with the changed .env, I did not try it in pre 1.6 docker as I only have the nightly image (updated via docker pull) The pre 1.6 instance is just a cloned repo with installed dependencies and running npm run start but with pm2 so it keeps running and being monitored even after closing the SSH. Iam currently at work, so no real way of doing further testing, will try when I get home.

AelithBlanchett commented 3 years ago

I've booted up another beefier droplet on DigitalOcean, reinstalled docker and ran the nightly docker build with the same parameters and it's still timing out at first with the same config, but I eventually get the OOS after a minute or two of failing, instead of nothing at all ever on the less powerful droplet. If I remove bestbuy-ca, it immediately pulls the data without any navigation errors.

It's probably just a performance issue maybe?

nlflint commented 3 years ago

I ran into this today too, after it ran for ~30mins. Ran great for 30mins, then a series of 30s timeout errors on each store shutdown all polling. Was able to repro it again but faster, see log below.

.env

PAGE_SLEEP_MIN=10000
PAGE_SLEEP_MAX=20000
SHOW_ONLY_SERIES=3080
STORES=amazon,newegg,officedepot,bestbuy

Here, the NewEgg gets an error (near bottom), then NewEgg is no longer polled Eventually all stores get an error and stop polling:

nathan@Mothra:~/repos/nvidia-snatcher (main) $ docker run --cap-add=SYS_ADMIN -it --rm --env-file ./.env ghcr.io/jef/nvidia-snatcher:nightly                          130 โ†ต

                    $$\       $$\ $$\                                                  $$\               $$\
                    \__|      $$ |\__|                                                 $$ |              $$ |
$$$$$$$\ $$\    $$\ $$\  $$$$$$$ |$$\  $$$$$$\           $$$$$$$\ $$$$$$$\   $$$$$$\ $$$$$$\    $$$$$$$\ $$$$$$$\   $$$$$$\   $$$$$$\
$$  __$$\\$$\  $$  |$$ |$$  __$$ |$$ | \____$$\ $$$$$$\ $$  _____|$$  __$$\  \____$$\\_$$  _|  $$  _____|$$  __$$\ $$  __$$\ $$  __$$\
$$ |  $$ |\$$\$$  / $$ |$$ /  $$ |$$ | $$$$$$$ |\______|\$$$$$$\  $$ |  $$ | $$$$$$$ | $$ |    $$ /      $$ |  $$ |$$$$$$$$ |$$ |  \__|
$$ |  $$ | \$$$  /  $$ |$$ |  $$ |$$ |$$  __$$ |         \____$$\ $$ |  $$ |$$  __$$ | $$ |$$\ $$ |      $$ |  $$ |$$   ____|$$ |
$$ |  $$ |  \$  /   $$ |\$$$$$$$ |$$ |\$$$$$$$ |        $$$$$$$  |$$ |  $$ |\$$$$$$$ | \$$$$  |\$$$$$$$\ $$ |  $$ |\$$$$$$$\ $$ |
\__|  \__|   \_/    \__| \_______|\__| \_______|        \_______/ \__|  \__| \_______|  \____/  \_______|\__|  \__| \_______|\__|
1.7.0

[6:18:13 PM] info :: โ„น selected stores: amazon, newegg, officedepot, bestbuy
[6:18:13 PM] info :: โ„น selected series: 3080
[6:18:30 PM] info :: โœ– [newegg] [asus (3080)] tuf :: OUT OF STOCK
[6:18:30 PM] info :: โœ– [officedepot] [pny (3080)] xlr8 :: OUT OF STOCK
[6:18:32 PM] info :: โœ– [newegg] [evga (3080)] ftw3 ultra :: OUT OF STOCK
[6:18:33 PM] info :: โœ– [officedepot] [pny (3080)] xlr8 rgb :: OUT OF STOCK
[6:18:35 PM] info :: โœ– [newegg] [evga (3080)] ftw3 :: OUT OF STOCK
[6:18:36 PM] info :: โœ– [bestbuy] [nvidia (3080)] founders edition :: OUT OF STOCK
[6:18:37 PM] info :: โœ– [amazon] [pny (3080)] xlr8 :: OUT OF STOCK
[6:18:38 PM] info :: โœ– [newegg] [evga (3080)] xc3 black :: OUT OF STOCK
[6:18:40 PM] info :: โœ– [newegg] [evga (3080)] xc3 :: OUT OF STOCK
[6:18:41 PM] info :: โœ– [bestbuy] [asus (3080)] rog strix :: OUT OF STOCK
[6:18:42 PM] info :: โœ– [amazon] [pny (3080)] xlr8 rgb :: OUT OF STOCK
[6:18:43 PM] info :: โœ– [newegg] [msi (3080)] ventus 3x :: OUT OF STOCK
[6:18:45 PM] info :: โœ– [bestbuy] [evga (3080)] xc3 black :: OUT OF STOCK
[6:18:46 PM] info :: โœ– [newegg] [msi (3080)] ventus 3x oc :: OUT OF STOCK
[6:18:46 PM] info :: โœ– [amazon] [msi (3080)] gaming x trio :: OUT OF STOCK
[6:18:49 PM] info :: โœ– [newegg] [msi (3080)] gaming x trio :: OUT OF STOCK
[6:18:49 PM] info :: โœ– [bestbuy] [evga (3080)] xc3 :: OUT OF STOCK
[6:18:51 PM] info :: โœ– [amazon] [evga (3080)] ftw3 ultra :: OUT OF STOCK
[6:18:51 PM] info :: โœ– [newegg] [gigabyte (3080)] gaming oc :: OUT OF STOCK
[6:18:53 PM] info :: โœ– [bestbuy] [evga (3080)] xc3 ultra :: OUT OF STOCK
[6:18:53 PM] info :: โœ– [officedepot] [pny (3080)] xlr8 :: OUT OF STOCK
[6:18:54 PM] info :: โœ– [newegg] [gigabyte (3080)] eagle oc :: OUT OF STOCK
[6:18:56 PM] info :: โœ– [amazon] [evga (3080)] xc3 ultra :: OUT OF STOCK
[6:18:56 PM] info :: โœ– [bestbuy] [evga (3080)] ftw3 ultra :: OUT OF STOCK
[6:18:57 PM] info :: โœ– [officedepot] [pny (3080)] xlr8 rgb :: OUT OF STOCK
[6:19:00 PM] info :: โœ– [amazon] [evga (3080)] ftw3 :: OUT OF STOCK
[6:19:00 PM] info :: โœ– [newegg] [gigabyte (3080)] aorus master :: OUT OF STOCK
[6:19:00 PM] info :: โœ– [bestbuy] [evga (3080)] ftw3 :: OUT OF STOCK
[6:19:03 PM] info :: โœ– [newegg] [zotac (3080)] trinity :: OUT OF STOCK
[6:19:04 PM] info :: โœ– [bestbuy] [gigabyte (3080)] gaming oc :: OUT OF STOCK
[6:19:04 PM] info :: โœ– [amazon] [evga (3080)] xc3 :: OUT OF STOCK
[6:19:05 PM] info :: โœ– [newegg] [asus (3080)] rog strix :: OUT OF STOCK
[6:19:08 PM] info :: โœ– [bestbuy] [gigabyte (3080)] eagle oc :: OUT OF STOCK
[6:19:09 PM] info :: โœ– [amazon] [evga (3080)] xc3 black :: OUT OF STOCK
[6:19:09 PM] info :: โœ– [newegg] [asus (3080)] tuf oc :: OUT OF STOCK
[6:19:12 PM] info :: โœ– [bestbuy] [gigabyte (3080)] vision oc :: OUT OF STOCK
[6:19:13 PM] info :: โœ– [officedepot] [pny (3080)] xlr8 :: OUT OF STOCK
[6:19:13 PM] info :: โœ– [amazon] [gigabyte (3080)] gaming oc :: OUT OF STOCK
[6:19:16 PM] info :: โœ– [bestbuy] [gigabyte (3080)] aorus master :: OUT OF STOCK
[6:19:16 PM] info :: โœ– [officedepot] [pny (3080)] xlr8 rgb :: OUT OF STOCK
[6:19:18 PM] info :: โœ– [amazon] [gigabyte (3080)] eagle oc :: OUT OF STOCK
[6:19:19 PM] info :: โœ– [bestbuy] [msi (3080)] ventus 3x oc :: OUT OF STOCK
[6:19:22 PM] info :: โœ– [amazon] [asus (3080)] tuf oc :: OUT OF STOCK
[6:19:23 PM] info :: โœ– [bestbuy] [pny (3080)] xlr8 rgb :: OUT OF STOCK
[6:19:26 PM] info :: โœ– [bestbuy] [pny (3080)] xlr8 rgb :: OUT OF STOCK
[6:19:27 PM] info :: โœ– [amazon] [asus (3080)] tuf :: OUT OF STOCK
[6:19:31 PM] info :: โœ– [officedepot] [pny (3080)] xlr8 :: OUT OF STOCK
[6:19:31 PM] info :: โœ– [amazon] [asus (3080)] rog strix oc :: OUT OF STOCK
[6:19:35 PM] info :: โœ– [officedepot] [pny (3080)] xlr8 rgb :: OUT OF STOCK
[6:19:36 PM] info :: โœ– [amazon] [msi (3080)] ventus 3x oc :: OUT OF STOCK
[6:19:39 PM] error :: โœ– [newegg] zotac 3080 trinity oc - Navigation timeout of 30000 ms exceeded
[6:19:41 PM] info :: โœ– [amazon] [zotac (3080)] trinity :: OUT OF STOCK
[6:19:42 PM] info :: โœ– [newegg] [pny (3080)] xlr8 rgb :: OUT OF STOCK
[6:19:43 PM] info :: โœ– [bestbuy] [nvidia (3080)] founders edition :: OUT OF STOCK
[6:19:47 PM] info :: โœ– [bestbuy] [asus (3080)] rog strix :: OUT OF STOCK
[6:19:51 PM] info :: โœ– [bestbuy] [evga (3080)] xc3 black :: OUT OF STOCK
[6:19:54 PM] info :: โœ– [bestbuy] [evga (3080)] xc3 :: OUT OF STOCK
[6:19:56 PM] info :: โœ– [amazon] [pny (3080)] xlr8 :: OUT OF STOCK
[6:19:56 PM] info :: โœ– [officedepot] [pny (3080)] xlr8 :: OUT OF STOCK
[6:19:58 PM] info :: โœ– [bestbuy] [evga (3080)] xc3 ultra :: OUT OF STOCK
[6:19:59 PM] info :: โœ– [officedepot] [pny (3080)] xlr8 rgb :: OUT OF STOCK
[6:20:01 PM] info :: โœ– [amazon] [pny (3080)] xlr8 rgb :: OUT OF STOCK
[6:20:01 PM] info :: โœ– [bestbuy] [evga (3080)] ftw3 ultra :: OUT OF STOCK
AelithBlanchett commented 3 years ago

I think this might be related to issue #676 If you look at it, everything starts to fail on the first error from newegg. I also encounter the same behavior on my machine.

jef commented 3 years ago

I would've guessed at first this:

I've booted up another beefier droplet on DigitalOcean

If you're using a cloud provider, more than likely, you'll run into issues because of a denylist of IPs.

But I think looking further into this problem is necessary.

Nick-Gatti commented 3 years ago

Can second this issue. Works fine on my Windows 10 machine with Powershell, but Ubuntu Server box times out.

Tomson124 commented 3 years ago

So did a little more testing on my side, seems to be the model parameter that messes it up for me. If I only search for 3070s it works fine, but when I am searching for 3080s or both at the same time, it only times times out. Couldn't be performance, as it is a Intel Xeon Golld (2 dedicated cores), 20GB RAM and RAID 10 SSDs. IP blacklsit or denylist is highly unlikly (in my case) as searching only for 3070s works and pinging the shops also works just fine.

Nick-Gatti commented 3 years ago

Iโ€™m having the issue with just the 3070 search so I wonder if itโ€™s an issue with the brand parameter.

On Wed, Nov 4, 2020 at 1:53 PM Thomas Wolfgruber notifications@github.com wrote:

So did a little more testing on my side, seems to be the model parameter that messes it up for me. If I only search for 3070s it works fine, but when I am searching for 3080s or both at the same time, it only times times out. Couldn't be performance, as it is a Intel Xeon Golld (2 dedicated cores), 20GB RAM and RAID 10 SSDs. IP blacklsit or denylist is highly unlikly (in my case) as searching only for 3070s works and pinging the shops also works just fine.

โ€” You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jef/nvidia-snatcher/issues/677#issuecomment-721995245, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJZWQTW55KEIZER5A7LA7NDSOHEOTANCNFSM4TJJGMQQ .

treydott commented 3 years ago

I am also having this issue. I am running the nightly docker. I tried adjusting the series, models, and page refresh wait times. The issue still seems to occur. I also tried using the normal install, same issue. Are there any docker versions that I can revert to which are pre v1.6.0?

Timeout starts after a few minutes for me and once a site times out it never tries to refresh the site again. When Newegg times out in the below code, it does not try to refresh again

๏ผณ๏ผด๏ผฒ๏ผฅ๏ผฅ๏ผด๏ผญ๏ผฅ๏ผฒ๏ผฃ๏ผจ๏ผก๏ผฎ๏ผด
,2.0.0
,
,[3:53:42 AM] info :: โ„น selected stores: adorama, amazon, bandh, bestbuy, newegg
,[3:53:42 AM] info :: โ„น selected brands: asus, gigabyte
,[3:53:42 AM] info :: โ„น selected series: 3080
,[3:53:57 AM] warn :: โœ– [amazon] [gigabyte (3080)] gaming oc :: STATUS CODE ERROR 503
,[3:54:04 AM] error :: Failed to parse card: Zotac ZT-A30800J-10P graphics card NVIDIA GeForce RTX 3080 10 GB GDDR6X ZT-A30800J-10P, GeForce RTX 3080, 10 GB, GDDR6X, 320 bit, 7680 x 4320 pixels, PCI Express x16 4.0
,[3:54:08 AM] error :: โœ– [newegg] asus 3080 tuf - Execution context was destroyed, most likely because of a navigation.
,[3:54:21 AM] warn :: โœ– [adorama] [asus (3080)] rog strix oc :: STATUS CODE ERROR 403
,[3:54:21 AM] error :: โœ– [newegg] gigabyte 3080 gaming oc - Execution context was destroyed, most likely because of a navigation.
,[3:54:23 AM] info :: โœ– [amazon] [gigabyte (3080)] eagle oc :: OUT OF STOCK
,[3:54:24 AM] warn :: โœ– [amazon] [asus (3080)] tuf oc :: STATUS CODE ERROR 503
,[3:54:25 AM] info :: โœ– [bandh] [gigabyte (3080)] gaming oc :: OUT OF STOCK
,[3:54:27 AM] error :: โœ– [newegg] gigabyte 3080 eagle oc - Execution context was destroyed, most likely because of a navigation.
,[3:54:28 AM] error :: โœ– [bestbuy] asus 3080 rog strix - Navigation timeout of 30000 ms exceeded
,[3:54:38 AM] warn :: โœ– [adorama] [asus (3080)] tuf oc :: STATUS CODE ERROR 403
,[3:54:39 AM] info :: โœ– [newegg] [gigabyte (3080)] aorus master :: OUT OF STOCK
,[3:54:42 AM] info :: โœ– [amazon] [asus (3080)] tuf :: OUT OF STOCK
,[3:54:44 AM] warn :: โœ– [amazon] [asus (3080)] rog strix oc :: STATUS CODE ERROR 503
,[3:54:46 AM] info :: โœ– [bestbuy] [gigabyte (3080)] gaming oc :: OUT OF STOCK
,[3:54:47 AM] info :: โœ– [bandh] [gigabyte (3080)] gaming oc :: OUT OF STOCK
,[3:54:51 AM] info :: โœ– [newegg] [asus (3080)] rog strix :: OUT OF STOCK
,[3:54:55 AM] warn :: โœ– [amazon] [gigabyte (3080)] gaming oc :: STATUS CODE ERROR 503
,[3:54:58 AM] info :: โœ– [bestbuy] [gigabyte (3080)] eagle oc :: OUT OF STOCK
,[3:55:05 AM] info :: โœ– [adorama] [asus (3080)] tuf :: OUT OF STOCK
,[3:55:09 AM] info :: โœ– [amazon] [gigabyte (3080)] eagle oc :: OUT OF STOCK
,[3:55:09 AM] info :: โœ– [bandh] [gigabyte (3080)] gaming oc :: OUT OF STOCK
,[3:55:10 AM] warn :: โœ– [amazon] [asus (3080)] tuf oc :: STATUS CODE ERROR 503
,[3:55:11 AM] warn :: โœ– [amazon] [asus (3080)] tuf :: STATUS CODE ERROR 503
,[3:55:14 AM] info :: โœ– [bestbuy] [gigabyte (3080)] vision oc :: OUT OF STOCK
,[3:55:17 AM] info :: โœ– [adorama] [asus (3080)] rog strix oc :: OUT OF STOCK
,[3:55:21 AM] error :: โœ– [newegg] asus 3080 tuf oc - Navigation timeout of 30000 ms exceeded
,[3:55:26 AM] info :: โœ– [amazon] [asus (3080)] rog strix oc :: OUT OF STOCK
,[3:55:29 AM] warn :: โœ– [adorama] [asus (3080)] tuf oc :: STATUS CODE ERROR 403
,[3:55:30 AM] info :: โœ– [bandh] [gigabyte (3080)] gaming oc :: OUT OF STOCK
,[3:55:30 AM] info :: โœ– [bestbuy] [gigabyte (3080)] aorus master :: OUT OF STOCK
,[3:55:33 AM] warn :: โœ– [amazon] [gigabyte (3080)] gaming oc :: STATUS CODE ERROR 503
,[3:55:35 AM] warn :: โœ– [amazon] [gigabyte (3080)] eagle oc :: STATUS CODE ERROR 503
,[3:55:46 AM] info :: โœ– [adorama] [asus (3080)] tuf :: OUT OF STOCK
,[3:55:49 AM] info :: โœ– [bandh] [gigabyte (3080)] gaming oc :: OUT OF STOCK
,[3:55:51 AM] info :: โœ– [amazon] [asus (3080)] tuf oc :: OUT OF STOCK
,

Unsure if this is related, but when I ran the logging level as debug, I am seeing constant lookups for stores not in the list.

Tested this on a windows VM and it does not occur. The issue only happens on my Linux box for both the Docker and native install

nlflint commented 3 years ago

When I run v1.7.0 outside a container (via pulling v1.7.0 tag locally and npm run start) then it works fine, no errors.

treydott commented 3 years ago

I tried it again outside of docker and it appears to be fine as well. Perhaps I was mistaken before. Seems docker specific

Nick-Gatti commented 3 years ago

Doesnโ€™t work outside docker for me. On Fri, Nov 6, 2020 at 6:50 PM Ozai notifications@github.com wrote:

I tried it again outside of docker and it appears to be fine as well. Perhaps I was mistaken before. Seems docker specific

โ€” You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jef/streetmerchant/issues/677#issuecomment-723379473, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJZWQTTXQ5B4ONDJIMTGBPLSOSYWJANCNFSM4TJJGMQQ .

jef commented 3 years ago

I've been running for a 24 hours with these stores and haven't had any problems. I'm going to close this out due to potentially being client sided. Feel free to open a ticket related to any questions on running the client.

Thanks!

reneruiz commented 3 years ago

Using docker here, same issue.

corvek97 commented 3 years ago

I stopped getting those errors after setting the USER_AGENT=

It is also parsing much faster than before after doing that.