Global Proxies net::ERR_FAILED

MidnightsFX commented 3 years ago

Expected Behavior

Global proxies rotate through the proxies available and successfully complete network calls.

Current Behavior

[12:35:20 PM] info :: ✖ [bestbuy] [nvidia (3080)] founders edition :: OUT OF STOCK
[12:35:20 PM] info :: ℹ [bestbuy] Next proxy index: 0 / Count: 1
[12:35:20 PM] error :: ✖ [newegg] gigabyte 3080 gaming oc - net::ERR_FAILED at https://www.newegg.com/gigabyte-geforce-rtx-3080-gv-n3080gaming-oc-10gd/p/N82E16814932329
[12:35:20 PM] info :: ℹ [newegg] Next proxy index: 0 / Count: 1
[12:35:21 PM] error :: ✖ [newegg] gigabyte 3080 eagle oc - net::ERR_FAILED at https://www.newegg.com/gigabyte-geforce-rtx-3080-gv-n3080eagle-oc-10gd/p/N82E16814932330
[12:35:21 PM] info :: ℹ [newegg] Next proxy index: 0 / Count: 1
[12:35:21 PM] error :: ✖ [newegg] gigabyte 3080 aorus master - net::ERR_FAILED at https://www.newegg.com/gigabyte-geforce-rtx-3080-gv-n3080aorus-m-10gd/p/N82E16814932336
[12:35:21 PM] info :: ℹ [newegg] Next proxy index: 0 / Count: 1
[12:35:21 PM] error :: ✖ [asus] asus 3080 strix oc - net::ERR_FAILED at https://store.asus.com/us/item/202009AM290000002
[12:35:21 PM] error :: ✖ [newegg] zotac 3080 trinity - net::ERR_FAILED at https://www.newegg.com/zotac-geforce-rtx-3080-zt-a30800d-10p/p/N82E16814500502

These Errors only happen when using the global proxies- defining a single proxy in the dotenv files does not result in this behavior

Steps to Reproduce

Ensure running commit 64898b8 or newer
Create a global.proxies
Run Node v15.4.0 OR v14.15.2
Customize stores allowed, the more the better (although for me this was reproducible with JUST newegg)

Environment

OS: Windows 10 - WSL - Ubuntu 20.04.1 LTS dotenv file:

I removed unsets

USER_AGENT="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
MAX_PRICE_SERIES_RX6800XT=850
MAX_PRICE_SERIES_RX6900XT=1200
PAGE_BACKOFF_MIN=900000
PAGE_BACKOFF_MAX=1800000
PAGE_TIMEOUT=30000
SHOW_ONLY_SERIES=ryzen5600,rx6800xt,rx6900xt,3080
STORES=newegg

Global.proxies

http://MY_PROXY_IP:8888

Proxy configuration- if you are interested is a vanilla tinyproxy with an allowlist for my IP.

Logs

Startup log

npm i && npm run start

up to date, audited 1424 packages in 2s

7 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

> start
> npm run build && node build/index.js

> build
> tsc

ＳＴＲＥＥＴＭＥＲＣＨＡＮＴ
3.3.0

[12:46:27 PM] info :: ℹ selected stores: newegg
[12:46:27 PM] info :: ℹ selected series: ryzen5600, rx6800xt, rx6900xt, 3080
[12:46:49 PM] info :: ℹ [newegg] Next proxy index: 0 / Count: 1
[12:46:49 PM] error :: ✖ [newegg] asus 3080 tuf - net::ERR_FAILED at https://www.newegg.com/asus-geforce-rtx-3080-tuf-rtx3080-10g-gaming/p/N82E16814126453
[12:46:49 PM] info :: ℹ [newegg] Next proxy index: 0 / Count: 1
[12:46:50 PM] error :: ✖ [newegg] evga 3080 ftw3 ultra - net::ERR_FAILED at https://www.newegg.com/evga-geforce-rtx-3080-10g-p5-3897-kr/p/N82E16814487518
[12:46:50 PM] info :: ℹ [newegg] Next proxy index: 0 / Count: 1
[12:46:50 PM] error :: ✖ [newegg] evga 3080 ftw3 - net::ERR_FAILED at https://www.newegg.com/evga-geforce-rtx-3080-10g-p5-3895-kr/p/N82E16814487519
[12:46:50 PM] info :: ℹ [newegg] Next proxy index: 0 / Count: 1
[12:46:56 PM] info :: ✖ [newegg] [evga (3080)] xc3 black :: OUT OF STOCK
[12:46:56 PM] info :: ℹ [newegg] Next proxy index: 0 / Count: 1
[12:46:56 PM] error :: ✖ [newegg] evga 3080 xc3 - net::ERR_FAILED at https://www.newegg.com/evga-geforce-rtx-3080-10g-p5-3883-kr/p/N82E16814487521
[12:46:56 PM] info :: ℹ [newegg] Next proxy index: 0 / Count: 1
[12:46:57 PM] error :: ✖ [newegg] msi 3080 ventus 3x - net::ERR_FAILED at https://www.newegg.com/msi-geforce-rtx-3080-rtx-3080-ventus-3x-10g/p/N82E16814137600
[12:46:57 PM] info :: ℹ [newegg] Next proxy index: 0 / Count: 1

larryskov commented 3 years ago

Newegg doesn't support most proxies. It fails specifically with them. You have to make a separate proxy for each store that you want to use it with and that is not already banned from, like Newegg.

MidnightsFX commented 3 years ago

These proxies are the same ones which work for newegg on a slightly older version of street merchant. The specific proxies are not the issue here.

dustinm commented 3 years ago

I'm having the same issue as well. Tested socks5 and http proxies that work fine outside of streetmerchant.

phtp commented 3 years ago

Having the same issue after today's update.

larryskov commented 3 years ago

Do the proxies still work if you test manually? Could be banned from the randomizing of user agents

dustinm commented 3 years ago

Yes the proxies work when testing manually. I have a static user agent set and that doesn't fix it. I've tested the proxies on selenium with no issues.

KalitheBirb commented 3 years ago

Can confirm this to, have tested proxies and they work fine just not in streetmerchant.

Yes the proxies work when testing manually. I have a static user agent set and that doesn't fix it. I've tested the proxies on selenium with no issues.

jef commented 3 years ago

Looks like @Doridian made some progress. @MidnightsFX, feel free to check and let us know.

Thanks!

MidnightsFX commented 3 years ago

Working much better! Thanks @jef & @Doridian!

jef commented 3 years ago

All @Doridian on this one. Thank you for the report!

jef / streetmerchant