search

jef / streetmerchant

🤖 The world's easiest, most powerful stock checker
https://jef.buzz/streetmerchant
MIT License
4.96k stars 1.31k forks source link

usage of proxies is broken in nightly release (net::ERR_FAILED) #2698

Open richie256 opened 3 years ago

richie256 commented 3 years ago

Expected Behavior

Using proxies with the docker nightly build or nodejs (using the latest code from github) should works fine without net::ERR_FAILED error.

Current Behavior

Using proxies with the nightly build or nodejs (using the latest code from github) return net::ERR_FAILED error. It works as expected using latest docker image.

Steps to Reproduce

  1. Configure the dotenv and global.proxies file with valid proxies
  2. Launch docker run -it --rm --env-file ./dotenv -v $(pwd)/global.proxies:/app/global.proxies ghcr.io/jef/streetmerchant:nightly

OR

  1. Clone the streetmerchant github.
  2. Configure the dotenv and global.proxies file with valid proxies
  3. Install and run: npm run start

Environment

dotenv ```dotenv Nothing Particular dotenv. Any should reproduce the same result. ```
global.proxies ```global.proxies socks5://:@:1085 socks5://:@:1085 socks5://:@:1085 socks5://:@:1085 socks5://:@:1085 socks5://:@:1085 socks5://:@:1085 socks5://:@:1085 socks5://:@:1085 socks5://:@:1085 socks5://:@:1085 socks5://:@:1085 socks5://:@:1085 socks5://:@:1085 socks5://:@:1085 socks5://:@:1085 socks5://:@:1085 socks5://:@:1085 socks5://:@:1085 socks5://:@:1085 socks5://:@:1085 socks5://:@:1085 socks5://:@:1085 ```

Logs

Using proxies with the nightly build we got net::ERR_FAILED error.

[1:38:42 AM] error :: ✖ [2/23] [newegg-ca] evga 3080 ftw3 - net::ERR_FAILED at https://www.newegg.ca/evga-geforce-rtx-3080-10g-p5-3895-kr/p/N82E16814487519
[1:38:42 AM] error :: ✖ [3/23] [canadacomputers] msi 3080 gaming x trio - net::ERR_FAILED at https://www.canadacomputers.com/product_info.php?cPath=43_557_559&item_id=181348&language=en
[1:38:42 AM] error :: ✖ [4/23] [memoryexpress] evga 3080 xc3 black - net::ERR_FAILED at https://www.memoryexpress.com/Products/MX00114094

Exactly the same config works as expected with the latest docker image

[1:47:52 AM] info :: ✖ [2/23] [memoryexpress] [gigabyte (3080)] eagle oc :: OUT OF STOCK
[1:47:52 AM] info :: ✖ [1/23] [canadacomputers] [evga (3080)] ftw3 ultra :: OUT OF STOCK
jef commented 3 years ago

Hi there, thanks for reporting. Is this still the case?

richie256 commented 3 years ago

hello @jef ,

Yes it is still Broken.

I've done some research: I've list all the commit between HEAD and Tag v3.5.0 and tested the versions. Based on my test, it doesn't work since the commit: 1d08d4c12292e1b98ad952fa4f88c1965fa7bd6a chore: update dependencies

Proxy Test Results ``` 00b468a73832b959e0c6ac1ed67b0b23856db76b (HEAD -> main, origin/main, origin/HEAD) docs: make templates required 74d71e457956743566796404458bd1d5d84598db docs: update issue templates 9aa3f164a59cd521701f55bfe4a54872b1c5fc4d docs: update issue templates 71711131ff4c57abe3016e543b7d0ce994e0ecd0 docs: update issue templates 9c346e70fac17854d347dfbdbc238bfb0056c6a8 fix(terraform): update IAM configuration for ECS deployment (#2675) 61af2a4325733463c918396868d313b7632bb310 fix: partial url regexp (#2382) f14a1ca76e41943e639e08a9c8a6299dfa362d9d chore: update dependencies ==> TEST: FAILED 0c589a71f259c80ff21a4bb623f9316cb7702a61 fix(bestbuy-ca): avoid false positive (#2694) 0b060e8f6aca1793d04474210aeec98212172014 feat: add a few stores for spain (#2646) 4bb58ac26192354ccbcd0c416aba78df2234cf19 chore(coolblue): add ps5 consoles (#2597) e0303106f725ee9726c0b7e2f29d97fe5137ca90 chore(deps): bump actions/stale from 3 to 4 (#2671) ==> TEST: FAILED 5e9b3db6f1630f2000da7f9a8b0dd0dc34c8341f chore: revert puppeteer, run linter 1d08d4c12292e1b98ad952fa4f88c1965fa7bd6a chore: update dependencies ==> TEST: FAILED 4e3eadea083071eba563dffc70d749db334dce69 chore: bump node 16.5.0 ==> TEST: OK 8c7b8aabe4633757dea836b7d11af2a407869e90 feat: add alternate-fr, 3070ti, change alternate to alternate.de (#2653). ==> TEST: OK c0f46d77cf9d4a3f7481aec5b0ddf33883b47d6e feat(stores-nz): update labels (#2486) cc0aaeca77dd18a2c9797055f26edf6935fa755b chore(bestbuy): add 3080ti models (#2670) b33f678090396b77a5765115d7a3fa42f6de06c8 chore: prune invalid links (#2592) 51c9b3ab2ac251c9c3abebf64025e1a03ecf4944 feat: add 3080ti (#2600) 4cfbd37ae0d28d1c2b8cf596b5dcb941daa7d9bc feat: add 3080ti (#2598) 52b5e5e79ac381b585ec1dd63288aefe4f9da9d1 ci: fix nightly 812dc5ea8d9830c251a526c02de05a15bfe20602 ci: change check-date name 3b43732f291d263792585fa05264dcf322b55620 chore: update dependencies and nightly workflow e4a267f675800b04d06ddb4212de1373a767a113 chore: update stale workflow da635f41dd3c3e949f3d77fc2b1bcbe860639179 build: remove smartthings 2e5b07c93fb02677596eb5a2f2ea857b99c2d4d9 chore(deps): bump nodemailer from 6.6.0 to 6.6.1 (#2568) a751cb11c53c7cf6b7883e69eb5e8e2fe1e57adf chore(deps): bump node-notifier from 9.0.1 to 10.0.0 (#2583) e4ebdc4719afdf390f7060bbc755229dcaba1410 chore(deps-dev): bump @types/cheerio from 0.22.28 to 0.22.29 (#2580) 1cb1787c27b2e7e37e3e989be686b635f98442e9 chore(deps-dev): bump @types/nodemailer from 6.4.1 to 6.4.2 (#2582) 9fc5ec06ff92e458ea9649f22af9df1ae5ba1d95 chore(deps-dev): bump webpack from 5.37.0 to 5.38.0 (#2579) ==> TEST: OK 9ba964a587158b551d7217764eb9799073aea21f chore: bump node version 9e13cd7dbb8026c53da4499309ebef03d51c3e92 chore(deps-dev): bump webpack from 5.37.0 to 5.37.1 (#2558) 3e74a4c7c41c939b9ce0952e089f36269f3fb46f chore(deps-dev): bump @types/node from 15.3.0 to 15.6.1 (#2571) 7d35c2b8f0620501061220baefeb499d6e66e2da chore(deps): bump twilio from 3.62.0 to 3.63.0 (#2561) 36e188fd36d31a915d9ac59016710b6f3fa4436d chore(deps): bump open from 8.0.2 to 8.2.0 (#2572) 11220750143df7ebdc011aafccb8e035bfd2bc5d chore(alternate): add PS5 bundle (#2562) 8854ce97bba7858dfc9385530e0c090200944ede fix: clear cookies when rotating proxies (#2575) ebf8b20071dc628b1f47b96ee2b03f888c056dd9 ci: fix documentation 729afd0ef3f4ceea1ae38b04e1a25bc4c4dcd3ec chore(deps): bump top-user-agents from 1.0.25 to 1.0.26 (#2556) 42a24a633c537941d63a6b76ed9598179dda9ea4 chore(computeruniverse): add backoff code (#2542) 32fe440049ce97c5683ca03276212899b2211af5 feat(gotify): add priority option (#2546) 0378a7cf0aa673933af35c3a3605f147615f6e96 chore(netonnet-no): update selectors (#2547) 1e62878c8d1b3c0b49189b05f846d555cfc205b3 chore(deps): bump node 16.1.0 9f58c22c0c6d14d0f0e59782bbe71a18b8329f6f style(gotify): linting d12639ee8561240d99ef5c2c364255ff808f78b0 chore(notification): add better logging, remove debug 8f2de5dc68667f6b9f0033bed129fd6cdbd9d8a7 feat(notification): add gotify 1ab0af234ca76b5db8602ffe04a1fc80b3c6515b chore(deps): bump dependencies 6e614a24918f38d189a3d3b70174fbd1590f78cc feat(captcha): add support for configurable capture type (#2468) b6af582f70843c5608249c5d629fdec8f10630dd chore: remove hooks d0a6c7e85950fe3cffbcb30662967a687aaf7347 chore(otto): add labels for xbox series x & s (#2515) c9049a3054d35f8a37340e552e19841e3ecaf631 chore(elkjop): update selectors (#2505) 64120e73d8f3a01dd5852044ffa0ef6b0ce31730 chore(mediamarkt): update captcha selector (#2496) 4d8930616d2296fcdd6c0d79bac2109f1388a0b0 chore(xbox): update oos selector 164902a587d80f0421b44079e0616ba37a7a743f fix(captcha): use Slack captcha handler token when provided (#2460) 790fe9b6cf4f9627e7f3dc7acc572737a739e4cc fix(amazon): false positives for xbox (#2497) 7c4c6ed79ecd89767ffbb3577716be28ce6c1bdc chore(multicom): update selectors (#2499) e397306f1a3b1576c0595dec5fae7be277e88f25 chore(deps-dev): bump webpack from 5.35.0 to 5.36.2 (#2480) b57905291df85ee2f722e5e3a81d1cecd0ab6ee4 feat: add INCOGNITO to dotenv-example and volta support (#2484) 441617f6729ab61174f7bcfe287c41a34a78728f feat(store): add amd-at (#2462) 5a744b1add24dd4d3c250f41382a034212d9c5b0 feat(store): add cyberport-at (#2463) eb76f0d2b78726c4380853bd47edcf9b1302e7bb fix: remove spaces from price (#2482) 41305f1ed5a81240ba7aa0dda1619d635d917371 feat(store): add multicom.no and more products to NO stores (#2483) 39fac90ba4feadb7e5c681db60e60a83749f5e62 fix(lookup): prevent terminal from being hidden (#2461) cac7c20193caaec5d41291104e66aa01a65adceb feat: add waitUntil and waitForSelctor to linksBuilder (#2437) 35f9eb43b6cca8a92f60ba2870e60a225b4c1da6 fix(alternate): use button selector instead of text (#2419) b6ac6c53ddc34ac35b537dd99415bbcb9eec0de7 fix(nodemon): use npx with ts-node (#2423) 092919ab349edac6a8b5b4747917a1a232235da7 fix(microcenter): add outOfStock label (#2432) d652b4936435377d36268141a2b63c8798f7f95b refactor(captcha): use seconds instead of ms (#2424) 2af153192c2cf3ac2bf9ba0d1d3201f250363c9b chore: change urls. ==> TEST: OK 5937d0141d1c34fdb90475bffbbe32a2977c789b chore(alternate): add new oos label (#2409) 3c00eca22378afb2a18aa76c1d4204107e6364c2 chore: bump dependencies (#2413) 409769e6dbd3add614fcd985731bb244297a2c2d feat: add interactive captcha handler with Slack and Discord (#2385) 8c39d0d30a9f8db7eda8665eb08d350fadf157e4 chore(materiel): add 3070 and 3080 cards (#2393) 4eb702b9a1187c8dcf159ea991ab1049c3c44bb2 fix(amazon-es): use euroFormat (#2371) 2be33ac6147f8f7460d108f361d954b17a335bd9 fix(web): add favicon to prevent missing icon console error (#2375) d4dc397e9a848d6d2ba1ed56188fb0ddda98e795 chore(store): add some PS5 bundles (#2021) d13f7f1ff91ada223e8ce7eb50d9a80030cbe668 fix(docker): disable opening browser f0e5c5d17bad282480da30cc68e5bd8eb9d80844 chore: update pr template 6e9b1a9640ca93f620623d53624230514fac6d40 feat(store): add amd-be store (#2338) 49c9e8c0dc5c3b25ce2aa20265ccd2865f229385 feat(store): add amd-nl store (#2072) bc59da11d512f6f55449781cb8f7f4bd1d3e7e57 fix(discord): properly wait for webhooks (#2352) aed9cfe98919e6524bfde0156a308c9a56d4dfc0 chore(deps): bump dependencies e2d804027a3ec6a19201ef9ce35dcb4c4dee7025 feat(store): add MightyApe, NoelLeeming and TheWarehouse (NZ) (#2133) c1c92bd25b7b744a05f21664f2029a0cb0683d35 chore(gamestop): update inventory (#2239) ae8e72a73044b8311add15d2b5c10936bae16e82 feat(store): add mediamarkt austria (#1836) 211e3e2692c1c97f607b39b73623a09958c69d81 chore(currys): add rx6800, rx6800xt and rx6900xt (#2236) 7ec8a160cf9844df34578fe9f413b88d6141178c refactor(materiel): update docs, change name 0ebe7451e4e4c6e46c64e3782bd51a92ce594dca feat(store): add materiel.net (french), 35 3060ti references (#2287) 5b2fe88a83f17ff98ebe5c91806c7eb5c1dc3bdc chore(amazon-fr): update evga 3080 ftw3 link (#2286) bf8f242f4b33796fcee91a863f0e086fcec29537 refactor(freemobile): remove local throw 6be2e0d491646d6e88f72f67c694874b11cda03a feat(notification): add support for Free mobile (#2285) a25b7f19cc8e403ffd7856e631cde7e9df63f1cc refactor: simplify maxPrice logic e006c48795b8fa104f0cee5584517a5f164d6512 chore: change verbiage to be more generic 9b02c36370aaf7fef20a90b2427273c2f7b30f00 feat: change max price behavior b4121463ed7f25566cfa4ff508fa0c4d03237a9d ci: update stale labels d03aea05a7783c9cc6a18c8baf0381050c105357 feat(pushover): ability to specify device (#2101) ea095a364c250f805704c1189c0f3e697e09673d chore(alternate): update links and selectors (#2282) c7e24e77d415819d27043d5814b2133b46549e57 chore(amd): add rx6800xt midnight black (#2321) 3c1419f170b1d00aa6ad08ad89752fc8f9134a93 chore: bump stale workflow to 60 days cd6e1f614bcaf00bc468c32a1b5a98cb51c7730f fix: add missing tests in Dockerfile (#2323) 91a0ff5e1ba83efabe5ef45c6a3e17e1c564f16b chore(deps): bump dependencies, node 15.14.0 bbfa808b820ee2d16c65dfdcecce58645f4899d0 feat: color empty stores (#2312) c2c8531ea26e2b75f50ed384f5128e5df54053e9 docs: update badges 113c9ce19f3ed1f140eec7cd555ec373f25d256f feat(series): support for rx6700xt (#2290) ba35706cca84a425da8d5a5bd6b86dbecbadffa1 docs: update Xbox Series S and X price limit desc (#2098) 2ebbf55004da325c38c1eb12bcd5dc3264791604 chore(deps): bump node from 15.11.0-alpine3.13 to 15.12.0-alpine3.13 (#2185) 1714e89af7964d06b01120fb75595ef8ec8177ba feat(store): Add AO and BT (UK) (#2089) 983cc525164744e4d3bff1475d12b5563d23200e chore(amazon-uk): add 3060 cards (#2229) a7a5e6954b3568b40d360982d5f11287c1b7f440 chore(deps-dev): bump @types/node from 14.14.35 to 14.14.36 (#2233) 2a1734a71d20e9a65c04edcd4318eaa3b5872a7d chore(deps): bump dependencies 83ce922761f85a13424ed43a1961668ffc870485 feat(pushover): add support for sound (#2203) 0a9ce0dd4c7681cba7681d570db44f65359d83cf chore(store): add Nvidia RTX Cards (#2204) b2ce7157d9631d6edd45debcb9ab85bc79241133 chore(walmart): update 5900x url (#2184) b0efc837bc554f5c8635e125b67775d8b1f8f6b3 fix(playstation): fix inStock selector (#2187) 54883f0ad7cf86052035d88ed2ffbc612c439725 chore(deps): bump open from 8.0.2 to 8.0.3 (#2186) 4bd048ff7a496e82e66293a9706581b8d10af6b3 chore: add APNS to dotenv-example (#2174) 99a274dbea088a6bcb053b1c1bf7812d0796442a fix(apns): use when specified 10f7e0cf5d8f633aa0195a1f9403965b4347e017 chore: bump dependencies 2775124d80f86466dbc0c3bcb227f489f35aab62 chore(deps-dev): bump webpack from 5.24.2 to 5.26.3 (#2164) a575143b244525ff87579b689ed90bf6ae251847 chore(deps-dev): bump mocha from 8.3.0 to 8.3.2 (#2167) 4cdda9890cb30443826e6df02b99d41dbbea026d chore(deps-dev): bump @types/sinon from 9.0.10 to 9.0.11 (#2166) 53e7a8d692b76155699b371e926a2b9687d4a081 chore: remove unused dependencies b516e268428219de70358821683e2dd74df59943 chore(deps-dev): bump typescript from 4.2.2 to 4.2.3 (#2092) 00e4ddce162877a4a08747955859508f5af0a802 chore(deps): bump open from 7.4.2 to 8.0.2 (#2078) eeab4b112a6482d4fd381014e44fd0d6e071c494 chore(deps): bump twitch-chat-client from 4.4.13 to 4.5.1 (#2127) 0ff8158511f4f90dc26407fffa22ab0a4ad3aa0b revert: replace euroFormat with regex (#2131) 91c4f12308905f4e6d65341ba54ae814c8ad94d6 revert: parsing prices with thousands separator and European format (#2153) 6b78b2ee4445ad2b868b6f848730a474ca12d657 feat(notifications): add apple push notifications (#2077) d46c6366b70ca668437bd2aab87c3c529f4c569b chore(microcenter): add 3090 cards 7fcce586bf03a573a1bba143c7d56e046418d31f chore(amd): update rx6900xt urls (#2094) 245d110d0a52df8b2d05f020aaaa1bfa8be7cab4 chore(asus): update urls (#2155) 61f8e3a9a03136682035881fce6d14356717d1ac fix: parsing prices with thousands separator and European format (#2153) 795c7230ad4632c81b17f5c2aeb3b0e88b2b04e3 refactor(store): replace euroFormat with regex (#2131) afe7a709009f43fba1cef22aa6c136976507c7bc chore(nodejs): bump version (#2088) f65aab87dade27c9f271ca816f83fe5a57ff7da6 docs: update node references c2a22716cce0de6cbdbfdf7ebdceb97844a22e0f ci: fix docker release ==> TEST: OK ```

thanks!

jef commented 3 years ago

I reverted that puppeteer change, which may have done some damage. Do you mind testing with this commit?

5e9b3db6f1630f2000da7f9a8b0dd0dc34c8341f chore: revert puppeteer, run linter

And to clarify, it works with the commit just prior to the one you posted?

4e3eadea083071eba563dffc70d749db334dce69 chore: bump node 16.5.0

richie256 commented 3 years ago

Hello @jef ,

And to clarify, it works with the commit just prior to the one you posted?

Exactly, it work with the commit just prior the one posted:

Same failure net::ERR_FAILED with the docker nightly build, as of now.

docker pull ghcr.io/jef/streetmerchant:nightly
docker run -it --rm --env-file ./dotenv -v $(pwd)/global.proxies:/app/global.proxies ghcr.io/jef/streetmerchant:nightly
richie256 commented 3 years ago

@jef

Do you mind testing with this commit?

5e9b3db chore: revert puppeteer, run linter

This fails net::ERR_FAILED as well using proxies.

Fmstrat commented 3 years ago

If it's of help, I had to roll back to https://github.com/jef/streetmerchant/pkgs/container/streetmerchant/3431703 to get things working.

Tiippex commented 2 years ago

Oh, I came across the same issue. It is present in 0906f64 (nightly).

I'm happy to help :) Anything we can do here to help you solve the issue?