Cloudflare pages aren't waiting long enough

TraeF1337 commented 3 years ago

Expected Behavior

Work as intended

Current Behavior

Always get error "Cloudflare, waiting"

Steps to Reproduce

just use the dotenv file

Environment

OS:Windows Version: git pull origin main

dotenv file:

COUNTRY="germany"
DESKTOP_NOTIFICATIONS="true"

AUTO_ADD_TO_CART=
BROWSER_TRUSTED="true"

HEADLESS=""
INCOGNITO=""
LOW_BANDWIDTH=""
IN_STOCK_WAIT_TIME="600"

MAX_PRICE_SERIES_3070=""
MAX_PRICE_SERIES_3080="900"
MAX_PRICE_SERIES_3090="1600"
MAX_PRICE_SERIES_RX6800=""
MAX_PRICE_SERIES_RX6800XT="1000"
MAX_PRICE_SERIES_RX6900XT="1250"
MAX_PRICE_SERIES_RYZEN5600=""
MAX_PRICE_SERIES_RYZEN5800=""
MAX_PRICE_SERIES_RYZEN5900=""
MAX_PRICE_SERIES_RYZEN5950=""
MAX_PRICE_SERIES_SONYPS5C=""
MAX_PRICE_SERIES_SONYPS5DE=""

NVIDIA_ADD_TO_CART_ATTEMPTS=""
NVIDIA_SESSION_TTL=""
OPEN_BROWSER=""
PAGE_BACKOFF_MIN=""
PAGE_BACKOFF_MAX=""
PAGE_SLEEP_MIN="30000"
PAGE_SLEEP_MAX="60000"
PAGE_TIMEOUT="60000"

#SHOW_ONLY_BRANDS="amd,asus,evga,gigabyte,msi,nvidia,sony,xfx,zotac"
#SHOW_ONLY_MODELS=""
#SHOW_ONLY_SERIES="rx6900xt,rx6800xt,3080,3090,sonyps5c"

SHOW_ONLY_BRANDS="amd,asus,evga,gigabyte,msi,nvidia,sony,xfx,zotac"
#SHOW_ONLY_MODELS="reference"
SHOW_ONLY_SERIES="rx6900xt,rx6800xt,3080,sonyps5c"

STORES="amd-de,nvidia,nvidia-api,arlt,asus-de,alternate,amazon-de,amazon-de-warehouse,caseking,coolblue,cyberport,euronics-de,evga-eu,expert,galaxus,gamestop-de,mediamarkt,medimax,mindfactory,notebooksbilliger,otto,proshop-de,saturn,spielegrotte"
#STORES="amd-de,nvidia,nvidia-api,asus-de,amazon-de,amazon-de-warehouse,computeruniverse,coolblue,cyberport,euronics-de,evga-eu,expert,galaxus,gamestop-de,mediamarkt,mindfactory,notebooksbilliger,otto,proshop-de,saturn,spielegrotte"

#USER_AGENT="Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4298.0 Safari/537.36"
WEB_PORT=""
"

Logs

[14:32:04] warn :: ✖ [caseking] [asus (3080)] tuf :: CLOUDFLARE, WAITING
[14:32:04] info :: ✖ [asus-de] [asus (3080)] strix oc :: OUT OF STOCK
[14:32:04] info :: ✖ [alternate] [xfx (rx6900xt)] merc :: OUT OF STOCK
[14:32:05] info :: ✖ [saturn] [asus (3080)] tuf oc :: OUT OF STOCK
[14:32:05] info :: ✖ [notebooksbilliger] [gigabyte (3080)] eagle oc :: OUT OF STOCK
[14:32:06] info :: ✖ [asus-de] [asus (3080)] tuf :: OUT OF STOCK
[14:32:07] info :: ✖ [alternate] [sony (sonyps5c)] ps5 console :: OUT OF STOCK
[14:32:07] info :: ✖ [saturn] [gigabyte (3080)] eagle oc :: OUT OF STOCK
[14:32:08] warn :: ✖ [caseking] [asus (3080)] tuf :: CLOUDFLARE, WAITING
[14:32:08] info :: ✖ [notebooksbilliger] [gigabyte (3080)] gaming oc :: OUT OF STOCK
[14:32:09] info :: ✖ [asus-de] [asus (3080)] tuf oc :: OUT OF STOCK
[14:32:09] info :: ✖ [proshop-de] [asus (3080)] strix :: OUT OF STOCK

jef commented 3 years ago

https://github.com/jef/streetmerchant/pull/1434 should fix this.

jef commented 3 years ago

Sorry, I see what you're saying now.

Perhaps https://github.com/jef/streetmerchant/pull/1434/files#diff-df5398bf4301cdf9301ed7143928adb248c73ce1bfefc70be38084c51580ec93R344 this is the issue and should wait longer.

neatchee commented 3 years ago

@jef Not quite right; cloudflare is actually repeatedly redirecting to another ddos protection page in some cases. The implementation actually waits for the page to navigate (page.waitForNavigation)

I have another pull request going up today that will resolve the infinite recursion scenario by aborting after 5 page loads.

jef / streetmerchant