TypeError: Cannot read property 'url' of null

[tarik0]

Hello, I am experiencing this issue. It would be great If you guys can help! PS: I did a exception handler to inspect the failed request. This is the picture of it. Failed requests are usually OPTION requests. Example website: https://antcpt.com/eng/information/demo-form/recaptcha-3-test-score.html

D:\reCaptchaV3\node_modules\puppeteer-page-proxy\src\lib\cookies.js:67
        this.url = request.isNavigationRequest() ? request.url() : request.frame().url();
                                                                                  ^

TypeError: Cannot read property 'url' of null
    at new CookieHandler (D:\reCaptchaV3\node_modules\puppeteer-page-proxy\src\lib\cookies.js:67:83)
    at requestHandler (D:\reCaptchaV3\node_modules\puppeteer-page-proxy\src\core\proxy.js:12:27)
    at $ppp_request_listener (D:\reCaptchaV3\node_modules\puppeteer-page-proxy\src\core\proxy.js:82:15)
    at D:\reCaptchaV3\node_modules\puppeteer\lib\cjs\vendor\mitt\src\index.js:51:62
    at Array.map (<anonymous>)
    at Object.emit (D:\reCaptchaV3\node_modules\puppeteer\lib\cjs\vendor\mitt\src\index.js:51:43)
    at Page.emit (D:\reCaptchaV3\node_modules\puppeteer\lib\cjs\puppeteer\common\EventEmitter.js:72:22)
    at D:\reCaptchaV3\node_modules\puppeteer\lib\cjs\puppeteer\common\Page.js:143:100
    at D:\reCaptchaV3\node_modules\puppeteer\lib\cjs\vendor\mitt\src\index.js:51:62
    at Array.map (<anonymous>)

My versions are;

  "dependencies": {
    "puppeteer": "^8.0.0",
    "puppeteer-extra": "^3.1.18",
    "puppeteer-extra-plugin-adblocker": "^2.11.11",
    "puppeteer-extra-plugin-recaptcha": "^3.3.7",
    "puppeteer-extra-plugin-stealth": "^2.7.6",
    "puppeteer-page-proxy": "^1.2.8"
  }

[wanhh]

+1

[sopparus]

Same here

[northmatt]

Same as well. Yall using proxy on request or proxy on page? Also what kind of proxy? Best to state so it can help them debuggers.

Am doing authed HTTPS proxy on an entire page.

edit: Also worth noting am using a userDataDir argument when launching browser edit2: Also tried newest version of Puppeteer with just page proxy and still got the error

"dependencies": {
    "log4js": "^6.3.0",
    "puppeteer": "^7.1.0",
    "puppeteer-extra": "^3.1.17",
    "puppeteer-extra-plugin-stealth": "^2.7.5",
    "puppeteer-page-proxy": "^1.2.8"
}

[SomethingWrong]

@tarik0 Hey, have you found out what's the problem is or any solution yet?

[northmatt]

@SomethingWrong There are at least two ways to fix this.

One: Various PRs have put in failsafes to not cause catastrophic errors. One of them switched one of the libraries entirely to fix the issue.

Two: Use per request proxies and have a try/catch on the UseProxy function.

For me, I liked the latter. I noticed in the former that these HTTPS requests can still fail it'll just not cause errors. The bad part is that the HTTPS requests dont get fully cleaned up after a "safe" error, dramatically slowing things down (I mean taking like 2 - 3 mins to load a site that can normally load in 8s with the proxy). Going with the latter allowed the HTTPS request to fully fail and get cleaned up, but not cause a catastrophic error which exits the program.

[shahzamanshaikh]

I don't fully grasp how I could use it on per request. Would I have to run the useproxy function for every action I do on the page or just when the URL change happens? Either way, I was able to find a workaround by adding an arg in the launch options: '--disable-web-security'. Idk how or why it works lol but I don't see that error anymore after adding this argument.

[mike503]

'--disable-web-security' from what @shahzamanshaikh said let me go further, eventually this happens though:

node:internal/errors:464
    ErrorCaptureStackTrace(err);
    ^

TypeError [ERR_INVALID_URL]: Invalid URL
    at new NodeError (node:internal/errors:371:5)
    at onParseError (node:internal/url:552:9)
    at new URL (node:internal/url:628:5)
    at new CookieHandler (C:\apps\node\node_modules\puppeteer-page-proxy\src\lib\cookies.js:68:23)

[Cuadrix]

fixed in v1.2.9