mxschmitt
commented
1 year ago This sounds like a bug! Sorry to hear. Do you have page.route by any chance in place?
You can debug which header this causes by setting the DEBUG=pw:api env var, this will then produce something like that, where you can share your set-cookie header with us so we can fix it.
pw:api → GET http://localhost:8907/setcookie.html +6ms
pw:api user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/117.0.5938.48 Safari/537.36 +0ms
pw:api accept: */* +0ms
pw:api accept-encoding: gzip,deflate,br +0ms
pw:api ← 200 OK +5ms
pw:api set-cookie: session=value,foo=bar; max-age=3600 +1ms
pw:api date: Sat, 09 Sep 2023 20:07:39 GMT +0ms
pw:api connection: close +0ms
pw:api content-length: 0 +0ms
db-do
bv-do
yury-s
System info
The below stack trace is displayed in our logs and the Thread that is running this never returns from response = self.page.goto(link.url, timeout=NAVIGATE_TIMEOUT_SECS*1000, wait_until=WAIT_UNTIL)
After crawling approximately 300 pages we receive this error. It will always happen on the same site, but on random pages (around the 300th page). We believe that the site owner is intentionally returning bad info which causes this stack trace.
Since the code gets stuck, we cannot see what "cookies" are causing it or prevent it from getting stuck.
Problematic Code
Log snippet
[DEBUG]|2023-09-06 11:49:00,416|common.playutil|64f8c59688daff0a9d514f9f|filter_response: fetching http://redactedsitedyavc6szj6ppbplfqoz3pgrk3zw57my4vybgblpfeayd.onion/Thread-Equifax-data-breach node:internal/process/promises:288 triggerUncaughtException(err, true / fromPromise /); ^
ProtocolError: Protocol error (Storage.setCookies): Invalid cookie fields at /Users/me/projects/proj/venv/lib/python3.8/site-packages/playwright/driver/package/lib/server/chromium/crConnection.js:169:16 at new Promise ()
at CRSession.send (/Users/me/projects/proj/venv/lib/python3.8/site-packages/playwright/driver/package/lib/server/chromium/crConnection.js:165:12)
at CRBrowserContext.addCookies (/Users/me/projects/proj/venv/lib/python3.8/site-packages/playwright/driver/package/lib/server/chromium/crBrowser.js:367:34)
at BrowserContextAPIRequestContext._addCookies (/Users/me/projects/proj/venv/lib/python3.8/site-packages/playwright/driver/package/lib/server/fetch.js:400:25)
at ClientRequest. (/Users/me/projects/proj/venv/lib/python3.8/site-packages/playwright/driver/package/lib/server/fetch.js:213:40)
at Object.onceWrapper (node:events:629:26)
at ClientRequest.emit (node:events:514:28)
at HTTPParser.parserOnIncomingClient [as onIncoming] (node:_http_client:700:27)
at HTTPParser.parserOnHeadersComplete (node:_http_common:119:17) {
sessionClosed: false
}
Node.js v18.17.0
Thread Dump
Thread 0x0000000174617000 (most recent call first): File "/Users/me/projects/proj/venv/lib/python3.8/site-packages/playwright/_impl/_sync_base.py", line 107 in _sync File "/Users/me/projects/proj/venv/lib/python3.8/site-packages/playwright/sync_api/_generated.py", line 9303 in goto File "/Users/me/projects/proj/src/crawling.py", line 400 in _goto File "/Users/me/projects/proj/src/crawling.py", line 377 in goto_with_retry File "/Users/me/projects/proj/src/crawling.py", line 327 in start_crawl_loop File "/Users/me/projects/proj/src/crawling.py", line 142 in _run File "/Users/me/projects/proj/src/crawling.py", line 119 in run File "/Users/me/.pyenv/versions/3.8.10/lib/python3.8/threading.py", line 932 in _bootstrap_inner File "/Users/me/.pyenv/versions/3.8.10/lib/python3.8/threading.py", line 890 in _bootstrap
Source code
Link to the GitHub repository with the repro
[https://github.com/your_profile/playwright_issue_title]
or
Test file (self-contained)
Steps
Expected
1) Make sure that if this happens the error percolates out. AND/OR 2) Make sure this problem correctly returns or raises a catchable error. OR 3) Honors the timeout.
Actual
Nothing. The code never returns.