search

miyakogi / pyppeteer

Headless chrome/chromium automation library (unofficial port of puppeteer)
Other
3.56k stars 370 forks source link

Compatibility with websockets 7.0 #158

Open jlamypoirier opened 5 years ago

jlamypoirier commented 5 years ago

With websockets 7.0 the connection to the browser is lost after some time, for me it is consistently after 20 seconds. The end result is unrelated error messages such as

 [...]
    return self._get_loop().run_until_complete(asyncio.wait_for(fn, self.timeout))
  File "/usr/lib/python3.6/asyncio/base_events.py", line 468, in run_until_complete
    return future.result()
  File "/usr/lib/python3.6/asyncio/tasks.py", line 358, in wait_for
    return fut.result()
  File "/eai-vision-generator/eai-data-generation/data_generation/html/generator.py", line 117, in generate_one
    await self.loadHtml(page, html_path)
  File "/eai-vision-generator/eai-data-generation/data_generation/html/generator.py", line 166, in loadHtml
    {'waitUntil': ['load', 'networkidle0']}
  File "/python/.venv/lib/python3.6/site-packages/pyppeteer/page.py", line 879, in goto
    result = await self._navigate(url, referrer)
  File "/python/.venv/lib/python3.6/site-packages/pyppeteer/page.py", line 894, in _navigate
    'Page.navigate', {'url': url, 'referrer': referrer})
pyppeteer.errors.NetworkError: Protocol error Page.navigate: Target closed.

The following is printed when re-raising the exception in connection.py, line 62. 

ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<Connection._async_send() done, defined at /python/.venv/lib/python3.6/site-packages/pyppeteer/connection.py:73> exception=InvalidStateError('invalid state',)>
Traceback (most recent call last):
  File "/python/.venv/lib/python3.6/site-packages/websockets/protocol.py", line 674, in transfer_data
    message = yield from self.read_message()
  File "/python/.venv/lib/python3.6/site-packages/websockets/protocol.py", line 742, in read_message
    frame = yield from self.read_data_frame(max_size=self.max_size)
  File "/python/.venv/lib/python3.6/site-packages/websockets/protocol.py", line 815, in read_data_frame
    frame = yield from self.read_frame(max_size)
  File "/python/.venv/lib/python3.6/site-packages/websockets/protocol.py", line 884, in read_frame
    extensions=self.extensions,
  File "/python/.venv/lib/python3.6/site-packages/websockets/framing.py", line 99, in read
    data = yield from reader(2)
  File "/usr/lib/python3.6/asyncio/streams.py", line 672, in readexactly
    raise IncompleteReadError(incomplete, n)
asyncio.streams.IncompleteReadError: 0 bytes read on a total of 2 expected bytes

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/python/.venv/lib/python3.6/site-packages/pyppeteer/connection.py", line 77, in _async_send
    await self.connection.send(msg)
  File "/python/.venv/lib/python3.6/site-packages/websockets/protocol.py", line 462, in send
    yield from self.ensure_open()
  File "/python/.venv/lib/python3.6/site-packages/websockets/protocol.py", line 646, in ensure_open
    ) from self.transfer_data_exc
websockets.exceptions.ConnectionClosed: WebSocket connection is closed: code = 1006 (connection closed abnormally [internal]), no reason

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/python/.venv/lib/python3.6/site-packages/pyppeteer/connection.py", line 84, in _async_send
    await self.dispose()
  File "/python/.venv/lib/python3.6/site-packages/pyppeteer/connection.py", line 175, in dispose
    await self._on_close()
  File "/python/.venv/lib/python3.6/site-packages/pyppeteer/connection.py", line 158, in _on_close
    f'Protocol error {cb.method}: Target closed.',  # type: ignore
asyncio.base_futures.InvalidStateError: invalid state
ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<Connection._recv_loop() done, defined at /python/.venv/lib/python3.6/site-packages/pyppeteer/connection.py:53> exception=ConnectionClosed('WebSocket connection is closed: code = 1006 (connection closed abnormally [internal]), no reason',)>
Traceback (most recent call last):
  File "/python/.venv/lib/python3.6/site-packages/websockets/protocol.py", line 674, in transfer_data
    message = yield from self.read_message()
  File "/python/.venv/lib/python3.6/site-packages/websockets/protocol.py", line 742, in read_message
    frame = yield from self.read_data_frame(max_size=self.max_size)
  File "/python/.venv/lib/python3.6/site-packages/websockets/protocol.py", line 815, in read_data_frame
    frame = yield from self.read_frame(max_size)
  File "/python/.venv/lib/python3.6/site-packages/websockets/protocol.py", line 884, in read_frame
    extensions=self.extensions,
  File "/python/.venv/lib/python3.6/site-packages/websockets/framing.py", line 99, in read
    data = yield from reader(2)
  File "/usr/lib/python3.6/asyncio/streams.py", line 672, in readexactly
    raise IncompleteReadError(incomplete, n)
asyncio.streams.IncompleteReadError: 0 bytes read on a total of 2 expected bytes

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/python/.venv/lib/python3.6/site-packages/pyppeteer/connection.py", line 59, in _recv_loop
    resp = await self.connection.recv()
  File "/python/.venv/lib/python3.6/site-packages/websockets/protocol.py", line 434, in recv
    yield from self.ensure_open()
  File "/python/.venv/lib/python3.6/site-packages/websockets/protocol.py", line 646, in ensure_open
    ) from self.transfer_data_exc
websockets.exceptions.ConnectionClosed: WebSocket connection is closed: code = 1006 (connection closed abnormally [internal]), no reason

This may be due to the change in how connections are terminated in version 7.0 https://websockets.readthedocs.io/en/stable/changelog.html

Mattwmaster58 commented 4 years ago

@jlamypoirier This is a bug in pyppeteer not websockets. Consider using pyppeteer2