Connection to Chrome/Chromium times out after 1k-ish screenshots?

[byt3bl33d3r]

So apparently, there's yet another websocket connection bug in pyppeteer. Seems that after 400ish screenshots the connection to chrome just dies. :( https://github.com/miyakogi/pyppeteer/issues/149

Tried some of the proposed fixes but none of them worked for me.

[E:pyppeteer.connection] connection unexpectedly closed
2019-11-10 23:19:09,304 [ERROR] - base_events.py: default_exception_handler - Task exception was never retrieved
future: <Task finished name='Task-7697' coro=<Connection._async_send() done, defined at /home/bhis/.local/share/virtualenvs/WitnessMe-QFMsedki/lib/python3.8/site-packages/pyppeteer/connection.py:69> exception=InvalidStateError('invalid state')>
Traceback (most recent call last):
  File "/home/bhis/.local/share/virtualenvs/WitnessMe-QFMsedki/lib/python3.8/site-packages/websockets/protocol.py", line 827, in transfer_data
    message = await self.read_message()
  File "/home/bhis/.local/share/virtualenvs/WitnessMe-QFMsedki/lib/python3.8/site-packages/websockets/protocol.py", line 895, in read_message
    frame = await self.read_data_frame(max_size=self.max_size)
  File "/home/bhis/.local/share/virtualenvs/WitnessMe-QFMsedki/lib/python3.8/site-packages/websockets/protocol.py", line 971, in read_data_frame
    frame = await self.read_frame(max_size)
  File "/home/bhis/.local/share/virtualenvs/WitnessMe-QFMsedki/lib/python3.8/site-packages/websockets/protocol.py", line 1047, in read_frame
    frame = await Frame.read(
  File "/home/bhis/.local/share/virtualenvs/WitnessMe-QFMsedki/lib/python3.8/site-packages/websockets/framing.py", line 105, in read
    data = await reader(2)
  File "/home/bhis/.pyenv/versions/3.8.0/lib/python3.8/asyncio/streams.py", line 738, in readexactly
    raise exceptions.IncompleteReadError(incomplete, n)
asyncio.exceptions.IncompleteReadError: 0 bytes read on a total of 2 expected bytes

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/bhis/.local/share/virtualenvs/WitnessMe-QFMsedki/lib/python3.8/site-packages/pyppeteer/connection.py", line 73, in _async_send
    await self.connection.send(msg)
  File "/home/bhis/.local/share/virtualenvs/WitnessMe-QFMsedki/lib/python3.8/site-packages/websockets/protocol.py", line 555, in send
    await self.ensure_open()
  File "/home/bhis/.local/share/virtualenvs/WitnessMe-QFMsedki/lib/python3.8/site-packages/websockets/protocol.py", line 803, in ensure_open
    raise self.connection_closed_exc()
websockets.exceptions.ConnectionClosedError: code = 1006 (connection closed abnormally [internal]), no reason

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/bhis/.local/share/virtualenvs/WitnessMe-QFMsedki/lib/python3.8/site-packages/pyppeteer/connection.py", line 79, in _async_send
    await self.dispose()
  File "/home/bhis/.local/share/virtualenvs/WitnessMe-QFMsedki/lib/python3.8/site-packages/pyppeteer/connection.py", line 170, in dispose
    await self._on_close()
  File "/home/bhis/.local/share/virtualenvs/WitnessMe-QFMsedki/lib/python3.8/site-packages/pyppeteer/connection.py", line 151, in _on_close
    cb.set_exception(_rewriteError(
asyncio.exceptions.InvalidStateError: invalid state

[byt3bl33d3r]

Wow so apparently installing pyppeteer from the dev branch:

python3 -m pip install -U git+https://github.com/miyakogi/pyppeteer.git@dev

plus applying the fix detailed here https://github.com/miyakogi/pyppeteer/pull/160#issuecomment-448886155 (which WitnessMe already does to solve another websocket timeout issue) seems to have solved the above error at least.

[byt3bl33d3r]

Ok, so doing the above seems to have at least fixed the error so the entire thing doesn't crash. However now it seems like after ~1k screenshots everything just stops. No errors, just everything starts timing out for some reason. Fun.

[byt3bl33d3r]

And I take everything back. Looks like even with the dev version things crash. FML

[arxenix]

Any solutions to this yet? WitnessMe is unusable for me because of this issue

[byt3bl33d3r]

Its an issue with the underlying pyppeteer library, I definitely don't have the bandwith to see if I can fix it. You're more then welcome to dig into it and submit a PR :)

[byt3bl33d3r]

Ok so the only way around this bug that I've managed to figure out is to open and close Chrome after a batch of screenshots have been taken (which is determined by the --threads argument). Commit e1b5ec1 does exactly this. So after 25 screenshots (by default) it will open and close Chrome for you automatically and keep scanning. This way the above error is avoided.

[byt3bl33d3r]

I'm closing this since this seems to be somewhat resolved as of v1.5.0.