search

alfred82santa / whalesong

Whalesong is an asyncio python library to manage WebApps remotely. Currently WhatsappWeb is implemented
https://whalesong.readthedocs.io
MIT License
50 stars 19 forks source link

Problems using pyppeteer and Chromium #115

Open dafner opened 4 years ago

dafner commented 4 years ago

Hi,

I am testing this library and found that after a few minutes of use (the amount varies and is not always fixed) the application stops receiving new messages or sending messages. When this happens, I stop the application with Ctrl-C and the following messages appear on the screen indicating a problem with the network connection:

Future exception was never retrieved future: <Future finished exception=NetworkError('Protocol error Target.sendMessageToTarget: Target closed.',)> pyppeteer.errors.NetworkError: Protocol error Target.sendMessageToTarget: Target closed. Future exception was never retrieved future: <Future finished exception=NetworkError('Protocol error Target.sendMessageToTarget: Target closed.',)> pyppeteer.errors.NetworkError: Protocol error Target.sendMessageToTarget: Target closed. Future exception was never retrieved future: <Future finished exception=NetworkError('Protocol error Target.sendMessageToTarget: Target closed.',)> pyppeteer.errors.NetworkError: Protocol error Target.sendMessageToTarget: Target closed. Future exception was never retrieved future: <Future finished exception=NetworkError('Protocol error Target.sendMessageToTarget: Target closed.',)> pyppeteer.errors.NetworkError: Protocol error Target.sendMessageToTarget: Target closed. Future exception was never retrieved future: <Future finished exception=NetworkError('Protocol error Target.sendMessageToTarget: Target closed.',)> pyppeteer.errors.NetworkError: Protocol error Target.sendMessageToTarget: Target closed. Future exception was never retrieved future: <Future finished exception=NetworkError('Protocol error Target.sendMessageToTarget: Target closed.',)> pyppeteer.errors.NetworkError: Protocol error Target.sendMessageToTarget: Target closed. Future exception was never retrieved future: <Future finished exception=NetworkError('Protocol error Target.sendMessageToTarget: Target closed.',)> pyppeteer.errors.NetworkError: Protocol error Target.sendMessageToTarget: Target closed. Future exception was never retrieved

Currently I am testing this library with pyppeteer and Chromium as a backend in order to test the asynchronous sending and receiving of messages.

I understand that this problem is not specific to this library and in the documentation mentions the application of a patch https://github.com/miyakogi/pyppeteer/pull/160 that should be applied to pyppeteer.

In the readme also mentioned that a patched version of Chromium must be downloaded from the following link: https://github.com/alfred82santa/whalesong/blob/master/using_chromium before use the backend. However, the link is not available.

I will thank anyone who can clarify how to apply the patch and where I can download the corrected version of Chromium, since the instrucctions indicated in https://github.com/miyakogi/pyppeteer/pull/160 are not very clear or give me the necessary instructions to be able to use this library with pyppeteer and Chromiun

Thanks very much

alfred82santa commented 4 years ago

using_chronium link must point to https://github.com/alfred82santa/whalesong/#using-chronium sorry...

On the other hand, your problem looks like you are using wrong websocket library version... it must be 6.0.0. Newer versions perform a ping/pong routine. But Chronium does not implement it on DevTool websocket. So, library closes socket.

dafner commented 4 years ago

Thanks for your prompt answer.

Now I testing my test application with firefox because the chromium backend are not very stable.

I don't know if anybody had the similar problem using chromium. I'm testing my test program using 2 cell phones and trying to send commands at the same time in order to verify the perfomance. While chromium works, the perfomance is much better than using Firefox.

I would like it, if possible; that someone share with me their experience with Chromiun working with some application.

By the way, I folowed all the step in order to install pyppeteer and whalesong for chromiun backend Also, I installed pyppdf (https://github.com/kiwi0fruit/pyppdf/tree/master/pyppdf) package that download a chrome patched version .local/share/pyppeteer/local-chromium/609904 and I linked to .local/share/pyppeteer/local-chromium/588429 which is the version that is installed using the command pyppeteer-install

In the following link (https://github.com/miyakogi/pyppeteer/pull/160) suggest to use the following code:

def patch_pyppeteer(): import pyppeteer.connection original_method = pyppeteer.connection.websockets.client.connect

def new_method(*args, **kwargs):
    kwargs['ping_interval'] = None
    kwargs['ping_timeout'] = None
    return original_method(*args, **kwargs)

pyppeteer.connection.websockets.client.connect = new_method

patch_pyppeteer()

And I modified the /usr/local/lib/python3.6/dist-packages/pyppeteer/connection.py in order to add those two params:

    self._ws = websockets.client.connect(
        self._url, max_size=None,
        ping_interval=None, ping_timeout=None,
        loop=self._loop)
    self._recv_fut = self._loop.create_task(self._recv_loop())
    self._closeCallback: Optional[Callable[[], None]] = None

Pero estos parámetros se reconocen en la última versión de websockets pero no en la versión 6:

File "/home/administrador/workspace/whatsappx/webot/apptest/whalesong/init.py", line 95, in start await self._driver.start_driver() File "/home/administrador/workspace/whatsappx/webot/apptest/whalesong/driver.py", line 49, in start_driver await self._fut_start File "/home/administrador/workspace/whatsappx/webot/apptest/whalesong/driver.py", line 53, in start_driver await self._fut_start File "/home/administrador/workspace/whatsappx/webot/apptest/whalesong/driver_chromium.py", line 72, in _internal_start_driver self.options) File "/usr/local/lib/python3.6/dist-packages/pyppeteer/launcher.py", line 330, in launch return await Launcher(options, kwargs).launch() File "/usr/local/lib/python3.6/dist-packages/pyppeteer/launcher.py", line 179, in launch connectionDelay, File "/usr/local/lib/python3.6/dist-packages/pyppeteer/connection.py", line 46, in init loop=self._loop) File "/usr/local/lib/python3.6/dist-packages/websockets/client.py", line 385, in init factory, host, port, **kwds) TypeError: create_connection() got an unexpected keyword argument 'ping_interval'

After all of these test; my test program based on minibot-chromium.py are not stable and stop to received and send messages after a random time. It's necesary to press the Ctrl-C keys to abort the application and them the following messages is displayed in the screen:

future: <Task finished coro=<WhalesongDriver._execute_command() done, defined at /home/administrador/workspace/whatsappx/webot/apptest/whalesong/driver_chromium.py:107> exception=NetworkError('Protocol error Runtime.evaluate: Target closed.',)> Traceback (most recent call last): File "/home/administrador/workspace/whatsappx/webot/apptest/whalesong/driver_chromium.py", line 109, in _execute_command f'(function() {{window.manager.executeCommand("{result_id}", "{command}", {dumps(params)})}})()' File "/usr/local/lib/python3.6/dist-packages/pyppeteer/page.py", line 1186, in evaluate return await frame.evaluate(pageFunction, args, force_expr=force_expr) File "/usr/local/lib/python3.6/dist-packages/pyppeteer/frame_manager.py", line 309, in evaluate pageFunction, args, force_expr=force_expr) File "/usr/local/lib/python3.6/dist-packages/pyppeteer/execution_context.py", line 54, in evaluate pageFunction, *args, force_expr=force_expr) File "/usr/local/lib/python3.6/dist-packages/pyppeteer/execution_context.py", line 88, in evaluateHandle _rewriteError(e) File "/usr/local/lib/python3.6/dist-packages/pyppeteer/execution_context.py", line 238, in _rewriteError raise error File "/usr/local/lib/python3.6/dist-packages/pyppeteer/execution_context.py", line 85, in evaluateHandle 'userGesture': True, pyppeteer.errors.NetworkError: Protocol error Runtime.evaluate: Target closed.

Partial list of the my installed packages

drwxr-sr-x 2 root staff 4096 Jul 9 10:45 nameparser-1.0.4.dist-info/ drwxr-sr-x 5 root staff 4096 Jul 8 19:05 openapi_client/ drwxr-sr-x 2 root staff 4096 Jul 8 19:05 openapi_client-1.0.0.dist-info/ drwxr-sr-x 21 root staff 4096 Apr 9 18:02 openpyxl/ drwxr-sr-x 2 root staff 4096 Apr 9 18:02 openpyxl-2.6.2-py3.6.egg-info/ drwxr-sr-x 5 root staff 4096 Jun 11 21:11 pip/ drwxr-sr-x 2 root staff 4096 Jun 11 21:11 pip-19.1.1.dist-info/ drwxr-sr-x 2 root staff 4096 Jun 11 18:18 pycache/ drwxr-sr-x 3 root staff 4096 Feb 28 14:36 pycard/ drwxr-sr-x 3 root staff 4096 Jul 8 17:09 pycares/ drwxr-sr-x 2 root staff 4096 Jul 8 17:09 pycares-3.0.0.dist-info/ drwxr-sr-x 3 root staff 4096 Jul 5 16:35 pydub/ drwxr-sr-x 2 root staff 4096 Jul 5 16:35 pydub-0.23.1.dist-info/ drwxr-sr-x 3 root staff 4096 Jul 16 14:23 pyee/ drwxr-sr-x 2 root staff 4096 Jul 16 14:23 pyee-5.0.0.dist-info/ drwxr-sr-x 3 root staff 4096 Jul 9 13:05 pypeln/ drwxr-sr-x 2 root staff 4096 Jul 9 13:05 pypeln-0.1.9.dist-info/ drwxr-sr-x 4 root staff 4096 Jul 17 00:08 pyppdf/ drwxr-sr-x 2 root staff 4096 Jul 17 00:08 pyppdf-0.0.10.dist-info/ drwxr-sr-x 3 root staff 4096 Jul 18 10:45 pyppeteer/ drwxr-sr-x 2 root staff 4096 Jul 17 12:16 pyppeteer-0.0.25.dist-info/ drwxr-sr-x 3 root staff 4096 Sep 6 2018 pytesseract/ drwxr-sr-x 2 root staff 4096 Sep 6 2018 pytesseract-0.2.4-py3.6.egg-info/ drwxr-sr-x 3 root staff 4096 Jul 5 18:51 redis/ drwxr-sr-x 2 root staff 4096 Jul 5 18:51 redis-3.2.1.dist-info/ drwxr-sr-x 3 root staff 4096 Jul 5 18:51 redis_collections/ drwxr-sr-x 2 root staff 4096 Jul 5 18:51 redis_collections-0.7.0.dist-info/ -rw-r--r-- 1 root staff 55 May 31 2018 setuptools.pth drwxr-sr-x 3 root staff 4096 Jul 8 19:05 test/ drwxr-sr-x 5 root staff 4096 Apr 16 16:18 uvloop/ drwxr-sr-x 2 root staff 4096 Apr 16 16:18 uvloop-0.12.2.dist-info/ drwxr-sr-x 2 root staff 4096 Jun 11 18:18 virtualenv-16.6.0.dist-info/ -rw-r--r-- 1 root staff 104065 Jun 11 18:18 virtualenv.py drwxr-sr-x 3 root staff 4096 Jun 11 18:18 virtualenv_support/ drwxr-sr-x 3 root staff 4096 Jul 1 18:32 vobject/ drwxr-sr-x 2 root staff 4096 Jul 1 18:32 vobject-0.9.6.1.dist-info/ drwxr-sr-x 6 root staff 4096 Jul 18 10:11 websockets/ drwxr-sr-x 2 root staff 4096 Jul 18 10:11 websockets-6.0.dist-info/

alfred82santa commented 4 years ago

You don't need to set that params if you use websocket 6.0