HENNGE / arsenic

Async WebDriver implementation for asyncio and asyncio-compatible frameworks
Other
349 stars 52 forks source link

compatibility issue when asyncio loop policy is not the default one #139

Closed nlgranger closed 2 years ago

nlgranger commented 2 years ago

get_session blocks when the loop policy is not the default one, for instance when using asyncio-glib:

import asyncio_glib
import asyncio

asyncio.set_event_loop_policy(asyncio_glib.GLibEventLoopPolicy())
from arsenic import services, browsers, get_session

async def loop():
    while True:
        await asyncio.sleep(0.5)
        print('.', end='', flush=True)

async def main():
    service = services.Geckodriver()
    browser = browsers.Firefox()
    print('ready?')
    async with get_session(service, browser) as d:
        await d.get('https://www.google.fr')
        print("done")

asyncio.run(asyncio.wait([main(), loop()]))
dimaqq commented 2 years ago

Interesting... any idea what's actually blocking? Is it trying to spawn geckodriver? Or trying to connect to its port? Or is it the .get(...) operation, which, in essence is aiohttp request?

nlgranger commented 2 years ago

I do not see any geckodriver process and I just noticed the python process uses 100% of a CPU while blocking on get_session.

nlgranger commented 2 years ago

It block when checking the version of geckodriver, here is the call stack:

run_process (.../venv/lib/python3.9/site-packages/arsenic/subprocess.py:46)
_check_version (.../venv/lib/python3.9/site-packages/arsenic/services.py:73)
start (.../venv/lib/python3.9/site-packages/arsenic/services.py:92)
start_session (.../venv/lib/python3.9/site-packages/arsenic/__init__.py:28)
__aenter__ (.../venv/lib/python3.9/site-packages/arsenic/__init__.py:16)
main (.../test.py:18)
_run (/usr/lib/python3.9/asyncio/events.py:80)
_run_once (/usr/lib/python3.9/asyncio/base_events.py:1882)
run_forever (/usr/lib/python3.9/asyncio/base_events.py:596)
run_until_complete (/usr/lib/python3.9/asyncio/base_events.py:629)
run (/usr/lib/python3.9/asyncio/runners.py:44)
<module> (.../test.py:24)
dimaqq commented 2 years ago

https://github.com/HDE/arsenic/blob/7323c550a46b94aa0dcda6659b9c3eff11bec3f0/src/arsenic/subprocess.py#L46 code in question...

I wonder if asyncio.create_subprocess_exec and process.communicate are supported in this custom event loop 🤔

dimaqq commented 2 years ago

It could be smth like https://github.com/jhenstridge/asyncio-glib/issues/5 🤔

dimaqq commented 2 years ago

Given that the last update to asyncio_glib was almost 2 years ago, I'm tempted to point my finger there.

nlgranger commented 2 years ago

Indeed, I will try something else for glib event loop interaction. Thanks for looking into this.