Closed JacobCallahan closed 3 years ago
mxschmitt
commented
4 years ago for iPython / Jupiter notebooks it seems like the current workaround would be to use that before you run your shell:
import nest_asyncio
nest_asyncio.apply()Out of curiosity to understand more the user case. You are trying out Playwright in a REPL or are you using it only with with the REPL in the end?
JacobCallahan
commented
4 years ago @mxschmitt thanks for the quick response! nest_asyncio isn't a standard package or one installed as a requirement for playwright. As such, I'm guessing most users wouldn't have it installed. However, that did solve the problem:
In [1]: import nest_asyncio
In [2]: nest_asyncio.apply()
In [3]: from playwright import sync_playwright
In [4]: playwright = sync_playwright().start()
In [5]: My use-case within IPython is largely API exploration and interactive process development before writing a script or integration with an application.
Additional note, your example works perfectly in the base python REPL.
bubthegreat
commented
4 years ago @mxschmitt - FYI, the nested async doesn't work for jupyter notebooks for at least the sync REPL example:
import nest_asyncio
nest_asyncio.apply()
from playwright import sync_playwright
playwright = sync_playwright().start()
---------------------------------------------------------------------------
NotImplementedError Traceback (most recent call last)
<ipython-input-3-85d403adca24> in <module>
----> 1 playwright = sync_playwright().start()
c:\users\bubth\development\playwright\venv\lib\site-packages\playwright\__init__.py in sync_playwright()
32
33 def sync_playwright() -> SyncPlaywrightContextManager:
---> 34 return SyncPlaywrightContextManager()
35
36
c:\users\bubth\development\playwright\venv\lib\site-packages\playwright\main.py in __init__(self)
81 class SyncPlaywrightContextManager:
82 def __init__(self) -> None:
---> 83 self._connection = run_driver()
84 self._playwright: SyncPlaywright
85
c:\users\bubth\development\playwright\venv\lib\site-packages\playwright\main.py in run_driver()
76 # if loop.is_running():
77 # raise Error("Can only run one Playwright at a time.")
---> 78 return loop.run_until_complete(run_driver_async())
79
80
c:\users\bubth\development\playwright\venv\lib\site-packages\nest_asyncio.py in run_until_complete(self, future)
93 raise RuntimeError(
94 'Event loop stopped before Future completed.')
---> 95 return f.result()
96 finally:
97 events._set_running_loop(old_running_loop)
C:\Program Files\Python37\lib\asyncio\futures.py in result(self)
176 self.__log_traceback = False
177 if self._exception is not None:
--> 178 raise self._exception
179 return self._result
180
C:\Program Files\Python37\lib\asyncio\tasks.py in __step(***failed resolving arguments***)
221 # We use the `send` method directly, because coroutines
222 # don't have `__iter__` and `__next__` methods.
--> 223 result = coro.send(None)
224 else:
225 result = coro.throw(exc)
c:\users\bubth\development\playwright\venv\lib\site-packages\playwright\main.py in run_driver_async()
62 stdout=asyncio.subprocess.PIPE,
63 stderr=_get_stderr_fileno(),
---> 64 limit=32768,
65 )
66 assert proc.stdout
C:\Program Files\Python37\lib\asyncio\subprocess.py in create_subprocess_exec(program, stdin, stdout, stderr, loop, limit, *args, **kwds)
215 program, *args,
216 stdin=stdin, stdout=stdout,
--> 217 stderr=stderr, **kwds)
218 return Process(transport, protocol, loop)
C:\Program Files\Python37\lib\asyncio\base_events.py in subprocess_exec(self, protocol_factory, program, stdin, stdout, stderr, universal_newlines, shell, bufsize, *args, **kwargs)
1531 transport = await self._make_subprocess_transport(
1532 protocol, popen_args, False, stdin, stdout, stderr,
-> 1533 bufsize, **kwargs)
1534 if self._debug and debug_log is not None:
1535 logger.info('%s: %r', debug_log, transport)
C:\Program Files\Python37\lib\asyncio\base_events.py in _make_subprocess_transport(self, protocol, args, shell, stdin, stdout, stderr, bufsize, extra, **kwargs)
461 extra=None, **kwargs):
462 """Create subprocess transport."""
--> 463 raise NotImplementedError
464
465 def _write_to_self(self):
NotImplementedError: 
pavelfeldman
commented
4 years ago @bubthegreat: This looks like a SelectorEventLoop's limitation on Python 3.7 on Windows. Starting with 3.8, a more complete implementation of the loop is used on Windows and it should support running subprocesses. See the platform support section for more details.
We do install this more capable ProactorEventLoop manually on 3.7 see here, but it might be that nest_asyncio does not respect our selection. So it is a combination of Windows + Python 3.7 + nest_asyncio that triggers it.
bubthegreat
commented
4 years ago I assume upgrading to 3.9 would improve this, still saw issues, and downgraded to 3.8.3 - still having issues with sync_playwright:
import nest_asyncio
nest_asyncio.apply()
from playwright import sync_playwright
playwright = sync_playwright().start()
---------------------------------------------------------------------------
Error Traceback (most recent call last)
<ipython-input-2-dfa883f2869c> in <module>
1 from playwright import sync_playwright
----> 2 playwright = sync_playwright().start()
c:\users\bubth\development\playwright-test\venv\lib\site-packages\playwright\__init__.py in sync_playwright()
32
33 def sync_playwright() -> SyncPlaywrightContextManager:
---> 34 return SyncPlaywrightContextManager()
35
36
c:\users\bubth\development\playwright-test\venv\lib\site-packages\playwright\main.py in __init__(self)
86 class SyncPlaywrightContextManager:
87 def __init__(self) -> None:
---> 88 self._connection = run_driver()
89 self._playwright: SyncPlaywright
90
c:\users\bubth\development\playwright-test\venv\lib\site-packages\playwright\main.py in run_driver()
80 loop = asyncio.get_event_loop()
81 if loop.is_running():
---> 82 raise Error("Can only run one Playwright at a time.")
83 return loop.run_until_complete(run_driver_async())
84
Error: Can only run one Playwright at a time.
ddelange
commented
4 years ago Currently, playwright disallows attaching to a running loop: https://github.com/microsoft/playwright-python/blob/d98d38b89b4864966218b1333c9399fc35487725/playwright/main.py#L79-L83
This should be fine in plain IPython (does not launch an asyncio event loop in the main thread), but will fail in Jupyter that uses tornado>=5 (launches asyncio event loop on startup under the hood).
As calling run_until_complete is disallowed on running loops, this will fail in Jupyter.
Minimal reproducible example that will run in latest IPython, but will break on Jupyter environments that use tornado>=5 under the hood:
import asyncio
print(asyncio.run(asyncio.sleep(0)))So for Jupyter, additionally a pip install 'tornado<5' 'notebook<5.7.9' should fix loop issues introduced in tornado>=5 (and if you need jupyterhub, use jupyterhub<0.9.0). refs https://github.com/jupyter/notebook/issues/3397#issuecomment-675566100 https://github.com/ipython/ipykernel/issues/548
pavelfeldman
commented
4 years ago My understanding is that resolving this requires cooperation between the embedder that controls the loop and sync Playwright that also wants to dispatch its messages forever. And I don't think such cooperation is the right way to go. Sync API is designed as self-contained, assumes it controls execution. Is there a way to use async in Jupyter?
bubthegreat
commented
4 years ago If I get time tonight ill play around with it- I dont see any reason for it not to run in the existing loop - or are there architectural challenges with it sharing the loop?
On Tue, Oct 6, 2020, 8:35 AM Pavel Feldman notifications@github.com wrote:
My understanding is that resolving this requires cooperation between the embedder that controls the loop and sync Playwright that also wants to dispatch its messages forever. And I don't think such cooperation is the right way to go. Sync API is designed as self-contained, assumes it controls execution. Is there a way to use async in Jupyter?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/microsoft/playwright-python/issues/178#issuecomment-704312511, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC5CHIMV52GR76C7R7XFYKTSJMTMLANCNFSM4QLBAFHQ .
ddelange
commented
4 years ago Executing a cell in Jupyter will actually require a lock on the main loop from ipykernel, so you could schedule execution of a Future in one cell (which will start execution once that cell has finished), and get the result in the next cell. Doing it all in one cell is certainly not trivial without nested loops.
tals
commented
3 years ago for iPython / Jupiter notebooks it seems like the current workaround would be to use that before you run your shell:
import nest_asyncio nest_asyncio.apply()Out of curiosity to understand more the user case. You are trying out Playwright in a REPL or are you using it only with with the REPL in the end?
Was just bitten by this.
It is very convenient to use Jupyter for explorative programming, especially when the thing you're working with is heavy (like a browser in a particular state). For my use case, I wanted to spin up a few browser instances and use them to test out a complicated feature that involves webrtc etc.
Also ran into this cute tutorial demonstrating why this is nice (he's using Selenium): https://medium.com/shakuro/adopting-ipython-jupyter-for-selenium-testing-d02309dd00b8
The stated workaround sadly didn't work for me, neither on sync nor async versions.
Hope this helps!
tals
commented
3 years ago Found a workaround for getting Playwright working within Jupyter.
It leverages an awesome python library called RPyc to run Playwright outside of the kernel while keeping the API the same.
!pip install rpyc
import rpyc
import subprocess
subprocess.Popen('rpyc_classic.py -m forking'.split()) # start RPyc worker
conn = rpyc.classic.connect('localhost')
rplaywright = conn.modules['playwright']
# rplaywright has the same API as playwright
pr = rplaywright.sync_playwright().start()
browser = pr.chromium.launch()
page = browser.newPage()
page.goto('http://whatsmyuseragent.org/')
page.screenshot(path='example.png')
browser.close()# Grab screen
from PIL import Image
from io import BytesIO
browser = pr.chromium.launch()
page = browser.newPage()
page.goto('http://localhost:4000/v/tf3nhh')
img=page.screenshot(path='example.png')
Image.open(BytesIO(img))
wohenniubi
commented
3 years ago Found a workaround for getting Playwright working within Jupyter.
It leverages an awesome python library called RPyc to run Playwright outside of the kernel while keeping the API the same.
!pip install rpyc import rpyc import subprocess subprocess.Popen('rpyc_classic.py -m forking'.split()) # start RPyc worker conn = rpyc.classic.connect('localhost') rplaywright = conn.modules['playwright'] # rplaywright has the same API as playwright pr = rplaywright.sync_playwright().start() browser = pr.chromium.launch() page = browser.newPage() page.goto('http://whatsmyuseragent.org/') page.screenshot(path='example.png') browser.close()# Grab screen from PIL import Image from io import BytesIO browser = pr.chromium.launch() page = browser.newPage() page.goto('http://localhost:4000/v/tf3nhh') img=page.screenshot(path='example.png') Image.open(BytesIO(img))
Hi tals, I follow the wolkaround that you mentioned, however, seems it can't help in win10. The error info shows "OSError: [WinError 193] %1 is not a valid Win32 application"
The complete error is as follows. Thanks for any further info.
OSError Traceback (most recent call last)
tals
commented
3 years ago @wohenniubi sorry don't have a windows machine to test this out, but try figuring out how to run the rpyc_classic.py program on command line first
wohenniubi
commented
3 years ago @tals Thanks for the response. After several trials without success, I might switch back to use selenium by now. Anyway, I'll keep on watching this jupyter issue and this promising tool, playwright.
@wohenniubi sorry don't have a windows machine to test this out, but try figuring out how to run the rpyc_classic.py program on command line first
mxschmitt
commented
3 years ago Closed for now as part of the triage. Please reopen if it's still persistent.
tals
commented
3 years ago @mxschmitt not sure why its closed - this problem still exists. You can reproduce this on the latest versions of jupyter+playwright
ddelange
commented
3 years ago The bug is upstream: https://github.com/ipython/ipykernel/issues/548
And there is a workaround: https://github.com/microsoft/playwright-python/issues/178#issuecomment-680249269
Maybe could warn the jupyter user with a verbose re-raising try-except?
tals
commented
3 years ago @ddelange the proposed workaround sadly doesn't work on jupyter: https://github.com/microsoft/playwright-python/issues/178#issuecomment-702504459
The only thing I could do to get it working was running it outside of the process with RPyc
bubthegreat
commented
2 years ago It's been almost two years and this still isn't fixed that I can find - playwright is explicitly stopping me from using nested loops:
import nest_asyncio
nest_asyncio.apply()
from playwright.sync_api import sync_playwright
playwright = sync_playwright().start()
---------------------------------------------------------------------------
Error Traceback (most recent call last)
Input In [3], in <cell line: 2>()
1 from playwright.sync_api import sync_playwright
----> 2 playwright = sync_playwright().start()
File c:\users\bubth\development\soc2\venv\lib\site-packages\playwright\sync_api\_context_manager.py:76, in PlaywrightContextManager.start(self)
75 def start(self) -> SyncPlaywright:
---> 76 return self.__enter__()
File c:\users\bubth\development\soc2\venv\lib\site-packages\playwright\sync_api\_context_manager.py:42, in PlaywrightContextManager.__enter__(self)
40 own_loop = loop
41 if loop.is_running():
---> 42 raise Error(
43 """It looks like you are using Playwright Sync API inside the asyncio loop.
44 Please use the Async API instead."""
45 )
47 def greenlet_main() -> None:
48 loop.run_until_complete(self._connection.run_as_sync())
Error: It looks like you are using Playwright Sync API inside the asyncio loop.
Please use the Async API instead.The workarounds don't work for those of us using windows, and they don't appear to work in WSL either - is there a reason we can't get more information on why this is still closed two years later with no traction and no functional workarounds for presumably a fairly normal use case of using a jupyter notebook for exploratory programming?
It seems hard to believe that we just assume everyone will use raw iPython for everything so I'm confused by the reluctance to actually fix this.
If I'm understanding the asyncio base_events.py error, it's because there's no implementation of the _make_subprocess_transport function - is that not something that would be on the playwright package to implement in it's async implementation?
hkfsc
commented
2 years ago @tals Rpyc will only works if we will not call any playwright sub-modules directly, https://github.com/tomerfiliba-org/rpyc/issues/496. In my case, it not works well, still finding a way to let playwright sync api work with jupyter now...
danielichis
commented
2 years ago still waiting i want to leave selenium but this stop me
ddelange
commented
2 years ago If you want to use an async framework (playwright-python is natively async) inside Jupyter Notebooks, the ipykernel maintainer suggests to use the async api directly (await & async def syntax), instead of the sync wrappers. They don't support sync wrappers around async libs...
liudonghua123
commented
2 years ago still does not work in jupyter (vscode interactive).
Sync API is not allowed!
import asyncio
from playwright.async_api import async_playwright
from playwright.sync_api import sync_playwright
# https://github.com/microsoft/playwright-python/issues/178#issuecomment-680249269
import nest_asyncio
nest_asyncio.apply()
# Error: It looks like you are using Playwright Sync API inside the asyncio loop.
# Please use the Async API instead.
p = sync_playwright().start()Async API is not worked!
import asyncio
from playwright.async_api import async_playwright
from playwright.sync_api import sync_playwright
# https://github.com/microsoft/playwright-python/issues/178#issuecomment-680249269
import nest_asyncio
nest_asyncio.apply()
async def main():
async with async_playwright() as p:
browser = await p.chromium.launch(channel="chrome",headless=False, slow_mo=100)
page = await browser.new_page()
await page.goto("https://cnki.net/")
await browser.close()
# NotImplementedError
asyncio.run(main())Maybe I should switch to selenium, see run-selenium-in-jupyter-notebook-on-wsl2-or-ubuntu。
ddelange
commented
2 years ago can you try the async snippet without nest_asyncio, and post the full error traceback here?
liudonghua123
commented
2 years ago can you try the async snippet without
nest_asyncio, and post the full error traceback here?
Here is my snapshot and error logs.
NotImplementedError Traceback (most recent call last)
Cell In [33], line 9
7 await browser.close()
8 # NotImplementedError
----> 9 asyncio.run(main())
File c:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\site-packages\nest_asyncio.py:35, in _patch_asyncio.<locals>.run(main, debug)
[33](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/site-packages/nest_asyncio.py?line=32) task = asyncio.ensure_future(main)
[34](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/site-packages/nest_asyncio.py?line=33) try:
---> [35](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/site-packages/nest_asyncio.py?line=34) return loop.run_until_complete(task)
[36](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/site-packages/nest_asyncio.py?line=35) finally:
[37](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/site-packages/nest_asyncio.py?line=36) if not task.done():
File c:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\site-packages\nest_asyncio.py:89, in _patch_loop.<locals>.run_until_complete(self, future)
[86](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/site-packages/nest_asyncio.py?line=85) if not f.done():
[87](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/site-packages/nest_asyncio.py?line=86) raise RuntimeError(
[88](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/site-packages/nest_asyncio.py?line=87) 'Event loop stopped before Future completed.')
---> [89](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/site-packages/nest_asyncio.py?line=88) return f.result()
File c:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\asyncio\futures.py:201, in Future.result(self)
[199](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/futures.py?line=198) self.__log_traceback = False
[200](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/futures.py?line=199) if self._exception is not None:
--> [201](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/futures.py?line=200) raise self._exception.with_traceback(self._exception_tb)
[202](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/futures.py?line=201) return self._result
...
[496](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/base_events.py?line=495) extra=None, **kwargs):
[497](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/base_events.py?line=496) """Create subprocess transport."""
--> [498](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/base_events.py?line=497) raise NotImplementedError
NotImplementedError:
this error is coming from nest_asyncio ^, can you run it in a fresh kernel without nest_asyncio?
liudonghua123
commented
2 years ago It seems sync API is not work in plain python script.
The following code just launch browser, open page then exit. I did not call browser.close().
from playwright.sync_api import sync_playwright
p = sync_playwright().start()
browser = p.chromium.launch(channel="chrome",headless=False, slow_mo=100)
page = browser.new_page()
page.goto('https://cnki.net/')Here is ipython error logs.
C:\Users\Liu.D.H>ipython
Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug 1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.5.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: from playwright.sync_api import sync_playwright
...:
...: p = sync_playwright().start()
...: browser = p.chromium.launch(channel="chrome",headless=False, slow_mo=100)
...: page = browser.new_page()
...: page.goto('https://cnki.net/')
Out[1]: <Response url='https://cnki.net/' request=<Request url='https://cnki.net/' method='GET'>>
Traceback (most recent call last):
File "C:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\Scripts\ipython.exe\__main__.py", line 7, in <module>
sys.exit(start_ipython())
File "C:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\site-packages\IPython\__init__.py", line 124, in start_ipython
return launch_new_instance(argv=argv, **kwargs)
File "C:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\site-packages\traitlets\config\application.py", line 976, in launch_instance
app.start()
File "C:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\site-packages\IPython\terminal\ipapp.py", line 318, in start
self.shell.mainloop()
File "C:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\site-packages\IPython\terminal\interactiveshell.py", line 685, in mainloop
self.interact()
File "C:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\site-packages\IPython\terminal\interactiveshell.py", line 670, in interact
code = self.prompt_for_code()
File "C:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\site-packages\IPython\terminal\interactiveshell.py", line 609, in prompt_for_code
text = self.pt_app.prompt(
File "C:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\site-packages\prompt_toolkit\shortcuts\prompt.py", line 1034, in prompt
return self.app.run(
File "C:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\site-packages\prompt_toolkit\application\application.py", line 937, in run
return loop.run_until_complete(
File "C:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\asyncio\base_events.py", line 622, in run_until_complete
self._check_running()
File "C:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\asyncio\base_events.py", line 582, in _check_running
raise RuntimeError('This event loop is already running')
RuntimeError: This event loop is already running
If you suspect this is an IPython 8.5.0 bug, please report it at:
https://github.com/ipython/ipython/issues
or send an email to the mailing list at ipython-dev@python.org
You can print a more detailed traceback right now with "%tb", or use "%debug"
to interactively debug it.
Extra-detailed tracebacks for bug-reporting purposes can be enabled via:
%config Application.verbose_crash=True
sys:1: RuntimeWarning: coroutine 'Application.run_async' was never awaited
C:\Users\Liu.D.H>this error is coming from
nest_asyncio^, can you run it in a fresh kernel withoutnest_asyncio?
I restarted the kernel, the errors changed.
I run the following code block.
async def main():
async with async_playwright() as p:
browser = await p.chromium.launch(channel="chrome",headless=False, slow_mo=100)
page = await browser.new_page()
await page.goto("https://cnki.net/")
await browser.close()
asyncio.run(main())RuntimeError Traceback (most recent call last)
d:\code\python\cnki_crawler_playwright\main.py in line 9
[35](file:///d%3A/code/python/cnki_crawler_playwright/main.py?line=34) await browser.close()
[36](file:///d%3A/code/python/cnki_crawler_playwright/main.py?line=35) # NotImplementedError
----> [37](file:///d%3A/code/python/cnki_crawler_playwright/main.py?line=36) asyncio.run(main())
File c:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\asyncio\runners.py:33, in run(main, debug)
[9](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/runners.py?line=8) """Execute the coroutine and return the result.
[10](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/runners.py?line=9)
[11](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/runners.py?line=10) This function runs the passed coroutine, taking care of
(...)
[30](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/runners.py?line=29) asyncio.run(main())
[31](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/runners.py?line=30) """
[32](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/runners.py?line=31) if events._get_running_loop() is not None:
---> [33](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/runners.py?line=32) raise RuntimeError(
[34](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/runners.py?line=33) "asyncio.run() cannot be called from a running event loop")
[36](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/runners.py?line=35) if not coroutines.iscoroutine(main):
[37](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/runners.py?line=36) raise ValueError("a coroutine was expected, got {!r}".format(main))
RuntimeError: asyncio.run() cannot be called from a running event loopinstead of asyncio.run(main()), try await main()
liudonghua123
commented
2 years ago instead of
asyncio.run(main()), tryawait main()
I tried, but still not work.
NotImplementedError Traceback (most recent call last)
d:\code\python\cnki_crawler_playwright\main.py in line 10
[34](file:///d%3A/code/python/cnki_crawler_playwright/main.py?line=33) await page.goto("https://cnki.net/")
[35](file:///d%3A/code/python/cnki_crawler_playwright/main.py?line=34) await browser.close()
---> [38](file:///d%3A/code/python/cnki_crawler_playwright/main.py?line=37) await main()
d:\code\python\cnki_crawler_playwright\main.py in line 3, in main()
[30](file:///d%3A/code/python/cnki_crawler_playwright/main.py?line=29) async def main():
----> [31](file:///d%3A/code/python/cnki_crawler_playwright/main.py?line=30) async with async_playwright() as p:
[32](file:///d%3A/code/python/cnki_crawler_playwright/main.py?line=31) browser = await p.chromium.launch(channel="chrome", headless=False, slow_mo=100)
[33](file:///d%3A/code/python/cnki_crawler_playwright/main.py?line=32) page = await browser.new_page()
File c:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\site-packages\playwright\async_api\_context_manager.py:46, in PlaywrightContextManager.__aenter__(self)
[44](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/site-packages/playwright/async_api/_context_manager.py?line=43) if not playwright_future.done():
[45](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/site-packages/playwright/async_api/_context_manager.py?line=44) playwright_future.cancel()
---> [46](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/site-packages/playwright/async_api/_context_manager.py?line=45) playwright = AsyncPlaywright(next(iter(done)).result())
[47](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/site-packages/playwright/async_api/_context_manager.py?line=46) playwright.stop = self.__aexit__ # type: ignore
[48](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/site-packages/playwright/async_api/_context_manager.py?line=47) return playwright
File c:\Users\Liu.D.H\AppData\Local\Programs\Python\Python310\lib\site-packages\playwright\_impl\_transport.py:121, in PipeTransport.connect(self)
[118](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/site-packages/playwright/_impl/_transport.py?line=117) if getattr(sys, "frozen", False):
[119](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/site-packages/playwright/_impl/_transport.py?line=118) env.setdefault("PLAYWRIGHT_BROWSERS_PATH", "0")
--> [121](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/site-packages/playwright/_impl/_transport.py?line=120) self._proc = await asyncio.create_subprocess_exec(
[122](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/site-packages/playwright/_impl/_transport.py?line=121) str(self._driver_executable),
...
[496](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/base_events.py?line=495) extra=None, **kwargs):
[497](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/base_events.py?line=496) """Create subprocess transport."""
--> [498](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/asyncio/base_events.py?line=497) raise NotImplementedError
NotImplementedError:I also tried asyncio.get_event_loop().run_until_complete(main()), however, I got RuntimeError: This event loop is already running.
ddelange
commented
2 years ago I have no problem running the example on a fresh jupyter server:
The fact that your traceback hangs here:
--> [121](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/site-packages/playwright/_impl/_transport.py?line=120) self._proc = await asyncio.create_subprocess_exec(
Makes me think you might not be running in a clean jupyter environment. Maybe vscode does some magic?
liudonghua123
commented
2 years ago I have no problem running the example on a fresh jupyter server:
The fact that your traceback hangs here:
--> [121](file:///c%3A/Users/Liu.D.H/AppData/Local/Programs/Python/Python310/lib/site-packages/playwright/_impl/_transport.py?line=120) self._proc = await asyncio.create_subprocess_exec(Makes me think you might not be running in a clean jupyter environment. Maybe vscode does some magic?
Hi, I tried to update jupyter via pip install -U jupyterlab notebook voila. Then tried the following code in jupyterlab. But still not works.
import asyncio
from playwright.async_api import async_playwright
async def run(playwright):
chromium = playwright.chromium
browser = await chromium.launch(channel="chrome",headless=False, slow_mo=100).launch()
page = await browser.new_page()
await page.goto("https://example.com")
await browser.close()
async def main():
async with async_playwright() as playwright:
await run(playwright)
await main()
ddelange
commented
2 years ago do you get the same traceback when you run the snippet in ipython instead of jupyter? if yes, we eliminated the ipykernel loop issue, and my next guess would be the fact that you're on windows...
ddelange
commented
2 years ago potentially indeed your OS ref https://stackoverflow.com/q/70349876
@mxschmitt would this be caught by the Windows CI?
edit: more specifically https://stackoverflow.com/a/44639711/5511061
liudonghua123
commented
2 years ago @ddelange Thanks. I found a workaround on windows now. NOT RECOMMEND!
From https://github.com/jupyter/notebook/issues/5916, https://github.com/minrk/ipykernel/commit/079f072a8e90422dc74270992589c56ad9f7f9f2, https://stackoverflow.com/questions/44633458/why-am-i-getting-notimplementederror-with-async-and-await-on-windows/74311290#74311290, https://github.com/ipython/ipykernel/blob/99a1becaa958b33d80fe337fdbc41305030fdb6d/ipykernel/kernelapp.py#L633-L637. I found a way to make it work hackly (not recommend) on windows.
Comment asyncio.set_event_loop_policy(WindowsSelectorEventLoopPolicy()) in %LOCALAPPDATA%\Programs\Python\Python310\Lib\site-packages\ipykernel\kernelapp.py (change to your path).
Then the following code will work in jupyter notebook.
import asyncio
from playwright.async_api import async_playwright
async def main():
async with async_playwright() as p:
browser = await p.chromium.launch(channel="chrome", headless=False, slow_mo=100)
page = await browser.new_page()
await page.goto("https://cnki.net/")
await browser. Close()
await main()still does not work in jupyter (vscode interactive).
Sync API is not allowed!
import asyncio from playwright.async_api import async_playwright from playwright.sync_api import sync_playwright # https://github.com/microsoft/playwright-python/issues/178#issuecomment-680249269 import nest_asyncio nest_asyncio.apply() # Error: It looks like you are using Playwright Sync API inside the asyncio loop. # Please use the Async API instead. p = sync_playwright().start()Async API is not worked!
import asyncio from playwright.async_api import async_playwright from playwright.sync_api import sync_playwright # https://github.com/microsoft/playwright-python/issues/178#issuecomment-680249269 import nest_asyncio nest_asyncio.apply() async def main(): async with async_playwright() as p: browser = await p.chromium.launch(channel="chrome",headless=False, slow_mo=100) page = await browser.new_page() await page.goto("https://cnki.net/") await browser.close() # NotImplementedError asyncio.run(main())Maybe I should switch to selenium, see run-selenium-in-jupyter-notebook-on-wsl2-or-ubuntu。
I have another question, why can't I use sync API in jupyter? I have to write a lot await in async API. The sync API is simple and straight, performance or jupyter does not allowed?
ddelange
commented
2 years ago If you want to use an async framework (playwright-python is natively async) inside Jupyter Notebooks, the ipykernel maintainer suggests to use the async api directly (
await&async defsyntax), instead of the sync wrappers. They don't support sync wrappers around async libs...
jup it's not allowed
liudonghua123
commented
2 years ago If you want to use an async framework (playwright-python is natively async) inside Jupyter Notebooks, the ipykernel maintainer suggests to use the async api directly (
await&async defsyntax), instead of the sync wrappers. They don't support sync wrappers around async libs...jup it's not allowed
Not allowed? However, I used a lot of packages like numpy, pandas, matplotlib, tensorflow, pytorch, and so on are sync style of API, most of them are async only.
Is there any ways to make sync API of playwright-python work on jupyter?
ddelange
commented
2 years ago I didn't double check, but I'm pretty sure those libraries don't use asyncio under the hood (they may still release the GIL however). For more information on the topic, see https://github.com/ipython/ipykernel/issues/548#issuecomment-713560735.
Each time I attempt the REPl example, I get the traceback below. Sync context-manager example worked though.
playwright==0.8.0REPL Example: https://github.com/microsoft/playwright-python#repl-support-without-context-managersIPython session