prompt-toolkit / ptpython

A better Python REPL
BSD 3-Clause "New" or "Revised" License
5.11k stars 280 forks source link

ptpython on Windows 11 in asyncio mode does not work #582

Open sockduct opened 3 weeks ago

sockduct commented 3 weeks ago

I have Windows 11 and Python 3.12.3 and wish to experiment with the asyncio REPL. It works fine with the built-in REPL:

PS C:\> python -m asyncio
asyncio REPL 3.12.3 (tags/v3.12.3:f6650f9, Apr  9 2024, 14:05:25) [MSC v.1938 64 bit (AMD64)] on win32
Use "await" directly instead of "asyncio.run()".
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncio
>>> await asyncio.sleep(1, 'Done!')
'Done!'
>>> exit()

However, I cannot get it to work with ptpython:

PS C:\> ptpython --asyncio

Starting ptpython asyncio REPL
Use "await" directly instead of "asyncio.run()".
In [1]: await asyncio.sleep(1, 'Done!')
Traceback (most recent call last):
  File "C:\Users\james\AppData\Roaming\Python\Python312\site-packages\ptpython\repl.py", line 183, in run_and_show_expression_async
    loop.add_signal_handler(signal.SIGINT, lambda *_: task.cancel())
  File "C:\Program Files\Python312\Lib\asyncio\events.py", line 582, in add_signal_handler
    raise NotImplementedError
NotImplementedError

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\james\AppData\Roaming\Python\Python312\Scripts\ptpython.exe\__main__.py", line 7, in <module>

  File "C:\Users\james\AppData\Roaming\Python\Python312\site-packages\ptpython\entry_points\run_ptpython.py", line 231, in run

    asyncio.run(embed_result)
  File "C:\Program Files\Python312\Lib\asyncio\runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python312\Lib\asyncio\runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python312\Lib\asyncio\base_events.py", line 687, in run_until_complete

    return future.result()
           ^^^^^^^^^^^^^^^
  File "C:\Users\james\AppData\Roaming\Python\Python312\site-packages\ptpython\repl.py", line 528, in coroutine

    await repl.run_async()
  File "C:\Users\james\AppData\Roaming\Python\Python312\site-packages\ptpython\repl.py", line 252, in run_async

    await self.run_and_show_expression_async(text)
  File "C:\Users\james\AppData\Roaming\Python\Python312\site-packages\ptpython\repl.py", line 206, in run_and_show_expression_async

    loop.remove_signal_handler(signal.SIGINT)
  File "C:\Program Files\Python312\Lib\asyncio\events.py", line 585, in remove_signal_handler

    raise NotImplementedError
NotImplementedError
Task exception was never retrieved
future: <Task finished name='Task-314' coro=<PythonRepl.run_and_show_expression_async.<locals>.eval() done, defined at C:\Users\james\AppData\Roaming\Python\Python312\site-packages\ptpython\repl.py:172> exception=NameError("name 'asyncio' is not defined")>

Traceback (most recent call last):
  File "C:\Users\james\AppData\Roaming\Python\Python312\site-packages\ptpython\repl.py", line 175, in eval

    return await self.eval_async(text)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\james\AppData\Roaming\Python\Python312\site-packages\ptpython\repl.py", line 329, in eval_async

    result = await result
             ^^^^^^^^^^^^
  File "<stdin>", line 1, in <module>
NameError: name 'asyncio' is not defined. Did you forget to import 'asyncio'

Note it also doesn't work with Python 3.8.10:

PS C:\> python -m asyncio
asyncio REPL 3.8.10 (tags/v3.8.10:3d8993a, May  3 2021, 11:34:34) [MSC v.1928 32 bit (Intel)] on win32
Use "await" directly instead of "asyncio.run()".
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncio
>>> await asyncio.sleep(1, 'Done!')
'Done!'
>>> exit()

C:\> ptpython --asyncio
Starting ptpython asyncio REPL
Use "await" directly instead of "asyncio.run()".
In [1]: await asyncio.sleep(1, 'Done!')
Traceback (most recent call last):
  File "c:\users\james\appdata\local\programs\python\python38-32\lib\site-packages\ptpython\repl.py", line 183, in run_
and_show_expression_async
    loop.add_signal_handler(signal.SIGINT, lambda *_: task.cancel())
  File "c:\users\james\appdata\local\programs\python\python38-32\lib\asyncio\events.py", line 536, in add_signal_handler
    raise NotImplementedError
NotImplementedError

Traceback (most recent call last):
  File "c:\users\james\appdata\local\programs\python\python38-32\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\users\james\appdata\local\programs\python\python38-32\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\users\james\appdata\local\Programs\Python\Python38-32\Scripts\ptpython.exe\__main__.py", line 7, in <module>
  File "c:\users\james\appdata\local\programs\python\python38-32\lib\site-packages\ptpython\entry_points\run_ptpython.py", line 231, in run
    asyncio.run(embed_result)
  File "c:\users\james\appdata\local\programs\python\python38-32\lib\asyncio\runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "c:\users\james\appdata\local\programs\python\python38-32\lib\asyncio\base_events.py", line 616, in run_until_complete
    return future.result()
  File "c:\users\james\appdata\local\programs\python\python38-32\lib\site-packages\ptpython\repl.py", line 528, in coroutine
    await repl.run_async()
  File "c:\users\james\appdata\local\programs\python\python38-32\lib\site-packages\ptpython\repl.py", line 252, in run_async
    await self.run_and_show_expression_async(text)
  File "c:\users\james\appdata\local\programs\python\python38-32\lib\site-packages\ptpython\repl.py", line 206, in run_and_show_expression_async
    loop.remove_signal_handler(signal.SIGINT)
  File "c:\users\james\appdata\local\programs\python\python38-32\lib\asyncio\events.py", line 539, in remove_signal_handler
    raise NotImplementedError
NotImplementedError
Task exception was never retrieved
future: <Task finished name='Task-61' coro=<PythonRepl.run_and_show_expression_async.<locals>.eval() done, defined at c:\users\james\appdata\local\programs\python\python38-32\lib\site-packages\ptpython\repl.py:172> exception=NameError("name 'asyncio' is not defined")>
Traceback (most recent call last):
  File "c:\users\james\appdata\local\programs\python\python38-32\lib\site-packages\ptpython\repl.py", line 175, in eval
    return await self.eval_async(text)
  File "c:\users\james\appdata\local\programs\python\python38-32\lib\site-packages\ptpython\repl.py", line 329, in eval_async
    result = await result
  File "<stdin>", line 1, in <module>
NameError: name 'asyncio' is not defined

Is it possible to get this working on Windows? I see on Stack Overflow, in some cases the event loop can be changed to support various use cases. Is there a configurable option to get this to work?

jupiterbjy commented 3 weeks ago

The other loop you probably meant is set by this:

import asyncio
asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())

But adding these at config.py doesn't make this work, nor that use of WindowsSelectorEventLoop is recommended neither, as it limit the functionality of asyncio.

sockduct commented 3 weeks ago

OK - thanks for the feedback. Any suggestions on approaching debugging this to see what's missing? Maybe I could at least document next steps???

jupiterbjy commented 2 weeks ago

I tried to figure out the internals but no luck so far, I'd need more time to look thru their source code, wish I had more time!

So far it seems to be already using asyncio internally:

# run_ptpython.py

def run() -> None:
    a = create_parser().parse_args()
    ...
    # When a file has been given, run that, otherwise start the shell.
    if a.args and not a.interactive:
        ...
    # Run interactive shell.
    else:
        enable_deprecation_warnings()

        # Apply config file
        def configure(repl: PythonRepl) -> None:
            ...

        embed_result = embed(  # type: ignore
            vi_mode=a.vi,
            history_filename=history_file,
            configure=configure,
            locals=__main__.__dict__,
            globals=__main__.__dict__,
            startup_paths=startup_paths,
            title="Python REPL (ptpython)",
            return_asyncio_coroutine=a.asyncio,
        )

        if a.asyncio:
            print("Starting ptpython asyncio REPL")
            print('Use "await" directly instead of "asyncio.run()".')
            asyncio.run(embed_result)

Yet can't debug further around line asyncio.run(), I'd try Pycharm later when I have time. I can only use vscode in company and it's so annoying to debug thru the builtin modules.

But so far feels like this section seems problematic:

    async def run_and_show_expression_async(self, text: str) -> Any:
        loop = asyncio.get_running_loop()
        system_exit: SystemExit | None = None

        try:
            try:
                # Create `eval` task. Ensure that control-c will cancel this
                # task.
                async def eval() -> Any:
                    nonlocal system_exit
                    try:
                        return await self.eval_async(text)
                    except SystemExit as e:
                        # Don't propagate SystemExit in `create_task()`. That
                        # will kill the event loop. We want to handle it
                        # gracefully.
                        system_exit = e

                task = asyncio.create_task(eval())
                loop.add_signal_handler(signal.SIGINT, lambda *_: task.cancel())
                # ^^^^^ here!
         ...

...as adding singal handler won't work on windows, potential hindsight of ptpython devs.

import signal
import asyncio

async def main():
    loop = asyncio.get_event_loop()
    loop.add_signal_handler(signal.SIGINT, lambda: print("SIGINT"))

    await asyncio.sleep(10)

asyncio.run(main())
Exception has occurred: NotImplementedError
exception: no description
  File "...\asyncio_task_count.py", line 7, in main
    loop.add_signal_handler(signal.SIGINT, lambda: print("SIGINT"))
  File ...\asyncio_task_count.py", line 12, in <module>
    asyncio.run(main())
NotImplementedError: 
jupiterbjy commented 2 weeks ago

Yeah I think speculation was right, so this section tries to handle ctrl+c but since it's not implemented in ProactorEventLoop it's failing. Windows triggers on KeyboardInterrupt instead as there's no signal.

(Line 183, 206)

https://github.com/prompt-toolkit/ptpython/blob/fb9bed1e5956ac5f109fd4cb401b3fae997efcd7/ptpython/repl.py#L164-L206

However as linked on TL;DR section of this SO answer it's almost impossibile to catch KeyboardInterrupt without singal handler by asyncio.Task side.

Looking at how asyncio/__main__.py handles the same work flawlessly - depending on structure of ptpython this might be plain impossible without overhaul, will need to look into it.

TL;DR: asyncio's REPL runs loop from non-async context, thus can resume loop after handling. Input handling is done by other thread, so this can be done.

https://github.com/python/cpython/blob/030b452e34bbb0096acacb70a31915b9590c8186/Lib/asyncio/__main__.py#L78-L137

https://github.com/python/cpython/blob/030b452e34bbb0096acacb70a31915b9590c8186/Lib/asyncio/__main__.py#L140-L195

https://github.com/python/cpython/blob/030b452e34bbb0096acacb70a31915b9590c8186/Lib/_pyrepl/simple_interact.py#L93-L171

But in ptpython seems like input handling is tied within async context. Would put some guard here and test if it help

https://github.com/prompt-toolkit/ptpython/blob/fb9bed1e5956ac5f109fd4cb401b3fae997efcd7/ptpython/entry_points/run_ptpython.py#L228-L231