A new Python REPL

[pablogsal]

This issue will track all the different steps to bootstrap a new Python REPL with many new features. The target of this issue is that we can reach a point where new contributions can be made easily to the REPL once is written in pure Python. This issue only coveres the initial ground work to reach this status.

Tasks to be done:

• [x] Redirect the parser and the main module to a new Python module.
• [x] Fallback to the previous tokenizer-based REPL if the terminal is not a tty (and for backwards compat reasons)
• [x] Bootstrap a new REPL module (based on a trimmed version pypy's pyrepl module).
• [x] Ensure that the basic features work:
  • [x] History
  • [x] Completions
  • [x] Ctrl-r and Ctrl-s

Linked PRs

• gh-111567
• gh-118616
• gh-118635
• gh-118638
• gh-118653
• gh-118700
• gh-118705
• gh-118712
• gh-119262
• gh-119274
• gh-119341
• gh-119348
• gh-119403
• gh-119405
• gh-119427
• gh-119432
• gh-119439
• gh-119559
• gh-119606
• gh-119833
• gh-119834
• gh-119848
• gh-119850
• gh-119924

[pablogsal]

If you want to help with this endeavour, please mention it here so we can coordinate everyone!

[pablogsal]

I have a initial version that we can start building upon, I will create a PR shortly.

[tomasr8]

I learned about this from the "core.py" podcast and it sounds like a great idea! I'd be happy help with this effort :)

[vstinner]

Fallback to the previous tokenizer-based REPL if the terminal is not a tty (and for backwards compat reasons)

Do you mean not using _pyrepl? Can _pyrepl be used without readline nor tty?

[pablogsal]

Can _pyrepl be used without readline nor tty?

It needs at least a tty yes. Readline can be lifted but I am not sure yet

[novaTopFlex]

I would like to see a new REPL (read, evaluate, print, loop) for the Python interpreter if the interpreter goes by a different name. Two other major Python interpreters also exist, and they already use other names ("bpython" and "ipython"). I would recommend forking the current REPL and renaming the interpreter with the new REPL to "zpython" instead. Also, I would like to see a built-in "clear()" function in the REPL as I already encounter with the "exit()" function and other REPL-specific functions.

[theLastOfCats]

Big feature without PEP?

[encukou]

Most buildbots are broken with:

Traceback (most recent call last):
  File ".../Lib/unittest/mock.py", line 1420, in patched
    return func(*newargs, **newkeywargs)
  File ".../Lib/test/test_pyrepl.py", line 643, in test_push_without_key_in_keymap
    eq = EventQueue(sys.stdout.fileno(), "utf-8")
                    ~~~~~~~~~~~~~~~~~^^
io.UnsupportedOperation: fileno

e.g.: https://buildbot.python.org/all/#/builders/725/builds/7939/steps/5/logs/stdio

[pablogsal]

Checking, will make a PR this morning

[lysnikolaou]

I already have something, will push in a bit.

[kulikjak]

Hi, I am seeing the following error:

======================================================================
FAIL: test_completion_with_many_options (test.test_pyrepl.TestPyReplCompleter.test_completion_with_many_options)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/..../cpython-main/Lib/test/test_pyrepl.py", line 586, in test_completion_with_many_options
    self.assertEqual(output, "os.O_ASYNC")
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'os.O_AS' != 'os.O_ASYNC'
- os.O_AS
+ os.O_ASYNC
?        +++

The reason is most likely that os.O_ASYNC is not necessarily available on all platforms (I am seeing this on Solaris where it's missing). I tried replacing it with O_AP and O_APPEND and then it works as expected.

[hauntsaninja]

(hmm there is only one constant starting with os.O_AS, so not sure where the "with_manyoptions" comes in. should it just be testing `os.O? if anythingtest_simple_completion` tests multiple options because of getenv and getenvb)

[pablogsal]

(hmm there is only one constant starting with os.O_AS, so not sure where the "with_manyoptions" comes in. should it just be testing `os.O? if anythingtest_simple_completion` tests multiple options because of getenv and getenvb)

Yes, but the "multiple options" comes from the fact that the test first checks what happens if you type os. and you press tab twice (which displays all options) and then you complete from one. Notice the sequence of characters is:

os.\t\tO_AP\t\n

[hauntsaninja]

• I think history is only separate when using help statement, not when calling help()
• When you use the help statement (but not when calling help()), there is an extra >>> when you exit
• Previously it was common for users to encounter "Type help(object) for help about object" message. I wonder if we should now mention this in the main help help
• Paste mode is awesome. Would be even more awesome if we can have it mostly work all the time, like IPython

[treyhunner]

• When you use the help statement (but not when calling help()), there is an extra >>> when you exit

Exiting history mode (hitting F2 and then q) also shows an extra >>>.

• Paste mode is awesome. Would be even more awesome if we can have it mostly work all the time, like IPython

IPython relies on prompt toolkit which relies on bracketed paste mode. Issue #84001 is about that.

Bracketed paste mode and Windows support (which is a much bigger ask than bracket paste mode I assume) would likely fully replace my tendency to explain pip and venv early for the sake of nudging brand new Python programmers toward IPython.

I am so excited to use this new REPL while teaching after 3.13's release. 👏

[cfbolz]

One behaviour I was pretty confused by right now is the "recursive" help mode. If you press F1 a bunch of times, it will start nested helps and each of them needs an exit/ctrl-d to leave. Is that intentional?

[ambv]

Keep the feedback coming, we'll be fixing those things for beta 2!

[pablogsal]

Bracketed paste coming right up: https://github.com/python/cpython/pull/118700

[barneygale]

Pressing Ctrl-R and then an arrow key trips an assertion for me. Logged as #118682, mentioning here for visibility.

[treyhunner]

Another issue, unfortunately caused by bracketed paste mode: pasting text is now about 1,000 times slower (by my estimate). Pasting the 441,033 character text of Frankenstein takes a couple seconds in the old REPL, but pasting just the first 7,628 characters took 39 seconds in the new REPL.

Hitting Ctrl+C while pasting also doesn't stop the pasting, so accidentally pasting a very large block is an unfortunate mistake to make currently.

[hugovk]

With the old REPL on macOS, sys.prefix.replace("\\", "\\\\") prints out a string. But the new one doesn't:

❯ PYTHON_BASIC_REPL=1 ./python.exe
Python 3.13.0a6+ (heads/main:fcf52d7cee, May  8 2024, 09:40:54) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.prefix
'/usr/local'
>>> sys.prefix.replace("\\", "\\\\")
'/usr/local'
>>> 

main on  main [$?⇡] via C v15.0.0-clang via 🐍 v3.12.3 via 💎 v3.1.3 took 36s
❯ ./python.exe
Python 3.13.0a6+ (heads/main:fcf52d7cee, May  8 2024, 09:40:54) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.prefix
'/usr/local'
>>> sys.prefix.replace("\\", "\\\\")
>>> 

[pablogsal]

With the old REPL on macOS, sys.prefix.replace("\\", "\\\\") prints out a string. But the new one doesn't:

❯ PYTHON_BASIC_REPL=1 ./python.exe
Python 3.13.0a6+ (heads/main:fcf52d7cee, May  8 2024, 09:40:54) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.prefix
'/usr/local'
>>> sys.prefix.replace("\\", "\\\\")
'/usr/local'
>>> 

main on  main [$?⇡] via C v15.0.0-clang via 🐍 v3.12.3 via 💎 v3.1.3 took 36s
❯ ./python.exe
Python 3.13.0a6+ (heads/main:fcf52d7cee, May  8 2024, 09:40:54) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.prefix
'/usr/local'
>>> sys.prefix.replace("\\", "\\\\")
>>> 

Hummm, are you using the latest main? It works for me:

main on  main [$?⇕] took 49s
❯ ./python.exe
Python 3.13.0a6+ (heads/main:c4f9823be27, May  8 2024, 10:02:25) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.prefix
'/usr/local'
>>> sys.prefix.replace("\\", "\\\\")
'/usr/local'

[pablogsal]

Oh, by any chance have you pasted the code into the REPL? Paste mode uses exec in some occasions so it may have a bug when it does not print the result

[hugovk]

Yes, I was on latest main and I did paste it in.

Yep, confirmed it reproduces when pasting, and works fine when typing or selecting from history.

[chgnrdv]

Since _pyrepl.__main__ is top-level now, its imports/definitions are exposed to user. Should we hide them?

Python 3.13.0a6+ (heads/main:c4f9823be2, May  8 2024, 21:42:56) [GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> dir()
['CAN_USE_PYREPL', '__annotations__', '__builtins__', '__cached__', '__doc__', '__file__',
'__loader__', '__name__', '__package__', '__spec__', 'interactive_console', 'os', 'sys']
>>>

[treyhunner]

When editing the last line in a function in the new REPL, hitting Enter doesn't end the block if the last line has indentation on it.

Demo of the bug, demo of what IPython does

[pablogsal]

When editing the last line in a function in the new REPL, hitting Enter doesn't end the block if the last line has indentation on it.

Demo of the bug, demo of what IPython does

We will look into it but that's by design I think. The block won't end unless you are in the last line at the start and you press enter. Otherwise it will think you are editing the block adding new lines

[sobolevn]

Now python and python -m asyncio works differently.

I want to give it a shot and fix it :)

[sobolevn]

Hm, right now I don't see any simple way of merging these two together: https://github.com/python/cpython/blob/c68acb1384a51eb745f572687eaf677371b9e765/Lib/asyncio/__main__.py#L15-L21

and

https://github.com/python/cpython/blob/c68acb1384a51eb745f572687eaf677371b9e765/Lib/_pyrepl/simple_interact.py#L81-L83

I think that this would require some API change. Not sure which one, though :( I would like pass it to someone else.

[AlexWaygood]

We've discovered that if you manually type out a condition that evaluates to False in the new REPL, False is printed as the result, as expected...

>>> (3, 13, 0, "final") < (3, 13, 0, "beta")
False

...but that if you copy and paste this condition into the REPL, nothing is printed (implying that the condition evaluates to None:

>>> (3, 13, 0, "final") < (3, 13, 0, "beta")
>>> 

Thanks @Eclips4 for realising that the difference in behaviour here was due to copying-and-pasting the condition rather than typing it out!

[Eclips4]

Additionally to Alex's message: There's another problem that probably has the same root cause:

>>> async with a: pass
  File "<python-input-2>", line 1
    async with a: pass
SyntaxError: invalid syntax

This was typed manually and raised an incorrect "version" of SyntaxError.

If I paste this (i.e async with a: pass) instead of typing it manually, the correct SyntaxError will be raised:

>>> async with a: pass
  File "<python-input-3>", line 1
SyntaxError: 'async with' outside async function
>>> 

[pablogsal]

We've discovered that if you manually type out a condition that evaluates to False in the new REPL, False is printed as the result, as expected...

>>> (3, 13, 0, "final") < (3, 13, 0, "beta")
False

...but that if you copy and paste this condition into the REPL, nothing is printed (implying that the condition evaluates to None:

>>> (3, 13, 0, "final") < (3, 13, 0, "beta")
>>> 

Thanks @Eclips4 for realising that the difference in behaviour here was due to copying-and-pasting the condition rather than typing it out!

This was already noted in https://github.com/python/cpython/issues/111201#issuecomment-2100108867

[pablogsal]

Maybe it's time to start opening individual issues for these things. @ambv @lysnikolaou thoughts?

[AlexWaygood]

This was already noted in #111201 (comment)

Ah, sorry for the noise :(

[pablogsal]

This was already noted in #111201 (comment)

Ah, sorry for the noise :(

No problem, but as we are only using one issue I am afraid that problems will be lost in the discussion ;)

[lysnikolaou]

Maybe it's time to start opening individual issues for these things. @ambv @lysnikolaou thoughts?

I agree. Opening new issues sounds better, but it'd be great if we could create a topic-repl label for easier grouping.

[cfbolz]

I agree. Opening new issues sounds better, but it'd be great if we could create a topic-repl label for easier grouping.

yes please, I'd like that too.

[hugovk]

Label requested at https://github.com/python/core-workflow/issues/536

[erlend-aasland]

Maybe it's time to start opening individual issues for these things. @ambv @lysnikolaou thoughts?

I agree. Opening new issues sounds better, but it'd be great if we could create a topic-repl label for easier grouping.

Consider also to use a GitHub Project.

[pablogsal]

Maybe it's time to start opening individual issues for these things. @ambv @lysnikolaou thoughts?

I agree. Opening new issues sounds better, but it'd be great if we could create a topic-repl label for easier grouping.

Consider also to use a GitHub Project.

What would be the advantage over a label?

[pablogsal]

Label created: https://github.com/python/cpython/labels/topic-repl

[pablogsal]

@treyhunner @AlexWaygood @chgnrdv @hugovk Do you mind creating individual issues and label them with the new label?

[pablogsal]

I'm closing this one for now, please lets continue with individual ones

[pablogsal]

Actually I am reopening until we land the tests PRs that we have pending

[erlend-aasland]

What would be the advantage over a label?

• you don't have to create an issue over at the core-workflow repo, you can just create the project
• you can add custom metadata (text fields, numeric fields, etc.) to further split the issues into groups

But a label is also fine.

[AlexWaygood]

• you don't have to create an issue over at the core-workflow repo, you can just create the project

Well, this is just our convention -- any core dev or triager can actually create a label by themselves at any time. We just choose to have a slightly more elaborate process around it :P

[AlexWaygood]

@treyhunner @AlexWaygood @chgnrdv @hugovk Do you mind creating individual issues and label them with the new label?

I created https://github.com/python/cpython/issues/118893

[treyhunner]

@treyhunner @AlexWaygood @chgnrdv @hugovk Do you mind creating individual issues and label them with the new label?

I created #118911