Unable to build for Python 3.11 on macOS

[milosivanovic]

Thanks a lot for making this package. Given that Homebrew might be removing GNU readline for good (as discussed here https://github.com/ludwigschwardt/python-gnureadline/issues/61), python-gnureadline seemed exactly like what I needed to hopefully get readline back into the brew-provided Python 3.11. I know that the README says it's only been tested up to Python 3.10, but I'm not sure if that's the reason for the compilation failure I'm getting.

From...

If you want to use this module as a drop-in replacement for readline in the standard Python shell, it has to be installed with the less polite easy_install script found in setuptools. Please take note that easy_install has been deprecated for a while and is about to be dropped from setuptools. Proceed at your own risk!

...I inferred that I should clone this repo and run python3.11 setup.py install since I want readline working in the Python REPL. Did I understand that correctly?

After running that command, the build process seemed to progress nicely until it reached this part:

============ Building the readline extension module ============

running install
running bdist_egg
running egg_info
creating gnureadline.egg-info
writing gnureadline.egg-info/PKG-INFO
writing dependency_links to gnureadline.egg-info/dependency_links.txt
writing top-level names to gnureadline.egg-info/top_level.txt
writing manifest file 'gnureadline.egg-info/SOURCES.txt'
reading manifest file 'gnureadline.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
writing manifest file 'gnureadline.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-12-arm64/egg
running install_lib
running build_py
creating build
creating build/lib.macosx-12-arm64-cpython-311
copying readline.py -> build/lib.macosx-12-arm64-cpython-311
warning: build_py: byte-compiling is disabled, skipping.

running build_ext
building 'gnureadline' extension
creating build/temp.macosx-12-arm64-cpython-311
creating build/temp.macosx-12-arm64-cpython-311/Modules
creating build/temp.macosx-12-arm64-cpython-311/Modules/3.x
clang -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -DHAVE_RL_APPEND_HISTORY -DHAVE_RL_CALLBACK -DHAVE_RL_CATCH_SIGNAL -DHAVE_RL_COMPLETION_APPEND_CHARACTER -DHAVE_RL_COMPLETION_DISPLAY_MATCHES_HOOK -DHAVE_RL_COMPLETION_MATCHES -DHAVE_RL_COMPLETION_SUPPRESS_APPEND -DHAVE_RL_PRE_INPUT_HOOK -DHAVE_RL_RESIZE_TERMINAL -I. -IModules/3.x -I/opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11/include/python3.11 -c Modules/3.x/readline.c -o build/temp.macosx-12-arm64-cpython-311/Modules/3.x/readline.o
Modules/3.x/readline.c:347:19: error: implicit declaration of function 'append_history' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
    errno = err = append_history(
                  ^
Modules/3.x/readline.c:1160:5: error: use of undeclared identifier 'rl_completion_suppress_append'
    rl_completion_suppress_append = 0;
    ^
Modules/3.x/readline.c:1317:5: error: use of undeclared identifier 'rl_catch_signals'
    rl_catch_signals = 0;
    ^
Modules/3.x/readline.c:1340:17: error: implicit declaration of function 'rl_resize_terminal' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
                rl_resize_terminal();
                ^
Modules/3.x/readline.c:1340:17: note: did you mean 'rl_reset_terminal'?
/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/include/editline/readline.h:181:8: note: 'rl_reset_terminal' declared here
void             rl_reset_terminal(const char *);
                 ^
4 errors generated.
error: command '/usr/bin/clang' failed with exit code 1

It looks like the first error is that append_history is missing, but I'm not sure where that's supposed to be defined. Could you let me know if this is just a 3.11 incompatibility that has yet to be fixed, or if I'm building it wrong?

[ludwigschwardt]

Thanks for reporting this! I’m also on Homebrew so you have got my attention 🙂 I’ll have some time in the next few days to try a 3.11 version of the readline module.

[milosivanovic]

@ludwigschwardt kindly following up - will it be possible to use this in Python 3.11 and later?

[ludwigschwardt]

Whoops Milos, I lost track of this - sincere apologies!

It's not a Python 3.11 issue per se... I already have 3.11 wheels on PyPI since October 2022 (#59). (Just checking: have you tried pip install gnureadline?)

I'm currently on Homebrew but only for non-Python libraries. I use pyenv to provide Python 3.11. The package also compiles fine on that - but then pyenv has readline...

I think the main compilation issue is that you don't have any readline headers around anymore. Try adding

CFLAGS=-Irl/readline-lib python3.11 setup.py install

to use the local readline headers inside the package.

In the meantime I'll fish out my old laptop that still has brew python and see if there is a better solution.

[milosivanovic]

@ludwigschwardt thanks a lot for responding. My use case is that I just want to have it in the shell so I can use CTRL+R and other readline features on macOS (as would be normal on Linux installs) since brew Python 3.11 removed readline support going forward, meaning Python 3.10 was the last version to be compiled against readline.

I tried pip3 install gnureadline but it didn't seem to have any effect on the shell. In the README I see the following:

If you want to use this module as a drop-in replacement for readline in the standard Python shell, it has to be installed with the less polite easy_install script found in [setuptools](https://pypi.python.org/pypi/setuptools). Please take note that easy_install has been deprecated for a while and is about to be dropped from setuptools. Proceed at your own risk!

I believe that's what I tried to do last time, which failed with the error in my original comment.

I just tried running CFLAGS=-Irl/readline-lib python3.11 setup.py install in the python-gnureadline directory but I got the same set of errors as before:

============ Building the readline extension module ============

running install
running bdist_egg
running egg_info
writing gnureadline.egg-info/PKG-INFO
writing dependency_links to gnureadline.egg-info/dependency_links.txt
writing top-level names to gnureadline.egg-info/top_level.txt
reading manifest file 'gnureadline.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
writing manifest file 'gnureadline.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-12-arm64/egg
running install_lib
running build_py
creating build
creating build/lib.macosx-12-arm64-cpython-311
copying readline.py -> build/lib.macosx-12-arm64-cpython-311
warning: build_py: byte-compiling is disabled, skipping.

running build_ext
building 'gnureadline' extension
creating build/temp.macosx-12-arm64-cpython-311
creating build/temp.macosx-12-arm64-cpython-311/Modules
creating build/temp.macosx-12-arm64-cpython-311/Modules/3.x
clang -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -Irl/readline-lib -DHAVE_RL_APPEND_HISTORY -DHAVE_RL_CALLBACK -DHAVE_RL_CATCH_SIGNAL -DHAVE_RL_COMPLETION_APPEND_CHARACTER -DHAVE_RL_COMPLETION_DISPLAY_MATCHES_HOOK -DHAVE_RL_COMPLETION_MATCHES -DHAVE_RL_COMPLETION_SUPPRESS_APPEND -DHAVE_RL_PRE_INPUT_HOOK -DHAVE_RL_RESIZE_TERMINAL -I. -IModules/3.x -I/opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11/include/python3.11 -c Modules/3.x/readline.c -o build/temp.macosx-12-arm64-cpython-311/Modules/3.x/readline.o
Modules/3.x/readline.c:347:19: error: implicit declaration of function 'append_history' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
    errno = err = append_history(
                  ^
Modules/3.x/readline.c:1160:5: error: use of undeclared identifier 'rl_completion_suppress_append'
    rl_completion_suppress_append = 0;
    ^
Modules/3.x/readline.c:1317:5: error: use of undeclared identifier 'rl_catch_signals'
    rl_catch_signals = 0;
    ^
Modules/3.x/readline.c:1340:17: error: implicit declaration of function 'rl_resize_terminal' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
                rl_resize_terminal();
                ^
Modules/3.x/readline.c:1340:17: note: did you mean 'rl_reset_terminal'?
/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/include/editline/readline.h:181:8: note: 'rl_reset_terminal' declared here
void             rl_reset_terminal(const char *);
                 ^
4 errors generated.
error: command '/usr/bin/clang' failed with exit code 1

Appreciate any further help.

[ludwigschwardt]

I tried pip3 install gnureadline but it didn't seem to have any effect on the shell.

Yes, it plays "nicely" with your Python installation and requires you to call it explicitly (unlike readline with its behind-the-scenes magic). This use case was originally developed for IPython, but they have since moved on to prompt_toolkit. I only ever use IPython to get readline features like Ctrl+R, which is why I've missed this issue so far :-)

We used to have a direct readline replacement package (still on PyPI at version 6.2.4) that could be installed via

easy_install readline

to do exactly what you require. This is what I meant by "easy_install" installation. It's different from python setup.py install. You shouldn't even be doing that these days 😅

The problem is that easy_install is not so easy anymore (ahem). It's still available but with dire warnings and discouragement. The command-line script is not installed anymore. You could try your luck with ez_setup to get the easy_install script - also deprecated. I couldn't resurrect it yet on my side.

On your build issue: the include directory flag should have been -I., but I see your clang line already has that... 🤔 Do you have a readline symlink in the same directory as setup.py?

The bigger challenge is to get the gnureadline package to replace the original readline module. Dirty tricks include easy_install, the easy-install.pth file, eggs (as opposed to wheels) and sys.modules['readline'] fiddling.

[ludwigschwardt]

For myself, I use pyenv and IPython for tab completion, for what it's worth.

[ludwigschwardt]

I have a new scheme to try out... 🙂

1. Install gnureadline by pip install gnureadline.
2. Create a config file like ~/.pythonrc:

   import sys
   try:
   import gnureadline as readline
   except ImportError:
   import readline
   sys.modules['readline'] = readline
   del readline
   del sys

3. Add the following to your ~/.profile:

   export PYTHONSTARTUP=~/.pythonrc

These commands will run at the start of every interactive Python session, which is when you need tab completion etc.

It also works in virtualenvs, if you remember to install gnureadline again...

Let me know if the resulting keystrokes are what you expect.

[ludwigschwardt]

Mmm, I see now that PYTHONSTARTUP is too late to the party. It happens right at the end of the startup sequence. The second-last step is to import rlcompleter, which pulls in the original readline. Presumably cmd also happens somewhere around there (see #61). See for yourself if you run python3 -v.

There are two earlier opportunities to tweak things: sitecustomize followed by usercustomize. The former is created and used by Homebrew, which makes it harder to fiddle with.

I had good success with usercustomize. Try the following:

USER_SITE=$(python3 -m site --user-site)  # just a temp variable - no need to set in profile
mkdir -p $USER_SITE
cp ~/.pythonrc $USER_SITE/usercustomize.py

This also works in virtualenvs 🙂

[milosivanovic]

@ludwigschwardt Wow, thank you so much! That works perfectly!

https://github.com/ludwigschwardt/python-gnureadline/assets/1274505/1b54a2f6-352a-429c-9105-9b26547c61e8

[keeely]

Whilst I appreciate you looking into this, I'll stick with the sys.modules['readline'] fiddling as it has worked so far and my users don't have to do anything (really important for me).

[milosivanovic]

For anyone else wanting this solution, here's a single command to copy-paste to set up the gnureadline injection:

mkdir -p $(python3 -m site --user-site) && cat << 'EOF' > $(python3 -m site --user-site)/usercustomize.py
import sys
try:
    import gnureadline as readline
except ImportError:
    import readline
sys.modules['readline'] = readline
del readline
del sys
EOF

After that, just install gnureadline if it isn't already: pip3 install gnureadline

Thanks again @ludwigschwardt!

[ludwigschwardt]

I'm very glad it worked for you, Milos! I know PYTHONSTARTUP but never used usercustomize before. I wish I knew about it sooner 😁

The del readline and del sys steps are now superfluous. Previously I was worried about polluting the namespace on startup but it looks like usercustomize cleans up afterwards so we can leave that out.

@keeely, glad you could sort out your users too. Your solution makes sense given the constraints. Do you import cmd in your own app or do your users interact with the usual Python REPL? I'm wondering if the meta_path trick can override rlcompleter too to fix the standard Python shell.

[ludwigschwardt]

Here is an example of a Python script that updates usercustomize.py in a reasonably safe way. I'm considering including such a script with gnureadline but I'm unsure if packages are "allowed" to modify user site-packages like this 🤔 😅

[onlynone]

For me, if all I want is my python interactive shell to use gnureadline, the following in ~/.pythonrc works fine:

import atexit
import os
import rlcompleter

try:
    # If we have gnureadline use it, and save history to ~/.python_history
    import gnureadline as readline
    history_path = "~/.python_history"
    use_gnureadline = True
except ImportError:
    # If we don't have gnureadline, import the regular readline and check if
    # it is gnu readline or bsd libedit
    import readline
    use_gnureadline = False

    if readline.__doc__.find("libedit") > 0:
        # We're probably on OS X (or maybe another BSD), the history file for
        # libedit is different from readline, so save it to another file to
        # prevent confusion
        history_path = "~/.python_history_libedit"
    else:
        # We're probably on linux, and readline is gnu readline, use the regular
        # history file
        history_path = "~/.python_history"

history_path = os.path.expanduser(history_path)

if os.path.exists(history_path):
    readline.read_history_file(history_path)

# Set the completer so we get tab completion
readline.set_completer(rlcompleter.Completer().complete)
readline.parse_and_bind('tab: complete')

# The function to run at exit to save the history
def save_history(history_path, use_gnureadline):
    if use_gnureadline:
        import gnureadline as readline
    else:
        import readline

    readline.write_history_file(history_path)

atexit.register(save_history, history_path, use_gnureadline)

# Remove all the vars from global namespace
del os, atexit, readline, rlcompleter, save_history, history_path, use_gnureadline

It's a little more complicated than the bare minimum because:

1. I use a different history file if the underlying library is libedit vs read readline because the file format used by libedit is (or at least was) incompatible with that used by readline. This saves me when I happen to use a python installation that doesn't use real readline (like when homebrew made the change to use libedit) so that my python history file doesn't get corrupted.
2. I add in tab completion

I don't have to mess with sys.modules or use usercustomize.py. I'm curious if this works for others too, or if those other hacks are necessary.

[ludwigschwardt]

Hi @onlynone, I had a good look at your PYTHONSTARTUP approach again. It makes sense what you are doing. The advantages of PYTHONSTARTUP over user/site customization are:

• it only affects interactive sessions in the interpreter (could be a disadvantage too, depending on use cases), and
• it fixes all "user" versions of Python once and for all (as long as you install gnureadline as needed).

I've just made a PR (#72) describing my latest approach to the problem.

I'm curious what the PYTHONSTARTUP approach does about the sys.__interactivehook__ that will be called shortly afterwards. Does it have any effect? This hook basically does the following internally:

import readline   # The old libedit readline returns...
readline.parse_and_bind('bind ^I rl_complete')

but it doesn't repeat this line:

readline.set_completer(rlcompleter.Completer().complete)

Is that sufficient to make things work out OK?

[onlynone]

I'm curious what the PYTHONSTARTUP approach does about the sys.__interactivehook__ that will be called shortly afterwards. Does it have any effect? This hook basically does the following internally:

import readline   # The old libedit readline returns...
readline.parse_and_bind('bind ^I rl_complete')

but it doesn't repeat this line:

readline.set_completer(rlcompleter.Completer().complete)

Is that sufficient to make things work out OK?

I was probably just copy-pasting stuff I found elsewhere when I came up with that. Looking at the rlcompleter docs now I see:

When this module is imported on a Unix platform with the readline module available, an instance of the Completer class is automatically created and its complete() method is set as the readline completer. The method provides completion of valid Python identifiers and keywords.

So maybe making the explicit call to set_completer is unnecessary. Although in my startup script, I import rlcompleter first before importing/checking for gnureadline. So if the built-in readline is libedit, then I guess that library would be setup with rlcompleter, but not gnureadline. So maybe you'd only need:

readline.set_completer(rlcompleter.Completer().complete)

inside the block that does:

try:
    # If we have gnureadline use it, and save history to ~/.python_history
    import gnureadline as readline
    history_path = "~/.python_history"
    use_gnureadline = True

[ludwigschwardt]

Hi Steven, I've now verified for myself that your .pythonrc works as expected. I checked it by replacing the line

readline.parse_and_bind('tab: complete')

with silly completion code customised for each library:

if "libedit" in readline.__doc__:
    readline.parse_and_bind("bind e rl_complete")
else:
    readline.parse_and_bind("r: complete")

So if thee key does tab completion, you are definitely using libedit, while tab completion via the r key confirms the presence of GNU readline. 😁

I've finally released 8.2.10 to PyPI now - thanks for your patience!

[ludwigschwardt]

This should be sorted with the 8.2.10 release.

[milosivanovic]

@ludwigschwardt It seems that in Python 3.13, gnureadline may have been replaced with a similar but not fully identical in-house implementation -- or, at least that's my initial assumption. I created a bug here to track: https://github.com/python/cpython/issues/125924

I wonder what you think of this. The import gnureadline trick with usercustomize/sitecustomize does not appear to work anymore. Do you see the same behaviour?

[ludwigschwardt]

I suspect it's part of the grand rewrite of the Python REPL in 3.13... I also see (r-search')` with both standard Python 3.13 and when installing gnureadline 8.2.13.

The funny thing is that I cannot find the code that emits that string. GNU Readline constructs the reverse-i-search prompt internally but has no r-search that I can see. And the Python 3.13 readline.c module still calls libreadline as before. I haven't found r-search in the cpython repo yet (although GitHub's so-called "exact" search doesn't help).

As far as I can tell the gnureadline import still works (looking at readline.__file__, for example, after the fix), but I might be missing something...

[ludwigschwardt]

Aha, it is the new 3.13 REPL. You can disable it like this to get the old behaviour (for now... 😅):

PYTHON_BASIC_REPL=1 python

Here is the code that emits that r-search prompt.

I suspect you are going to have a hard time getting Python folks to revert one of their shiny new features two weeks after release 😂

[ludwigschwardt]

This comment seems to suggest that the PYTHON_BASIC_REPL override will be here for a while.

[ludwigschwardt]

Hi @milosivanovic, I've opened #77 to mention this in the docs. Could you have a look at it please? Unfortunately I can't add you as a reviewer, it seems...