interactive mode TAB does not insert on OS X built with editline instead of GNU readline

[ned-deily]

BPO	9907
Nosy	@birkenfeld, @ronaldoussoren, @benjaminp, @ned-deily, @bitdancer, @zvezdan
Files	issue9907-py3k.patch issue9907-27.patch issue9907-py3k-ronald.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = 'https://github.com/ronaldoussoren' closed_at = created_at = labels = ['OS-mac', 'type-bug', 'release-blocker'] title = 'interactive mode TAB does not insert on OS X built with editline instead of GNU readline' updated_at = user = 'https://github.com/ned-deily' ``` bugs.python.org fields: ```python activity = actor = 'r.david.murray' assignee = 'ronaldoussoren' closed = True closed_date = closer = 'r.david.murray' components = ['macOS'] creation = creator = 'ned.deily' dependencies = [] files = ['18945', '18946', '19045'] hgrepos = [] issue_num = 9907 keywords = ['patch'] message_count = 8.0 messages = ['116981', '116986', '117042', '117288', '117529', '117991', '124243', '124272'] nosy_count = 6.0 nosy_names = ['georg.brandl', 'ronaldoussoren', 'benjamin.peterson', 'ned.deily', 'r.david.murray', 'zvezdan'] pr_nums = [] priority = 'release blocker' resolution = 'fixed' stage = 'resolved' status = 'closed' superseder = None type = 'behavior' url = 'https://bugs.python.org/issue9907' versions = ['Python 2.7', 'Python 3.2'] ```

[ned-deily]

When building Python on OS X, there is now support for linking Python with the readline compatibility interface of the OS X supplied BSD editline library rather than using the GNU readline library. Because of deficiencies in the version in earlier OS X releases, this support is only enabled for builds with a deployment target of 10.5 or higher. With the python 2.7 release, for the first time a python.org installer for OS X is available that uses this capability: the 10.5 and higher 32-bit/64-bit version. The 10.3 and higher 32-bit-only installer uses GNU readline as do previous installers. There is a behavior regression in the editline-linked versions: when started in interactive mode, the TAB key does not insert, rather it inserts a "./" file spec in the command buffer and a second TAB causes a completion search of files in the current directory.

With readline and typing \<TAB> \<CR>:

  $ unset PYTHONSTARTUP 
  $ python2.7
  Python 2.7 (r27:82508, Jul  3 2010, 20:17:05) 
  [GCC 4.0.1 (Apple Inc. build 5493)] on darwin
  Type "help", "copyright", "credits" or "license" for more information.
  >>>     
  ... 
  $ 

With editline and \<TAB> \<CR>:

  $ unset PYTHONSTARTUP 
  $ python2.7
  Python 2.7 (r27:82508, Jul  3 2010, 21:12:11) 
  [GCC 4.0.1 (Apple Inc. build 5493)] on darwin
  Type "help", "copyright", "credits" or "license" for more information.
  >>> ./
    File "<stdin>", line 1
      ./
      ^
  SyntaxError: invalid syntax
  >>> ^D

Two workarounds for python2.7 until the problem is addressed in a future installer:

(1) either install the 10.3 and up python 2.7

or (2) add or edit a python startup file for python2.7:

       $ cat > $HOME/.pystartup
       import readline
       if 'libedit' in readline.__doc__:
           readline.parse_and_bind("bind ^I ed-insert")
       ^D
       $ export PYTHONSTARTUP=$HOME/.pystartup

Since whitespace is significant in Python, Modules/readline.c initialization attempts to override TAB behavior by forcing TAB to "insert" by default (unless overridden by later readline module calls). Somehow that is failing when going through editline's readline compatibility layer.

[ned-deily]

[Thanks to Nik Krumm for reporting the problem on python-list/comp.lang.python]

[ned-deily]

The problem is due to a difference in the behavior of the rl_initialize function between the editline readline emulation and the real GNU libreadline. Modules/readline.c setup_readline calls several rl functions to create various default bindings, including overriding TAB to "insert" rather than to "trigger completion", then calls rl_initialize allowing the users defaults from .inputrc to override. It seems the emulated rl_initialize causes all the modified bindings to be discarded, causing TAB to revert to its default "trigger file completion". The solution in the attached patches is to conditionally call rl_initialize at the beginning of setup_readline, rather than at the end, if the editline emulation is in use. Patches supplied for py3k and 27 (but not 31 since the feature was never backported there even though it was to 26). I did not supply any additional tests since I can't think of a straightforward way to simulate the condition in the test framework; suggestions welcome.

[ronaldoussoren]

Patch looks fine and should IMO be applied

[ronaldoussoren]

On second thought, the patch isn't quite as harmless as I first thought: the default key-bindings that are created after the call to rl_initialize will replace custom bindings in the users .editrc file.

I've attached a new version of the py3k patch that works around this problem by calling rl_read_init_file(NULL) after setting the default bindings.

This allows me to override the bindings for TAB in \~/.editrc as before.

[ned-deily]

The modified patch looks OK to me and tests OK. The rl_read_init_file call seems like a reasonable thing for users who are used to using libedit's .editrc. As a practical matter, though, I think the only thing that would be affected is an .editrc TAB binding. Some of the initializations done in Modules/readline.c, like rl_bind_key_in_map (for sure) and rl_completer_word_break_characters are silently ignored by the libedit readline-compatibility layer; it does not implement features like the emacs_meta_keymap.

[ned-deily]

I believe this fix should go into 3.2 (and 2.7) as it has been reported by a number of people in various places and the fix risk is low.

[bitdancer]

Committed to py3k in 87356 and 2.7 in r87358.