search

astral-sh / rye

a Hassle-Free Python Experience
https://rye.astral.sh
MIT License
13.79k stars 467 forks source link

Automatically scan for terminfo files if env var TERMINFO is blank #250

Open Woovie opened 1 year ago

Woovie commented 1 year ago

Steps to Reproduce

In my Ubuntu environments, TERMINFO is blank as an environment variable. I've tested this in Ubuntu 20.04 natively running on a computer, as well as in WSL with Ubuntu 20.04. Setting your TERMINFO env var to nothing should replicate the environment.

Expected Result

Python versions included with Ubuntu work as expected:

% /usr/bin/python3
Python 3.8.10 (default, Nov 14 2022, 12:59:47)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import curses
>>> curses.initscr()
<_curses.window object at 0x7f072692add0>
                                         >>>

Actual Result

~
2 % rye init my_project && cd my_project
success: Initialized project in /home/jbanasik/my_project
  Run `rye sync` to get started

~/my_project on master [+]
% rye sync
Initializing new virtualenv in /home/jbanasik/my_project/.venv
Python version: cpython@3.11.3
Generating production lockfile: /home/jbanasik/my_project/requirements.lock
Generating dev lockfile: /home/jbanasik/my_project/requirements-dev.lock
Installing dependencies
Looking in indexes: https://pypi.org/simple/
Obtaining file:///. (from -r /tmp/tmptnk3ysno (line 1))
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Building wheels for collected packages: my-project
  Building editable for my-project (pyproject.toml) ... done
  Created wheel for my-project: filename=my_project-0.1.0-py3-none-any.whl size=1051 sha256=ebb046f48d026f966598b83b7993d4762d1db7bee381c002ee43211e00d1fbec
  Stored in directory: /tmp/pip-ephem-wheel-cache-2iul7ln0/wheels/97/54/f5/d849319cdfa096e074df352654ee2e7c919da8951f090690c6
Successfully built my-project
Installing collected packages: my-project
Successfully installed my-project-0.1.0
Done!

~/my_project on master [+] took 4s
% python
Cannot read termcap database;
using dumb terminal settings.
Python 3.11.3 (main, May  7 2023, 19:33:53) [Clang 16.0.3 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>import curses
>>> curses.initscr()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jbanasik/.rye/py/cpython@3.11.3/install/lib/python3.11/curses/__init__.py", line 29, in initscr
    setupterm(term=_os.environ.get("TERM", "unknown"),
_curses.error: setupterm: could not find terminfo database
>>> import os
>>> os.environ.get("TERM", "unknown")
'xterm-256color'

Version Info

% rye --version
rye 0.4.0
commit: 0.4.0 (cdc5c37bc 2023-05-29)
platform: linux (x86_64)
self-python: cpython@3.10
symlink support: true

Stacktrace

No response

Woovie commented 1 year ago

The solution we came to on Discord was forcing my TERMINFO:

~/my_project on master [+] took 2s
% TERMINFO=/lib/terminfo python
Python 3.11.3 (main, May  7 2023, 19:33:53) [Clang 16.0.3 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

I feel that perhaps this should be shimmed for the user. I checked other binaries via strace and other tools, they seem to scan /lib/terminfo by default. Can this path be added as part of compilation of Python? I am unfamiliar, just guessing here.

Thanks again for rye, it's been an amazing tool to use.

mitsuhiko commented 1 year ago

It does sound like something that needs improving but i would need to further understand how it currently works and it should work. I would have expected the default logic to kick in if that variable is empty but apparently it's not. Would be interesting to check if unsetting TERMINFO makes it works or causes the same problem.

bluss commented 1 year ago

@Woovie see also indygreg's docs on this https://gregoryszorc.com/docs/python-build-standalone/main/quirks.html#backspace-key-doesn-t-work-in-python-repl

Woovie commented 1 year ago

I've gone over that document and as well dug into some of the terminfo and ncurses manpages over the past few days. I think this particular section of terminfo's man page is most relevant here.

Analyzing an strace on /usr/bin/python3 provided by Ubuntu on version 20.04, we see the following:

13:13:33 (+     0.000072) openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libtinfo.so.6", O_RDONLY|O_CLOEXEC) = 3</usr/lib/x86_64-linux-gnu/libtinfo.so.6.2>
13:13:33 (+     0.000071) read(3</usr/lib/x86_64-linux-gnu/libtinfo.so.6.2>, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240\346\0\0\0\0\0\0"..., 832) = 832
13:13:33 (+     0.000067) fstat(3</usr/lib/x86_64-linux-gnu/libtinfo.so.6.2>, {st_dev=makedev(0x8, 0x20), st_ino=11539, st_mode=S_IFREG|0644, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=376, st_size=192032, st_atime=1685502389 /* 2023-05-30T20:06:29.370679002-0700 */, st_atime_nsec=370679002, st_mtime=1582701266 /* 2020-02-25T23:14:26-0800 */, st_mtime_nsec=0, st_ctime=1654881887 /* 2022-06-10T10:24:47.265000347-0700 */, st_ctime_nsec=265000347}) = 0
13:13:33 (+     0.000079) mmap(NULL, 194944, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3</usr/lib/x86_64-linux-gnu/libtinfo.so.6.2>, 0) = 0x7ff6bde28000
13:13:33 (+     0.000084) mmap(0x7ff6bde36000, 61440, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3</usr/lib/x86_64-linux-gnu/libtinfo.so.6.2>, 0xe000) = 0x7ff6bde36000
13:13:33 (+     0.000079) mmap(0x7ff6bde45000, 57344, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3</usr/lib/x86_64-linux-gnu/libtinfo.so.6.2>, 0x1d000) = 0x7ff6bde45000
13:13:33 (+     0.000075) mmap(0x7ff6bde53000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3</usr/lib/x86_64-linux-gnu/libtinfo.so.6.2>, 0x2a000) = 0x7ff6bde53000
13:13:33 (+     0.000089) close(3</usr/lib/x86_64-linux-gnu/libtinfo.so.6.2>) = 0
13:13:33 (+     0.000131) mprotect(0x7ff6bde53000, 16384, PROT_READ) = 0
13:13:33 (+     0.000184) mprotect(0x7ff6bde9f000, 8192, PROT_READ) = 0
13:13:33 (+     0.000141) mprotect(0x7ff6bdebe000, 4096, PROT_READ) = 0
13:13:33 (+     0.000112) munmap(0x7ff6bdea8000, 50790) = 0
13:13:33 (+     0.000134) rt_sigaction(SIGWINCH, {sa_handler=0x7ff6bdeb8a40, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7ff6be60c090}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
13:13:33 (+     0.000073) ioctl(1<pipe:[1451618]>, TCGETS, 0x7ffefbb946f0) = -1 ENOTTY (Inappropriate ioctl for device)
13:13:33 (+     0.000073) ioctl(1<pipe:[1451618]>, TCGETS, 0x7ffefbb945c0) = -1 ENOTTY (Inappropriate ioctl for device)
13:13:33 (+     0.000072) stat("/home/jbanasik/.terminfo", 0x23a0140) = -1 ENOENT (No such file or directory)
13:13:33 (+     0.000067) stat("/etc/terminfo", {st_dev=makedev(0x8, 0x20), st_ino=1472, st_mode=S_IFDIR|0755, st_nlink=2, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=8, st_size=4096, st_atime=1684437320 /* 2023-05-18T12:15:20.170654026-0700 */, st_atime_nsec=170654026, st_mtime=1587624027 /* 2020-04-22T23:40:27.053345494-0700 */, st_mtime_nsec=53345494, st_ctime=1654881888 /* 2022-06-10T10:24:48.905000355-0700 */, st_ctime_nsec=905000355}) = 0
13:13:33 (+     0.000085) stat("/lib/terminfo", {st_dev=makedev(0x8, 0x20), st_ino=10431, st_mode=S_IFDIR|0755, st_nlink=16, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=8, st_size=4096, st_atime=1684437436 /* 2023-05-18T12:17:16.672621255-0700 */, st_atime_nsec=672621255, st_mtime=1582701266 /* 2020-02-25T23:14:26-0800 */, st_mtime_nsec=0, st_ctime=1654881888 /* 2022-06-10T10:24:48.905000355-0700 */, st_ctime_nsec=905000355}) = 0
13:13:33 (+     0.000103) stat("/usr/share/terminfo", {st_dev=makedev(0x8, 0x20), st_ino=30125, st_mode=S_IFDIR|0755, st_nlink=44, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=8, st_size=4096, st_atime=1684436982 /* 2023-05-18T12:09:42.737833221-0700 */, st_atime_nsec=737833221, st_mtime=1587624120 /* 2020-04-22T23:42:00.132104361-0700 */, st_mtime_nsec=132104361, st_ctime=1654881888 /* 2022-06-10T10:24:48.895000355-0700 */, st_ctime_nsec=895000355}) = 0
13:13:33 (+     0.000105) access("/etc/terminfo/x/xterm-256color", R_OK) = -1 ENOENT (No such file or directory)
13:13:33 (+     0.000090) access("/lib/terminfo/x/xterm-256color", R_OK) = 0
13:13:33 (+     0.000102) openat(AT_FDCWD, "/lib/terminfo/x/xterm-256color", O_RDONLY) = 3</usr/lib/terminfo/x/xterm-256color>
13:13:33 (+     0.000102) fstat(3</usr/lib/terminfo/x/xterm-256color>, {st_dev=makedev(0x8, 0x20), st_ino=10483, st_mode=S_IFREG|0644, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=8, st_size=3503, st_atime=1685502389 /* 2023-05-30T20:06:29.370679002-0700 */, st_atime_nsec=370679002, st_mtime=1582701266 /* 2020-02-25T23:14:26-0800 */, st_mtime_nsec=0, st_ctime=1654881886 /* 2022-06-10T10:24:46.505000343-0700 */, st_ctime_nsec=505000343}) = 0
13:13:33 (+     0.000099) read(3</usr/lib/terminfo/x/xterm-256color>, "\36\2%\0&\0\17\0\235\1\356\5xterm-256color|xterm"..., 32768) = 3503
13:13:33 (+     0.000098) read(3</usr/lib/terminfo/x/xterm-256color>, "", 28672) = 0
13:13:33 (+     0.000104) close(3</usr/lib/terminfo/x/xterm-256color>) = 0

Unfortunately, standalone Python builds seem to do their own thing.

I am guessing here that Ubuntu's included ncurses library is built correctly for the environment and therefore checks these paths. Doing an equivalent strace on these python versions, the only file I see checked at all is $HOME/.terminfo which is by default not populated. Same goes for $TERMINFO and $TERMINFO_DIRS.

I am okay with manually populating one of these in my .*rc or some other solution. At minimum, we should definitely have this documented for other users. I think potential solutions could be:

Of course, I am open to other ideas as well. Anything to help others avoid this error.

mitsuhiko commented 1 year ago

So for sure the current state is not great. I also do not quite like that readline is not being used but libedit is, but that's a licensing issue for the standalone builds that's harder to get rid off. So at least some of those problems might be indirectly be caused by the difference choice of line edit library.

I wonder if there is a way to reconfigure where it looks for the terminfo files without setting environment variables. Because if there is, then I would probably patch into that automatically for the user.