python3.10+: global pip install layout does not match debian's due to sysconfig changes

abersheeran commented 2 years ago

description

Why /usr/lib/python3.10/site-packages not in sys.path?

I use the following code to install python3.10 in Ubuntu18.04.

apt-get update
apt-get install -y software-properties-common
add-apt-repository ppa:deadsnakes/ppa
apt-get update
apt-get install -y python3.10 python3.10-distutils

And then run python3.10 -c 'import sys; print(sys.path)', I got:

['', '/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '/usr/local/lib/python3.10/dist-packages', '/usr/lib/python3/dist-packages']

But if run python3.10 get-pip.py --disable-pip-version-check to install pip, pip would be in /usr/lib/python3.10/site-packages.

os information

lsb_release -a

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.6 LTS
Release:        18.04
Codename:       bionic

uname -a

Linux fc5a25148e85 5.4.72-microsoft-standard-WSL2 #1 SMP Wed Oct 28 23:40:43 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

asottile commented 2 years ago

the paths are ~a little non standard because we follow what debian chooses to do there (dist-packages instead of site-packages, and usr/local for user-installed things)

as for the pip install location, that looks a little bit broken -- it should end up in /usr/local/lib/python3.10/dist-packages

abersheeran commented 2 years ago

Use python3.10 get-pip.py --user can solve it.

asottile commented 2 years ago

please don't close this, there's a bug that needs fixing

asottile commented 2 years ago

@uranusjr -- debian's patch is currently insufficient, I'm working with them to figure out the best way to actually solve this since it seems patching sysconfig is insufficient:

$ docker run --rm -ti debian:testing bash -euxc 'apt-get update >& /dev/null && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends python3.10 python3-distutils curl ca-certificates >& /dev/null && dpkg -l | grep python3.10 && curl --location --silent https://bootstrap.pypa.io/get-pip.py | python3.10 - && ls /usr/local/bin && ls /usr/lib/python3.10/site-packages && python3.10 -m pip --help'
Resolved "debian" as an alias (/etc/containers/registries.conf.d/shortnames.conf)
Trying to pull docker.io/library/debian:testing...
Getting image source signatures
Copying blob e3875f34eeed done  
Copying config 9ef6d00d6c done  
Writing manifest to image destination
Storing signatures
+ apt-get update
+ DEBIAN_FRONTEND=noninteractive
+ apt-get install -y --no-install-recommends python3.10 python3-distutils curl ca-certificates
+ dpkg -l
+ grep python3.10
ii  libpython3.10-minimal:amd64 3.10.1-1                          amd64        Minimal subset of the Python language (version 3.10)
ii  libpython3.10-stdlib:amd64  3.10.1-1                          amd64        Interactive high-level object-oriented language (standard library, version 3.10)
ii  python3.10                  3.10.1-1                          amd64        Interactive high-level object-oriented language (version 3.10)
ii  python3.10-minimal          3.10.1-1                          amd64        Minimal subset of the Python language (version 3.10)
+ curl --location --silent https://bootstrap.pypa.io/get-pip.py
+ python3.10 -
Collecting pip
  Downloading pip-21.3.1-py3-none-any.whl (1.7 MB)
     |████████████████████████████████| 1.7 MB 4.5 MB/s            
Collecting setuptools
  Downloading setuptools-59.5.0-py3-none-any.whl (952 kB)
     |████████████████████████████████| 952 kB 50.5 MB/s            
Collecting wheel
  Downloading wheel-0.37.0-py2.py3-none-any.whl (35 kB)
Installing collected packages: wheel, setuptools, pip
Successfully installed pip-21.3.1 setuptools-59.5.0 wheel-0.37.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
+ ls /usr/local/bin
+ ls /usr/lib/python3.10/site-packages
_distutils_hack       pip-21.3.1.dist-info  setuptools-59.5.0.dist-info
distutils-precedence.pth  pkg_resources     wheel
pip           setuptools        wheel-0.37.0.dist-info
+ python3.10 -m pip --help
/usr/bin/python3.10: No module named pip

note that we do patch sysconfig (similar to debian) but it doesn't seem to be enough for pip:

uranusjr commented 2 years ago

You need to patch Lib/sysconfig.py, not Lib/distutils/sysconfig.py (although the latter probably also needs to be patched for other tools that have not migrate away from distutils to continue working). Your (latter) patch adds two entries under _INSTALL_SCHEMES, but I think you also need some logic to actually choose posix_local when appropriate. See lines 47 to 71 in this patch. You probably don’t need deb_system; from what I know it’s used to build .deb packages for Python libs, which deadsnakes don’t distribute.

asottile commented 2 years ago

yes, but even that patch is insufficient as shown above (debian:testing has the latest packaged and patched version of debian's python3.10) -- deb_system can still be relevant if packages are built against it (some companies do)

mirekphd commented 2 years ago

@asottile it apparently can work in Debian and here is how they made it work: https://stackoverflow.com/a/70138402/9962007

asottile commented 2 years ago

@mirekphd no that's off topic, they built python from source into /usr/local

uranusjr commented 2 years ago

I debugged a bit in debian-testing, it seems that sysconfig.get_preferred_scheme() needs to be patched like the _get_default_scheme() in 3.9 and prior, so get_preferred_scheme("prefix") returns either posix_local or deb_system (depending on the situation), instead of posix_prefix as it currently does.

asottile commented 2 years ago

I believe this is fixed now, the apt version for these will look like 3.10.1-1+focal2 -- the 2 being the updated version (for 3.11 it'll be 3 because there was another revision there as well)

frostming commented 2 years ago

I debugged a bit in debian-testing, it seems that sysconfig.get_preferred_scheme() needs to be patched like the _get_default_scheme() in 3.9 and prior, so get_preferred_scheme("prefix") returns either posix_local or deb_system (depending on the situation), instead of posix_prefix as it currently does.

This behavior is not correct, now the install scheme is wrong when --prefix option is passed. It should be posix_prefix while posix_local is used.

EDIT: It might need some discussion. I think --prefix should use posix_prefix scheme, if not otherwise. And posix_local should be the default scheme.

uranusjr commented 2 years ago

What is the behaviour on Debian Testing’s built-in Python? If they have the exact same problem, this should be raised to Debian first to discuss.

asottile commented 2 years ago

we are unaffiliated with debian, please raise any issues with debian to their bug tracker

deadsnakes / issues