Open njsmith opened 1 year ago
As part of the unpacking process, PyPy can call python -m sysconfig --generate-posix-vars
which will generate a file in a setuptools-compatible build directory like 'build/lib.linux-x86_64-3.9/_sysconfigdata__linux_x86_64-linux-gnu.py'
. Then this file is copied into the stdlib directory. PyPy will not read the Makefile (which it does not use) to get these values, rather it will take them from the current sysconfig
set. This solves the problem of location-specific-values in the file. However it leaves the problem of where to get the rest of the values. Over time PyPy has collected a minimum set of values that must appear in _sysconfig_*.py
to support the build tools that use it (CMake, meson, cross-compile, ...). Since PyPy is not used on as many exotic platforms as CPython, it can hard-code some of the values. This is what running that command generates on a PyPy on my linux x86_64 system (this is about the minimum set of values needed for a posix system):
# system configuration generated and used by the sysconfig module
build_time_vars = {'ABIFLAGS': '',
'AR': 'ar',
'ARFLAGS': 'rc',
'CC': 'gcc -pthread',
'CCSHARED': '-fPIC',
'CFLAGS': '-DNDEBUG -O2',
'CONFINCLUDEPY': '/home/matti/oss/pypy3.9-HEAD/include/pypy3.9',
'CXX': 'g++ -pthread',
'EXE': '',
'EXT_SUFFIX': '.pypy39-pp73-x86_64-linux-gnu.so',
'GNULD': 'yes',
'INCLUDEPY': '/home/matti/oss/pypy3.9-HEAD/include/pypy3.9',
'LDFLAGS': '-Wl,-Bsymbolic-functions',
'LDLIBRARY': 'libpypy3.9-c.so',
'LDSHARED': 'gcc -pthread -shared -Wl,-Bsymbolic-functions',
'LDVERSION': '3.9',
'LIBDIR': '/home/matti/oss/pypy3.9-HEAD/bin',
'MULTIARCH': 'x86_64-linux-gnu',
'OPT': '-DNDEBUG -O2',
'Py_DEBUG': 0,
'Py_ENABLE_SHARED': 0,
'SHLIB_SUFFIX': '.so',
'SIZEOF_VOID_P': 8,
'SO': '.pypy39-pp73-x86_64-linux-gnu.so',
'SOABI': 'pypy39-pp73',
'TZPATH': '/home/matti/oss/pypy3.9-HEAD/share/zoneinfo:/home/matti/oss/pypy3.9-HEAD/lib/zoneinfo:/home/matti/oss/pypy3.9-HEAD/share/lib/zoneinfo:/home/matti/oss/pypy3.9-HEAD/../etc/zoneinfo:/usr/share/zoneinfo:/usr/lib/zoneinfo:/usr/share/lib/zoneinfo:/etc/zoneinfo',
'VERSION': '3.9'}
Additionally, PyPy extends the command line interface of python -m sysconfig --generate-posix-vars
by modifying sysconfig.py
to allow including additional key,value pairs. This is used to add HOST_GNU_TYPE
which is needed for cross-compile. On CPython this can be obtained from the Makefile. So on my system I would run
pypy -m sysconfig --generate-posix-vars HOST_GNU_TYPE x86_64-pc-linux-gnu
PyPy has the luxury of shipping its own stdlib, so it can be lazy about patching the upstream (CPython) stdlib with changes it made. Conda does not have that luxury, so it has a patching mechanism to download the CPython sources and patch them. Maybe posy could patch pybi where needed.
Oh yeah, I was wondering what conda did! Looks like it's.... here?
or maybe here?
Maybe I should just look at the package instead of deciphering build scripts.
...oh fun, latest miniconda installer gives errors on Ubuntu because it has bashisms in a #!/bin/sh
, what the heck:
./Miniconda3-latest-Linux-x86_64.sh: 438: [[: not found
Installing * environment...
./Miniconda3-latest-Linux-x86_64.sh: 444: [[: not found
CondaFileIOError: '/home/njs/miniconda3/pkgs/envs/*/env.txt'. [Errno 2] No such file or directory: '/home/njs/miniconda3/pkgs/envs/*/env.txt'
It also has a symlink from lib/python3.1 -> lib/python3.10
? That's weird.
~/miniconda3/lib
❯ ls -ld python*
lrwxrwxrwx 1 njs njs 10 Feb 2 00:39 python3.1 -> python3.10/
drwxrwxr-x 36 njs njs 13k Feb 2 00:39 python3.10/
Well... anyway... it looks like they rely on conda's trick where it can rewrite files after installing to substitute in the installation path. Which is cool and all, but it would be Extremely Nice™ if pybis could be relocatable without that extra step, and it seems pretty viable given that sysconfig
gets to execute arbitrary python code at runtime. So I guess conda is a dead end.
It also has a symlink from
lib/python3.1 -> lib/python3.10
? That's weird.
I'd have to look up the details, but basically that's a disgusting hack that became necessary for reasons™ after python's minor version gained a second digit.
There are two parts to this issue:
The regeneration is available when calling python -m sysconfig --generate-posix-vars
, plus minus details. I don't know where would be the right place to inject this into the workflow.
It seems like the viable alternatives for patching sysconfig.py are to submit an upstream PR (problematic for older Pythons) or by some magic done when packaging.
Possibly relevant: https://github.com/python/cpython/issues/99942
Currently, distutils/setuptools are buggy when run inside a relocatable Python. For example, https://github.com/pypa/setuptools/issues/3786, though there may be others.
Ultimately we should fix distutils/setuptools. And as a temporary workaround, posy's PEP 517 frontend currently monkeypatches distutils to paper over that specific bug. But in the discourse thread, @mattip mentioned another possible workaround: changing the
_sysconfig_*.py
files that are shipped inside the relocatable python distributions. This is attractive because it would work for anyone who uses our pybis; not just posy. And apparently pypy has experience with this that we might be able to steal? Matti, maybe you can explain more about how pypy does it?