search

python / cpython

The Python programming language
https://www.python.org
Other
62.45k stars 29.98k forks source link

Cross compiles try to load libraries for target python when the version + SOABI combo match the host and can crash #115382

Open vfazio opened 7 months ago

vfazio commented 7 months ago

Bug report

Bug description:

Note: I'm using 3.11.6 to showcase the behavior because that's easiest, but the build problems exist in 3.12+

When cross compiling Python, typically the foreign build is targeting a different architecture, but this is not always the case.

It's possible that an x86-64 host may be building a Python for a "foreign" x86-64 machine. This typically means that there may be some difference in libc version or CPU instruction support.

When performing a cross compile for the same architecture (by this I mean the combination of Python version + SOABI), Python 3.12+ will attempt to load foreign libraries as part of some of the target dependencies for the build_all make target and will potentially fail.

When cross compiling, builds specify --with-build-python during configure which specifies a host-safe version of python to use to perform python based tasks on behalf of the foreign python build. When configured, PYTHON_FOR_BUILD will be set to a rather complex command that generally evaluates to something like:

_PYTHON_PROJECT_BASE=/mnt/development/buildroot/output/build/python3-3.11.6 _PYTHON_HOST_PLATFORM=linux-x86_64 PYTHONPATH=/mnt/development/buildroot/output/build/python3-3.11.6/build/lib.linux-x86_64-3.11/:./Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata__linux_x86_64-linux-gnu /mnt/development/buildroot/output/host/bin/python3

This is currently a problem for checksharedmods:

# dependency on BUILDPYTHON ensures that the target is run last
.PHONY: checksharedmods
checksharedmods: sharedmods $(PYTHON_FOR_BUILD_DEPS) $(BUILDPYTHON)
    @$(RUNSHARED) $(PYTHON_FOR_BUILD) $(srcdir)/Tools/build/check_extension_modules.py

When run, it tries to run check_extension_modules via PYTHON_FOR_BUILD, however this script has nested in its import dependencies a dependency on externally built modules which poses a problem.

vfazio@vfazio4 ~/development/buildroot/output/build/python3-3.11.6 $ _PYTHON_PROJECT_BASE=/mnt/development/buildroot/output/build/python3-3.11.6 _PYTHON_HOST_PLATFORM=linux-x86_64 PYTHONPATH=/mnt/development/buildroot/output/build/python3-3.11.6/build/lib.linux-x86_64-3.11/:./Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata__linux_x86_64-linux-uclibc /mnt/development/buildroot/output/host/bin/python3 -c "import sys; print(sys.version); import math"
3.11.6 (main, Feb 12 2024, 09:05:53) [GCC 11.4.0]
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: libc.so.0: cannot open shared object file: No such file or directory

Note, this was introduced as part of 3.12:

https://github.com/python/cpython/commit/7bd67d1d88383bb6d156ac9ca816e56085ca5ec8 https://github.com/python/cpython/commit/81dca70d704

This also can show up with glibc like so

When the PYTHON_FOR_BUILD command is updated to not include the local build directory in PYTHONPATH, everything works fine.

This makes sense because it will use the host's libraries instead of the target's libraries... Inserting that path into PYTHONPATH alters the search order for extensions, and because the python versions match (this is a requirement for cross builds) and SOABI matches (which is due to the triplets being the same), it will try to use the libraries from the target path:

>>> sys.path
['', '/mnt/development/buildroot/output/build/python3-3.11.6/build/lib.linux-x86_64-3.11', '/mnt/development/buildroot/output/build/python3-3.11.6/Lib', '/mnt/development/buildroot/output/host/lib/python311.zip', '/mnt/development/buildroot/output/host/lib/python3.11', '/mnt/development/buildroot/output/host/lib/python3.11/lib-dynload', '/home/vfazio/.local/lib/python3.11/site-packages', '/mnt/development/buildroot/output/host/lib/python3.11/site-packages']

>>> sys.meta_path
[<_distutils_hack.DistutilsMetaFinder object at 0x7f031311cf50>, <class '_frozen_importlib.BuiltinImporter'>, <class '_frozen_importlib.FrozenImporter'>, <class '_frozen_importlib_external.PathFinder'>]
>>> sys.meta_path[3]
<class '_frozen_importlib_external.PathFinder'>
>>> sys.meta_path[3].find_spec("math")
ModuleSpec(name='math', loader=<_frozen_importlib_external.ExtensionFileLoader object at 0x7f0313092d50>, origin='/mnt/development/buildroot/output/build/python3-3.11.6/build/lib.linux-x86_64-3.11/math.cpython-311-x86_64-linux-gnu.so')

>>> sys.meta_path[3].find_module("math").load_module()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<frozen importlib._bootstrap_external>", line 605, in _check_name_wrapper
  File "<frozen importlib._bootstrap_external>", line 1120, in load_module
  File "<frozen importlib._bootstrap_external>", line 945, in load_module
  File "<frozen importlib._bootstrap>", line 290, in _load_module_shim
  File "<frozen importlib._bootstrap>", line 721, in _load
  File "<frozen importlib._bootstrap>", line 676, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 573, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1233, in create_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
ImportError: libc.so.0: cannot open shared object file: No such file or directory

So, the "easy solution" is to remove:

diff --git a/configure.ac b/configure.ac
index 384718db1f..6b01083336 100644
--- a/configure.ac
+++ b/configure.ac
@@ -164,7 +164,7 @@ AC_ARG_WITH([build-python],
     dnl Build Python interpreter is used for regeneration and freezing.
     ac_cv_prog_PYTHON_FOR_REGEN=$with_build_python
     PYTHON_FOR_FREEZE="$with_build_python"
-    PYTHON_FOR_BUILD='_PYTHON_PROJECT_BASE=$(abs_builddir) _PYTHON_HOST_PLATFORM=$(_PYTHON_HOST_PLATFORM) PYTHONPATH=$(shell test -f pybuilddir.txt && echo $(abs_builddir)/`cat pybuilddir.txt`:)$(srcdir)/Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_$(ABIFLAGS)_$(MACHDEP)_$(MULTIARCH) '$with_build_python
+    PYTHON_FOR_BUILD='_PYTHON_PROJECT_BASE=$(abs_builddir) _PYTHON_HOST_PLATFORM=$(_PYTHON_HOST_PLATFORM) PYTHONPATH=$(srcdir)/Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_$(ABIFLAGS)_$(MACHDEP)_$(MULTIARCH) '$with_build_python
     AC_MSG_RESULT([$with_build_python])
   ], [
     AS_VAR_IF([cross_compiling], [yes],

However, I assume that was added for a reason and is maybe necessary for other steps (though in testing, it didn't appear to be necessary for the compile to complete).

I'm not sure it ever makes sense to load external modules from the foreign target python when cross compiling, This only works currently because for foreign architecture builds, the triplet will mismatch and cause it to fall back to the host's versions of the libraries. It'd only be safe to do this if the modules were completely source based.

If PYTHONPATH has to be set, then there's maybe an argument that python version and SOABI do not provide enough differentiation and I'm not sure of what the best solution is there.

Alternatively, the check_extension_modules script could be rewritten to reduce the use of external modules to perform its parsing and temporarily work around this issue, but a similar issue could be reintroduced in a subsequent script without policies on what can and can't be used in scripts called during build.

CPython versions tested on:

3.12 3.11

Operating systems tested on:

Linux

Linked PRs

vfazio commented 7 months ago

@tiran Apologies in advance for tagging you, but would like your input as the developer who introduced the script as a dependency to the default build target.

vfazio commented 7 months ago

I forgot to actually run autoreconf after patching configure.ac so the updated PYTHONPATH didn't actually stick. Builds will definitely fail as part of check_extension_modules.py if the local path is not there for foreign architecture builds, so this doesn't seem like an actual solution.

Traceback (most recent call last):
  File "/work/.cache/build/rpi3/python/3.12.2/python-3.12.2/Tools/build/check_extension_modules.py", line 484, in <module>
    main()
  File "/work/.cache/build/rpi3/python/3.12.2/python-3.12.2/Tools/build/check_extension_modules.py", line 466, in main
    checker = ModuleChecker(
              ^^^^^^^^^^^^^^
  File "/work/.cache/build/rpi3/python/3.12.2/python-3.12.2/Tools/build/check_extension_modules.py", line 143, in __init__
    self.ext_suffix = sysconfig.get_config_var("EXT_SUFFIX")
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/.cache/build/rpi3/python/3.12.2/python-3.12.2/Lib/sysconfig.py", line 740, in get_config_var
    return get_config_vars().get(name)
           ^^^^^^^^^^^^^^^^^
  File "/work/.cache/build/rpi3/python/3.12.2/python-3.12.2/Lib/sysconfig.py", line 723, in get_config_vars
    _init_config_vars()
  File "/work/.cache/build/rpi3/python/3.12.2/python-3.12.2/Lib/sysconfig.py", line 670, in _init_config_vars
    _init_posix(_CONFIG_VARS)
  File "/work/.cache/build/rpi3/python/3.12.2/python-3.12.2/Lib/sysconfig.py", line 536, in _init_posix
    _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named '_sysconfigdata__linux_aarch64-linux-gnu'
make: *** [Makefile:1144: checksharedmods] Error 1

So the challenge is we need the local path to find the foreign libraries to check they exist, but specifying the local path in PYTHONPATH causes scripts used during the build process to attempt to load external modules, like math, from the target instead of the host.

Maybe one option is to not specify the path in PYTHONPATH, so that the initial module imports work and use the host's libraries, but then as part of check_extension_modules.py append the build directory + cat pybuilddir.txt into the head of sys.path prior to calling anything from sysconfig.

citrus-it commented 7 months ago

I am seeing something very similar when cross compiling from illumos x86-64 to illumos aarch64.

aarch64-unknown-solaris2.11-ar rcs libpython3.12.a Modules/getbuildinfo.o Parser/token.o  Parser/pegen.o Parser/pegen_errors.o Parser/action_helpers.o Parser/parser.o Parser/string_parser.o Parser/peg_api.o Parser/myreadline.o Parser/tokenizer.o Objects/abstract.o Objects/boolobject.o Objects/bytes_methods.o Objects/bytearrayobject.o Objects/bytesobject.o Objects/call.o Objects/capsule.o Objects/cellobject.o Objects/classobject.o Objects/codeobject.o Objects/complexobject.o Objects/descrobject.o Objects/enumobject.o Objects/exceptions.o Objects/genericaliasobject.o Objects/genobject.o Objects/fileobject.o Objects/floatobject.o Objects/frameobject.o Objects/funcobject.o Objects/interpreteridobject.o Objects/iterobject.o Objects/listobject.o Objects/longobject.o Objects/dictobject.o Objects/odictobject.o Objects/memoryobject.o Objects/methodobject.o Objects/moduleobject.o Objects/namespaceobject.o Objects/object.o Objects/obmalloc.o Objects/picklebufobject.o Objects/rangeobject.o Objects/setobject.o Objects/sliceobject.o Objects/structseq.o Objects/tupleobject.o Objects/typeobject.o Objects/typevarobject.o Objects/unicodeobject.o Objects/unicodectype.o Objects/unionobject.o Objects/weakrefobject.o  Python/_warnings.o Python/Python-ast.o Python/Python-tokenize.o Python/asdl.o Python/assemble.o Python/ast.o Python/ast_opt.o Python/ast_unparse.o Python/bltinmodule.o Python/ceval.o Python/codecs.o Python/compile.o Python/context.o Python/dynamic_annotations.o Python/errors.o Python/flowgraph.o Python/frame.o Python/frozenmain.o Python/future.o Python/getargs.o Python/getcompiler.o Python/getcopyright.o Python/getplatform.o Python/getversion.o Python/ceval_gil.o Python/hamt.o Python/hashtable.o Python/import.o Python/importdl.o Python/initconfig.o Python/instrumentation.o Python/intrinsics.o Python/legacy_tracing.o Python/marshal.o Python/modsupport.o Python/mysnprintf.o Python/mystrtoul.o Python/pathconfig.o Python/preconfig.o Python/pyarena.o Python/pyctype.o Python/pyfpe.o Python/pyhash.o Python/pylifecycle.o Python/pymath.o Python/pystate.o Python/pythonrun.o Python/pytime.o Python/bootstrap_hash.o Python/specialize.o Python/structmember.o Python/symtable.o Python/sysmodule.o Python/thread.o Python/traceback.o Python/tracemalloc.o Python/getopt.o Python/pystrcmp.o Python/pystrtod.o Python/pystrhex.o Python/dtoa.o Python/formatter_unicode.o Python/fileutils.o Python/suggestions.o Python/perf_trampoline.o Python/dynload_shlib.o     Modules/config.o Modules/main.o Modules/gcmodule.o Modules/ucred.o  Modules/dlpimodule.o  Modules/privileges.o  Modules/pyrbac.o Modules/authattr.o Modules/execattr.o Modules/userattr.o  Modules/atexitmodule.o  Modules/faulthandler.o  Modules/posixmodule.o  Modules/signalmodule.o  Modules/_tracemalloc.o  Modules/_codecsmodule.o  Modules/_collectionsmodule.o  Modules/errnomodule.o  Modules/_io/_iomodule.o Modules/_io/iobase.o Modules/_io/fileio.o Modules/_io/bytesio.o Modules/_io/bufferedio.o Modules/_io/textio.o Modules/_io/stringio.o  Modules/itertoolsmodule.o  Modules/_sre/sre.o  Modules/_threadmodule.o  Modules/timemodule.o  Modules/_typingmodule.o  Modules/_weakref.o  Modules/_abc.o  Modules/_functoolsmodule.o  Modules/_localemodule.o  Modules/_operator.o  Modules/_stat.o  Modules/symtablemodule.o  Modules/pwdmodule.o Python/deepfreeze/deepfreeze.o Modules/getpath.o Python/frozen.o
/opt/cross/aarch64/bin/gcc --sysroot=/data/omnios-build/omniosorg/bloody/_build/sysroot.aarch64 -shared      -o libpython3.so -Wl,-hlibpython3.so libpython3.12.so
/opt/cross/aarch64/bin/gcc --sysroot=/data/omnios-build/omniosorg/bloody/_build/sysroot.aarch64      -o python Programs/python.o -Wl,-R,/usr/lib -L. -lpython3.12 -lsocket -lnsl -lintl -ldl -lsendfile  -lpthread -ltsol  -ldlpi   -lnsl -lsocket -lsecdb                        -lm
/opt/cross/aarch64/bin/gcc --sysroot=/data/omnios-build/omniosorg/bloody/_build/sysroot.aarch64      -o Programs/_testembed Programs/_testembed.o -Wl,-R,/usr/lib -L. -lpython3.12 -lsocket -lnsl -lintl -ldl -lsendfile  -lpthread -ltsol  -ldlpi   -lnsl -lsocket -lsecdb                        -lm
Traceback (most recent call last):
  File "/data/omnios-build/omniosorg/bloody/_build/Python-3.12.2/Python-3.12.2/./Tools/build/check_extension_modules.py", line 25, in <module>
    import pathlib
  File "/data/omnios-build/omniosorg/bloody/_build/Python-3.12.2/Python-3.12.2/Lib/pathlib.py", line 20, in <module>
    from urllib.parse import quote_from_bytes as urlquote_from_bytes
  File "/data/omnios-build/omniosorg/bloody/_build/Python-3.12.2/Python-3.12.2/Lib/urllib/parse.py", line 36, in <module>
    import math
ImportError: ld.so.1: python3.12: fatal: libm.so.0: open failed: No such file or directory
make: *** [Makefile:1157: checksharedmods] Error 1
--- Make failed

The link worked, the checksharedmods target tries to load libm.so.0 which only exists on illumos aarch64.

vfazio commented 7 months ago

@citrus-it

I am seeing something very similar when cross compiling from illumos x86-64 to illumos aarch64.

The link worked, the checksharedmods target tries to load libm.so.0 which only exists on illumos aarch64.

That's pretty odd. It shouldn't be doing this for cross architecture builds, though my testing was limited to Debian and Buildroot based distros.

Can you look at your config.log and find the PLATFORM_TRIPLET and MULTIARCH variables?

https://github.com/python/cpython/blob/ae6c01d9d27dd6fb0805340a34f5011b7c1d5e7e/configure#L6752

Also cat the pybuilddir.txt file in the build directory.

My guess is that something is not being correctly identified because it's an unknown libc or somehow the architecture isn't being set correctly and the import code thinks it can load the foreign architecture libs from the current build directory.

citrus-it commented 7 months ago

Can you look at your config.log and find the PLATFORM_TRIPLET and MULTIARCH variables?

PLATFORM_TRIPLET is none and MULTIARCH is empty,

% cat tmp/src/pybuilddir.txt
build/lib.sunos5-arm-3.12%

I'll go and look more closely at configure.ac and the existing cross patch that we apply - there are probably some adjustments we need to make in there. Thanks, and sorry for chipping into what I thought was a related issue.

vfazio commented 7 months ago

No apologies necessary, I just hadn't seen this variant before. It could still be related (odd that it only fails at the very end when running this script)

citrus-it commented 7 months ago

No apologies necessary, I just hadn't seen this variant before. It could still be related (odd that it only fails at the very end when running this script)

With a patched configure.ac, so that the triplet is now populated, I get this instead. I will look into it properly later today, and should probably get these local patches upstreamed too.

ModuleNotFoundError: No module named '_sysconfigdata__sunos5_aarch64-unknown-solaris2'
vfazio commented 7 months ago

Just as an FYI, platform checking was reworked recently https://github.com/python/cpython/commit/c163d7f0b67a568e9b64eeb9c1cbbaa127818596

For us, it's a bit of a bummer that uClibc is now explicitly disabled. We may try to get that support added in at some point,

citrus-it commented 7 months ago

After correcting the host triple detection, everything's working fine for my cross compilation case. Thanks for the pointers @vfazio !

vfazio commented 7 months ago

I was thinking about this last night and I think the options are:

1) change check_extension_modules.py to not use pathlib which is importing math which is an external module and is impacted by the adjusted import priority due to PYTHONPATH. This fixes the immediate issue but doesn't prevent other build scripts from possibly making the same mistake.

2) Update PYTHON_FOR_BUILD to not use PYTHONPATH and force any build scripts that need to interrogate the build to manually adjust sys.path (or pass the build path as an argument). Scripts should be running out of the build directory so should know where to look for adjusting the import path. This should only be necessary for cross compile scenarios I think?

3) Specify yet another PYTHON variable in the Makefile that does not specify PYTHONPATH for "special cases" and use that for scripts that are "build directory aware" and which manually adjust sys.path "when appropriate"

4) Make the SOABIs more unique. This is difficult because of what needs to be encoded. It's not just the architecture and libc, but the libc version (host may be on glibc 2.34 but the cross build may target 2.38), architecture IP/instruction support (psABI https://gitlab.com/x86-psABIs/x86-64-ABI) etc.

5) <other magic>

In support of option 2 above...

Setting PYTHONPATH feels like we're abusing the import mechanism a bit "because it works" in most cases. These build time scripts are special in that they need to run on the host and may want to leverage host provided libraries but perform parsing on files generated by the target build. The mechanism used in check_extension_modules.py is to interrogate sysconfig for modules but sysconfig is a fun hodge-podge of information; sometimes it reflects the host python data (see sysconfig.get_platform), sometimes it reflects the target.

For this script in particular, we rely on get_config_var to query the modules specified in the target Makefile, meaning if we were to drop the build directory from PYTHONPATH we cannot reliably get information if sysconfig._init_config_vars was called because it would reflect the values from the host and not the target.

Luckily, this does not (currently) get called by any dependency within the script, so we could reliably inject the build path into sys.path but best is maybe to check as a precaution:

diff --git a/Tools/build/check_extension_modules.py b/Tools/build/check_extension_modules.py
index 59239c62e2..fe3423e4aa 100644
--- a/Tools/build/check_extension_modules.py
+++ b/Tools/build/check_extension_modules.py
@@ -140,9 +140,17 @@ class ModuleChecker:
     def __init__(self, cross_compiling: bool = False, strict: bool = False):
         self.cross_compiling = cross_compiling
         self.strict_extensions_build = strict
+        self.builddir = self.get_builddir()
+
+        # Add path for cross modules prior to sysconfig parsing the makefile.
+        if self.cross_compiling:
+            if sysconfig._CONFIG_VARS_INITIALIZED:
+                sysconfig._CONFIG_VARS_INITIALIZED = False
+                sysconfig._CONFIG_VARS = None
+            sys.path.insert(1, os.path.join(sysconfig._PROJECT_BASE, self.builddir))
+
         self.ext_suffix = sysconfig.get_config_var("EXT_SUFFIX")
         self.platform = sysconfig.get_platform()
-        self.builddir = self.get_builddir()
         self.modules = self.get_modules()

         self.builtin_ok = []
diff --git a/configure b/configure
index e962a6aed1..467962f75b 100755
--- a/configure
+++ b/configure
@@ -3686,7 +3686,7 @@ fi
     fi
         ac_cv_prog_PYTHON_FOR_REGEN=$with_build_python
     PYTHON_FOR_FREEZE="$with_build_python"
-    PYTHON_FOR_BUILD='_PYTHON_PROJECT_BASE=$(abs_builddir) _PYTHON_HOST_PLATFORM=$(_PYTHON_HOST_PLATFORM) PYTHONPATH=$(shell test -f pybuilddir.txt && echo $(abs_builddir)/`cat pybuilddir.txt`:)$(srcdir)/Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_$(ABIFLAGS)_$(MACHDEP)_$(MULTIARCH) '$with_build_python
+    PYTHON_FOR_BUILD='_PYTHON_PROJECT_BASE=$(abs_builddir) _PYTHON_HOST_PLATFORM=$(_PYTHON_HOST_PLATFORM) PYTHONPATH=$(srcdir)/Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_$(ABIFLAGS)_$(MACHDEP)_$(MULTIARCH) '$with_build_python
     { printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $with_build_python" >&5
 printf "%s\n" "$with_build_python" >&6; }

diff --git a/configure.ac b/configure.ac
index 384718db1f..6b01083336 100644
--- a/configure.ac
+++ b/configure.ac
@@ -164,7 +164,7 @@ AC_ARG_WITH([build-python],
     dnl Build Python interpreter is used for regeneration and freezing.
     ac_cv_prog_PYTHON_FOR_REGEN=$with_build_python
     PYTHON_FOR_FREEZE="$with_build_python"
-    PYTHON_FOR_BUILD='_PYTHON_PROJECT_BASE=$(abs_builddir) _PYTHON_HOST_PLATFORM=$(_PYTHON_HOST_PLATFORM) PYTHONPATH=$(shell test -f pybuilddir.txt && echo $(abs_builddir)/`cat pybuilddir.txt`:)$(srcdir)/Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_$(ABIFLAGS)_$(MACHDEP)_$(MULTIARCH) '$with_build_python
+    PYTHON_FOR_BUILD='_PYTHON_PROJECT_BASE=$(abs_builddir) _PYTHON_HOST_PLATFORM=$(_PYTHON_HOST_PLATFORM) PYTHONPATH=$(srcdir)/Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_$(ABIFLAGS)_$(MACHDEP)_$(MULTIARCH) '$with_build_python
     AC_MSG_RESULT([$with_build_python])
   ], [
     AS_VAR_IF([cross_compiling], [yes],

I tested this on a cross architecture and "same architecture but foreign libc" build and both builds completed though i haven't run the test suites.

Changing PYTHONPATH obviously would affect more than this script, so all other callers would need to be evaluated to ensure they're still working as intended. This change is almost a decade old so there may be some ingrained assumptions issue 15484 commit: 9731330d6f4d63d6d57c8e99c8d11ef1def42858

This doesn't mean we can't do some combination of all options. Option 1 may be the easiest bandaid and has less chance of impacting other scripts until we've had time to vet them.

bkuhls commented 7 months ago

I tested this on a cross architecture and "same architecture but foreign libc" build and both builds completed though i haven't run the test suites.

Thanks for the patch, it fixes the problem

    import math
ImportError: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.38' not found

when cross-compiling for x86-64-v2/glibc-2.38:

$ grep "PLATFORM_TRIPLET\|MULTIARCH" config.log
MULTIARCH='x86_64-linux-gnu'
PLATFORM_TRIPLET='x86_64-linux-gnu'
$ cat pybuilddir.txt
build/lib.linux-x86_64-3.12
$ uname -a
Linux debian 6.7.5-x64v3-xanmod1 #0~20240216.gf29bf01 SMP PREEMPT_DYNAMIC Fri Feb 16 20:35:14 UTC x86_64 GNU/Linux
$ /usr/lib/x86_64-linux-gnu/libc.so.6
GNU C Library (Debian GLIBC 2.36-9+deb12u4) stable release version 2.36.
vfazio commented 7 months ago

A patch to convert check_extension_modules.py to drop pathlib. I have not tested this for cross builds, just the basic functionality, but I think it should work. I may try to submit this as it's a more "isolated" change than changing PYTHONPATH for all make targets, knowing that the long term goal should be to drop PYTHONPATH where possible for cross compile scenarios, which could lead to the revert of this commit.

diff --git a/Tools/build/check_extension_modules.py b/Tools/build/check_extension_modules.py
index a9fee4981e..ab6138750c 100644
--- a/Tools/build/check_extension_modules.py
+++ b/Tools/build/check_extension_modules.py
@@ -22,7 +22,6 @@
 import enum
 import logging
 import os
-import pathlib
 import re
 import sys
 import sysconfig
@@ -33,7 +32,7 @@
 from importlib.util import spec_from_file_location, spec_from_loader
 from typing import Iterable

-SRC_DIR = pathlib.Path(__file__).parent.parent.parent
+SRC_DIR = os.path.dirname(os.path.dirname(os.path.dirname(__file__)))

 # core modules, hard-coded in Modules/config.h.in
 CORE_MODULES = {
@@ -133,7 +132,7 @@ class ModuleChecker:
         "Modules/Setup.local",
         "Modules/Setup.stdlib",
         "Modules/Setup.bootstrap",
-        SRC_DIR / "Modules/Setup",
+        os.path.join(SRC_DIR, "Modules/Setup"),
     )

     def __init__(self, cross_compiling: bool = False, strict: bool = False):
@@ -263,14 +262,13 @@ def list_module_names(self, *, all: bool = False) -> set:
             names.update(WINDOWS_MODULES)
         return names

-    def get_builddir(self) -> pathlib.Path:
+    def get_builddir(self) -> str:
         try:
             with open(self.pybuilddir_txt, encoding="utf-8") as f:
                 builddir = f.read()
         except FileNotFoundError:
             logger.error("%s must be run from the top build directory", __file__)
             raise
-        builddir = pathlib.Path(builddir)
         logger.debug("%s: %s", self.pybuilddir_txt, builddir)
         return builddir

@@ -338,7 +336,7 @@ def get_sysconfig_modules(self) -> Iterable[ModuleInfo]:
             logger.debug("Found %s in Makefile", modinfo)
             yield modinfo

-    def parse_setup_file(self, setup_file: pathlib.Path) -> Iterable[ModuleInfo]:
+    def parse_setup_file(self, setup_file: str) -> Iterable[ModuleInfo]:
         """Parse a Modules/Setup file"""
         assign_var = re.compile(r"^\w+=")  # EGG_SPAM=foo
         # default to static module
@@ -382,10 +380,10 @@ def get_spec(self, modinfo: ModuleInfo) -> ModuleSpec:
         else:
             raise ValueError(modinfo)

-    def get_location(self, modinfo: ModuleInfo) -> pathlib.Path:
+    def get_location(self, modinfo: ModuleInfo) -> str:
         """Get shared library location in build directory"""
         if modinfo.state == ModuleState.SHARED:
-            return self.builddir / f"{modinfo.name}{self.ext_suffix}"
+            return os.path.join(self.builddir, f"{modinfo.name}{self.ext_suffix}")
         else:
             return None

@@ -430,23 +428,33 @@ def rename_module(self, modinfo: ModuleInfo) -> None:

         failed_name = f"{modinfo.name}_failed{self.ext_suffix}"
         builddir_path = self.get_location(modinfo)
-        if builddir_path.is_symlink():
+        if os.path.islink(builddir_path):
             symlink = builddir_path
-            module_path = builddir_path.resolve().relative_to(os.getcwd())
-            failed_path = module_path.parent / failed_name
+            real_path = os.path.realpath(builddir_path)
+            cwd = os.getcwd()
+            if not real_path.startswith(cwd):
+                raise ValueError(f"{real_path} is not in the subpath of {cwd}")
+            module_path = real_path.partition(cwd)[2]
+            failed_path = os.path.join(os.path.dirname(module_path), failed_name)
         else:
             symlink = None
             module_path = builddir_path
-            failed_path = self.builddir / failed_name
+            failed_path = os.path.join(self.builddir, failed_name)

         # remove old failed file
-        failed_path.unlink(missing_ok=True)
+        try:
+            os.unlink(failed_path)
+        except FileNotFoundError:
+            pass
         # remove symlink
         if symlink is not None:
-            symlink.unlink(missing_ok=True)
+            try:
+                os.unlink(symlink)
+            except FileNotFoundError:
+                pass
         # rename shared extension file
         try:
-            module_path.rename(failed_path)
+            os.rename(module_path, failed_path)
         except FileNotFoundError:
             logger.debug("Shared extension file '%s' does not exist.", module_path)
         else:
vfazio commented 6 months ago

I'm not sure 3.13 is currently affected. 15de493395c3251b8b82063bbe22a379792b9404 changed pathlib to delay import of quote_from_bytes so math doesn't get imported when pathlib does.

However, the root problem is still there. If a library is compiled as a shared module and it's a dependency of a build script, there's a risk of the foreign module being imported.

Build targets that use PYTHON_FOR_BUILD:

  1. platform
  2. pybuilddir.txt
  3. wasm_stdlib
  4. checksharedmods
  5. install
  6. altinstall
  7. libinstall

I don't know anything about wasm_stdlib to make a call there, but besides checksharedmods, builds configured --with-ensurepip will likely fail as part of install/altinstall and libinstall will probably fail when compiling the optimizations.

The reason Buildroot doesn't see more failures is because we specifically configure --without-ensurepip and we patch out the compilation of optimizations.

After review, I don't feel confident in dropping the build directory from PYTHONPATH for these other build targets, especially in a patch to 3.12.x

Honestly, as much of a hack as it may look like, it may be easiest to delay the import of from urllib.parse import quote_from_bytes as urlquote_from_bytes into PurePath.as_uri like the above commit to work around the issue without needing to rework all of these targets in 3.12.x. This is untested, but i may try it out tomorrow.

diff --git a/Lib/pathlib.py b/Lib/pathlib.py
index bd5a096f9e..544290d718 100644
--- a/Lib/pathlib.py
+++ b/Lib/pathlib.py
@@ -17,7 +17,6 @@
 from _collections_abc import Sequence
 from errno import ENOENT, ENOTDIR, EBADF, ELOOP
 from stat import S_ISDIR, S_ISLNK, S_ISREG, S_ISSOCK, S_ISBLK, S_ISCHR, S_ISFIFO
-from urllib.parse import quote_from_bytes as urlquote_from_bytes

 __all__ = [
@@ -479,7 +478,8 @@ def as_uri(self):
             # It's a posix path => 'file:///etc/hosts'
             prefix = 'file://'
             path = str(self)
-        return prefix + urlquote_from_bytes(os.fsencode(path))
+        from urllib.parse import quote_from_bytes
+        return prefix + quote_from_bytes(os.fsencode(path))

     @property
     def _str_normcase(self):
bendebled commented 6 months ago

@vfazio If it helps... I have used your patch in a Buildroot build with success!

The build was done from Buildroot commit 8ab4a0a348 which is of course a commit above 36e635d2d5c0166476858aa239ccbe78e8f2af14 (package/python3: bump version to 3.12.1).

Without the patch, I get the exact error than the one described here. With the patch, no error!

vfazio commented 6 months ago

@vfazio If it helps... I have used your patch in a Buildroot build with success!

The build was done from Buildroot commit 8ab4a0a348 which is of course a commit above 36e635d2d5c0166476858aa239ccbe78e8f2af14 (package/python3: bump version to 3.12.1).

Without the patch, I get the exact error than the one described here. With the patch, no error!

@bendebled which patch? i've unfortunately proposed 3 of them and even I'm starting to lose track.

bendebled commented 6 months ago

@vfazio If it helps... I have used your patch in a Buildroot build with success! The build was done from Buildroot commit 8ab4a0a348 which is of course a commit above 36e635d2d5c0166476858aa239ccbe78e8f2af14 (package/python3: bump version to 3.12.1). Without the patch, I get the exact error than the one described here. With the patch, no error!

@bendebled which patch? i've unfortunately proposed 3 of them and even I'm starting to lose track.

Your latest one:

diff --git a/Lib/pathlib.py b/Lib/pathlib.py
index bd5a096f9e..544290d718 100644
--- a/Lib/pathlib.py
+++ b/Lib/pathlib.py
@@ -17,7 +17,6 @@
 from _collections_abc import Sequence
 from errno import ENOENT, ENOTDIR, EBADF, ELOOP
 from stat import S_ISDIR, S_ISLNK, S_ISREG, S_ISSOCK, S_ISBLK, S_ISCHR, S_ISFIFO
-from urllib.parse import quote_from_bytes as urlquote_from_bytes

 __all__ = [
@@ -479,7 +478,8 @@ def as_uri(self):
             # It's a posix path => 'file:///etc/hosts'
             prefix = 'file://'
             path = str(self)
-        return prefix + urlquote_from_bytes(os.fsencode(path))
+        from urllib.parse import quote_from_bytes
+        return prefix + quote_from_bytes(os.fsencode(path))

     @property
     def _str_normcase(self):
vfazio commented 6 months ago

Looks like OE has a patch that reinjects the host libraries into sys.path: https://git.openembedded.org/openembedded-core/tree/meta/recipes-devtools/python/python3/crosspythonpath.patch

They rely on a special CROSSPYTHONPATH variable exported in their recipe, but that could probably be emulated with something like (untested):

diff --git a/Makefile.pre.in b/Makefile.pre.in
index dd5e69f7ab..c2b94c2cbf 100644
--- a/Makefile.pre.in
+++ b/Makefile.pre.in
@@ -786,6 +786,10 @@ $(BUILDPYTHON):    Programs/python.o $(LINK_PYTHON_DEPS)
 platform: $(PYTHON_FOR_BUILD_DEPS) pybuilddir.txt
        $(RUNSHARED) $(PYTHON_FOR_BUILD) -c 'import sys ; from sysconfig import get_platform ; print("%s-%d.%d" % (get_platform(), *sys.version_info[:2]))' >platform

+# Must be generated before pybuilddir.txt otherwise sys.path includes foreign libraries
+crosspath.txt:
+    $(RUNSHARED) $(PYTHON_FOR_BUILD) -c 'import sys ; print(":".join([p for p in sys.path if p]))' >crosspath.txt
+
 # Create build directory and generate the sysconfig build-time data there.
 # pybuilddir.txt contains the name of the build dir and is used for
 # sys.path fixup -- see Modules/getpath.c.
@@ -793,7 +797,7 @@ platform: $(PYTHON_FOR_BUILD_DEPS) pybuilddir.txt
 # problems by creating a dummy pybuilddir.txt just to allow interpreter
 # initialization to succeed.  It will be overwritten by generate-posix-vars
 # or removed in case of failure.
-pybuilddir.txt: $(PYTHON_FOR_BUILD_DEPS)
+pybuilddir.txt: $(PYTHON_FOR_BUILD_DEPS) crosspath.txt
        @echo "none" > ./pybuilddir.txt
        $(RUNSHARED) $(PYTHON_FOR_BUILD) -S -m sysconfig --generate-posix-vars ;\
        if test $$? -ne 0 ; then \
diff --git a/configure b/configure
index e962a6aed1..dbce87ac68 100755
--- a/configure
+++ b/configure
@@ -3686,7 +3686,7 @@ fi
     fi
         ac_cv_prog_PYTHON_FOR_REGEN=$with_build_python
     PYTHON_FOR_FREEZE="$with_build_python"
-    PYTHON_FOR_BUILD='_PYTHON_PROJECT_BASE=$(abs_builddir) _PYTHON_HOST_PLATFORM=$(_PYTHON_HOST_PLATFORM) PYTHONPATH=$(shell test -f pybuilddir.txt && echo $(abs_builddir)/`cat pybuilddir.txt`:)$(srcdir)/Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_$(ABIFLAGS)_$(MACHDEP)_$(MULTIARCH) '$with_build_python
+    PYTHON_FOR_BUILD='_PYTHON_PROJECT_BASE=$(abs_builddir) _PYTHON_HOST_PLATFORM=$(_PYTHON_HOST_PLATFORM) PYTHONPATH=$(shell test -f crosspath.txt && echo $(abs_builddir)/`cat crosspath.txt`:)$(shell test -f pybuilddir.txt && echo $(abs_builddir)/`cat pybuilddir.txt`:)$(srcdir)/Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_$(ABIFLAGS)_$(MACHDEP)_$(MULTIARCH) '$with_build_python
     { printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $with_build_python" >&5
 printf "%s\n" "$with_build_python" >&6; }

diff --git a/configure.ac b/configure.ac
index 384718db1f..69db53b1bc 100644
--- a/configure.ac
+++ b/configure.ac
@@ -164,7 +164,7 @@ AC_ARG_WITH([build-python],
     dnl Build Python interpreter is used for regeneration and freezing.
     ac_cv_prog_PYTHON_FOR_REGEN=$with_build_python
     PYTHON_FOR_FREEZE="$with_build_python"
-    PYTHON_FOR_BUILD='_PYTHON_PROJECT_BASE=$(abs_builddir) _PYTHON_HOST_PLATFORM=$(_PYTHON_HOST_PLATFORM) PYTHONPATH=$(shell test -f pybuilddir.txt && echo $(abs_builddir)/`cat pybuilddir.txt`:)$(srcdir)/Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_$(ABIFLAGS)_$(MACHDEP)_$(MULTIARCH) '$with_build_python
+    PYTHON_FOR_BUILD='_PYTHON_PROJECT_BASE=$(abs_builddir) _PYTHON_HOST_PLATFORM=$(_PYTHON_HOST_PLATFORM) PYTHONPATH=$(shell test -f crosspath.txt && echo $(abs_builddir)/`cat crosspath.txt`:)$(shell test -f pybuilddir.txt && echo $(abs_builddir)/`cat pybuilddir.txt`:)$(srcdir)/Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_$(ABIFLAGS)_$(MACHDEP)_$(MULTIARCH) '$with_build_python
     AC_MSG_RESULT([$with_build_python])
   ], [
     AS_VAR_IF([cross_compiling], [yes],

However, I'm not a big fan of trying to reinsert the host libraries to the top, it just proves that PYTHONPATH should not have the foreign libraries in the path and that build scripts need to accept some flag that a cross compile is happening and can accept a path for the local build directory if they need to amend sys.path or perform some operation relative to the build directory

vfazio commented 6 months ago

@vfazio If it helps... I have used your patch in a Buildroot build with success! The build was done from Buildroot commit 8ab4a0a348 which is of course a commit above 36e635d2d5c0166476858aa239ccbe78e8f2af14 (package/python3: bump version to 3.12.1). Without the patch, I get the exact error than the one described here. With the patch, no error!

@bendebled which patch? i've unfortunately proposed 3 of them and even I'm starting to lose track.

Your latest one:

diff --git a/Lib/pathlib.py b/Lib/pathlib.py
index bd5a096f9e..544290d718 100644
--- a/Lib/pathlib.py
+++ b/Lib/pathlib.py
@@ -17,7 +17,6 @@
 from _collections_abc import Sequence
 from errno import ENOENT, ENOTDIR, EBADF, ELOOP
 from stat import S_ISDIR, S_ISLNK, S_ISREG, S_ISSOCK, S_ISBLK, S_ISCHR, S_ISFIFO
-from urllib.parse import quote_from_bytes as urlquote_from_bytes

 __all__ = [
@@ -479,7 +478,8 @@ def as_uri(self):
             # It's a posix path => 'file:///etc/hosts'
             prefix = 'file://'
             path = str(self)
-        return prefix + urlquote_from_bytes(os.fsencode(path))
+        from urllib.parse import quote_from_bytes
+        return prefix + quote_from_bytes(os.fsencode(path))

     @property
     def _str_normcase(self):

As we discussed on IRC, this won't fix all targets

Traceback (most recent call last):
  File "<frozen runpy>", line 189, in _run_module_as_main
  File "<frozen runpy>", line 148, in _get_module_details
  File "<frozen runpy>", line 112, in _get_module_details
  File "xxx/build/python3-3.12.1/Lib/ensurepip/__init__.py", line 4, in <module>
    import subprocess
  File "xxx/build/python3-3.12.1/Lib/subprocess.py", line 119, in <module>
    import selectors
  File "xxx/build/python3-3.12.1/Lib/selectors.py", line 11, in <module>
    import math
ImportError: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.38' not found (required by xxx/build/python3-3.12.1/build/lib.linux-x86_64-3.12/math.cpython-312-x86_64-linux-gnu.so)
make[2]: *** [Makefile:2030: install] Error 1
make[1]: *** [package/pkg-generic.mk:322: xxx/build/python3-3.12.1/.stamp_staging_installed] Error 2
make: *** [Makefile:82: _all] Error 2
make: Leaving directory 'xxx/buildroot'
Done in 12min 29s
bkuhls commented 6 months ago

A patch to convert check_extension_modules.py to drop pathlib. I have not tested this for cross builds

Tested using buildroot and it also fixes the "libm.so.6: version `GLIBC_2.38' not found" cross-compile error, thanks!

vfazio commented 6 months ago

A patch to convert check_extension_modules.py to drop pathlib. I have not tested this for cross builds

Tested using buildroot and it also fixes the "libm.so.6: version `GLIBC_2.38' not found" cross-compile error, thanks!

@bkuhls

I expect Buildroot will need to carry a unique patch to address the build issue there. As mentioned in https://github.com/python/cpython/issues/115382#issuecomment-1963248712, BR disables ensurepip and does not compile target pyc files via the Makefile so the other problematic paths are obviated. I think we can either go the pathlib -> os.path route or go the delayed import route via https://github.com/python/cpython/issues/115382#issuecomment-1964242439 and call out in Upstream: that the fix is hyper-specific to the BR build process. Feel free to message me on IRC or email me if you want. I don't know how Yann and Arnout want to handle this. I know Peter held the bump back from 2024.02 because of this issue.

The general issue still stands, however, and I haven't had time to think about the best way to approach resolving it. The whole cross compile and build script situation is predicated on a series of environment variables and assumptions.

Things that need to be taken into consideration:

Now, with all of that said, I think we could do something akin to:

diff --git a/Tools/build/check_extension_modules.py b/Tools/build/check_extension_modules.py
index 59239c62e2..45fdd7fe09 100644
--- a/Tools/build/check_extension_modules.py
+++ b/Tools/build/check_extension_modules.py
@@ -125,6 +125,23 @@ def __bool__(self):

 ModuleInfo = collections.namedtuple("ModuleInfo", "name state")

+class SysConfigShim:
+
+    def __init__(self, path: pathlib.Path):
+        data_file = path / (os.environ.get("_PYTHON_SYSCONFIGDATA_NAME") + ".py")
+        self.data: dict[str, str] = {}
+        exec(data_file.read_text(), globals(), self.data)
+
+    def get_config_var(self, name: str):
+        return self.get_config_vars().get(name)
+
+    def get_config_vars(self, *args):
+        if args:
+            vals = []
+            for name in args:
+                vals.append(self.data['build_time_vars'].get(name))
+            return vals
+        return self.data['build_time_vars']

 class ModuleChecker:
     pybuilddir_txt = "pybuilddir.txt"
@@ -139,10 +159,15 @@ class ModuleChecker:

     def __init__(self, cross_compiling: bool = False, strict: bool = False):
         self.cross_compiling = cross_compiling
+        self.builddir = self.get_builddir()
+        if self.cross_compiling:
+            shim = SysConfigShim(self.builddir)
+            sysconfig.get_config_var = shim.get_config_var
+            sysconfig.get_config_vars = shim.get_config_vars
+
         self.strict_extensions_build = strict
         self.ext_suffix = sysconfig.get_config_var("EXT_SUFFIX")
         self.platform = sysconfig.get_platform()
-        self.builddir = self.get_builddir()
         self.modules = self.get_modules()

         self.builtin_ok = []

The goal here is to replace the problematic sysconfig calls by utilizing the shim that, when cross-compiling, will load the sysconfig data from the target manually.

ensurepip could encapsulate similar logic possibly:

if '_PYTHON_HOST_PLATFORM' in os.environ:
    build_dir_base = os.environ.get("_PYTHON_PROJECT_BASE")
    pybuilddir = os.path.join(build_dir_base, "pybuilddir.txt")
    with open(pybuilddir, encoding="utf-8") as f:
        builddir = f.read()
    sysconfig_path = os.path.join(build_dir_base, builddir, os.environ.get("_PYTHON_SYSCONFIGDATA_NAME") + ".py")

    with open(sysconfig_path) as f:
        data = f.read()
    loc = {}
    exec(data, globals(), loc)

    _WHEEL_PKG_DIR = loc['build_time_vars'].get('WHEEL_PKG_DIR')
else:
    _WHEEL_PKG_DIR = sysconfig.get_config_var('WHEEL_PKG_DIR')

This may allow us to drop the target's path from PYTHONPATH in configure.

From what i can tell, compileall and the wasm_assets likely won't be impacted by any changes to PYTHONPATH

vfazio commented 6 months ago

I have two branches that I'm going to try to test out. Anyone else is welcome to try the patch and provide feedback.

https://github.com/vfazio/cpython/tree/vfazio-fix-cross-compile-main

https://github.com/vfazio/cpython/tree/vfazio-fix-cross-compile-3.12

bkuhls commented 6 months ago

https://github.com/vfazio/cpython/tree/vfazio-fix-cross-compile-3.12

A buildroot build is fixed by the patch https://github.com/vfazio/cpython/commit/1ee82310c5438dee539bf6ce35a52c39f5d2c27d from this branch, thanks!

vfazio commented 6 months ago

https://github.com/vfazio/cpython/tree/vfazio-fix-cross-compile-3.12

A buildroot build is fixed by the patch vfazio@1ee8231 from this branch, thanks!

@bkuhls thanks for confirming. I just confirmed it doesn't break anything for an actual foreign (target arch != host arch) cross compile. My cross compile is without any other patches, so the pyc compile stage does run and it did so successfully. I haven't run the actual test suite on the generated installation yet, so will do that soon.

I do need to test the ensurepip stuff is working still

vfazio commented 6 months ago

This wont work either... ensurepip will fail when it tries to bootstrap pip into DESTDIR. I could hack it and append the target's sysconfigdata path to the end of sys.path

def _run_pip(args, additional_paths=None):
    # Run the bootstrapping in a subprocess to avoid leaking any state that happens
    # after pip has executed. Particularly, this avoids the case when pip holds onto
    # the files in *additional_paths*, preventing us to remove them at the end of the
    # invocation.
    target_lib_dir = []
    if '_PYTHON_HOST_PLATFORM' in os.environ:
        # If executing in a cross-compile environment, the target's sysconfigdata
        # directory needs to be in sys.path so the child process can import it.
        target_lib_dir = [_find_target_sysconfigdata_dir()]

    code = f"""
import runpy
import sys
sys.path = {additional_paths or []} + sys.path + {target_lib_dir}

However, what this means is:

Putting the path at the beginning for host == target arch replicates the issue we're trying to fix.

For this to work, the checks for a cross build environment would have to extend into pip which sounds absolutely arduous.

I'm starting to run dry of ideas on how to cleanly handle this case...

My last bad idea is to shift the path to pybuilddir.txt from PYTHONPATH to _PYTHON_SYSCONFIGDATA_PATH and then in sysconfig._init_posix do some quackery to import based on name and path.

In testing, this works for a foreign cross build with ensurepip, so maybe it solves all of our problems.

vfazio commented 6 months ago

So, I've pushed an MR that I believe should address this issue.

The commit message sort of speaks for itself, but I'll detail what's going on just as a summary post.

When --with-build-python is specified, which is mandatory for cross compiles but could be used in non-cross compile scenarios, an external but "compatible" host interpreter is used to perform certain build stages. There is a minimal version specified https://github.com/python/cpython/issues/104487 to account for required language features for certain build targets.

When a host interpreter is used, the command line to invoke that interpreter includes PYTHONPATH which points to the target build. This puts the target's paths higher in the sys.path import search order so the target's libraries are prioritized over the host's versions.

For foreign architecture builds (think build=x86_64, host=arm64), when imports are searched and the module is a compiled module, the ExtensionFileLoader searches for the specific SOABI+file extension:

const char *_PyImport_DynLoadFiletab[] = {
#ifdef __CYGWIN__
    ".dll",
#else  /* !__CYGWIN__ */
    "." SOABI ".so",
#ifdef ALT_SOABI
    "." ALT_SOABI ".so",
#endif
    ".abi" PYTHON_ABI_STRING ".so",
    ".so",
#endif  /* __CYGWIN__ */
    NULL,
};

This means that the host interpreter will never load the target compiled libraries so long as they disagree on SOABI.

However, when the host interpreter and the target have the same SOABI (so host=x86_64, build=x86_64), there's a risk that due to their precedence in sys.path that the target's libraries will be loaded and/or executed.

As seen in this issue, this is problematic because the target may have been built and linked against a different libc implementation which the host cannot load. Even if the libc matched, there's a chance that the instruction set for the generated binaries may not be executable on the host (x86 psABI differences).

The only time it's "OK" to run the target libraries is if the imports and their dependencies are pure Python.

From what I can naively tell, the only real reason it's necessary to include the target's path in sys.path via PYTHONPATH is so the target's generated sysconfigdata module can be interrogated to check for built modules, compile options, etc.

If this is indeed the case, it should be sufficient to drop the path from PYTHONPATH and build an override mechanism for its path similar to how the name is already handled via _PYTHON_SYSCONFIGDATA_NAME.

In build testing, this works fine. The Python CI pipelines also seem to be ok with this.

vfazio commented 5 months ago

Note that CPython 3.11 also fails when running make sharedmods if the host doesn't support instructions generated by the target toolchain:

An example build: http://autobuild.buildroot.net/results/c854080e003e9a7d525325073190b472a8f982aa/build-end.log

make[1]: *** [Makefile:868: sharedmods] Illegal instruction (core dumped)
make[1]: *** Waiting for unfinished jobs....

I tested the patch in #116294 and adapted it for 3.11 and it seemed to resolve the issue.