sysconfig --generate-posix-vars creates wrong file when cross compiling

[tiran]

BPO	45909
Nosy	@tiran, @moreati, @ethanhs

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = None created_at = labels = ['type-bug', 'build', '3.11'] title = 'sysconfig --generate-posix-vars creates wrong file when cross compiling' updated_at = user = 'https://github.com/tiran' ``` bugs.python.org fields: ```python activity = actor = 'christian.heimes' assignee = 'none' closed = False closed_date = None closer = None components = ['Build', 'Cross-Build'] creation = creator = 'christian.heimes' dependencies = [] files = [] hgrepos = [] issue_num = 45909 keywords = [] message_count = 2.0 messages = ['407141', '407163'] nosy_count = 3.0 nosy_names = ['christian.heimes', 'Alex.Willmer', 'ethan smith'] pr_nums = [] priority = 'normal' resolution = None stage = None status = 'open' superseder = None type = 'behavior' url = 'https://bugs.python.org/issue45909' versions = ['Python 3.11'] ```

[tiran]

"sysconfig --generate-posix-vars" creates pybuilddir.txt and a platform-specific sysconfig data file like build/lib.linux-x86_64-3.11/_sysconfigdata__linux_x86_64-linux-gnu.py

When creating a cross-compile build of Python, sysconfig mixes data from the cross compile build and the host build. It creates a pybuilddir.txt and build/lib directory with correct values (e.g. wasm32-unknown-emscripten) but sysconfigdata file with name values from the host Python PYTHON_FOR_BUILD (e.g x86_64-unknown-linux-gnu).

$ cat pybuilddir.txt 
build/lib.wasm32-unknown-emscripten-3.11

$ ls build/lib.wasm32-unknown-emscripten-3.11/_sysconfigdata*                           
build/lib.wasm32-unknown-emscripten-3.11/_sysconfigdata__linux_x86_64-linux-gnu.py

$ grep CC build/lib.wasm32-unknown-emscripten-3.11/_sysconfigdata__linux_x86_64-linux-gnu.py 
 'CC': 'gcc',

$ grep ^CC Makefile
CC=             emcc

[tiran]

I just realized that PYTHON_FOR_BUILD is far more complicated. Our WASM build system sets the env var to "$(pwd)/cpython/builddir/build/python". The configure script expects us to set several additional env vars.

The logic goes like this for cross compiling:

if PYTHON_FOR_BUILD is empty:
    find Python interpreter with same version as PACKAGE_VERSION
    set PYTHON_FOR_BUILD='_PYTHON_PROJECT_BASE=$(abs_builddir) _PYTHON_HOST_PLATFORM=$(_PYTHON_HOST_PLATFORM) PYTHONPATH=$(shell test -f pybuilddir.txt && echo $(abs_builddir)/`cat pybuilddir.txt`:)$(srcdir)/Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_$(ABIFLAGS)_$(MACHDEP)_$(MULTIARCH) '$interp
else:
    use user-supplied PYTHON_FOR_BUILD

Since we don't set any of the variables _PYTHON_PROJECT_BASE, _PYTHON_HOST_PLATFORM, and _PYTHON_SYSCONFIGDATA_NAME, we get wrong sysconfig data file.

[Osyotr]

I can't cross-compile python because of this. Lib/sysconfig.py uses os.name and sys.platform almost everywhere which are wrong in cross-compile scenario. For example, when building on windows for non-windows, the build will fail here: https://github.com/python/cpython/blob/d8c7a1174cc182668085b10aab4049f6a2794c2f/Lib/sysconfig.py#L590-L593 _PYTHON_HOST_PLATFORM should probably be used instead.

Cmdline is like this: _PYTHON_PROJECT_BASE=/d/vcpkg/buildtrees/python3/ntoarmv7-qnx-rel _PYTHON_HOST_PLATFORM=qnx-arm PYTHONPATH=./../src/v3.11.0rc1-bb53a96c2e.clean/Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata__qnx_ C:/Users/Osyotr/Desktop/bb_gcc_test/out/vcpkg_installed/x64-windows/tools/python3/python.exe -S -m sysconfig --generate-posix-vars

[tiran]

Cross compiling from Windows to a POSIX-like OS is uncharted territory. I suggest that you open a new ticket. Your use case needs feedback from core devs who are familiar with Windows build system, too.

[FFY00]

@tiran I don't know much about the WASM builds, but would it be possible for you to set the missing environment variables? Long term, I think we definitely want something better, but I think that requires standardizing and documenting cross-builds.

Do you have any alternative proposal?