search

indygreg / python-build-standalone

Produce redistributable builds of Python
Mozilla Public License 2.0
1.98k stars 127 forks source link

Unexpected output of 'arch' on macOS when cross-compiling to x86_64 #150

Closed jaraco closed 1 year ago

jaraco commented 1 year ago

Attempting to build Python for macOS on an M1 Macbook Pro running Ventura (macOS 13.0.1) and targeting x86_64 fails during the configure:

 python-build-standalone main $ py build-macos.py --python=cpython-3.9 --optimizations=pgo --target=x86_64-apple-darwin
...
cpython-3.9> checking whether pthread_key_t is compatible with int... no
cpython-3.9> configure: error: Unexpected output of 'arch' on macOS
process exited 1
make: *** [/Users/jaraco/draft/python-build-standalone/build/cpython-3.9.15-x86_64-apple-darwin-pgo.tar] Error 1

Since the docs say it should be possible to build for x86 on ARM, I'm guessing this is a bug. I reproduced it against both main (a07a392) and 20221106.

For reference, here's the full output of a run and the corresponding invocation/stderr:

$ py build-macos.py --target x86_64-apple-darwin | gist

[notice] A new release of pip available: 22.2.2 -> 22.3.1
[notice] To update, run: /Users/jaraco/draft/python-build-standalone/build/venv.macos/bin/python3.10 -m pip install --upgrade pip
/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/subprocess.py:1014: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used
  self.stdout = io.open(c2pread, 'rb', bufsize)
make: *** [/Users/jaraco/draft/python-build-standalone/build/cpython-3.10.8-x86_64-apple-darwin-noopt.tar] Error 1
jaraco commented 1 year ago

Side note, it's a shame that the error message doesn't indicate what output was received or what was expected.

jaraco commented 1 year ago

Same error occurs without --optimizations and without --python and with --python=cpython-3.10.

jaraco commented 1 year ago

Looking at this SO question, it seems it was a known issue with older Pythons prior to Apple Silicon support, but that should be irrelevant for Python 3.9+. I am using py, which was resolving to Python 3.12.0a2, but I'm guessing the version of Python running the build script is not relevant to the configure step, and confirmed (issue occurs with py -3.10 ...).

jaraco commented 1 year ago

The error message apparently comes from:

https://github.com/python/cpython/blob/cd67c1bb30eccd0c6fd1386405df225aed4c91a9/configure#L10953

Suggesting that /usr/bin/arch is emitting something other than i386, ppc, or arm64... or maybe the word size detection is breaking and causing the configure script to use the wrong cases.

I have confirmed that i386 is what appears from arch when running under x86_64:

$ arch -x86_64 sh -c arch
i386
jaraco commented 1 year ago

I think I've observed that the configure is failing during a second attempt. Is that because this project first builds Python for the host system, then performs the cross-compilation?

indygreg commented 1 year ago

CPython's configure script is notoriously bad about handling cross-compilation on macOS. It makes a ton of assumptions about Apple targeting. We have to carry a handful of patches to essentially remove all these assumptions. It looks like this arch issue somehow fell through the cracks. We'll need to carry a patch to configure.ac to fix this busted logic.

indygreg commented 1 year ago

The failing code hasn't been unused since 2011. Lol. https://github.com/python/cpython/pull/101048

jaraco commented 1 year ago

I can confirm the 20230116 tag is building now. Thank you, thank you, thank you.