beeware / Python-Apple-support

A meta-package for building a version of Python that can be embedded into a macOS, iOS, tvOS or watchOS project.
MIT License
1.08k stars 155 forks source link

macOS 3.11 not building #180

Closed shakfu closed 1 year ago

shakfu commented 1 year ago

Describe the bug

trying to build a macOS 3.11 beware patched python:

....
link $HOME/Downloads/src/Python-Apple-support/install/macOS/macosx.arm64/openssl-3.0.5/lib/libcrypto.dylib -> $HOME/Downloads/src/Python-Apple-support/install/macOS/macosx.arm64/openssl-3.0.5/lib/libcrypto.3.dylib
link $HOME/Downloads/src/Python-Apple-support/install/macOS/macosx.arm64/openssl-3.0.5/lib/libssl.dylib -> $HOME/Downloads/src/Python-Apple-support/install/macOS/macosx.arm64/openssl-3.0.5/lib/libssl.3.dylib
created directory `$HOME/Downloads/src/Python-Apple-support/install/macOS/macosx.arm64/openssl-3.0.5/lib/pkgconfig'
install libcrypto.pc -> $HOME/Downloads/src/Python-Apple-support/install/macOS/macosx.arm64/openssl-3.0.5/lib/pkgconfig/libcrypto.pc
install libssl.pc -> $HOME/Downloads/src/Python-Apple-support/install/macOS/macosx.arm64/openssl-3.0.5/lib/pkgconfig/libssl.pc
install openssl.pc -> $HOME/Downloads/src/Python-Apple-support/install/macOS/macosx.arm64/openssl-3.0.5/lib/pkgconfig/openssl.pc
/Applications/Xcode.app/Contents/Developer/usr/bin/make depend && /Applications/Xcode.app/Contents/Developer/usr/bin/make _build_modules
make[2]: Nothing to be done for `_build_modules'.
created directory `$HOME/Downloads/src/Python-Apple-support/install/macOS/macosx.arm64/openssl-3.0.5/lib/engines-3'
*** Installing engines
install engines/capi.dylib -> $HOME/Downloads/src/Python-Apple-support/install/macOS/macosx.arm64/openssl-3.0.5/lib/engines-3/capi.dylib
install engines/loader_attic.dylib -> $HOME/Downloads/src/Python-Apple-support/install/macOS/macosx.arm64/openssl-3.0.5/lib/engines-3/loader_attic.dylib
install engines/padlock.dylib -> $HOME/Downloads/src/Python-Apple-support/install/macOS/macosx.arm64/openssl-3.0.5/lib/engines-3/padlock.dylib
created directory `$HOME/Downloads/src/Python-Apple-support/install/macOS/macosx.arm64/openssl-3.0.5/lib/ossl-modules'
*** Installing modules
install providers/legacy.dylib -> $HOME/Downloads/src/Python-Apple-support/install/macOS/macosx.arm64/openssl-3.0.5/lib/ossl-modules/legacy.dylib
/Applications/Xcode.app/Contents/Developer/usr/bin/make depend && /Applications/Xcode.app/Contents/Developer/usr/bin/make _build_programs
make[2]: Nothing to be done for `_build_programs'.
created directory `$HOME/Downloads/src/Python-Apple-support/install/macOS/macosx.arm64/openssl-3.0.5/bin'
*** Installing runtime programs
install apps/openssl -> $HOME/Downloads/src/Python-Apple-support/install/macOS/macosx.arm64/openssl-3.0.5/bin/openssl
install tools/c_rehash -> $HOME/Downloads/src/Python-Apple-support/install/macOS/macosx.arm64/openssl-3.0.5/bin/c_rehash
>>> Copy OpenSSL headers from the first target associated with the SDK
mkdir -p $HOME/Downloads/src/Python-Apple-support/merge/macOS/macosx/openssl-3.0.5
cp -r $HOME/Downloads/src/Python-Apple-support/install/macOS/macosx.x86_64/openssl-3.0.5/include $HOME/Downloads/src/Python-Apple-support/merge/macOS/macosx/openssl-3.0.5
>>> Build OpenSSL ssl fat library for macosx
mkdir -p $HOME/Downloads/src/Python-Apple-support/merge/macOS/macosx/openssl-3.0.5/lib
lipo -create -output $HOME/Downloads/src/Python-Apple-support/merge/macOS/macosx/openssl-3.0.5/lib/libssl.a $HOME/Downloads/src/Python-Apple-support/install/macOS/macosx.x86_64/openssl-3.0.5/lib/libssl.a $HOME/Downloads/src/Python-Apple-support/install/macOS/macosx.arm64/openssl-3.0.5/lib/libssl.a 2>&1 | tee -a merge/macOS/macosx/openssl-3.0.5.ssl.lipo.log
>>> Build OpenSSL crypto fat library for macosx
mkdir -p $HOME/Downloads/src/Python-Apple-support/merge/macOS/macosx/openssl-3.0.5/lib
lipo -create -output $HOME/Downloads/src/Python-Apple-support/merge/macOS/macosx/openssl-3.0.5/lib/libcrypto.a $HOME/Downloads/src/Python-Apple-support/install/macOS/macosx.x86_64/openssl-3.0.5/lib/libcrypto.a $HOME/Downloads/src/Python-Apple-support/install/macOS/macosx.arm64/openssl-3.0.5/lib/libcrypto.a 2>&1 | tee -a merge/macOS/macosx/openssl-3.0.5.crypto.lipo.log
>>> Download Python sources
curl --fail --location --create-dirs --progress-bar -o downloads/Python-3.11.0.tar.gz \
        https://www.python.org/ftp/python/3.11.0/Python-3.11.0.tgz
######################################################################### 100.0%
>>> Unpack and configure Python for macosx
mkdir -p build/macOS/macosx/python-3.11.0
tar zxf downloads/Python-3.11.0.tar.gz --strip-components 1 -C build/macOS/macosx/python-3.11.0
# Apply target Python patches
cd build/macOS/macosx/python-3.11.0 && patch -p1 < $HOME/Downloads/src/Python-Apple-support/patch/Python/Python.patch
patching file 'Lib/_ios_support.py'
patching file 'Lib/ctypes/util.py'
patching file 'Lib/distutils/tests/test_cygwinccompiler.py'
patching file 'Lib/distutils/unixccompiler.py'
patching file 'Lib/distutils/util.py'
patching file 'Lib/importlib/_bootstrap_external.py'
patching file 'Lib/platform.py'
patching file 'Lib/site.py'
patching file 'Lib/subprocess.py'
patching file 'Lib/sysconfig.py'
patching file 'Lib/test/support/__init__.py'
patching file 'Lib/test/test_asyncio/test_events.py'
patching file 'Lib/test/test_asyncio/test_streams.py'
patching file 'Lib/test/test_asyncio/test_unix_events.py'
patching file 'Lib/test/test_fcntl.py'
patching file 'Lib/test/test_httpservers.py'
patching file 'Lib/test/test_io.py'
patching file 'Lib/test/test_logging.py'
patching file 'Lib/test/test_marshal.py'
patching file 'Lib/test/test_mmap.py'
patching file 'Lib/test/test_platform.py'
patching file 'Lib/test/test_posix.py'
patching file 'Lib/test/test_shutil.py'
patching file 'Lib/test/test_socket.py'
patching file 'Lib/test/test_socketserver.py'
patching file 'Lib/test/test_sundry.py'
patching file 'Lib/test/test_sysconfig.py'
patching file 'Lib/test/test_threading.py'
patching file 'Lib/test/test_venv.py'
1 out of 2 hunks failed--saving rejects to 'Lib/test/test_venv.py.rej'
patching file 'Lib/test/test_zipfile.py'
patching file 'Lib/webbrowser.py'
patching file Makefile.pre.in
patching file 'Modules/_posixsubprocess.c'
patching file 'Modules/faulthandler.c'
patching file 'Modules/mathmodule.c'
patching file 'Modules/posixmodule.c'
patching file 'Modules/pwdmodule.c'
patching file 'Modules/timemodule.c'
patching file 'Python/bootstrap_hash.c'
patching file 'Python/marshal.c'
patching file aclocal.m4
patching file config.sub
patching file configure
patching file configure.ac
patching file setup.py
make: *** [build/macOS/macosx/python-3.11.0/Makefile] Error 1

Steps to reproduce

git clone https://github.com/beeware/Python-Apple-support.git
cd Python-Apple-support
make macOS

Expected behavior

To build without errors

Screenshots

No response

Environment

Logs

Additional context

No response

freakboy3742 commented 1 year ago

Based on that stack trace, there's two likely causes:

  1. It's a second build, and the build is dirty, and the patch can't re-apply. If you run make clean-Python and re-run, you might have success.

  2. If you've tried to update the Python version, the script will fail because the patch is very specific to the Python version.

The former is more likely based on the log you've provided, but I can't be completely sure.

shakfu commented 1 year ago

Well, I cloned it afresh and then did a make macOS and got the same error so that rules out (1)

The only thing I can think of is that I'm running python 3.11.2:

% python3 --version
Python 3.11.2
shakfu commented 1 year ago

I tried adjusting the version of python to 3.11.2 in the Makefile and got a similar error:

...
>>> Download Python sources
curl --fail --location --create-dirs --progress-bar -o downloads/Python-3.11.2.tar.gz \
        https://www.python.org/ftp/python/3.11.2/Python-3.11.2.tgz
######################################################################### 100.0%
>>> Unpack and configure Python for macosx
mkdir -p build/macOS/macosx/python-3.11.2
tar zxf downloads/Python-3.11.2.tar.gz --strip-components 1 -C build/macOS/macosx/python-3.11.2
# Apply target Python patches
cd build/macOS/macosx/python-3.11.2 && patch -p1 < $HOME/Downloads/src/Python-Apple-support/patch/Python/Python.patch
patching file 'Lib/_ios_support.py'
patching file 'Lib/ctypes/util.py'
patching file 'Lib/distutils/tests/test_cygwinccompiler.py'
patching file 'Lib/distutils/unixccompiler.py'
patching file 'Lib/distutils/util.py'
patching file 'Lib/importlib/_bootstrap_external.py'
patching file 'Lib/platform.py'
patching file 'Lib/site.py'
patching file 'Lib/subprocess.py'
patching file 'Lib/sysconfig.py'
patching file 'Lib/test/support/__init__.py'
patching file 'Lib/test/test_asyncio/test_events.py'
patching file 'Lib/test/test_asyncio/test_streams.py'
patching file 'Lib/test/test_asyncio/test_unix_events.py'
patching file 'Lib/test/test_fcntl.py'
patching file 'Lib/test/test_httpservers.py'
patching file 'Lib/test/test_io.py'
patching file 'Lib/test/test_logging.py'
patching file 'Lib/test/test_marshal.py'
patching file 'Lib/test/test_mmap.py'
patching file 'Lib/test/test_platform.py'
patching file 'Lib/test/test_posix.py'
patching file 'Lib/test/test_shutil.py'
patching file 'Lib/test/test_socket.py'
patching file 'Lib/test/test_socketserver.py'
patching file 'Lib/test/test_sundry.py'
patching file 'Lib/test/test_sysconfig.py'
patching file 'Lib/test/test_threading.py'
patching file 'Lib/test/test_venv.py'
1 out of 2 hunks failed--saving rejects to 'Lib/test/test_venv.py.rej'
patching file 'Lib/test/test_zipfile.py'
patching file 'Lib/webbrowser.py'
patching file Makefile.pre.in
patching file 'Modules/_posixsubprocess.c'
patching file 'Modules/faulthandler.c'
patching file 'Modules/mathmodule.c'
patching file 'Modules/posixmodule.c'
patching file 'Modules/pwdmodule.c'
patching file 'Modules/timemodule.c'
patching file 'Python/bootstrap_hash.c'
patching file 'Python/marshal.c'
1 out of 2 hunks failed--saving rejects to 'Python/marshal.c.rej'
patching file aclocal.m4
patching file config.sub
patching file configure
patch: **** misordered hunks! output would be garbled
make: *** [build/macOS/macosx/python-3.11.2/Makefile] Error 2
freakboy3742 commented 1 year ago

My apologies - I wasn't suggesting changing the Python version as a fix, but as a potential cause of the problem. If you've changed the Python version and haven't also updated the patch, you're definitely going to hit problems.

One other possible problem - are you running this on the main branch of the repo, or the 3.11 branch? If you're using the main branch, that won't be 100% reliable because the patch won't always align with the release (which would be consistent with the problem you're seeing). If you aren't already on the 3.11 branch, running git checkout 3.11 then make clean && make macOS should work.

shakfu commented 1 year ago

@freakboy3742

One other possible problem - are you running this on the main branch of the repo, or the 3.11 branch? If you're using the main branch, that won't be 100% reliable because the patch won't always align with the release (which would be consistent with the problem you're seeing). If you aren't already on the 3.11 branch, running git checkout 3.11 then make clean && make macOS should work.

This exactly was the problem. After doing a git checkout 3.11 after a fresh clone, the compilation worked without issues.

Thanks for your help!