Python build failed on MacOs Big Sur 11.2.3

asdf-community / asdf-python

Python plugin for the asdf version manager

https://github.com/asdf-vm/asdf

MIT License

651 stars 56 forks source link

Python build failed on MacOs Big Sur 11.2.3 #98

Open smedegaard opened 3 years ago

smedegaard commented 3 years ago

Relates to #93 and #94

After updating to MacOs 11.2.3 I cannot build python.

I've reinstalled all Brew dependencies:

brew reinstall bzip2 lbzip2 lzlib openssl zlib

I've set environment variables, and run asdf install with variables directly:

LDFLAGS="-L/usr/local/opt/bzip2/lib -L/usr/local/opt/zlib/lib -L/usr/local/opt/openssl@1.1/lib" CFLAGS="-I/usr/local/opt/bzip2/include -I/usr/local/opt/zlib/include -I/usr/local/opt/openssl@1.1/include -I$(xcrun --show-sdk-path)/usr/include -Wno-implicit-function-declaration" asdf install python 3.8.8

This is the current result of running with variables;

python-build 3.8.8 /Users/anders/.asdf/installs/python/3.8.8
python-build: use openssl@1.1 from homebrew
python-build: use readline from homebrew
Downloading Python-3.8.8.tar.xz...
-> https://www.python.org/ftp/python/3.8.8/Python-3.8.8.tar.xz
Installing Python-3.8.8...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk

BUILD FAILED (OS X 11.2.3 using python-build 1.2.23-75-g80e418ec)

Inspect or clean up the working tree at /var/folders/tl/06ddx4rj5tj_5h3gpvvtxtbm0000gn/T/python-build.20210317113113.24543
Results logged to /var/folders/tl/06ddx4rj5tj_5h3gpvvtxtbm0000gn/T/python-build.20210317113113.24543.log

Last 10 log lines:
  File "/private/var/folders/tl/06ddx4rj5tj_5h3gpvvtxtbm0000gn/T/python-build.20210317113113.24543/Python-3.8.8/Lib/distutils/command/build_ext.py", line 340, in run
    self.build_extensions()
  File "./setup.py", line 320, in build_extensions
    self.detect_modules()
  File "./setup.py", line 1695, in detect_modules
    self.detect_readline_curses()
  File "./setup.py", line 930, in detect_readline_curses
    (tuple(int(n) for n in dep_target.split('.')[0:2])
AttributeError: 'int' object has no attribute 'split'
make: *** [sharedmods] Error 1

cho-m commented 3 years ago

Looks like Python bug: https://bugs.python.org/issue42504 So, I don't think this is an asdf-python or python-build issue.

It was fixed in both latest Python and at least backported to 3.9. Fix exists in CPython v3.9.2: https://github.com/python/cpython/blob/v3.9.2/Lib/sysconfig.py#L22-L24

Not too sure if they plan to backport to 3.8.

EDIT: @smedegaard if you need Python 3.8 and cannot use 3.9, you may want to try manual patch. Homebrew uses patch file: https://github.com/python/cpython/commit/09a698b4.patch?full_index=1 as seen https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/python@3.8.rb#L74

ASDF supports a single patch file: https://github.com/danhper/asdf-python/blob/8ab052fced78b2c68c8fe17d5811d0ebae06eee8/bin/install#L19

So, maybe try something like

export ASDF_PYTHON_PATCHES_DIRECTORY=/tmp
curl -sSL "https://github.com/python/cpython/commit/09a698b4.patch?full_index=1" > "$ASDF_PYTHON_PATCHES_DIRECTORY/3.8.8.patch"

curtisault commented 3 years ago

I'm having a related problem but for 3.7.3. Here's the information from my log:

python-build 3.7.3 /Users/me/.asdf/installs/python/3.7.3
python-build: use openssl@1.1 from homebrew
python-build: use readline from homebrew
Downloading Python-3.7.3.tar.xz...
-> https://www.python.org/ftp/python/3.7.3/Python-3.7.3.tar.xz
Installing Python-3.7.3...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk

BUILD FAILED (OS X 11.2.3 using python-build 1.2.24.1-6-g132d5464)

Inspect or clean up the working tree at /var/folders/l3/plnsfnc91qd2_3dh7v5b1x180000gn/T/python-build.20210331134326.79346
Results logged to /var/folders/l3/plnsfnc91qd2_3dh7v5b1x180000gn/T/python-build.20210331134326.79346.log

Last 10 log lines:
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include   -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include   -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration   -I. -I./Include -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/Users/me/.asdf/installs/python/3.7.3/include  -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/Users/me/.asdf/installs/python/3.7.3/include   -DPy_BUILD_CORE_BUILTIN  -c ./Modules/errnomodule.c -o Modules/errnomodule.o
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include   -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include   -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration   -I. -I./Include -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/Users/me/.asdf/installs/python/3.7.3/include  -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/Users/me/.asdf/installs/python/3.7.3/include   -DPy_BUILD_CORE_BUILTIN  -c ./Modules/pwdmodule.c -o Modules/pwdmodule.o
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include   -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include   -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration   -I. -I./Include -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/Users/me/.asdf/installs/python/3.7.3/include  -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/Users/me/.asdf/installs/python/3.7.3/include   -DPy_BUILD_CORE_BUILTIN  -c ./Modules/_sre.c -o Modules/_sre.o
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include   -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include   -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration   -I. -I./Include -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/Users/me/.asdf/installs/python/3.7.3/include  -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/Users/me/.asdf/installs/python/3.7.3/include   -DPy_BUILD_CORE_BUILTIN  -c ./Modules/_codecsmodule.c -o Modules/_codecsmodule.o
./Modules/posixmodule.c:8431:15: error: implicit declaration of function 'sendfile' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
        ret = sendfile(in, out, offset, &sbytes, &sf, flags);
              ^
1 error generated.
make: *** [Modules/posixmodule.o] Error 1
make: *** Waiting for unfinished jobs....

cho-m commented 3 years ago

@curtisault The error looks similar to https://bugs.python.org/issue43470, which was closed (i.e. won't fix)

From response,

While the particular error you ran into is easily worked around, the resulting Python will still not execute correctly on macOS 11 Big Sur as there are a number of other run-time issues that would need to be fixed. At the moment, only Python 3.9.2 is fully supported on Big Sur.

Looking at Python status at https://devguide.python.org/#status-of-python-branches,

Python 3.7 will only accept security fixes, so it probably will never work on Big Sur unless you manually patch it.
Python 3.8 accepts bugfixes, but it is still up to release manager to decide if they want to backport fixes for Big Sur.
Python 3.9 is current stable, so it works on Big Sur

EDIT: if you seriously need an older version of Python and don't care about the fact that Python devs consider it unsupported, then you can one of following:

try hunting down alternative libraries. Try the Homebrew versions like shown in first post (or https://github.com/danhper/asdf-python/issues/93#issuecomment-791737833) and work from there. You may also have to hunt down patches to apply during build.
you can always try a Pre-Built. As asdf-python needs to build Python using python-build, it probably doesn't support this. If you are on Intel Mac, you can give Homebrew's python@3.7 formula or MacPort's python37 a try

Also, see if you can use a newer minor version. I have been able to build v3.7.8+ locally with XCode 12.4 and Homebrew's readline/openssl on M1 Rosetta2 (v3.7.7 and older had similar issue).

ev-dev commented 3 years ago

I believe this issue may not be limited to prior versions of python, I'm also running into similar python-build errors while trying to install the latest v3.9.4 (also MacOS 11.2.3).

I'm trying to understand more what the problem is, but I believe there may be some compat issues goinig on with my xcode tooling. I've tried to do the install using latest xcode v12 and beta v12.5 via xcode-select --switch but no difference.

Last 10 log lines:

checking for --with-universal-archs... no
checking MACHDEP... "darwin"
checking for gcc... clang
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... configure: error: in '/var/folders/01/197gjn9100312t8k_61655tr0000gn/T/python-build.20210405194509.72821/Python-3.9.4':
configure: error: cannot run C compiled programs.
If you meant to cross compile, use '--host'.
See 'config.log' for more details

I wish I could provide more details; does anyone know how to access the config.log this refers too?

ev-dev commented 3 years ago

I'm pretty sure this is not relevant but during asdf's install via python-build it shows that while it's recognizes the readline dep installed via homebrew, it uses zlib from my xcode sdk instead of homebrews.

Both of these homebrew packages are installed and relevant env vars are global in my shell (LDFLAGS/CPPFLAGS/PKG_CONFIG_PATH), is there any way to force python-build to use dependencies installed from Homebrew's (like zlib)?

cho-m commented 3 years ago

I believe this issue may not be limited to prior versions of python, I'm also running into similar python-build errors while trying to install the latest v3.9.4 (also MacOS 11.2.3).

@ev-dev there are many reasons why a build/compilation can fail. Each failure will often need to be analyzed separately for root cause. Based on glance, I don't see a reason in your build errors to assume it is similar to previously discussed issues. Not sure what exact reason for failure is though.

The other failures shown in this issue have known upstream bugs in CPython that were fixed in later versions of python (or alternatively worked-around by linking other libraries / applying patches).

Your failure seems to be happening during CPython's configure script that should create Makefile: https://github.com/python/cpython/blob/v3.9.4/configure#L4067

There was at least one person who saw similar issue for python-build of 3.9.2: https://github.com/pyenv/pyenv/issues/1768#issuecomment-808604291

Someone else will need to comment if log file is still available. If log was stored inside install path, then it may be deleted during ASDF failure cleanup: https://github.com/asdf-vm/asdf/blob/b60c08acb2d7b39159eae7dfa2622289537a5304/lib/commands/command-install.bash#L5

EDIT: You might be able to find log inside directory in log lines you posted: /var/folders/01/197gjn9100312t8k_61655tr0000gn/T/python-build.20210405194509.72821/Python-3.9.4

install via python-build it shows that while it's recognizes the readline dep installed via homebrew, it uses zlib from my xcode sdk instead of homebrews.

In the case of zlib, python-build is coded to prioritize XCode zlib on macOS/OSX 10.14+: https://github.com/pyenv/pyenv/blob/4f93366be8060e2b096e8136ffeedf9cbd050bd8/plugins/python-build/bin/python-build#L771

I don't think this would be an issue.

There are many factors that can impact a build.

In my case, I am able to build both 3.9.2 & 3.9.4 on M1 (arm64) macOS 11.2.3 using stable XCode 12.4 without exporting any flags and python-build automatically using Homebrew's readline & openssl.

It is hard to tell the exact state of your environment/setup.

You can try rerunning XCode CLI install command to make sure header files are all available with xcode-select --install

Otherwise, you might want to try seeing if issue occurs on pyenv and report issue there since asdf-python is just wrapping around pyenv's python-build so ASDF is often too high of a layer to analyze failures. Dropping down to python-build or even lower to direct Python source code would give more details.

smedegaard commented 3 years ago

Looks like Python bug: https://bugs.python.org/issue42504 So, I don't think this is an asdf-python or python-build issue.

It was fixed in both latest Python and at least backported to 3.9. Fix exists in CPython v3.9.2: https://github.com/python/cpython/blob/v3.9.2/Lib/sysconfig.py#L22-L24

Not too sure if they plan to backport to 3.8.

EDIT: @smedegaard if you need Python 3.8 and cannot use 3.9, you may want to try manual patch. Homebrew uses patch file: https://github.com/python/cpython/commit/09a698b4.patch?full_index=1 as seen https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/python@3.8.rb#L74

ASDF supports a single patch file:

https://github.com/danhper/asdf-python/blob/8ab052fced78b2c68c8fe17d5811d0ebae06eee8/bin/install#L19

So, maybe try something like
export ASDF_PYTHON_PATCHES_DIRECTORY=/tmp
curl -sSL "https://github.com/python/cpython/commit/09a698b4.patch?full_index=1" > "$ASDF_PYTHON_PATCHES_DIRECTORY/3.8.8.patch"

Thanks, I needed 3.8.

I'll test again at a later point in time

ev-dev commented 3 years ago

Hey @cho-m really appreciate all the details, was looking forward to grasping the install process more and you provided lots of areas of research for me so thank you!

First, I just wanted to clarify the recommended pre-reqs for using asdf python plugin:

Python build tools (ie. https://github.com/pyenv/pyenv/wiki#suggested-build-environment)
Existing/functionalpyenv installed on system as a prerequisite? If yes:
- Are the most common package managers per major OS the suggested resource to install pyenv like brew on Mac & apt on Ubuntu/Deb?
- Is it best to have a bare-install of pyenv (no existing configuration of the tool) when you want asdf to manage your default (first in $PATH) python dist?
Other pre-installed tooling? (within reasonable scope, sorry if this one is too big a rabbit hole)

I may try to submit a small update to the readme on this detail in pursuit of providing others with a bit more definitive info like is found in some other popular asdf plugin repos.

cho-m commented 3 years ago

@ev-dev

Python build tools (ie. https://github.com/pyenv/pyenv/wiki#suggested-build-environment)

You probably should be fine with those. The exact packages can vary based on what libraries you want to have available (which may require additional CFLAGS/CPPFLAGS/LDFLAGS).

The essential build dependencies on macOS are from Xcode Command Line Tools as required by official docs.

For libraries that are automatically detected by python-build (no need to add flags):

Homebrew openssl is good to have. If not installed, python-build will use a fixed version, so security issues may not be patched.
Homebrew readline is also good to have. Probably also defaults to fixed version.
For zlib, as previously mentioned, on newer macOS building newer Python versions the existing zlib is usually fine. This is also what is used by Homebrew.
- NOTE: For older Python versions on newer macOS, some people have had to use Homebrew zlib/bzip2.

For other packages listed in link:

Homebrew xz should add libzma library (discussed in Python docs) and can also be used during decompressing source code (e.g. Python-3.9.4.tar.xz).
For sqlite, I don't think it is used by default. May be useful if building Python with --enable-loadable-sqlite-extensions as it seems to be missing in default macOS version.

Existing/functionalpyenv installed on system as a prerequisite? If yes:

No. asdf-python automatically pulls a copy of pyenv into "${ASDF_DATA_DIR:-$HOME/.asdf}/plugins/python/pyenv"

It really doesn't use pyenv, but the underlying python-build. Sadly, they didn't split off repo unlike rbenv and underlying ruby-build.

Other pre-installed tooling? (within reasonable scope, sorry if this one is too big a rabbit hole)

In general, not really much else once you have working ASDF install and XCode Tools.

Most of extra dependencies are C libraries that are used by some modules in CPython.

katsusuke commented 2 years ago

macOS BigSur 11.6 (20G165)

$ asdf version
v0.8.1-fe5b71d

I have successfully installed it with the following command.

CFLAGS="-I$(brew --prefix openssl)/include -I$(brew --prefix bzip2)/include -I$(brew --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include" LDFLAGS="-L$(brew --prefix openssl)/lib -L$(brew --prefix readline)/lib -L$(brew --prefix zlib)/lib -L$(brew --prefix bzip2)/lib" ASDF_PYTHON_PATCH_URL=https://github.com/python/cpython/commit/8ea6353.patch\?full_index\=1 asdf install python 3.7.3

s3rgeym commented 2 years ago

macOS BigSur 11.6 (20G165)

$ asdf version
v0.8.1-fe5b71d

I have successfully installed it with the following command.

CFLAGS="-I$(brew --prefix openssl)/include -I$(brew --prefix bzip2)/include -I$(brew --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include" LDFLAGS="-L$(brew --prefix openssl)/lib -L$(brew --prefix readline)/lib -L$(brew --prefix zlib)/lib -L$(brew --prefix bzip2)/lib" ASDF_PYTHON_PATCH_URL=https://github.com/python/cpython/commit/8ea6353.patch\?full_index\=1 asdf install python 3.7.3

too hard

typkrft commented 2 years ago

I had this warning, using 3.10.0 on Apple Silicon. I don't want to diverge too the conversation as I was on Monterey. But after the installing the latest version of Command Line Tools, it seems that uninstalling python through asdf and reinstalling resolved the issue for me.