Open smedegaard opened 3 years ago
Looks like Python bug: https://bugs.python.org/issue42504
So, I don't think this is an asdf-python
or python-build
issue.
It was fixed in both latest Python and at least backported to 3.9. Fix exists in CPython v3.9.2: https://github.com/python/cpython/blob/v3.9.2/Lib/sysconfig.py#L22-L24
Not too sure if they plan to backport to 3.8.
EDIT: @smedegaard if you need Python 3.8 and cannot use 3.9, you may want to try manual patch. Homebrew uses patch file: https://github.com/python/cpython/commit/09a698b4.patch?full_index=1 as seen https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/python@3.8.rb#L74
ASDF supports a single patch file: https://github.com/danhper/asdf-python/blob/8ab052fced78b2c68c8fe17d5811d0ebae06eee8/bin/install#L19
So, maybe try something like
export ASDF_PYTHON_PATCHES_DIRECTORY=/tmp
curl -sSL "https://github.com/python/cpython/commit/09a698b4.patch?full_index=1" > "$ASDF_PYTHON_PATCHES_DIRECTORY/3.8.8.patch"
I'm having a related problem but for 3.7.3. Here's the information from my log:
python-build 3.7.3 /Users/me/.asdf/installs/python/3.7.3
python-build: use openssl@1.1 from homebrew
python-build: use readline from homebrew
Downloading Python-3.7.3.tar.xz...
-> https://www.python.org/ftp/python/3.7.3/Python-3.7.3.tar.xz
Installing Python-3.7.3...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
BUILD FAILED (OS X 11.2.3 using python-build 1.2.24.1-6-g132d5464)
Inspect or clean up the working tree at /var/folders/l3/plnsfnc91qd2_3dh7v5b1x180000gn/T/python-build.20210331134326.79346
Results logged to /var/folders/l3/plnsfnc91qd2_3dh7v5b1x180000gn/T/python-build.20210331134326.79346.log
Last 10 log lines:
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -I. -I./Include -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/Users/me/.asdf/installs/python/3.7.3/include -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/Users/me/.asdf/installs/python/3.7.3/include -DPy_BUILD_CORE_BUILTIN -c ./Modules/errnomodule.c -o Modules/errnomodule.o
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -I. -I./Include -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/Users/me/.asdf/installs/python/3.7.3/include -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/Users/me/.asdf/installs/python/3.7.3/include -DPy_BUILD_CORE_BUILTIN -c ./Modules/pwdmodule.c -o Modules/pwdmodule.o
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -I. -I./Include -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/Users/me/.asdf/installs/python/3.7.3/include -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/Users/me/.asdf/installs/python/3.7.3/include -DPy_BUILD_CORE_BUILTIN -c ./Modules/_sre.c -o Modules/_sre.o
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -I. -I./Include -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/Users/me/.asdf/installs/python/3.7.3/include -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/Users/me/.asdf/installs/python/3.7.3/include -DPy_BUILD_CORE_BUILTIN -c ./Modules/_codecsmodule.c -o Modules/_codecsmodule.o
./Modules/posixmodule.c:8431:15: error: implicit declaration of function 'sendfile' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
ret = sendfile(in, out, offset, &sbytes, &sf, flags);
^
1 error generated.
make: *** [Modules/posixmodule.o] Error 1
make: *** Waiting for unfinished jobs....
@curtisault The error looks similar to https://bugs.python.org/issue43470, which was closed (i.e. won't fix)
From response,
While the particular error you ran into is easily worked around, the resulting Python will still not execute correctly on macOS 11 Big Sur as there are a number of other run-time issues that would need to be fixed. At the moment, only Python 3.9.2 is fully supported on Big Sur.
Looking at Python status at https://devguide.python.org/#status-of-python-branches,
EDIT: if you seriously need an older version of Python and don't care about the fact that Python devs consider it unsupported, then you can one of following:
asdf-python
needs to build Python using python-build
, it probably doesn't support this. If you are on Intel Mac, you can give Homebrew's python@3.7
formula or MacPort's python37
a tryAlso, see if you can use a newer minor version. I have been able to build v3.7.8+ locally with XCode 12.4 and Homebrew's readline/openssl on M1 Rosetta2 (v3.7.7 and older had similar issue).
I believe this issue may not be limited to prior versions of python, I'm also running into similar python-build
errors while trying to install the latest v3.9.4 (also MacOS 11.2.3).
I'm trying to understand more what the problem is, but I believe there may be some compat issues goinig on with my xcode tooling. I've tried to do the install using latest xcode v12 and beta v12.5 via xcode-select --switch
but no difference.
Last 10 log lines:
checking for --with-universal-archs... no
checking MACHDEP... "darwin"
checking for gcc... clang
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... configure: error: in '/var/folders/01/197gjn9100312t8k_61655tr0000gn/T/python-build.20210405194509.72821/Python-3.9.4':
configure: error: cannot run C compiled programs.
If you meant to cross compile, use '--host'.
See 'config.log' for more details
I wish I could provide more details; does anyone know how to access the config.log this refers too?
I'm pretty sure this is not relevant but during asdf's install via python-build
it shows that while it's recognizes the readline
dep installed via homebrew, it uses zlib
from my xcode sdk instead of homebrews.
Both of these homebrew packages are installed and relevant env vars are global in my shell (LDFLAGS/CPPFLAGS/PKG_CONFIG_PATH), is there any way to force python-build
to use dependencies installed from Homebrew's (like zlib)?
I believe this issue may not be limited to prior versions of python, I'm also running into similar
python-build
errors while trying to install the latest v3.9.4 (also MacOS 11.2.3).
@ev-dev there are many reasons why a build/compilation can fail. Each failure will often need to be analyzed separately for root cause. Based on glance, I don't see a reason in your build errors to assume it is similar to previously discussed issues. Not sure what exact reason for failure is though.
The other failures shown in this issue have known upstream bugs in CPython that were fixed in later versions of python (or alternatively worked-around by linking other libraries / applying patches).
Your failure seems to be happening during CPython's configure
script that should create Makefile: https://github.com/python/cpython/blob/v3.9.4/configure#L4067
There was at least one person who saw similar issue for python-build of 3.9.2: https://github.com/pyenv/pyenv/issues/1768#issuecomment-808604291
Someone else will need to comment if log file is still available. If log was stored inside install path, then it may be deleted during ASDF failure cleanup: https://github.com/asdf-vm/asdf/blob/b60c08acb2d7b39159eae7dfa2622289537a5304/lib/commands/command-install.bash#L5
EDIT: You might be able to find log inside directory in log lines you posted:
/var/folders/01/197gjn9100312t8k_61655tr0000gn/T/python-build.20210405194509.72821/Python-3.9.4
install via
python-build
it shows that while it's recognizes thereadline
dep installed via homebrew, it useszlib
from my xcode sdk instead of homebrews.
In the case of zlib
, python-build is coded to prioritize XCode zlib on macOS/OSX 10.14+:
https://github.com/pyenv/pyenv/blob/4f93366be8060e2b096e8136ffeedf9cbd050bd8/plugins/python-build/bin/python-build#L771
I don't think this would be an issue.
There are many factors that can impact a build.
In my case, I am able to build both 3.9.2 & 3.9.4 on M1 (arm64) macOS 11.2.3 using stable XCode 12.4 without exporting any flags and python-build
automatically using Homebrew's readline & openssl.
It is hard to tell the exact state of your environment/setup.
You can try rerunning XCode CLI install command to make sure header files are all available with xcode-select --install
Otherwise, you might want to try seeing if issue occurs on pyenv
and report issue there since asdf-python
is just wrapping around pyenv
's python-build
so ASDF is often too high of a layer to analyze failures. Dropping down to python-build or even lower to direct Python source code would give more details.
Looks like Python bug: https://bugs.python.org/issue42504 So, I don't think this is an
asdf-python
orpython-build
issue.It was fixed in both latest Python and at least backported to 3.9. Fix exists in CPython v3.9.2: https://github.com/python/cpython/blob/v3.9.2/Lib/sysconfig.py#L22-L24
Not too sure if they plan to backport to 3.8.
EDIT: @smedegaard if you need Python 3.8 and cannot use 3.9, you may want to try manual patch. Homebrew uses patch file: https://github.com/python/cpython/commit/09a698b4.patch?full_index=1 as seen https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/python@3.8.rb#L74
ASDF supports a single patch file:
https://github.com/danhper/asdf-python/blob/8ab052fced78b2c68c8fe17d5811d0ebae06eee8/bin/install#L19
So, maybe try something like
export ASDF_PYTHON_PATCHES_DIRECTORY=/tmp curl -sSL "https://github.com/python/cpython/commit/09a698b4.patch?full_index=1" > "$ASDF_PYTHON_PATCHES_DIRECTORY/3.8.8.patch"
Thanks, I needed 3.8.
I'll test again at a later point in time
Hey @cho-m really appreciate all the details, was looking forward to grasping the install process more and you provided lots of areas of research for me so thank you!
First, I just wanted to clarify the recommended pre-reqs for using asdf python plugin:
pyenv
installed on system as a prerequisite? If yes:
pyenv
like brew on Mac & apt on Ubuntu/Deb? pyenv
(no existing configuration of the tool) when you want asdf
to manage your default (first in $PATH) python dist? I may try to submit a small update to the readme on this detail in pursuit of providing others with a bit more definitive info like is found in some other popular asdf plugin repos.
@ev-dev
- Python build tools (ie. https://github.com/pyenv/pyenv/wiki#suggested-build-environment)
You probably should be fine with those. The exact packages can vary based on what libraries you want to have available (which may require additional CFLAGS/CPPFLAGS/LDFLAGS).
The essential build dependencies on macOS are from Xcode Command Line Tools as required by official docs.
For libraries that are automatically detected by python-build
(no need to add flags):
openssl
is good to have. If not installed, python-build
will use a fixed version, so security issues may not be patched.readline
is also good to have. Probably also defaults to fixed version.zlib
, as previously mentioned, on newer macOS building newer Python versions the existing zlib is usually fine. This is also what is used by Homebrew.
zlib
/bzip2
.For other packages listed in link:
xz
should add libzma
library (discussed in Python docs) and can also be used during decompressing source code (e.g. Python-3.9.4.tar.xz).sqlite
, I don't think it is used by default. May be useful if building Python with --enable-loadable-sqlite-extensions
as it seems to be missing in default macOS version.
- Existing/functional
pyenv
installed on system as a prerequisite? If yes:
No.
asdf-python
automatically pulls a copy of pyenv into "${ASDF_DATA_DIR:-$HOME/.asdf}/plugins/python/pyenv"
It really doesn't use pyenv
, but the underlying python-build
.
Sadly, they didn't split off repo unlike rbenv
and underlying ruby-build
.
- Other pre-installed tooling? (within reasonable scope, sorry if this one is too big a rabbit hole)
In general, not really much else once you have working ASDF install and XCode Tools.
Most of extra dependencies are C libraries that are used by some modules in CPython.
macOS BigSur 11.6 (20G165)
$ asdf version
v0.8.1-fe5b71d
I have successfully installed it with the following command.
CFLAGS="-I$(brew --prefix openssl)/include -I$(brew --prefix bzip2)/include -I$(brew --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include" LDFLAGS="-L$(brew --prefix openssl)/lib -L$(brew --prefix readline)/lib -L$(brew --prefix zlib)/lib -L$(brew --prefix bzip2)/lib" ASDF_PYTHON_PATCH_URL=https://github.com/python/cpython/commit/8ea6353.patch\?full_index\=1 asdf install python 3.7.3
macOS BigSur 11.6 (20G165)
$ asdf version v0.8.1-fe5b71d
I have successfully installed it with the following command.
CFLAGS="-I$(brew --prefix openssl)/include -I$(brew --prefix bzip2)/include -I$(brew --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include" LDFLAGS="-L$(brew --prefix openssl)/lib -L$(brew --prefix readline)/lib -L$(brew --prefix zlib)/lib -L$(brew --prefix bzip2)/lib" ASDF_PYTHON_PATCH_URL=https://github.com/python/cpython/commit/8ea6353.patch\?full_index\=1 asdf install python 3.7.3
too hard
I had this warning, using 3.10.0 on Apple Silicon. I don't want to diverge too the conversation as I was on Monterey. But after the installing the latest version of Command Line Tools, it seems that uninstalling python through asdf and reinstalling resolved the issue for me.
Relates to #93 and #94
After updating to MacOs 11.2.3 I cannot build python.
I've reinstalled all Brew dependencies:
brew reinstall bzip2 lbzip2 lzlib openssl zlib
I've set environment variables, and run
asdf install
with variables directly:This is the current result of running with variables;