Closed punowo closed 6 months ago
I have the same problem. I installed pyenv
with git clone
.
It seems there is a similar problem with Mac M1.
The suggestions there didn't helped me...
I can install every other version without problems...
I have the same problem. I installed
pyenv
withgit clone
. It seems there is a similar problem with Mac M1. The suggestions there didn't helped me... I can install every other version without problems...
pyenv install 3.11
Downloading Python-3.11.6.tar.xz...
-> https://www.python.org/ftp/python/3.11.6/Python-3.11.6.tar.xz
Installing Python-3.11.6...
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/pun/.pyenv/versions/3.11.6/lib/python3.11/curses/__init__.py", line 13, in <module>
from _curses import *
ModuleNotFoundError: No module named '_curses'
WARNING: The Python curses extension was not compiled. Missing the ncurses lib?
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/pun/.pyenv/versions/3.11.6/lib/python3.11/ctypes/__init__.py", line 8, in <module>
from _ctypes import Union, Structure, Array
ModuleNotFoundError: No module named '_ctypes'
WARNING: The Python ctypes extension was not compiled. Missing the libffi lib?
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'readline'
WARNING: The Python readline extension was not compiled. Missing the GNU readline lib?
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/pun/.pyenv/versions/3.11.6/lib/python3.11/tkinter/__init__.py", line 38, in <module>
import _tkinter # If this fails your Python may not be configured for Tk
^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named '_tkinter'
WARNING: The Python tkinter extension was not compiled and GUI subsystem has been detected. Missing the Tk toolkit?
Installed Python-3.11.6 to /home/pun/.pyenv/versions/3.11.6
You're right. I seem to have some warnings but other versions of python seem to work.
@punowo this is the reason:
checking for pkg-config... /home/linuxbrew/.linuxbrew/bin/pkg-config
Homebrew installs pkg-config
under a name that shadows the system's one.
This stops one from being able to build software using the system toolchain -- because the flags it produces only work out of the box if using a Homebrew-provided compiler.
Pyenv doesn't use packages from Homebrew in Linux by default. That's because Homebrew is more problematic to use than distro-provided packages. Unlike most distros, Homebrew is rolling-release -- and as such, its dependency package upgrades break existing Pyenv-provided installations whenever their .so
names change.
To build with the system's toolchain,
brew unlink
Homebrew-provided pkg-config
; orPATH
To build with Homebrew's toolchain,
export CC=<compiler executable to use>
@native-api Thank you for your explanation. I understand what is happening but I'm not really sure on how to deal with it. I'll give a brief explanation as to what I'm trying to do. I use homebrew on my Mac to handle pretty much everything package related, so for this machine I'm currently encountering this type of problem I wanted to have the baseline offered by Ubuntu and some other stuff, like go, fnm ( for node ), pyenv ( for python ) handled by homebrew similarly to how I handle it on my Mac. I opted for installing pyenv using homebrew since it will automatically get updated as I update other homebrew packages.
Correct me if I'm wrong but unlinking pkg-config would be a temporary solution right, same goes for using a homebrew provided compiler ( my current mental model as a non C/C++ dev is that in case some component is provided by Ubuntu, but was not installed by homebrew I'm going to have the same issue ) ? Is it better for my use case to not use homebrew at all for pyenv, or better remove homebrew from PATH during it's runtime ?
Sorry but I couldn't attach this log, I uploaded here: https://0x0.st/HJAk.log
The command I executed was:
CC=/home/linuxbrew/.linuxbrew/bin/gcc-13 pyenv install 3.12
Downloading Python-3.12.0.tar.xz...
-> https://www.python.org/ftp/python/3.12.0/Python-3.12.0.tar.xz
Installing Python-3.12.0...
BUILD FAILED (Ubuntu 23.10 using python-build 20180424)
Inspect or clean up the working tree at /tmp/python-build.20231018212653.4871
Results logged to /tmp/python-build.20231018212653.4871.log
Last 10 log lines:
/usr/bin/install -c -m 755 Modules/_lzma.cpython-312-x86_64-linux-gnu.so /home/pun/.pyenv/versions/3.12.0/lib/python3.12/lib-dynload/_lzma.cpython-312-x86_64-linux-gnu.so
/usr/bin/install -c -m 755 Modules/zlib.cpython-312-x86_64-linux-gnu.so /home/pun/.pyenv/versions/3.12.0/lib/python3.12/lib-dynload/zlib.cpython-312-x86_64-linux-gnu.so
/usr/bin/install -c -m 755 Modules/readline.cpython-312-x86_64-linux-gnu.so /home/pun/.pyenv/versions/3.12.0/lib/python3.12/lib-dynload/readline.cpython-312-x86_64-linux-gnu.so
/usr/bin/install -c -m 755 Modules/_md5.cpython-312-x86_64-linux-gnu.so /home/pun/.pyenv/versions/3.12.0/lib/python3.12/lib-dynload/_md5.cpython-312-x86_64-linux-gnu.so
/usr/bin/install -c -m 755 Modules/_sha1.cpython-312-x86_64-linux-gnu.so /home/pun/.pyenv/versions/3.12.0/lib/python3.12/lib-dynload/_sha1.cpython-312-x86_64-linux-gnu.so
/usr/bin/install -c -m 755 Modules/_sha2.cpython-312-x86_64-linux-gnu.so /home/pun/.pyenv/versions/3.12.0/lib/python3.12/lib-dynload/_sha2.cpython-312-x86_64-linux-gnu.so
/usr/bin/install -c -m 755 Modules/_sha3.cpython-312-x86_64-linux-gnu.so /home/pun/.pyenv/versions/3.12.0/lib/python3.12/lib-dynload/_sha3.cpython-312-x86_64-linux-gnu.so
/usr/bin/install -c -m 755 Modules/_blake2.cpython-312-x86_64-linux-gnu.so /home/pun/.pyenv/versions/3.12.0/lib/python3.12/lib-dynload/_blake2.cpython-312-x86_64-linux-gnu.so
/usr/bin/install: cannot stat 'Modules/_blake2.cpython-312-x86_64-linux-gnu.so': No such file or directory
make: *** [Makefile:2074: sharedinstall] Error 1
[ERROR] _blake2 failed to import: libb2.so.1: cannot open shared object file: No such file or directory
[ERROR] _crypt failed to import: libcrypt.so.2: cannot open shared object file: No such file or directory
[ERROR] nis failed to import: libnsl.so.3: cannot open shared object file: No such file or directory
This looks like https://github.com/pyenv/pyenv/issues/2693#issuecomment-1531613792 (cited below).
The compiler uses symlinks libxxx.so -> libxxx.so.<version>
to decide which specific version to link against when none is explicitly specified (which is the typical case; to link against a specific version, -l:libxxx.so.<version>
needs to be used instead of -lxxx
which I've never seen in the wild so far).
Check if you have invalid symlinks named libssl.so on your system pointing to a nonexistent "libssl.so.1.1".
You have OpenSSL 3 installed but the build tries to link against OpenSSL 1.1. The above can be the cause.
If one is in /usr/lib, fix it to point to the corresponding OpenSSL 3 file (reinstalling the OpenSSL package may do this for you). If it's in /usr/local/lib, delete it (this most likely means you have leftovers from a source installation of OpenSSL 1.1).
To build with the system's toolchain,
* uninstall or at least `brew unlink` Homebrew-provided `pkg-config`; or * remove Homebrew from `PATH`
removing Homebrew from PATH
worked for me. Ty
[ERROR] _blake2 failed to import: libb2.so.1: cannot open shared object file: No such file or directory [ERROR] _crypt failed to import: libcrypt.so.2: cannot open shared object file: No such file or directory [ERROR] nis failed to import: libnsl.so.3: cannot open shared object file: No such file or directory
This looks like #2693 (comment) (cited below). The compiler uses symlinks
libxxx.so -> libxxx.so.<version>
to decide which specific version to link against when none is explicitly specified (which is the typical case; to link against a specific version,-l:libxxx.so.<version>
needs to be used instead of-lxxx
which I've never seen in the wild so far).Check if you have invalid symlinks named libssl.so on your system pointing to a nonexistent "libssl.so.1.1". You have OpenSSL 3 installed but the build tries to link against OpenSSL 1.1. The above can be the cause. If one is in /usr/lib, fix it to point to the corresponding OpenSSL 3 file (reinstalling the OpenSSL package may do this for you). If it's in /usr/local/lib, delete it (this most likely means you have leftovers from a source installation of OpenSSL 1.1).
pun@owo:/home/linuxbrew$ fd -H "libcrypt.so"
.linuxbrew/Cellar/libxcrypt/4.4.36/lib/libcrypt.so
.linuxbrew/Cellar/libxcrypt/4.4.36/lib/libcrypt.so.2
.linuxbrew/Cellar/libxcrypt/4.4.36/lib/libcrypt.so.2.0.0
.linuxbrew/lib/libcrypt.so
.linuxbrew/lib/libcrypt.so.2
.linuxbrew/lib/libcrypt.so.2.0.0
pun@owo:/home/linuxbrew$ fd -H "libnsl.so"
.linuxbrew/Cellar/libnsl/2.0.0_1/lib/libnsl.so
.linuxbrew/Cellar/libnsl/2.0.0_1/lib/libnsl.so.3
.linuxbrew/Cellar/libnsl/2.0.0_1/lib/libnsl.so.3.0.0
.linuxbrew/lib/libnsl.so
.linuxbrew/lib/libnsl.so.3
.linuxbrew/lib/libnsl.so.3.0.0
pun@owo:/home/linuxbrew$ fd -H "libb2"
.linuxbrew/Cellar/libb2/
.linuxbrew/Cellar/libb2/0.98.1/.brew/libb2.rb
.linuxbrew/Cellar/libb2/0.98.1/lib/libb2.a
.linuxbrew/Cellar/libb2/0.98.1/lib/libb2.so
.linuxbrew/Cellar/libb2/0.98.1/lib/libb2.so.1
.linuxbrew/Cellar/libb2/0.98.1/lib/libb2.so.1.0.4
.linuxbrew/Cellar/libb2/0.98.1/lib/pkgconfig/libb2.pc
.linuxbrew/lib/libb2.a
.linuxbrew/lib/libb2.so
.linuxbrew/lib/libb2.so.1
.linuxbrew/lib/libb2.so.1.0.4
.linuxbrew/lib/pkgconfig/libb2.pc
.linuxbrew/opt/libb2
.linuxbrew/var/homebrew/linked/libb2
The following are not symlinks:
.linuxbrew/Cellar/libxcrypt/4.4.36/lib/libcrypt.so.2.0.0
.linuxbrew/Cellar/libnsl/2.0.0_1/lib/libnsl.so.3.0.0
.linuxbrew/Cellar/libb2/0.98.1/lib/libb2.so.1.0.4
The command should be something like this right ?
pun@owo:/$ CC=/home/linuxbrew/.linuxbrew/bin/gcc-13 LDFLAGS="-l:libb2.so.1.0.4 -l:libnsl.so.3.0.0 -l:libcrypt.so.2.0.0" pyenv install 3.12
Downloading Python-3.12.0.tar.xz...
-> https://www.python.org/ftp/python/3.12.0/Python-3.12.0.tar.xz
Installing Python-3.12.0...
BUILD FAILED (Ubuntu 23.10 using python-build 20180424)
Inspect or clean up the working tree at /tmp/python-build.20231019101334.19932
Results logged to /tmp/python-build.20231019101334.19932.log
Last 10 log lines:
checking for pkg-config... /home/linuxbrew/.linuxbrew/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for --enable-universalsdk... no
checking for --with-universal-archs... no
checking MACHDEP... "linux"
checking for gcc... /home/linuxbrew/.linuxbrew/bin/gcc-13
checking whether the C compiler works... no
configure: error: in `/tmp/python-build.20231019101334.19932/Python-3.12.0':
configure: error: C compiler cannot create executables
See `config.log' for more details
This worked:
~> CC=/home/linuxbrew/.linuxbrew/bin/gcc-13 CPPFLAGS="-I$(brew --prefix)/include" LDFLAGS="-L$(brew --prefix)/lib" pyenv install 3.12
Downloading Python-3.12.0.tar.xz...
-> https://www.python.org/ftp/python/3.12.0/Python-3.12.0.tar.xz
Installing Python-3.12.0...
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/pun/.pyenv/versions/3.12.0/lib/python3.12/sqlite3/__init__.py", line 57, in <module>
from sqlite3.dbapi2 import *
File "/home/pun/.pyenv/versions/3.12.0/lib/python3.12/sqlite3/dbapi2.py", line 27, in <module>
from _sqlite3 import *
ModuleNotFoundError: No module named '_sqlite3'
WARNING: The Python sqlite3 extension was not compiled. Missing the SQLite3 lib?
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/pun/.pyenv/versions/3.12.0/lib/python3.12/tkinter/__init__.py", line 38, in <module>
import _tkinter # If this fails your Python may not be configured for Tk
^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named '_tkinter'
WARNING: The Python tkinter extension was not compiled and GUI subsystem has been detected. Missing the Tk toolkit?
Installed Python-3.12.0 to /home/pun/.pyenv/versions/3.12.0
Should I worry about the warnings ?
Should I worry about the warnings ?
The mentioned modules will be missing so the associated Python functionality won't work. This may or may not be significant for you. Normally, you want to install all the dependencies as per https://github.com/pyenv/pyenv/wiki#suggested-build-environment before building to get a fully functional Python.
This worked:
~> CC=/home/linuxbrew/.linuxbrew/bin/gcc-13 CPPFLAGS="-I$(brew --prefix)/include" LDFLAGS="-L$(brew --prefix)/lib" py
Strange. For me, it worked with just CC=<...>
. I have installed all the required packages from Homebrew as per the wiki though.
Without the required logs as per the issue template, I cannot say anything for sure anyway.
Without the required logs as per the issue template, I cannot say anything for sure anyway.
Sorry I left my laptop at work. I'll post the version only with CC and the one with everything on Monday.
As a resolution to this topic, it seems we'd need to add an exception to Pyenv to use Homebrew in Linux by default if Pyenv itself has been installed with Homebrew. (Then you would not need to use a Homebrew-provided compiler 'cuz Pyenv itself would be adding the compiler options that those add implicitly.) This has been an idea ever since we turned that off by default due to the problems it was causing.
Shall we also print a warning if Homebrew is not going to be used but a Homebrew-provided pkg-config
is detected on PATH
?
Hello. I've updated pyenv.
pun@owo:~$ pyenv --version
pyenv 2.3.31
Here is the first command I tried running which failed:
env PYTHON_CONFIGURE_OPTS='--enable-optimizations --with-lto' PYTHON_CFLAGS='-march=native -mtune=native' pyenv install 3.12
Here are the logs:
python-build.20231023084224.9101.log
And here is the second command I tried which was successful:
env PYTHON_CONFIGURE_OPTS='--enable-optimizations --with-lto' PYTHON_CFLAGS='-march=native -mtune=native' CC=/home/linuxbrew/.linuxbrew/bin/gcc-13 CPPFLAGS="-I$(brew --prefix)/include" LDFLAGS="-L$(brew --prefix)/lib" pyenv install 3.12
Downloading Python-3.12.0.tar.xz...
-> https://www.python.org/ftp/python/3.12.0/Python-3.12.0.tar.xz
Installing Python-3.12.0...
Installed Python-3.12.0 to /home/pun/.pyenv/versions/3.12.0
@punowo this is the reason:
checking for pkg-config... /home/linuxbrew/.linuxbrew/bin/pkg-config
Homebrew installs
pkg-config
under a name that shadows the system's one.This stops one from being able to build software using the system toolchain -- because the flags it produces only work out of the box if using a Homebrew-provided compiler.
Pyenv doesn't use packages from Homebrew in Linux by default. That's because Homebrew is more problematic to use than distro-provided packages. Unlike most distros, Homebrew is rolling-release -- and as such, its dependency package upgrades break existing Pyenv-provided installations whenever their
.so
names change.To build with the system's toolchain,
- uninstall or at least
brew unlink
Homebrew-providedpkg-config
; or- remove Homebrew from
PATH
To build with Homebrew's toolchain,
- install a Homebrew-provided compiler and
- direct any builds to use it with
export CC=<compiler executable to use>
Installing python 3.12.0 raised /usr/bin/install: cannot stat 'Modules/_crypt.cpython-312-x86_64-linux-gnu.so': No such file or directory
, unlike any other Python version.
Using Ubuntu 22.04 LTS and brew unlink pkg-config
solved the issue for me.
brew unlink pkg-config
also works for macOS
ginstall: cannot stat 'Modules/_dbm.cpython-312-darwin.so': No such file or directory
In my case i was missing gdbm
So i had to brew install gdbm
brew unlink pkg-config
also works for macOS
PSA this can be easily undone by the counterpart brew link pkg-config
afterwards 👍🏼
And here is the second command I tried which was successful:
env PYTHON_CONFIGURE_OPTS='--enable-optimizations --with-lto' PYTHON_CFLAGS='-march=native -mtune=native' CC=/home/linuxbrew/.linuxbrew/bin/gcc-13 CPPFLAGS="-I$(brew --prefix)/include" LDFLAGS="-L$(brew --prefix)/lib" pyenv install 3.12
this did not work for me. I'm on a Mac, so I wanted to avoid the brew toolchain, so my cmd was:
env PYTHON_BUILD_SKIP_HOMEBREW=1 PYTHON_CONFIGURE_OPTS="--enable-framework --enable-optimizations --with-lto" PYTHON_CFLAGS='-march=native -mtune=native' CC=/opt/homebrew/bin/gcc-13 pyenv install 3.12.0
where that CC came from brew install gcc, in case that's not obvious:
robbie@macbook-pro ~ % ls -la /opt/homebrew/bin/gcc-13
lrwxr-xr-x 1 macadmin staff 31 Nov 24 15:05 /opt/homebrew/bin/gcc-13 -> ../Cellar/gcc/13.2.0/bin/gcc-13
the error:
Downloading openssl-3.1.2.tar.gz...
-> https://www.openssl.org/source/openssl-3.1.2.tar.gz
Installing openssl-3.1.2...
Installed openssl-3.1.2 to /Users/robbie/.pyenv/versions/3.12.0
Downloading readline-8.2.tar.gz...
-> https://ftpmirror.gnu.org/readline/readline-8.2.tar.gz
Installing readline-8.2...
Installed readline-8.2 to /Users/robbie/.pyenv/versions/3.12.0
Downloading Python-3.12.0.tar.xz...
-> https://www.python.org/ftp/python/3.12.0/Python-3.12.0.tar.xz
Installing Python-3.12.0...
BUILD FAILED (OS X 14.1.1 using python-build 20180424)
Inspect or clean up the working tree at /var/folders/9h/_r86cs8177x6hmpxy0fg473h0000gp/T/python-build.20231124154047.93268
Results logged to /var/folders/9h/_r86cs8177x6hmpxy0fg473h0000gp/T/python-build.20231124154047.93268.log
Last 10 log lines:
gcc-13: error: unrecognized command-line option '-fprofile-instr-generate'; did you mean '-fprofile-generate'?
gcc-13: error: unrecognized command-line option '-fprofile-instr-generate'; did you mean '-fprofile-generate'?
make[2]: *** [Parser/action_helpers.o] Error 1
make[2]: *** [Parser/token.o] Error 1
gcc-13: error: unrecognized command-line option '-fprofile-instr-generate'; did you mean '-fprofile-generate'?
make[2]: *** [Objects/bytes_methods.o] Error 1
gcc-13: error: unrecognized command-line option '-fprofile-instr-generate'; did you mean '-fprofile-generate'?
make[2]: *** [Parser/parser.o] Error 1
make[1]: *** [profile-gen-stamp] Error 2
make: *** [profile-run-stamp] Error 2
why does python assume my gcc is really clang?
@robbiemu Sorry I don't know how to help. I didn't have any problems on Mac OS at all. I haven't had any problems but then again when I'm on Mac OS I've learned to just use the clang it arrives. By default, on a Mac, gcc points to it.
❯ gcc --version
Apple clang version 15.0.0 (clang-1500.0.40.1)
Target: arm64-apple-darwin23.1.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
For the record I have this in my .zprofile.
# Setup pyenv ( python version manager )
if [ -x "$HOMEBREW_PATH/bin/pyenv" ]; then
export PYENV_ROOT="$HOME/.pyenv"
export PYTHON_CONFIGURE_OPTS='--enable-optimizations --with-lto'
export PYTHON_CFLAGS='-march=native -mtune=native'
export LDFLAGS="-L/opt/homebrew/opt/zlib/lib"
export CPPFLAGS="-I/opt/homebrew/opt/zlib/include"
command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"
fi
@punowo - hey, thank you for answering my question! the fact that gcc by default points to clang in the Mac framework is a great reason why it would do that. I went back to the other way to install and eventually also solved my problem. When you get a new Mac, you might want to try out the game porting toolkit and play a game or two, instead of immediately setting up your coding env, I'm sure I have some sympathy in that. I used a pattern at first that installed the x64 version of homebrew to set up windows games (later migrated to whisky, because it is cleaner). So I have two homebrews, one of which is for the x64 arch instead of arm64. It was being picked up and preventing me from success using the default method. uninstalling libb2 with the x64 version of homebrew solved my problems. I'm good, using 3.12 now :)
brew unlink pkg-config
worked a treat; it would have been nice to get this as a warning or note on the command line. Ubuntu 22.04.03 + homebrew installed pyenv
brew unlink pkg-config
also works for macOS
This also worked for me using linux mint, installed at first using homebrew, then i uninstalled and installed pyenv manually, still got the problem. So if you use homebrew once you have to do this before installing latest python version
I don't even have brew installed, any tips?
Ubuntu Linux 23.10 amd64 pyenv version: 2.3.30 gcc (Ubuntu 13.2.0-4ubuntu3) 13.2.0
trace.log python-build.20231018124421.17515.log
I installed pyenv using homebrew on Linux, but I also had a similar issue when using the pyenv installer on another Ubuntu 23.10 machine.