[python3] Build error

[byxrolland]

Package: python3:x64-linux -> 3.11.4

Host Environment

• Host: x64-linux
• Compiler: GNU 11.4.0
• vcpkg-tool version: 2023-08-02-6d13efa755f9b5e101712d210199e4139b4c29f6 vcpkg-scripts version: 247662ef3 2023-08-08 (12 hours ago)

To Reproduce

vcpkg install boost[mpi]

Failure logs

-- Using cached python-cpython-v3.11.4.tar.gz.
-- Cleaning sources at /home/byxrolland/vcpkg/buildtrees/python3/src/v3.11.4-c76ea88c37.clean. Use --editable to skip cleaning for the packages you specify.
-- Extracting source /home/byxrolland/vcpkg/downloads/python-cpython-v3.11.4.tar.gz
-- Applying patch 0001-only-build-required-projects.patch
-- Applying patch 0003-use-vcpkg-zlib.patch
-- Applying patch 0004-devendor-external-dependencies.patch
-- Applying patch 0005-dont-copy-vcruntime.patch
-- Applying patch 0008-python.pc.patch
-- Applying patch 0010-dont-skip-rpath.patch
-- Applying patch 0012-force-disable-curses.patch
-- Applying patch 0014-fix-get-python-inc-output.patch
-- Applying patch 0002-static-library.patch
-- Applying patch 0011-gcc-ldflags-fix.patch
-- Using source at /home/byxrolland/vcpkg/buildtrees/python3/src/v3.11.4-c76ea88c37.clean
-- Getting CMake variables for x64-linux-dbg
-- Getting CMake variables for x64-linux-rel
-- Generating configure for x64-linux
-- Finished generating configure for x64-linux
-- Configuring x64-linux-dbg
-- Configuring x64-linux-rel
-- Building x64-linux-dbg
-- Installing x64-linux-dbg
-- Building x64-linux-rel
-- Installing x64-linux-rel
-- Installing: /home/byxrolland/vcpkg/packages/python3_x64-linux/share/python3/copyright
-- Fixing pkgconfig file: /home/byxrolland/vcpkg/packages/python3_x64-linux/lib/pkgconfig/python-3.11-embed.pc
-- Fixing pkgconfig file: /home/byxrolland/vcpkg/packages/python3_x64-linux/lib/pkgconfig/python-3.11.pc
-- Fixing pkgconfig file: /home/byxrolland/vcpkg/packages/python3_x64-linux/debug/lib/pkgconfig/python-3.11-embed.pc
-- Fixing pkgconfig file: /home/byxrolland/vcpkg/packages/python3_x64-linux/debug/lib/pkgconfig/python-3.11.pc
CMake Error at ports/python3/portfile.cmake:294 (message):
  There should be no modules with "_failed" suffix:

  /home/byxrolland/vcpkg/packages/python3_x64-linux/debug/lib/python3.11/lib-dynload/_crypt.cpython-311d-x86_64-linux-gnu_failed.so

  /home/byxrolland/vcpkg/packages/python3_x64-linux/lib/python3.11/lib-dynload/_crypt.cpython-311-x86_64-linux-gnu_failed.so
Call Stack (most recent call first):
  scripts/ports.cmake:147 (include)

Additional context

[FrankXie05]

Duplicate of https://github.com/microsoft/vcpkg/issues/29447

[byxrolland]

I didn't have this problem when building Python 3.10, only 3.11. At first it required autoconf. After I installed autoconf it said there was a macro not defined. After I installed automake and libtool, it showed the error as above.

Duplicate of #29447

[Osyotr]

@FrankXie05 this does not look like a duplicate of #29447. Please reopen the ticket.

@byxrolland please upload more logs.

[FrankXie05]

@byxrolland Could you please run command sudo apt-get install libcrypt-dev to re-install crypt?

[byxrolland]

@FrankXie05 this does not look like a duplicate of #29447. Please reopen the ticket.

@byxrolland please upload more logs.

Where can I find more logs?

[byxrolland]

@byxrolland Could you please run command sudo apt-get install libcrypt-dev to re-install crypt?

It is already installed.

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
libcrypt-dev is already the newest version (1:4.4.27-1).
libcrypt-dev set to manually installed.

[Osyotr]

Where can I find more logs?

/home/byxrolland/vcpkg/buildtrees/python3

[byxrolland]

Where can I find more logs?

/home/byxrolland/vcpkg/buildtrees/python3

In autoconf-x64-linux-err.log:

autoreconf: export WARNINGS=
autoreconf: Entering directory '.'
autoreconf: configure.ac: not using Gettext
autoreconf: running: aclocal --force 
autoreconf: configure.ac: tracing
autoreconf: configure.ac: not using Libtool
autoreconf: configure.ac: not using Intltool
autoreconf: configure.ac: not using Gtkdoc
autoreconf: running: /home/linuxbrew/.linuxbrew/Cellar/autoconf/2.71/bin/autoconf --force
autoreconf: running: /home/linuxbrew/.linuxbrew/Cellar/autoconf/2.71/bin/autoheader --force
autoreconf: configure.ac: not using Automake
autoreconf: Leaving directory '.'

The following messages occurred frequently in several err.logs:

*** WARNING: renaming "_crypt" since importing it failed: libcrypt.so.2: cannot open shared object file: No such file or directory
renaming build/scripts-3.11/pydoc3 to build/scripts-3.11/pydoc3.11
renaming build/scripts-3.11/idle3 to build/scripts-3.11/idle3.11
renaming build/scripts-3.11/2to3 to build/scripts-3.11/2to3-3.11

configure: WARNING: unrecognized options: --disable-silent-rules, --enable-static
configure: WARNING: unrecognized options: --disable-silent-rules, --enable-static

CMake Warning:
  Manually-specified variables were not used by the project:

    BUILD_SHARED_LIBS
    CMAKE_INSTALL_BINDIR
    CMAKE_INSTALL_LIBDIR
    VCPKG_PLATFORM_TOOLSET
    VCPKG_SET_CHARSET_FLAG
    _VCPKG_ROOT_DIR

Hope these could be helpful.

[Osyotr]

Would you kindly upload config-x64-linux-rel-config.log and build-x64-linux-rel-out.log?

[byxrolland]

Here you are:

build-x64-linux-rel-out.log config-x64-linux-rel-config.log

[Osyotr]

Ok, here are some hints:

PKG_CONFIG='/home/linuxbrew/.linuxbrew/bin/pkg-config'
pkg_cv_LIBCRYPT_CFLAGS=-I/home/linuxbrew/.linuxbrew/Cellar/libxcrypt/4.4.36/include
pkg_cv_LIBCRYPT_LIBS='-L/home/linuxbrew/.linuxbrew/Cellar/libxcrypt/4.4.36/lib -lcrypt'

You use pkg-config from linuxbrew which then finds libxcrypt in /home/linuxbrew/.linuxbrew/Cellar/libxcrypt/4.4.36/lib, but this path is not in your library path. Consider setting LD_LIBRARY_PATH prior to running vcpkg.

[byxrolland]

Ok, here are some hints:

PKG_CONFIG='/home/linuxbrew/.linuxbrew/bin/pkg-config'
pkg_cv_LIBCRYPT_CFLAGS=-I/home/linuxbrew/.linuxbrew/Cellar/libxcrypt/4.4.36/include
pkg_cv_LIBCRYPT_LIBS='-L/home/linuxbrew/.linuxbrew/Cellar/libxcrypt/4.4.36/lib -lcrypt'

You use pkg-config from linuxbrew which then finds libxcrypt in /home/linuxbrew/.linuxbrew/Cellar/libxcrypt/4.4.36/lib, but this path is not in your library path. Consider setting LD_LIBRARY_PATH prior to running vcpkg.

But the linuxbrew has libxcrypt installed. Why is that a problem?

Also, could you tell how to set LD_LIBRARY_PATH?

[byxrolland]

Also, I noticed other problems:

The linuxbrew has gcc-13 installed but vcpkg only detected the system gcc.

Also, there is an error that appears frequently in config-x64-linux-rel-config.log, namely

cc1: error: argument to '-O' should be a non-negative integer, 'g', 's' or 'fast'

Is that going to be a problem?

[Osyotr]

The linuxbrew has gcc-13 installed but vcpkg only detected the system gcc.

How do you invoke gcc that is installed from linuxbrew? I don't have any experience with linuxbrew but I guess there's some kind of environment activation like in SCL or venv (UPD: looks like that's true https://docs.brew.sh/Homebrew-on-Linux). In that case you need to run vcpkg from the same environment.

But the linuxbrew has libxcrypt installed. Why is that a problem?

The problem is that the path to the library is not in your LD_LIBRARY_PATH so any program that links to libxcrypt.so will not be able to start. It might be the same issue I explained above.

Also, there is an error that appears frequently in config-x64-linux-rel-config.log

I'd say this is https://github.com/python/cpython/pull/105062, not really critical to build process.

[byxrolland]

How do you invoke gcc that is installed from linuxbrew? I don't have any experience with linuxbrew but I guess there's some kind of environment activation like in SCL or venv (UPD: looks like that's true https://docs.brew.sh/Homebrew-on-Linux). In that case you need to run vcpkg from the same environment.

It is in LINE 201 of bootstrap.sh. Your script doesn't accept gcc-13. It stops at g++-12.

[Osyotr]

It is in LINE 201 of bootstrap.sh. Your script doesn't accept gcc-13. It stops at g++-12.

This code https://github.com/microsoft/vcpkg/blob/master/scripts/bootstrap.sh#L201 is used only if you build vcpkg tool from source and does not affect building of packages.

[github-actions[bot]]

This is an automated message. Per our repo policy, stale issues get closed if there has been no activity in the past 28 days. The issue will be automatically closed in 14 days. If you wish to keep this issue open, please add a new comment.

[byxrolland]

I'm waiting for g++-13 support in the script.

[dg0yt]

I'm waiting for g++-13 support in the script.

The support is already there: Set CXX to the desired compiler. The if stuff is to be removed, https://github.com/microsoft/vcpkg/pull/33306#issuecomment-1687955723.

[byxrolland]

I'm waiting for g++-13 support in the script.

The support is already there: Set CXX to the desired compiler. The if stuff is to be removed, #33306 (comment).

Could you tell me how to set it?

[byxrolland]

OK. I set CXX and CC to the linuxbrew location (which will use its own autoconf and automake) and the problem is solved. But I really would like vcpkg to auto-detect the latest C and C++ compiler available on my machine. Otherwise, the mismatch of compilers and autoconf is really hard to find.

[dg0yt]

I think you are mixing up things.

• The bootstrap script is for building the vcpkg tool (when it isn't downloaded as a binary). It doesn't change what is used to build ports like python3.
  As written in https://github.com/microsoft/vcpkg/issues/33068#issuecomment-1685299679
• Port builds are controlled by triplet and toolchain files. If you don't want to touch these files you can simply setup your PATH to start with th directories which provide the desired g++, gcc, autoconf etc. You can create a single directory with symlinks for convenience. (I use that for arm64-linux on x64-linux.) This is a standard pattern. Setting CC/CXX won't work well for cross-builds.
• I don't think there is a general mismatch of compilers and autoconf. But with different package providers, it is easy to end up with mixing incompatible libraries at runtime.

[PhoebeHui]

I don't think there is a general mismatch of compilers and autoconf. But with different package providers, it is easy to end up with mixing incompatible libraries at runtime.

I totally agree.

@byxrolland, it looks the problem was solved now?

[byxrolland]

It works for me know but I wouldn't consider it as solved as you changed the behavior of vcpkg and don't regard that as your problem now.

[PhoebeHui]

@byxrolland, good to know it doesn't block you now。

For autoconf automake failures, we have another issue https://github.com/microsoft/vcpkg/issues/33080 to track.

For your suggestions ' I really would like vcpkg to auto-detect the latest C and C++ compiler available on my machine', I would say it's not a python3 issue, AFAIK vcpkg assumes that a useable compiler exists in your system https://github.com/microsoft/vcpkg/blob/master/scripts/detect_compiler/CMakeLists.txt, which from CMake CMAKE_CXX_COMPILER and CMAKE_C_COMPILER(linux.cmake), that means vcpkg doesn't auto detect or change enviroment variable about compiler, but rely on these values from build system or user manually passed. Anyway, in my opinion, it's out of a scope for a specific port issue, but could be improved in the long term, it would be great if you could open a discussion issue with details suggestions.

[asclearuc]

Looks like I have similar problem, and don't know how to solve it...

• Host: Ubuntu 20/WSL (under Windows 11)
• Compiler: clang-9
• vcpkg: 2023.06.20 (however the same is true for the master branch of vcpkg)

ATTN: the same build runs OK on Ubuntu 18/WSL (under the same Windows 11)

Trying to build python using command vcpkg install python3, and here is the output:

-- Fixing pkgconfig file: /home/asclearuc/development/engine/vcpkg/packages/python3_x64-linux-clang9-aparavi/debug/lib/pkgconfig/python-3.10.pc
CMake Error at /home/asclearuc/development/engine/apLib/cmake/ports/python3/portfile.cmake:299 (message):
  There should be no modules with "_failed" suffix:
 /home/asclearuc/development/engine/vcpkg/packages/python3_x64-linux-clang9-aparavi/debug/lib/python3.10/lib-dynload/_crypt.cpython-310d-x86_64-linux-gnu_failed.so
  /home/asclearuc/development/engine/vcpkg/packages/python3_x64-linux-clang9-aparavi/debug/lib/python3.10/lib-dynload/nis.cpython-310d-x86_64-linux-gnu_failed.so
  /home/asclearuc/development/engine/vcpkg/packages/python3_x64-linux-clang9-aparavi/lib/python3.10/lib-dynload/_crypt.cpython-310-x86_64-linux-gnu_failed.so
  /home/asclearuc/development/engine/vcpkg/packages/python3_x64-linux-clang9-aparavi/lib/python3.10/lib-dynload/nis.cpython-310-x86_64-linux-gnu_failed.so
Call Stack (most recent call first):
  scripts/ports.cmake:170 (include)

error: building python3:x64-linux-clang9-aparavi failed with: BUILD_FAILED

Investigating a little bit log files:

• from build-x64-linux-clang9-aparavi-dbg-err.log - there are 2 undefined symbols, one of them is crypt_r

  *** WARNING: renaming "_crypt" since importing it failed: /home/asclearuc/development/engine.latest-vcpkg/vcpkg/buildtrees/python3/x64-linux-clang9-aparavi-dbg/build/lib.linux-x86_64-3.10-pydebug/_crypt.cpython-310d-x86_64-linux-gnu.so: undefined symbol: crypt_r
  *** WARNING: renaming "nis" since importing it failed: /home/asclearuc/development/engine.latest-vcpkg/vcpkg/buildtrees/python3/x64-linux-clang9-aparavi-dbg/build/lib.linux-x86_64-3.10-pydebug/nis.cpython-310d-x86_64-linux-gnu.so: undefined symbol: yp_get_default_domain
  renaming build/scripts-3.10/pydoc3 to build/scripts-3.10/pydoc3.10
  renaming build/scripts-3.10/idle3 to build/scripts-3.10/idle3.10
  renaming build/scripts-3.10/2to3 to build/scripts-3.10/2to3-3.10

• looking at the binaries:

On Ubuntu 18 (please notice presence of crypt in the output):

vcpkg/buildtrees/python3$ ldd ./x64-linux-clang9-aparavi-dbg/build/lib.linux-x86_64-3.10-pydebug/_crypt.cpython-310d-x86_64-linux-gnu.so
linux-vdso.so.1 (0x00007ffdb9df5000)
libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f0b2390a000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f0b236eb000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0b232fa000)
/lib64/ld-linux-x86-64.so.2 (0x00007f0b23b42000)

On Ubuntu 20 (libcrypt is not present :( ):

vcpkg/buildtrees/python3$ ldd ./x64-linux-clang9-aparavi-rel/build/lib.linux-x86_64-3.10/_crypt.cpython-310-x86_64-linux-gnu_failed.so
linux-vdso.so.1 (0x00007ffdcf273000)
libpthread.so.0 => /usr/lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f660b4c8000)
libc.so.6 => /usr/lib/x86_64-linux-gnu/libc.so.6 (0x00007f660b2d6000)
/lib64/ld-linux-x86-64.so.2 (0x00007f660b4f1000)

• command line to link _crypt

on Ubuntu 20:

/usr/bin/clang-9 -pthread -shared -L/home/asclearuc/development/engine/vcpkg/installed/x64-linux-clang9-aparavi/debug/lib -stdlib=libc++ -Wl,--no-export-dynamic /usr/lib/llvm-9/lib/libc++.a /usr/lib/llvm-9/lib/libc++abi.a -fuse-ld=lld-9 -L/home/asclearuc/development/engine/vcpkg/installed/x64-linux-clang9-aparavi/debug/lib -L/home/asclearuc/development/engine/vcpkg/installed/x64-linux-clang9-aparavi/debug/lib/manual-link -L/home/asclearuc/development/engine/vcpkg/installed/x64-linux-clang9-aparavi/debug/lib -L/home/asclearuc/development/engine/vcpkg/installed/x64-linux-clang9-aparavi/debug/lib/manual-link -fPIC -g -DCATCH_CONFIG_NO_POSIX_SIGNALS build/temp.linux-x86_64-3.10-pydebug/home/asclearuc/development/engine/vcpkg/buildtrees/python3/src/v3.10.7-e072c2af43.clean/Modules/_cryptmodule.o -L/home/asclearuc/development/engine/vcpkg/installed/x64-linux-clang9-aparavi/debug/lib -L/home/asclearuc/development/engine/vcpkg/installed/x64-linux-clang9-aparavi/debug/lib/manual-link -L/usr/local/lib -Wl,--enable-new-dtags,-R/home/asclearuc/development/engine/vcpkg/installed/x64-linux-clang9-aparavi/debug/lib -Wl,--enable-new-dtags,-R/home/asclearuc/development/engine/vcpkg/installed/x64-linux-clang9-aparavi/debug/lib/manual-link -Wl,--enable-new-dtags,-R/usr/local/lib -Wl,--enable-new-dtags,-R/lib64 -Wl,--enable-new-dtags,-R/usr/lib64 -Wl,--enable-new-dtags,-R/lib -Wl,--enable-new-dtags,-R/usr/lib -o build/lib.linux-x86_64-3.10-pydebug/_crypt.cpython-310d-x86_64-linux-gnu.so

and corresponding line on Ubuntu 18 (-lcrypt is present):

/usr/bin/clang-9 -pthread -shared -L/home/asclearuc/development/engine/vcpkg/installed/x64-linux-clang9-aparavi/debug/lib -stdlib=libc++ -Wl,--no-export-dynamic /usr/lib/llvm-9/lib/libc++.a /usr/lib/llvm-9/lib/libc++abi.a -fuse-ld=lld-9 -L/home/asclearuc/development/engine/vcpkg/installed/x64-linux-clang9-aparavi/debug/lib -L/home/asclearuc/development/engine/vcpkg/installed/x64-linux-clang9-aparavi/debug/lib/manual-link -L/home/asclearuc/development/engine/vcpkg/installed/x64-linux-clang9-aparavi/debug/lib -L/home/asclearuc/development/engine/vcpkg/installed/x64-linux-clang9-aparavi/debug/lib/manual-link -fPIC -g -DCATCH_CONFIG_NO_POSIX_SIGNALS build/temp.linux-x86_64-3.10-pydebug/home/asclearuc/development/engine/vcpkg/buildtrees/python3/src/v3.10.7-e072c2af43.clean/Modules/_cryptmodule.o -L/home/asclearuc/development/engine/vcpkg/installed/x64-linux-clang9-aparavi/debug/lib -L/home/asclearuc/development/engine/vcpkg/installed/x64-linux-clang9-aparavi/debug/lib/manual-link -L/usr/lib/x86_64-linux-gnu -L/usr/local/lib -Wl,--enable-new-dtags,-R/home/asclearuc/development/engine/vcpkg/installed/x64-linux-clang9-aparavi/debug/lib -Wl,--enable-new-dtags,-R/home/asclearuc/development/engine/vcpkg/installed/x64-linux-clang9-aparavi/debug/lib/manual-link -Wl,--enable-new-dtags,-R/usr/lib/x86_64-linux-gnu -Wl,--enable-new-dtags,-R/usr/local/lib -Wl,--enable-new-dtags,-R/lib64 -Wl,--enable-new-dtags,-R/usr/lib64 -Wl,--enable-new-dtags,-R/lib -Wl,--enable-new-dtags,-R/usr/lib -lcrypt -o build/lib.linux-x86_64-3.10-pydebug/_crypt.cpython-310d-x86_64-linux-gnu.so

• config-x64-linux-clang9-aparavi-dbg-config.log has the same information for both U18/U20

  ac_cv_search_crypt=-lcrypt
  ac_cv_search_crypt_r='none required'

So, for me - it looks like configuration is generally the same. However, it results in different compile/link options :(

Any advice?

PS. Different scenarios were tried for U20, like setting CC/CXX to the corresponding values, setting export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu:$LD_LIBRARY_PATH - no matter result was the same

Happy Winter Holidays!

[Osyotr]

Any advice?

If you don't need _crypt and nis modules, you can create an overlay port and remove the post-build check.