[BUG]: Cannot find Python on manylinux with CMake>=3.27

burgholzer commented 1 year ago

Required prerequisites

[X] Make sure you've read the documentation. Your issue may be addressed there.
[X] Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
[X] Consider asking first in the Gitter chat room or in a Discussion.

What version (or hash if on master) of pybind11 are you using?

2.11.1 (also tested with latest master https://github.com/pybind/pybind11/commit/b9359ceadbf4d4b22509b684eea107d055b77901)

Problem description

Pybind11 cannot find Python on manylinux containers (such as within cibuidwheel) when running in new FindPython mode (set(PYBIND11_FINDPYTHON ON)) or under CMake>=3.27 (where that new mode is the default).

Traces always have the following form:

CMake Error at /opt/_internal/pipx/venvs/cmake/lib/python3.10/site-packages/cmake/data/share/cmake-3.27/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find Python (missing: Python_LIBRARIES Development
  Development.Embed) (found suitable version "3.12.0", minimum required is
  "3.6")
Call Stack (most recent call first):
  /opt/_internal/pipx/venvs/cmake/lib/python3.10/site-packages/cmake/data/share/cmake-3.27/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)
  /opt/_internal/pipx/venvs/cmake/lib/python3.10/site-packages/cmake/data/share/cmake-3.27/Modules/FindPython/Support.cmake:3824 (find_package_handle_standard_args)
  /opt/_internal/pipx/venvs/cmake/lib/python3.10/site-packages/cmake/data/share/cmake-3.27/Modules/FindPython.cmake:574 (include)
  /opt/python/cp38-cp38/lib/python3.8/site-packages/pybind11/share/cmake/pybind11/pybind11NewTools.cmake:35 (find_package)
  /opt/python/cp38-cp38/lib/python3.8/site-packages/pybind11/share/cmake/pybind11/pybind11Common.cmake:183 (include)
  /opt/python/cp38-cp38/lib/python3.8/site-packages/pybind11/share/cmake/pybind11/pybind11Config.cmake:250 (include)

Adding a

find_package(Python REQUIRED COMPONENTS Interpreter Development.Module)

statement before the pybind11 import fixes the problem, but I fail to understand why that should be required and would consider this a bug.

Reproducible example code

Adapted loosely from what is happening inside cibuildwheel

docker run -it quay.io/pypa/manylinux2014_x86_64:latest
echo -e "cmake_minimum_required(VERSION 3.18...3.26)\nproject(test)\nset(PYBIND11_FINDPYTHON ON)\nfind_package(pybind11 CONFIG REQUIRED)\n" > tmp/CMakeLists.txt
/opt/python/cp38-cp38/bin/python -m pip install pybind11
export pybind11_DIR=$(/opt/python/cp38-cp38/bin/python -m pybind11 --cmakedir)
cmake tmp/

Yields
```console
-- The C compiler identification is GNU 10.2.1
-- The CXX compiler identification is GNU 10.2.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/rh/devtoolset-10/root/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /opt/rh/devtoolset-10/root/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Error at /opt/_internal/pipx/venvs/cmake/lib/python3.10/site-packages/cmake/data/share/cmake-3.27/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find Python (missing: Python_LIBRARIES Development
  Development.Embed) (found suitable version "3.12.0", minimum required is
  "3.6")
Call Stack (most recent call first):
  /opt/_internal/pipx/venvs/cmake/lib/python3.10/site-packages/cmake/data/share/cmake-3.27/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)
  /opt/_internal/pipx/venvs/cmake/lib/python3.10/site-packages/cmake/data/share/cmake-3.27/Modules/FindPython/Support.cmake:3824 (find_package_handle_standard_args)
  /opt/_internal/pipx/venvs/cmake/lib/python3.10/site-packages/cmake/data/share/cmake-3.27/Modules/FindPython.cmake:574 (include)
  /opt/python/cp38-cp38/lib/python3.8/site-packages/pybind11/share/cmake/pybind11/pybind11NewTools.cmake:35 (find_package)
  /opt/python/cp38-cp38/lib/python3.8/site-packages/pybind11/share/cmake/pybind11/pybind11Common.cmake:183 (include)
  /opt/python/cp38-cp38/lib/python3.8/site-packages/pybind11/share/cmake/pybind11/pybind11Config.cmake:250 (include)
  CMakeLists.txt:4 (find_package)

-- Configuring incomplete, errors occurred!

The same error occurs with the even simpler

echo -e "cmake_minimum_required(VERSION 3.18...3.27)\nproject(test)\nfind_package(pybind11 CONFIG REQUIRED)\n" > tmp/CMakeLists.txt

Running

 echo -e "cmake_minimum_required(VERSION 3.18...3.26)\nproject(test)\nfind_package(Python REQUIRED COMPONENTS Interpreter Development.Module)\nset(PYBIND11_FINDPYTHON ON)\nfind_package(pybind11 CONFIG REQUIRED)\n" > tmp/CMakeLists.txt

instead, yields

-- The C compiler identification is GNU 10.2.1
-- The CXX compiler identification is GNU 10.2.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/rh/devtoolset-10/root/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /opt/rh/devtoolset-10/root/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Python: /usr/local/bin/python3.12 (found version "3.12.0") found components: Interpreter Development.Module 
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- Found pybind11: /opt/python/cp38-cp38/lib/python3.8/site-packages/pybind11/include (found version "2.11.1")
-- Configuring done (5.8s)
-- Generating done (0.0s)
-- Build files have been written to: /tmp



### Is this a regression? Put the last known working version here if it is.

Not a regression

LecrisUT commented 1 year ago

Some context, it is better to use find_package(Python3) instead of find_package(Python)

henryiii commented 1 year ago

Not really, if you set a minimum version (which you should), than they are the same (and I prefer the one without the 3, since IMO Python == Python 3, and I don't want to start using the 3 everywhere in case 4 ever comes.)

LecrisUT commented 1 year ago

Oh right, I keep forgetting that FindPython works with minimum set. I just remember that without setting it you would only be searching for python and not python3 in PATH.

henryiii commented 1 year ago

Hmm, might be worth checking, but I think it's identical with the minimum, especially since this is common on Unix without venvs. But on Windows, you only have python and not python3 (the opposite of Unix, where you might have python3 and not python). IMO everyone should use venvs, but I can't force that. :)

pybind / pybind11