If python2 is installed in ubuntu pybind11 bindings does not compile

[GiulioRomualdi]

With @isorrentino we experienced the following issue:

If python2 is installed in the system and python command is the alias of python2 the bindings compilation fails with the following error:

/usr/include/pybind11/detail/common.h:112:10: fatal error: Python.h: No such file or directory
  112 | #include <Python.h>
      |          ^~~~~~~~~~
compilation terminated.
In file included from /usr/include/pybind11/pytypes.h:12,
                 from /usr/include/pybind11/cast.h:13,
                 from /usr/include/pybind11/attr.h:13,
                 from /usr/include/pybind11/pybind11.h:44,
                 from /home/isorrentino/dev/robotology-superbuild/src/bipedal-locomotion-framework/bindings/python/System/src/VariablesHandler.cpp:8:
/usr/include/pybind11/detail/common.h:112:10: fatal error: Python.h: No such file or directory
  112 | #include <Python.h>
      |          ^~~~~~~~~~
compilation terminated.
In file included from /usr/include/pybind11/

cc @traversaro

[GiulioRomualdi]

Probably something related to this file https://github.com/ami-iit/bipedal-locomotion-framework/blob/924760208473c889bc48c4c487906af1cbcc7657/cmake/FindPython3.cmake

Indeed if the Python3_EXECUTABLE cmake variable is manually set to /usr/bin/python3 everything works fine

cc @diegoferigo

[traversaro]

Can you specify with which distro are you experiencing this?

[traversaro]

Just to understand, you would like CMake to automatically set Python3_EXECUTABLE to /usr/bin/python3, as it should understand that /usr/bin/python is actually Python2 and so it is not supported?

[GiulioRomualdi]

Can you specify with which distro are you experiencing this?

Ubuntu 20.04

Just to understand, you would like CMake to automatically set Python3_EXECUTABLE to /usr/bin/python3, as it should understand that /usr/bin/python is actually Python2 and so it is not supported?

exactly or at least print a warning

[traversaro]

I am testing in https://github.com/ami-iit/bipedal-locomotion-framework/pull/452, but it seems that python2 package was already installed in CI. I guess the problem is that the python-is-python2 package was installed?

[GiulioRomualdi]

I think so

[GiulioRomualdi]

Probably @isorrentino can give us a better explanation

[traversaro]

The CI indeed fails if we install python-is-python2, but I guess this is expected as we explicitly pass DPython3_ROOT_DIR=$(python -c "import sys; print(sys.prefix)") in the CI.

[traversaro]

I checked the output from https://github.com/ami-iit/bipedal-locomotion-framework/pull/452, and I think the problematic part is:

2021-11-16T15:34:03.0806037Z -- Found PythonInterp: /usr/bin/python (found version "2.7.18") 
2021-11-16T15:34:03.1842872Z -- Found PythonLibs: python2.7

Basically, some part of the code (probably some <pkg>Config.cmake file) is calling the legacy find_package(PythonLibs) that prefers python over python3.

[traversaro]

Bingo, the problem is in pybind11 2.4.3's pybind11Config.cmake (the version shipped with apt from Ubuntu 20.04), https://github.com/pybind/pybind11/blob/v2.4.3/tools/pybind11Config.cmake.in#L86 :

find_package(PythonLibsNew ${PYBIND11_PYTHON_VERSION} MODULE REQUIRED)

It is not something that it is easy to fix, probably the easiest solution if you just to have a user-friendly error message is to check (if it exists, it does not exists if the used pybind11 version is more modern) the PYTHON_VERSION variable and print a clean erro if it contains the 2.7 string.

[isorrentino]

Sorry for not responding soon. I just installed python3-dev and pybind11-dev by using

sudo apt install python3-dev
sudo apt install pybind11-dev

and actually I don't know what has installed python2. Maybe it is something realted to what you (@traversaro) wrote in the previous comment.

Bingo, the problem is in pybind11 2.4.3's pybind11Config.cmake (the version shipped with apt from Ubuntu 20.04),

[diegoferigo]

I didn't go too deep in the problem. For what I remember, if the CMake project already called FindPython3 before calling FindPybind11, the pybind11 logic will use the interpreter already detected. I think that, if blf uses FindPython3, there's no way that python 2 gets detected, regardless of pybind11's logic.

Alternatively, you can try with -DPYBIND11_FINDPYTHON=ON as documented in the FAQ.

Also have a look to this FAQ that might be relevant.

[traversaro]

I didn't go too deep in the problem. For what I remember, if the CMake project already called FindPython3 before calling FindPybind11, the pybind11 logic will use the interpreter already detected. I think that, if blf uses FindPython3, there's no way that python 2 gets detected, regardless of pybind11's logic.

Alternatively, you can try with -DPYBIND11_FINDPYTHON=ON as documented in the FAQ.

Also have a look to this FAQ that might be relevant.

All of this (including the FAQ) are true but apply to recent releases of pybind121, but the problem described in this issue happens for pybind11 installed from apt in Ubuntu 20.04 that is version 2.4.3 for which the logic was different.