Python version 3 is not carried into the pybind11 compilation

guru-florida commented 2 years ago

Bug report

During compilation of rclpy I was getting this error repeated:

/opt/ros/latest/src/ros2/rclpy/rclpy/src/rclpy/context.hpp: In destructor ‘rclpy::InitOptions::~InitOptions()’:
/opt/ros/latest/src/ros2/rclpy/rclpy/src/rclpy/context.hpp:48:7: error: ‘PyErr_WarnFormat’ was not declared in this scope; did you mean ‘PyErr_Format’?
   48 |       PyErr_WarnFormat(
      |       ^~~~~~~~~~~~~~~~
      |       PyErr_Format

The PyErr_WarnFormat function exists in the python3 C API (ex. /usr/include/python3.8/warnings.h). I have python 2.7 and 3.8 installed. The rclpy cmake configure step does in fact find python3.8. Here are two relevant lines from the rclpy configure step:

-- Found Python3: /usr/bin/python3.8 (found version "3.8.10") found components: Interpreter Development 
-- Found pybind11_vendor: 2.4.0 (/opt/ros/latest/install/pybind11_vendor/share/pybind11_vendor/cmake)

So clearly rclpy should be adding the proper python3.8 include directory to compilation. This led me down the rabbit hole of why the proper (warnings.h) file wasn't being included. I used VERBOSE compilation of rclpy and it is clear the pybind11 source files are including the python2.7 directory instead. Example build output of one source file:

[ 27%] Building CXX object CMakeFiles/_rclpy_pybind11.dir/src/rclpy/graph.cpp.o
/usr/bin/c++
    -DDEFAULT_RMW_IMPLEMENTATION=rmw_fastrtps_cpp
    -DRCUTILS_ENABLE_FAULT_INJECTION
    -D_rclpy_pybind11_EXPORTS
    -I/usr/include/python2.7     <=== here
    ...[snip]...

So the configure step in rclpy finding python3 does not seem to transition build flags into the pybind11 builds and pybind11 configure step isn't enforcing python3. Is there a way we can fix this? I don't think we should support python2 anymore, but we can cmake-find python3 instead of just python or at least error out and say "python2 not supported".

Python3 is what I normally use but I didn't realize my python default (/usr/bin/python) was set to 2.7 since Ubuntu 20.04 default has been 3 for a while. So I switched python default using the recommended "sudo apt install python-is-python3" and the build is working fine now. I figure we should still enforce python3 in the pybind11 cmake file, thoughts?

Required Info:

Operating System:
- Ubuntu 20.04 LTS
Installation type:
- source, rolling
Version or commit hash:
- master, 3.3.1 (8e8c978908f36e4cb5f34b3417eb6f912789f76d)
DDS implementation:
- rmw_fastrtps_cpp
Client library (if applicable):
- rclpy

Steps to reproduce issue

install both python 2 and 3 interpreter
set /usr/bin/python to python2 interpreter
compile rcply: VERBOSE=1 colcon build --symlink-install --packages-select rclpy --event-handlers console_cohesion+

Expected behavior

pybind11 build steps should inherit C/CXX build flags from rclpy configure step, or otherwise enforce python3 version using the same cmake find step as the main rclpy CMakeLists.txt.

Actual behavior

pybind11 build step includes python2 API if /usr/bin/python points to version 2.

Additional information

Full example compile line of pybind11 source file:

[ 27%] Building CXX object CMakeFiles/_rclpy_pybind11.dir/src/rclpy/graph.cpp.o
/usr/bin/c++  -DDEFAULT_RMW_IMPLEMENTATION=rmw_fastrtps_cpp -DRCUTILS_ENABLE_FAULT_INJECTION -D_rclpy_pybind11_EXPORTS -I/usr/include/python2.7 -I/opt/ros/latest/src/ros2/rclpy/rclpy/src/rclpy -isystem /opt/ros/latest/install/lifecycle_msgs/include/lifecycle_msgs -isystem /opt/ros/latest/install/rosidl_runtime_c/include/rosidl_runtime_c -isystem /opt/ros/latest/install/rcutils/include/rcutils -isystem /opt/ros/latest/install/rosidl_typesupport_interface/include/rosidl_typesupport_interface -isystem /opt/ros/latest/install/rcl/include/rcl -isystem /opt/ros/latest/install/rcl_interfaces/include/rcl_interfaces -isystem /opt/ros/latest/install/builtin_interfaces/include/builtin_interfaces -isystem /opt/ros/latest/install/fastcdr/include -isystem /opt/ros/latest/install/rosidl_runtime_cpp/include/rosidl_runtime_cpp -isystem /opt/ros/latest/install/rosidl_typesupport_fastrtps_cpp/include/rosidl_typesupport_fastrtps_cpp -isystem /opt/ros/latest/install/rmw/include/rmw -isystem /opt/ros/latest/install/rosidl_typesupport_fastrtps_c/include/rosidl_typesupport_fastrtps_c -isystem /opt/ros/latest/install/rosidl_typesupport_introspection_c/include/rosidl_typesupport_introspection_c -isystem /opt/ros/latest/install/rosidl_typesupport_introspection_cpp/include/rosidl_typesupport_introspection_cpp -isystem /opt/ros/latest/install/rcl_logging_interface/include/rcl_logging_interface -isystem /opt/ros/latest/install/rcl_yaml_param_parser/include/rcl_yaml_param_parser -isystem /opt/ros/latest/install/libyaml_vendor/include/libyaml_vendor -isystem /opt/ros/latest/install/tracetools/include/tracetools -isystem /opt/ros/latest/install/rcl_action/include/rcl_action -isystem /opt/ros/latest/install/action_msgs/include/action_msgs -isystem /opt/ros/latest/install/unique_identifier_msgs/include/unique_identifier_msgs -isystem /opt/ros/latest/install/rcl_lifecycle/include/rcl_lifecycle -isystem /opt/ros/latest/install/rcpputils/include/rcpputils  -fPIC -fvisibility=hidden   -Wall -Wextra -flto -fno-fat-lto-objects -std=gnu++17 -o CMakeFiles/_rclpy_pybind11.dir/src/rclpy/action_server.cpp.o -c /opt/ros/latest/src/ros2/rclpy/rclpy/src/rclpy/action_server.cpp

zhj-buffer commented 2 years ago

I laso met this issue on Jetson nx. Could anyone take a look at this?

FelipeGdM commented 2 years ago

I faced the same issue when compiling Humble from source on Jetson Xavier NX with Ubuntu 20.04 and Jetpack 5.0.1. I solved it by removing python2.7. completely, but I understand it may not be an option on al situations. Before going to the hammer solution, I found some info that may be relevant

This StackOverflow question talks about the exact issue with pybind11 using inconsistent versions of python when building. The proposed solution is setting the config variable PYBIND11_PYTHON_VERSION before calling find_package(pybind11), so pybin11 is forced to use a specific version of python

It may be sufficient to add the following line before the find_package() call

set(PYBIND11_PYTHON_VERSION 3)

Or add the command line argument --catkin-cmake-args -DPYBIND11_PYTHON_VERSION=3 when calling colcon build

I didn't tested this solution myself due to time constrains, so I won't open a pull request, but this may be useful for the next person that bumps into this problem.

clalancette commented 2 years ago

FYI, I believe https://github.com/ros2/pybind11_vendor/pull/15 should fix this issue.

clalancette commented 2 years ago

Fixed by https://github.com/ros2/pybind11_vendor/pull/15, so closing.

shtern commented 1 year ago

@FelipeGdM Thank you very much, adding --cmake-args -DPYBIND11_PYTHON_VERSION=3 helped me. It's a solution for those who're running the ROS versions that are older than the release of the fix

ros2 / rclpy