rosidl_python installs two almost identical versions of lib<PACKAGE_NAME>__rosidl_generator_py.so

DasRoteSkelett commented 2 years ago

Bug report

Required Info:

Operating System: Ubuntu 22.04, yocto honister
Installation type: binaries, from source
Version or commit hash:
- humble, rolling, 20bcc4ca118145a8a2914fa20aaef20a8e3ce72d
DDS implementation:
- default

Steps to reproduce issue

If you look for files that end with __rosidl_generator_py.so you will find two versions of the same library. The only difference seems to be the RUNPATH that is set to /tmp/binarydeb/ros-humble-builtin-interfaces-1.2.0/.obj-x86_64-linux-gnu:/opt/ros/humble/lib: (ymmv here)

cd /opt/ros/humble/
ROSIDL_FROM_LIB=./lib/libbuiltin_interfaces__rosidl_generator_py.so 
ROSIDL_FROM_DIST_PACKAGES=./local/lib/python3.10/dist-packages/builtin_interfaces/libbuiltin_interfaces__rosidl_generator_py.so
ls -l ${ROSIDL_FROM_LIB} ${ROSIDL_FROM_DIST_PACKAGES}
echo "Comparing the contents:"
diff <(nm -D  ${ROSIDL_FROM_LIB}) <(nm -D ${ROSIDL_FROM_DIST_PACKAGES})
md5sum ${ROSIDL_FROM_LIB} ${ROSIDL_FROM_DIST_PACKAGES}
diff <(readelf -a ${ROSIDL_FROM_LIB}) <(readelf -a ${ROSIDL_FROM_DIST_PACKAGES})

Expected behavior

Only one version gets installed (e.g. foxy)

Actual behavior

Two almost identical versions get installed

Additional information

This popped up while trying to get humble to work with meta-ros where this is a QA issue

ERROR: builtin-interfaces-1.2.0-2-r0 do_package: builtin-interfaces: Multiple shlib providers for libbuiltin_interfaces__rosidl_generator_py.so: builtin-interfaces, builtin-interfaces (used by files: /yocto/sf-build/BUILD-ros2-humble-honister/work/core2-64-oe-linux/builtin-interfaces/1.2.0-2-r0/packages-split/builtin-interfaces/usr/lib/python3.9/site-packages/builtin_interfaces/builtin_interfaces_s__rosidl_typesupport_fastrtps_c.cpython-39-x86_64-linux-gnu.so)

I will have a look into this, but since I am not very familiar with this, I do not think I get this done quick or without side effects.

ErickKramer commented 2 years ago

It seems that the libbuiltin_interfaces__rosidl_generator_py.so within the .local/lib/python3.10/dist-packages is not needed.

I used the following setup for both testing scenarios

Started from the latest ros:humble-ros-base docker image
Created workspace
Created Interface package with a simple Num.msg msg based on Custom ROS2 interfaces tutorial
Created minimal rclpy package with a publisher node using the generated Num.msg

Library in dist-packages dir

Removed the /opt/ros/humble/local/lib/python3.10/dist-packages/builtin_interfaces/libbuiltin_interfaces__rosidl_generator_py.so file
Compiled workspace
Ran the publisher node
Output:
```
ros2 run min_publisher hello 
```
Library in lib dir
Removed the /opt/ros/humble/lib/libbuiltin_interfaces__rosidl_generator_py.so file
Compiled workspace
Ran the publisher node
Output

ros2 run min_publisher hello 
Traceback (most recent call last):
  File "/opt/ros/humble/local/lib/python3.10/dist-packages/rosidl_generator_py/import_type_support_impl.py", line 46, in import_type_support
    return importlib.import_module(module_name, package=pkg_name)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 674, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 571, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1176, in create_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
ImportError: libbuiltin_interfaces__rosidl_generator_py.so: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/ws/install/min_publisher/lib/min_publisher/hello", line 33, in <module>
    sys.exit(load_entry_point('min-publisher', 'console_scripts', 'hello')())
  File "/ws/build/min_publisher/min_publisher/publisher.py", line 27, in main
    minimal_publisher = MinimalPublisher()
  File "/ws/build/min_publisher/min_publisher/publisher.py", line 10, in __init__
    super().__init__('minimal_publisher')
  File "/opt/ros/humble/local/lib/python3.10/dist-packages/rclpy/node.py", line 199, in __init__
    self._parameter_event_publisher = self.create_publisher(
  File "/opt/ros/humble/local/lib/python3.10/dist-packages/rclpy/node.py", line 1290, in create_publisher
    check_is_valid_msg_type(msg_type)
  File "/opt/ros/humble/local/lib/python3.10/dist-packages/rclpy/type_support.py", line 35, in check_is_valid_msg_type
    check_for_type_support(msg_type)
  File "/opt/ros/humble/local/lib/python3.10/dist-packages/rclpy/type_support.py", line 29, in check_for_type_support
    msg_or_srv_type.__class__.__import_type_support__()
  File "/opt/ros/humble/local/lib/python3.10/dist-packages/rcl_interfaces/msg/_parameter_event.py", line 29, in __import_type_support__
    module = import_type_support('rcl_interfaces')
  File "/opt/ros/humble/local/lib/python3.10/dist-packages/rosidl_generator_py/import_type_support_impl.py", line 48, in import_type_support
    raise UnsupportedTypeSupport(pkg_name)
rosidl_generator_py.import_type_support_impl.UnsupportedTypeSupport: Could not import 'rosidl_typesupport_c' for package 'rcl_interfaces'
[ros2run]: Process exited with failure 1

ghost commented 1 year ago

Any progress or statement on this and / or on the pull request?

ros-discourse commented 1 year ago

This issue has been mentioned on ROS Discourse. There might be relevant details there:

https://discourse.ros.org/t/improving-the-developer-experience/29774/18

cottsay commented 1 year ago

I haven't verified this, but I believe that this is where the library gets installed to lib: https://github.com/ros2/rosidl_python/blob/52eaf342deea5fda49d9589deaad6fbb481a6221/rosidl_generator_py/cmake/rosidl_generator_py_generate_interfaces.cmake#L302-L306

...and I believe that it gets installed into site-packages because the library output directory is getting overridden, and I'm not sure it should be: https://github.com/ros2/rosidl_python/blob/52eaf342deea5fda49d9589deaad6fbb481a6221/rosidl_generator_py/cmake/rosidl_generator_py_generate_interfaces.cmake#L156-L159

So figuring out why the output directory is getting overridden is the path forward.

ghost commented 1 year ago

I made a pull request that fixes this:

https://github.com/ament/ament_cmake/pull/413

So far, nobody cared to look at it.

clalancette commented 1 year ago

ament/ament_cmake#413

I don't think that is the right solution. In particular, it seems to fix the symptom (an extra library), rather than the cause (the extra library being created in the first place). I think we probably need to investigate deeper into rosidl_python itself and just not generate that extra library to begin with.

ghost commented 1 year ago

I don't think that is the right solution. In particular, it seems to fix the symptom (an extra library), rather than the cause (the extra library being created in the first place). I think we probably need to investigate deeper into rosidl_python itself and just not generate that extra library to begin with.

Well, I know I have spend some time back then, but since it is some time ago, I do not remember exactly the reasoning for this solution. Interestingly it is some kind of regression, as foxy did not have those extra libraries. Therefore, if you have other suggestions or concrete files / functions to look at I will be happy to improve the PR to a state that it can considered to be merged.

brta-jc commented 11 months ago

This is still an issue. We need to split packages into '_msgs' packages, which for some packages is really unnecessary and cumbersome. Any updates on the PR's? Can we have the pre foxy behavior back please?

ros2 / rosidl_python