CommonplaceRobotics / iRC_ROS

ROS2 packages for the igus Robot Control
Apache License 2.0
16 stars 7 forks source link

CycloneDDS causes errors in controller startup #105

Open TheUncivil opened 1 year ago

TheUncivil commented 1 year ago

Your environment

ROS Distro: Humble
OS Version: Ubuntu 22.04
Branch/Commit hash: humble #1699a1b
Protocol: CPRCANv2
Module Firmware version(s): 03.03

Description

When launching the moveit control interface for the igus ReBel cobot, the system throws many errors, then fails to load the robot description (presumably because it cannot access the robot state). This in turn causes all controllers to fail to launch.

Modules 0x70 and 0x80, which represent the DIO pins for the base and arm respectively, seem to report every error they can report, which makes me think there is something else going on.

I have two ReBel arms available and tried this with both. The issue occurs in both cases

Some parts of the terminal output are shown below. All output was generated in a single attempt to launch; I've simply removed the lines that I didn't think were relevant.

[ros2_control_node-3] [INFO] [1691745360.761066082] [resource_manager]: Loading hardware 'igus_rebel_6dof' 
[ros2_control_node-3] [INFO] [1691745360.762750944] [resource_manager]: Initialize hardware 'igus_rebel_6dof' 
[ros2_control_node-3] [INFO] [1691745360.763139776] [iRC_ROS]: CAN socket set as can0 in the urdf file
[ros2_control_node-3] [INFO] [1691745360.763413919] [iRC_ROS]: Joint joint1: Referencing_required: 0
[ros2_control_node-3] [INFO] [1691745360.763429334] [iRC_ROS]: Joint "joint1" specified with CAN_ID: 0x10 Controller type: closed_loop Gear scale: 1443.550000
[ros2_control_node-3] [INFO] [1691745360.763548552] [iRC_ROS]: Joint joint2: Referencing_required: 0
[ros2_control_node-3] [INFO] [1691745360.763557923] [iRC_ROS]: Joint "joint2" specified with CAN_ID: 0x20 Controller type: closed_loop Gear scale: 1443.550000
[ros2_control_node-3] [INFO] [1691745360.763585142] [iRC_ROS]: Joint joint3: Referencing_required: 0
[ros2_control_node-3] [INFO] [1691745360.763592194] [iRC_ROS]: Joint "joint3" specified with CAN_ID: 0x30 Controller type: closed_loop Gear scale: 1443.550000
[ros2_control_node-3] [INFO] [1691745360.763609902] [iRC_ROS]: Joint joint4: Referencing_required: 0
[ros2_control_node-3] [INFO] [1691745360.763615794] [iRC_ROS]: Joint "joint4" specified with CAN_ID: 0x40 Controller type: closed_loop Gear scale: -1443.550000
[ros2_control_node-3] [INFO] [1691745360.763638985] [iRC_ROS]: Joint joint5: Referencing_required: 0
[ros2_control_node-3] [INFO] [1691745360.763645734] [iRC_ROS]: Joint "joint5" specified with CAN_ID: 0x50 Controller type: closed_loop Gear scale: 1031.110000
[ros2_control_node-3] [INFO] [1691745360.763663678] [iRC_ROS]: Joint joint6: Referencing_required: 0
[ros2_control_node-3] [INFO] [1691745360.763670149] [iRC_ROS]: Joint "joint6" specified with CAN_ID: 0x60 Controller type: closed_loop Gear scale: -1031.110000
[ros2_control_node-3] [INFO] [1691745360.763739885] [iRC_ROS]: DIO "dio_base" specified with CAN_ID: 0x70
[ros2_control_node-3] [INFO] [1691745360.763757851] [iRC_ROS]: DIO "dio_arm" specified with CAN_ID: 0x80
[ros2_control_node-3] [INFO] [1691745360.763778523] [resource_manager]: Successful initialization of hardware 'igus_rebel_6dof'
[ros2_control_node-3] [INFO] [1691745360.764203763] [resource_manager]: 'configure' hardware 'igus_rebel_6dof' 
[ros2_control_node-3] [INFO] [1691745360.764228301] [iRC_ROS::CAN]: Connecting
[ros2_control_node-3] [INFO] [1691745360.764363700] [iRC_ROS::CAN]: Connected to CAN module via SocketCAN...
[robot_state_publisher-1] [INFO] [1691745360.786733672] [robot_state_publisher]: got segment base_link
[robot_state_publisher-1] [INFO] [1691745360.787464266] [robot_state_publisher]: got segment box_frame
[robot_state_publisher-1] [INFO] [1691745360.787889111] [robot_state_publisher]: got segment cam0
[robot_state_publisher-1] [INFO] [1691745360.788279253] [robot_state_publisher]: got segment charuco
[robot_state_publisher-1] [INFO] [1691745360.788668872] [robot_state_publisher]: got segment flange
[robot_state_publisher-1] [INFO] [1691745360.789056473] [robot_state_publisher]: got segment gripper_base
[robot_state_publisher-1] [INFO] [1691745360.789443883] [robot_state_publisher]: got segment gripper_tcp
[robot_state_publisher-1] [INFO] [1691745360.789877176] [robot_state_publisher]: got segment gripper_virtual
[robot_state_publisher-1] [INFO] [1691745360.790316720] [robot_state_publisher]: got segment link_1
[robot_state_publisher-1] [INFO] [1691745360.790795277] [robot_state_publisher]: got segment link_2
[robot_state_publisher-1] [INFO] [1691745360.791295409] [robot_state_publisher]: got segment link_5
[robot_state_publisher-1] [INFO] [1691745360.791770628] [robot_state_publisher]: got segment link_6
[robot_state_publisher-1] [INFO] [1691745360.792209565] [robot_state_publisher]: got segment link_7
[robot_state_publisher-1] [INFO] [1691745360.792642034] [robot_state_publisher]: got segment link_8
[robot_state_publisher-1] [INFO] [1691745360.793086350] [robot_state_publisher]: got segment world
[move_group-10] Warning: Link 'cam0_link' is not known to URDF. Cannot disable/enable collisons.
[move_group-10]          at line 577 in ./src/model.cpp
[move_group-10] Warning: Link 'cam0_link' is not known to URDF. Cannot disable/enable collisons.
[move_group-10]          at line 577 in ./src/model.cpp
[move_group-10] Warning: Link 'cam1_link' is not known to URDF. Cannot disable/enable collisons.
[move_group-10]          at line 572 in ./src/model.cpp
[move_group-10] Warning: Link 'cam1_link' is not known to URDF. Cannot disable/enable collisons.
[move_group-10]          at line 577 in ./src/model.cpp
[move_group-10] Warning: Link 'cam1_link' is not known to URDF. Cannot disable/enable collisons.
[move_group-10]          at line 577 in ./src/model.cpp
[move_group-10] Warning: Link 'cam1_link' is not known to URDF. Cannot disable/enable collisons.
[move_group-10]          at line 577 in ./src/model.cpp
[move_group-10] [INFO] [1691745360.811196244] [moveit_rdf_loader.rdf_loader]: Loaded robot model in 0.0447321 seconds
[move_group-10] [INFO] [1691745360.811264762] [moveit_robot_model.robot_model]: Loading robot model 'igus_rebel_6dof'...
[move_group-10] [INFO] [1691745360.811273012] [moveit_robot_model.robot_model]: No root/virtual joint specified in SRDF. Assuming fixed joint
[move_group-10] [WARN] [1691745360.815026884] [moveit_robot_model.robot_model]: Link gripper_virtual has visual geometry but no collision geometry. Collision geometry will be left empty. Fix your URDF file by explicitly specifying collision geometry.
[move_group-10] [INFO] [1691745360.832788091] [moveit_ros.planning_scene_monitor.planning_scene_monitor]: Publishing maintained planning scene on 'monitored_planning_scene'
[move_group-10] [INFO] [1691745360.832906759] [moveit.ros_planning_interface.moveit_cpp]: Listening to 'joint_states' for joint states
[move_group-10] [INFO] [1691745360.833281130] [moveit_ros.current_state_monitor]: Listening to joint states on topic 'joint_states'
[move_group-10] [INFO] [1691745360.833693502] [moveit_ros.planning_scene_monitor.planning_scene_monitor]: Listening to '/attached_collision_object' for attached collision objects
[move_group-10] [INFO] [1691745360.833705202] [moveit_ros.planning_scene_monitor.planning_scene_monitor]: Starting planning scene monitor
[move_group-10] [INFO] [1691745360.834565459] [moveit_ros.planning_scene_monitor.planning_scene_monitor]: Listening to '/planning_scene'
[move_group-10] [INFO] [1691745360.834575728] [moveit_ros.planning_scene_monitor.planning_scene_monitor]: Starting world geometry update monitor for collision objects, attached objects, octomap updates.
[move_group-10] [INFO] [1691745360.834875250] [moveit_ros.planning_scene_monitor.planning_scene_monitor]: Listening to 'collision_object'
[move_group-10] [INFO] [1691745360.835200602] [moveit_ros.planning_scene_monitor.planning_scene_monitor]: Listening to 'planning_scene_world' for planning scene world geometry
[move_group-10] [WARN] [1691745360.836202015] [moveit.ros.occupancy_map_monitor.middleware_handle]: Resolution not specified for Octomap. Assuming resolution = 0.1 instead
[move_group-10] [ERROR] [1691745360.836218564] [moveit.ros.occupancy_map_monitor.middleware_handle]: No 3D sensor plugin(s) defined for octomap updates
[ros2_control_node-3] [INFO] [1691745361.264725764] [resource_manager]: Successful 'configure' of hardware 'igus_rebel_6dof'
[ros2_control_node-3] [INFO] [1691745361.264818620] [resource_manager]: 'activate' hardware 'igus_rebel_6dof' 
[ros2_control_node-3] [INFO] [1691745361.265032829] [resource_manager]: Successful 'activate' of hardware 'igus_rebel_6dof'
[ros2_control_node-3] [INFO] [1691745361.268177416] [controller_manager]: update rate is 100 Hz
[ros2_control_node-3] [INFO] [1691745361.268434157] [controller_manager]: RT kernel is recommended for better performance
[ros2_control_node-3] [INFO] [1691745361.268611343] [iRC_ROS]: Module 0x60: Startup message received
[ros2_control_node-3] [INFO] [1691745361.268635753] [iRC_ROS]: Module 0x60: Hardware Ident 0x52
[ros2_control_node-3] [INFO] [1691745361.268639877] [iRC_ROS]: Module 0x60: Firmware version is 03.03
[ros2_control_node-3] [INFO] [1691745361.268658419] [iRC_ROS]: Module 0x50: Startup message received
[ros2_control_node-3] [INFO] [1691745361.268662609] [iRC_ROS]: Module 0x50: Hardware Ident 0x52
[ros2_control_node-3] [INFO] [1691745361.268665363] [iRC_ROS]: Module 0x50: Firmware version is 03.03
[ros2_control_node-3] [INFO] [1691745361.268674849] [iRC_ROS]: Module 0x30: Startup message received
[ros2_control_node-3] [INFO] [1691745361.268678026] [iRC_ROS]: Module 0x30: Hardware Ident 0x52
[ros2_control_node-3] [INFO] [1691745361.268680474] [iRC_ROS]: Module 0x30: Firmware version is 03.03
[ros2_control_node-3] [INFO] [1691745361.268687360] [iRC_ROS]: Module 0x20: Startup message received
[ros2_control_node-3] [INFO] [1691745361.268690502] [iRC_ROS]: Module 0x20: Hardware Ident 0x52
[ros2_control_node-3] [INFO] [1691745361.268693130] [iRC_ROS]: Module 0x20: Firmware version is 03.03
[ros2_control_node-3] [INFO] [1691745361.268704429] [iRC_ROS]: Module 0x40: Startup message received
[ros2_control_node-3] [INFO] [1691745361.268707604] [iRC_ROS]: Module 0x40: Hardware Ident 0x52
[ros2_control_node-3] [INFO] [1691745361.268710209] [iRC_ROS]: Module 0x40: Firmware version is 03.03
[ros2_control_node-3] [INFO] [1691745361.268718509] [iRC_ROS]: Module 0x10: Startup message received
[ros2_control_node-3] [INFO] [1691745361.268721679] [iRC_ROS]: Module 0x10: Hardware Ident 0x52
[ros2_control_node-3] [INFO] [1691745361.268724283] [iRC_ROS]: Module 0x10: Firmware version is 03.03
[ros2_control_node-3] [INFO] [1691745361.268739643] [iRC_ROS]: Module 0x80: Errors TEMP ESTOP MNE COM LAG LAG DRV OC detected, resetting
[ros2_control_node-3] [INFO] [1691745361.268743957] [iRC_ROS]: Module 0x80: Resetting
[ros2_control_node-3] [INFO] [1691745361.268830206] [iRC_ROS]: Module 0x70: Errors TEMP ESTOP MNE COM LAG LAG DRV OC detected, resetting
[ros2_control_node-3] [INFO] [1691745361.268835415] [iRC_ROS]: Module 0x70: Resetting
[ros2_control_node-3] [INFO] [1691745361.278752075] [iRC_ROS]: Module 0x70: Errorcode: 00000100 
[ros2_control_node-3] [INFO] [1691745361.278810921] [iRC_ROS]: Module 0x60: Errorcode: 00011100 
[ros2_control_node-3] [INFO] [1691745361.278883945] [iRC_ROS]: Module 0x50: Errorcode: 00011100 
[ros2_control_node-3] [INFO] [1691745361.278899910] [iRC_ROS]: Module 0x30: Errorcode: 00011100 
[ros2_control_node-3] [INFO] [1691745361.278913241] [iRC_ROS]: Module 0x20: Errorcode: 00011100 
[ros2_control_node-3] [INFO] [1691745361.278925576] [iRC_ROS]: Module 0x40: Errorcode: 00011100 
[ros2_control_node-3] [INFO] [1691745361.278938618] [iRC_ROS]: Module 0x10: Errorcode: 00011100 
[ros2_control_node-3] [INFO] [1691745361.278956478] [iRC_ROS]: Module 0x80: Errors TEMP ESTOP MNE COM LAG LAG DRV OC detected, resetting
[ros2_control_node-3] [INFO] [1691745361.279043274] [iRC_ROS]: Module 0x70: Errors MNE detected, resetting
[ros2_control_node-3] [INFO] [1691745361.279052556] [iRC_ROS]: Module 0x70: Enabling motor
[ros2_control_node-3] [INFO] [1691745361.288600708] [iRC_ROS]: Module 0x80: Errorcode: 00000100 
[ros2_control_node-3] [INFO] [1691745361.288659755] [iRC_ROS]: Module 0x70: Errorcode: 00000000 
[ros2_control_node-3] [INFO] [1691745361.288766063] [iRC_ROS]: Module 0x80: Errors MNE detected, resetting
[ros2_control_node-3] [INFO] [1691745361.288775228] [iRC_ROS]: Module 0x80: Enabling motor
[ros2_control_node-3] [INFO] [1691745361.298661194] [iRC_ROS]: Module 0x80: Errors MNE detected, resetting
[ros2_control_node-3] [INFO] [1691745361.308594833] [iRC_ROS]: Module 0x80: Errorcode: 00000000 
[spawner-6] [INFO] [1691745363.076718708] [spawner_vacuum_controller]: Waiting for '/controller_manager' node to exist
[spawner-5] [INFO] [1691745363.159620855] [spawner_joint_trajectory_controller]: Waiting for '/controller_manager' node to exist
[spawner-8] [INFO] [1691745363.163450759] [spawner_dashboard_controller]: Waiting for '/controller_manager' node to exist
[spawner-4] [INFO] [1691745363.166874073] [spawner_joint_state_broadcaster]: Waiting for '/controller_manager' node to exist
[spawner-7] [INFO] [1691745363.231795846] [spawner_dio_controller]: Waiting for '/controller_manager' node to exist
[spawner-6] [INFO] [1691745365.083582072] [spawner_vacuum_controller]: Waiting for '/controller_manager' node to exist
[spawner-5] [INFO] [1691745365.166338656] [spawner_joint_trajectory_controller]: Waiting for '/controller_manager' node to exist
[spawner-8] [INFO] [1691745365.170150253] [spawner_dashboard_controller]: Waiting for '/controller_manager' node to exist
[spawner-4] [INFO] [1691745365.173247678] [spawner_joint_state_broadcaster]: Waiting for '/controller_manager' node to exist
[spawner-7] [INFO] [1691745365.238686089] [spawner_dio_controller]: Waiting for '/controller_manager' node to exist
[spawner-6] [INFO] [1691745367.090569157] [spawner_vacuum_controller]: Waiting for '/controller_manager' node to exist
[spawner-5] [INFO] [1691745367.173283498] [spawner_joint_trajectory_controller]: Waiting for '/controller_manager' node to exist
[spawner-8] [INFO] [1691745367.176683849] [spawner_dashboard_controller]: Waiting for '/controller_manager' node to exist
[spawner-4] [INFO] [1691745367.179767581] [spawner_joint_state_broadcaster]: Waiting for '/controller_manager' node to exist
[spawner-7] [INFO] [1691745367.245532210] [spawner_dio_controller]: Waiting for '/controller_manager' node to exist
[spawner-6] [INFO] [1691745369.097643292] [spawner_vacuum_controller]: Waiting for '/controller_manager' node to exist
[spawner-5] [INFO] [1691745369.180048186] [spawner_joint_trajectory_controller]: Waiting for '/controller_manager' node to exist
[spawner-8] [INFO] [1691745369.183312788] [spawner_dashboard_controller]: Waiting for '/controller_manager' node to exist
[spawner-4] [INFO] [1691745369.186560076] [spawner_joint_state_broadcaster]: Waiting for '/controller_manager' node to exist
[spawner-7] [INFO] [1691745369.252335394] [spawner_dio_controller]: Waiting for '/controller_manager' node to exist
[spawner-6] [ERROR] [1691745371.104647851] [spawner_vacuum_controller]: Controller manager not available
[spawner-5] [ERROR] [1691745371.187015631] [spawner_joint_trajectory_controller]: Controller manager not available
[spawner-8] [ERROR] [1691745371.189642297] [spawner_dashboard_controller]: Controller manager not available
[spawner-4] [ERROR] [1691745371.192943318] [spawner_joint_state_broadcaster]: Controller manager not available
[spawner-7] [ERROR] [1691745371.259660410] [spawner_dio_controller]: Controller manager not available
[rviz2-11] [ERROR] [1691745371.281271062] [rviz]: Could not find parameter robot_description and did not receive robot_description via std_msgs::msg::String subscription within 10.000000 seconds.
[ERROR] [spawner-6]: process has died [pid 31176, exit code 1, cmd '/opt/ros/humble/lib/controller_manager/spawner vacuum_controller -c /controller_manager --ros-args -r __ns:=/'].
[ERROR] [spawner-5]: process has died [pid 31174, exit code 1, cmd '/opt/ros/humble/lib/controller_manager/spawner joint_trajectory_controller -c /controller_manager --ros-args -r __ns:=/'].
[ERROR] [spawner-8]: process has died [pid 31180, exit code 1, cmd '/opt/ros/humble/lib/controller_manager/spawner dashboard_controller -c /controller_manager --ros-args -r __ns:=/'].
[ERROR] [spawner-4]: process has died [pid 31172, exit code 1, cmd '/opt/ros/humble/lib/controller_manager/spawner joint_state_broadcaster -c /controller_manager --ros-args -r __ns:=/'].
[ERROR] [spawner-7]: process has died [pid 31178, exit code 1, cmd '/opt/ros/humble/lib/controller_manager/spawner dio_controller -c /controller_manager --ros-args -r __ns:=/']
TheUncivil commented 1 year ago

I've also tried to control the robot with the igus Robot Controller, which works normally.

TheUncivil commented 1 year ago

I've attempted to reproduce this on a separate machine, but the system worked normally. This leads me to believe the problem is caused by something else on my computer.

Ultimately, it turns out that the issue was only occurring when I am using CycloneDDS for ROS2. When I don't use CycloneDDS, the system works normally.

Any idea why CycloneDDS would cause this issue?

cpr-fer commented 1 year ago

Hi @TheUncivil ,

thank you for the bug report and already finding the root of the problem yourself! I am not sure why CycloneDDS causes the issue with only the DIO modules, but not the motor modules.

The only differrence between those are the ros2_controllers, where the DIO modules are using their own controller (found in the controllers package of this repo). Still there is nothing special going on in them, the communication over DDS happening there are 2 publishers, one subscriber and one service, all using default settings.

I'll google around a bit and see if other people ran into similar issues with CycloneDDS, until then I'll add a comment in the readme about it. Can you tell me which DDS implementation you are using instead?

Oh and btw, you might want to remove the controllers you are not using from your launch file to get less spam on startup :sweat_smile:

TheUncivil commented 1 year ago

Can you tell me which DDS implementation you are using instead?

I'm currently using the ROS default DDS implementation, which is eProsima Fast DDS according to this page: https://docs.ros.org/en/humble/Concepts/Intermediate/About-Different-Middleware-Vendors.html

Oh and btw, you might want to remove the controllers you are not using from your launch file to get less spam on startup

I am launching too many controllers, yes. I was planning to remove the ones I don't use this week.

cpr-fer commented 1 year ago

I'm currently using the ROS default DDS implementation, which is eProsima Fast DDS according to this page: https://docs.ros.org/en/humble/Concepts/Intermediate/About-Different-Middleware-Vendors.html

Okay that is the same that I've been using the most. I remember also running into some other issues with CycloneDDS at the beginning of the project but don't remember exactly what it was...

Flauriane commented 5 months ago

Hi, please did anyone face this issues MicrosoftTeams-image (3) MicrosoftTeams-image (2) MicrosoftTeams-image (1) MicrosoftTeams-image