search

UniversalRobots / Universal_Robots_ROS2_Driver

Universal Robots ROS2 driver supporting CB3 and e-Series
BSD 3-Clause "New" or "Revised" License
436 stars 226 forks source link

Pipeline producer overflowed! <RTDE Data Pipeline> #911

Open Hytac opened 10 months ago

Hytac commented 10 months ago

Affected ROS2 Driver version(s)

017a54c3991cb3f576d1d6dc0e7d039ef0770826

Used ROS distribution.

Humble

Which combination of platform is the ROS driver running on.

Ubuntu Linux with standard kernel

How is the UR ROS2 Driver installed.

Build both the ROS driver and UR Client Library from source

Which robot platform is the driver connected to.

UR E-series robot, URSim in docker

Robot SW / URSim version(s)

(Baseline/Polyscope/Controller/Firmware/Platofrm) UR10e 11.3.1001/74.11.1642/41.6.15/32.3.151/0.1.1 URSim 12.3.417/74.13.232/66.7.16/38.7.85/0.1.1

How is the ROS driver used.

Through the robot teach pendant using External Control URCap

Issue details

Summary

My initial_joint_controllers file includes the following boradcaster and controllers:

joint_state_broadcaster:
      type: joint_state_broadcaster/JointStateBroadcaster

io_and_status_controller:
      type: ur_controllers/GPIOController

speed_scaling_state_broadcaster:
      type: ur_controllers/SpeedScalingStateBroadcaster

force_torque_sensor_broadcaster:
      type: force_torque_sensor_broadcaster/ForceTorqueSensorBroadcaster

joint_trajectory_controller:
      type: joint_trajectory_controller/JointTrajectoryController

scaled_joint_trajectory_controller:
      type: ur_controllers/ScaledJointTrajectoryController

forward_velocity_controller:
      type: velocity_controllers/JointGroupVelocityController

forward_position_controller:
      type: position_controllers/JointGroupPositionController

forward_position_controller_zstage:
      type: forward_command_controller/ForwardCommandController

joint_trajectory_controller_zstage:
      type: joint_trajectory_controller/JointTrajectoryController

Last two entries are related to a prismatic axis where the UR10e is mounted. This file is pass to the ur_robot_driver ur_ros2_control_node as shown in here everything is configred

[ur_ros2_control_node-1] [INFO 1706012200.571395246] [controller_manager]: Loading controller 'scaled_joint_trajectory_controller' (load_controller() at ./src/controller_manager.cpp:279)
[ur_ros2_control_node-1] [INFO 1706012200.651275989] [controller_manager]: Configuring controller 'scaled_joint_trajectory_controller' (configure_controller() at ./src/controller_manager.cpp:418)
[ur_ros2_control_node-1] [INFO 1706012200.661763773] [controller_manager]: Configuring controller 'speed_scaling_state_broadcaster' (configure_controller() at ./src/controller_manager.cpp:418)
[ur_ros2_control_node-1] [INFO 1706012200.681164248] [controller_manager]: Configuring controller 'joint_state_broadcaster' (configure_controller() at ./src/controller_manager.cpp:418)
[spawner-4] [INFO 1706012200.701698345] [spawner_scaled_joint_trajectory_controller]: Configured and activated scaled_joint_trajectory_controller (main() at /opt/ros/humble/local/lib/python3.10/dist-packages/controller_manager/spawner.py:241)
[ur_ros2_control_node-1] [INFO 1706012204.312974819] [controller_manager]: Loading controller 'joint_trajectory_controller_zstage' (load_controller() at ./src/controller_manager.cpp:279)
[spawner-32] [INFO 1706012204.351622674] [spawner_joint_trajectory_controller_zstage]: Loaded joint_trajectory_controller_zstage (main() at /opt/ros/humble/local/lib/python3.10/dist-packages/controller_manager/spawner.py:209)
[spawner-32] [INFO 1706012204.371540252] [spawner_joint_trajectory_controller_zstage]: Configured and activated joint_trajectory_controller_zstage (main() at /opt/ros/humble/local/lib/python3.10/dist-packages/controller_manager/spawner.py:241)
[INFO] [spawner-32]: process has finished cleanly [pid 237301]
[move_group-16] [INFO 1706012199.286637592] [moveit.plugins.moveit_simple_controller_manager]: Added FollowJointTrajectory controller for scaled_joint_trajectory_controller (initialize() at ./src/moveit_simple_controller_manager.cpp:182)
[move_group-16] [INFO 1706012199.290903332] [moveit.plugins.moveit_simple_controller_manager]: Added FollowJointTrajectory controller for joint_trajectory_controller_zstage (initialize() at ./src/moveit_simple_controller_manager.cpp:182)
[move_group-16] [INFO 1706012199.291091017] [moveit.plugins.moveit_simple_controller_manager]: Returned 2 controllers in list (getControllersList() at ./src/moveit_simple_controller_manager.cpp:233)
[move_group-16] [INFO 1706012199.291121931] [moveit.plugins.moveit_simple_controller_manager]: Returned 2 controllers in list (getControllersList() at ./src/moveit_simple_controller_manager.cpp:233)

Issue details

When using URSim (use_fake_harfware is false) i can command both ur and prismatic axis with no problem. Simply by using this: 

ros2 topic pub /joint_trajectory_controller_zstage/joint_trajectory trajectory_msgs/msg/JointTrajectory "{header: {stamp: {sec: 0, nanosec: 0}, frame_id: 'base_link'}, joint_names: ['ewellix_lift_top_joint'], points: [{positions: [0.0], velocities: [], accelerations: [], effort: [], time_from_start: {sec: 1, nanosec: 0}}]}" -1

ros2 topic pub /scaled_joint_trajectory_controller/joint_trajectory trajectory_msgs/msg/JointTrajectory "{header: {stamp: {sec: 0, nanosec: 0}, frame_id: 'base_link'}, joint_names: ['ur_shoulder_pan_joint','ur_shoulder_lift_joint','ur_elbow_joint','ur_wrist_1_joint','ur_wrist_2_joint','ur_wrist_3_joint'], points: [{positions: [-1.58, -1.60, -2.19, 0.0, 0.0, 0.0], velocities: [], accelerations: [], effort: [], time_from_start: {sec: 1, nanosec: 0}}]}" -1

Both move fine.

The problem appear when the system is connected to the real thing. I'm still able to move both but while the prismatic axis is moving i got the following error:

[ur_ros2_control_node-1] [INFO 1706012225.218901302] [UR_Client_Library:ur_]: Robot requested program (() at ./src/control/script_sender.cpp:58)
[ur_ros2_control_node-1] [INFO 1706012225.218952951] [UR_Client_Library:ur_]: Sent program to robot (() at ./src/control/script_sender.cpp:71)
[ur_ros2_control_node-1] [INFO 1706012225.267773292] [UR_Client_Library:ur_]: Robot connected to reverse interface. Ready to receive control commands. (() at ./src/control/reverse_interface.cpp:219)
[ur_ros2_control_node-1] SerialComTlt::setColumnSize 0.0435531 PRISMATIC AXIS IS MOVING NOW
[ur_ros2_control_node-1] [ERROR 1706012261.287439809] [UR_Client_Library:ur_]: Pipeline producer overflowed! <RTDE Data Pipeline> (() at ./include/ur_client_library/comm/pipeline.h:406)
[ur_ros2_control_node-1] [ERROR 1706012261.289402578] [UR_Client_Library:ur_]: Pipeline producer overflowed! <RTDE Data Pipeline> (() at ./include/ur_client_library/comm/pipeline.h:406)
[ur_ros2_control_node-1] [ERROR 1706012261.291397575] [UR_Client_Library:ur_]: Pipeline producer overflowed! <RTDE Data Pipeline> (() at ./include/ur_client_library/comm/pipeline.h:406)
[ur_ros2_control_node-1] [ERROR 1706012261.293390850] [UR_Client_Library:ur_]: Pipeline producer overflowed! <RTDE Data Pipeline> (() at ./include/ur_client_library/comm/pipeline.h:406)
[ur_ros2_control_node-1] [ERROR 1706012261.295394266] [UR_Client_Library:ur_]: Pipeline producer overflowed! <RTDE Data Pipeline> (() at ./include/ur_client_library/comm/pipeline.h:406)
[ur_ros2_control_node-1] [ERROR 1706012261.297405407] [UR_Client_Library:ur_]: Pipeline producer overflowed! <RTDE Data Pipeline> (() at ./include/ur_client_library/comm/pipeline.h:406)
[ur_ros2_control_node-1] [ERROR 1706012261.299419470] [UR_Client_Library:ur_]: Pipeline producer overflowed! <RTDE Data Pipeline> (() at ./include/ur_client_library/comm/pipeline.h:406)
[ur_ros2_control_node-1] [ERROR 1706012261.301414504] [UR_Client_Library:ur_]: Pipeline producer overflowed! <RTDE Data Pipeline> (() at ./include/ur_client_library/comm/pipeline.h:406)
[ur_ros2_control_node-1] [ERROR 1706012261.303436438] [UR_Client_Library:ur_]: Pipeline producer overflowed! <RTDE Data Pipeline> (() at ./include/ur_client_library/comm/pipeline.h:406)
[ur_ros2_control_node-1] [ERROR 1706012261.305388572] [UR_Client_Library:ur_]: Pipeline producer overflowed! <RTDE Data Pipeline> (() at ./include/ur_client_library/comm/pipeline.h:406)
[ur_ros2_control_node-1] [ERROR 1706012261.307448800] [UR_Client_Library:ur_]: Pipeline producer overflowed! <RTDE Data Pipeline> (() at ./include/ur_client_library/comm/pipeline.h:406)
[ur_ros2_control_node-1] [ERROR 1706012261.309384373] [UR_Client_Library:ur_]: Pipeline producer overflowed! <RTDE Data Pipeline> (() at ./include/ur_client_library/comm/pipeline.h:406)
[ur_ros2_control_node-1] [ERROR 1706012261.311449759] [UR_Client_Library:ur_]: Pipeline producer overflowed! <RTDE Data Pipeline> (() at ./include/ur_client_library/comm/pipeline.h:406)
[ur_ros2_control_node-1] [ERROR 1706012261.313388964] [UR_Client_Library:ur_]: Pipeline producer overflowed! <RTDE Data Pipeline> (() at ./include/ur_client_library/comm/pipeline.h:406)

While the prismatic axis is moving the errors appear in a loop. when moving the real UR the error does not appear. After the error i can keep using both controllers.

Relevant log output

No response

Accept Public visibility

fmauch commented 6 months ago

This sounds like the hardware interface of your prismatic axis is slowing down the control loop while executing motions.

If the control loop isn't keeping up with the 500Hz, the robot is sending in data while it has not been processed by the hardware interface, hence the Pipeline producer overflowed! error.

ros2_control has a relatively new feature for running components and controllers asynchronously, that could probably help you there, but I didn't test that, yet.

Elfits commented 2 months ago

Any news? I am using a gripper through RTDE and facing the same problem. The asynchronous controllers did not work...