search

UniversalRobots / Universal_Robots_ROS2_Driver

Universal Robots ROS2 driver supporting CB3 and e-Series
BSD 3-Clause "New" or "Revised" License
399 stars 209 forks source link

No realtime capabilities found - Connection to reverse interface dropped #382

Closed TE-2 closed 2 years ago

TE-2 commented 2 years ago

Hi, I am currently working on a ROS2 Foxy Snap (Snapcraft) with UR driver included. The snap should run on a ctrlx CORE device which uses a Linux arm64 system with realtime kernel (5.4.143-rt63 SMP PREEMPT_RT).

I have already created and executed the snap on the system and connected it to a robot (UR CB3). [INFO] [UR_Client_Library]: Robot connected to reverse interface. Ready to receive control commands.

Unfortunately, the connection to the robot breaks off again after a few moments. [INFO] [UR_Client_Library]: Connection to reverse interface dropped.

The system's computing power should be equivalent to that of a Raspberry Pi 3 or 4. I have read that without real-time capability on a Raspberry Pi, the same problem exists: Connection to reverse interface dropped - UR10e

However, although my operating system is real-time capable, I get the following message when I start the driver: [WARN] [UR_Client_Library]: No realtime capabilities found. Consider using a realtime system for better performance

Is there a way to enable the realtime capability of the ROS2 UR driver on my system?

I have read in this issue that you can add the following code to the script of the ur_ros2_control_node to enable realtime, but I have not found this script under the Foxy Branch.

danzimmerman commented 2 years ago

Is there a way to enable the realtime capability of the ROS2 UR driver on my system?

I wonder if there's more system configuration you need to do to get your Snap to run on real-time cores.

A glance at the CtrlX Core manual makes it seem like there are a mix of realtime-enabled cores and not. I'm looking on page 10/28 here

I have read in this https://github.com/UniversalRobots/Universal_Robots_ROS2_Driver/issues/381 that you can add the following code to the script of the ur_ros2_control_node to enable realtime, but I have not found this script under the Foxy Branch.

I don't think this is the issue. I didn't modify any code to add additional real-time capabilities yet as discussed there. I'm just running the unmodified driver on the galactic branch and my system is trying to use RT priority on some threads (On an ordinary laptop running the PREEMPT_RT kernel by default)

I had to deal with some permissions issues. I was getting different real-time related errors from UR Client Library during the driver bringup, like

[UR_Client_Library]: Unsuccessful in setting producer thread realtime priority. Error code: 1

etc.

To resolve these errors, I had to follow the instructions to make a realtime group and adjust permissions as detailed here:

https://github.com/UniversalRobots/Universal_Robots_ROS_Driver/blob/master/ur_robot_driver/doc/real_time.md#setup-user-privileges-to-use-real-time-scheduling

https://forum.universal-robots.com/t/produce-thread-realtime-priority-error-message/6805

Now I get 

[ur_ros2_control_node-1] [INFO] [1654113047.211602604] [UR_Client_Library]: Producer thread: SCHED_FIFO OK
[ur_ros2_control_node-1] [INFO] [1654113047.211647061] [UR_Client_Library]: Thread priority is 99

But this is pretty different from what you're seeing, which seems like the software does not know at all that you have the real-time kernel available. When I get a chance I'll reboot into a non-rt kernel and see if I get a similar message to yours.

danzimmerman commented 2 years ago

Yes if I reboot into the generic non-realtime kernel

dan@computer:~$ uname -r
5.13.0-44-generic

I get the same warning you get

[ur_ros2_control node-1] [WARN] [1654117225.812684613] [UR_Client_Library]: No realtime capabilities found. Consider using a realtime system for better performance

and it looks like after a short time I also drop the connection to the reverse interface.

So I'd probably look at additional config steps to allow your Snap to use your realtime kernel if that's possible.

TE-2 commented 2 years ago

Sorry for my late feedback

I now have sudo privileges and have been able to try several things.

But first of all thanks for your effort @danzimmerman, the tutorial setup user privileges to use real-time scheduling was very helpful. However, I could not edit the file /etc/security/limits.conf on the controller.

Realtime is only recognised when the snap is executed in devmode and with sudo rights.

[ros2_control_node-1] [INFO] [1654701213.079833517] [URPositionHardwareInterface]: Initializing driver...
[ros2_control_node-1] [INFO] [1654701213.108062648] [UR_Client_Library]: Producer thread: SCHED_FIFO OK
[ros2_control_node-1] [INFO] [1654701213.108234728] [UR_Client_Library]: Thread priority is 99
[ros2_control_node-1] [INFO] [1654701213.110869029] [UR_Client_Library]: Negotiated RTDE protocol version to 2.
[ros2_control_node-1] [INFO] [1654701213.111776349] [UR_Client_Library]: Setting up RTDE communication with frequency 500.000000
[ros2_control_node-1] [INFO] [1654701214.185424372] [URPositionHardwareInterface]: Calibration checksum: 'calib_16756443741236045476'.
[ros2_control_node-1] [INFO] [1654701214.201952269] [UR_Client_Library]: Producer thread: SCHED_FIFO OK
[ros2_control_node-1] [INFO] [1654701214.202122459] [UR_Client_Library]: Thread priority is 99
[ros2_control_node-1] [INFO] [1654701215.214450668] [URPositionHardwareInterface]: Calibration checked successfully.
[ros2_control_node-1] [INFO] [1654701215.232781735] [UR_Client_Library]: Producer thread: SCHED_FIFO OK
[ros2_control_node-1] [INFO] [1654701215.232989335] [UR_Client_Library]: Thread priority is 99
[ros2_control_node-1] [INFO] [1654701215.235067996] [URPositionHardwareInterface]: System successfully started!

Running the snap in devmode without sudo I get:

[ros2_control_node-1] [INFO] [1654775823.293365592] [URPositionHardwareInterface]: Initializing driver...
[ros2_control_node-1] [ERROR] [1654775823.311075973] [UR_Client_Library]: Unsuccessful in setting producer thread realtime priority. Error code: 1
[ros2_control_node-1] [ERROR] [1654775823.311280183] [UR_Client_Library]: Producer thread: Scheduling is NOT SCHED_FIFO!
[ros2_control_node-1] [INFO] [1654775823.311316233] [UR_Client_Library]: Thread priority is 0
[ros2_control_node-1] [INFO] [1654775823.533821518] [UR_Client_Library]: Negotiated RTDE protocol version to 2.
[ros2_control_node-1] [INFO] [1654775823.538187293] [UR_Client_Library]: Setting up RTDE communication with frequency 500.000000
[ros2_control_node-1] [INFO] [1654775824.562015967] [URPositionHardwareInterface]: Calibration checksum: 'calib_16756443741236045476'.
[ros2_control_node-1] [ERROR] [1654775824.580467031] [UR_Client_Library]: Unsuccessful in setting producer thread realtime priority. Error code: 1
[ros2_control_node-1] [ERROR] [1654775824.587760366] [UR_Client_Library]: Producer thread: Scheduling is NOT SCHED_FIFO!
[ros2_control_node-1] [INFO] [1654775824.587842566] [UR_Client_Library]: Thread priority is 0
[ros2_control_node-1] [INFO] [1654775825.646512340] [URPositionHardwareInterface]: Calibration checked successfully.
[ros2_control_node-1] [ERROR] [1654775825.656785716] [UR_Client_Library]: Unsuccessful in setting producer thread realtime priority. Error code: 1
[ros2_control_node-1] [ERROR] [1654775825.663735970] [UR_Client_Library]: Producer thread: Scheduling is NOT SCHED_FIFO!
[ros2_control_node-1] [INFO] [1654775825.669299869] [UR_Client_Library]: Thread priority is 0
[ros2_control_node-1] [INFO] [1654775825.675921542] [URPositionHardwareInterface]: System successfully started!

(This error is probably caused by the missing settings for the user group in /etc/security/limits.conf)

I also noticed that an RTPS error occurs when the snap is not run in devmode (sudo doesn't matter):

[ros2_control_node-1] 2022-06-09 11:52:28.413 [RTPS_TRANSPORT_SHM Error] Failed to create segment 15e0bd55f1dcc25d: Permission denied -> Function compute_per_allocation_extra_size
[ros2_control_node-1] 2022-06-09 11:52:28.414 [RTPS_MSG_OUT Error] Permission denied -> Function init
[dashboard_client-2] 2022-06-09 11:52:28.451 [RTPS_TRANSPORT_SHM Error] Failed to create segment 53dff7d440af4f72: Permission denied -> Function compute_per_allocation_extra_size
[dashboard_client-2] 2022-06-09 11:52:28.451 [RTPS_MSG_OUT Error] Permission denied -> Function init
[dashboard_client-2] [INFO] [1654775548.980310232] [UR_Client_Library]: Connected: Universal Robots Dashboard Server
[dashboard_client-2]
[robot_state_publisher-3] 2022-06-09 11:52:29.479 [RTPS_TRANSPORT_SHM Error] Failed to create segment ca4de7c49658c82a: Permission denied -> Function compute_per_allocation_extra_size
[robot_state_publisher-3] 2022-06-09 11:52:29.479 [RTPS_MSG_OUT Error] Permission denied -> Function init

Summary:

If I run my snap with sudo rights and Devmode, everything works without problems.

The problem is that the snap must later function without sudo rights and devmode, for normal user installation. If I knew which accesses the UR driver exactly needs, I maybe can fix the RTPS and the No realtime capabilities found. Consider using a realtime system for better performance - Error. The only problem would then be the user group with the authorisation to assign priorities.

fmauch commented 2 years ago

From reading through this issue it seems that the (permission) problem has already been identified and from my perspective I would say this isn't inside this driver's scope, but rather due to the permission management on the kernel's realtime capabilities. Please comment / contradict if you think different.

TE-2 commented 2 years ago

Yes exactly these problems are purely snapcraft specific. It would be good to know where in the source code the realtime login was implemented and what permissions are needed (also for RTPS). This information would help me a lot.

danzimmerman commented 2 years ago

It would be good to know where in the source code the realtime login was implemented

I don't think there's anything like that, instead there are unusual permissions required for using all the features of the PREEMPT_RT kernel.

Some more info here:

https://rt.wiki.kernel.org/index.php/Frequently_Asked_Questions#Do_I_need_root_privileges_to_start_a_realtime_application.3F

sumit280188 commented 1 month ago

Model - UR5e

Working platform wsl2 ubuntu 22.04.4 ROS2 humble also tried in Ubuntu machine with Ubuntu 22.04 LTS humble also tried in Ubuntu machine with Ubuntu 24.04 LTS rolling

Network is connected.

Since 2 weeks, I am trying to execute plan on moveit, but it’s not working.

build is clean. 8 packages buid

ros2 launch ur_calibration calibration_correction.launch.py robot_ip:=192.168.1.102 target_filename:=“${HOME}/my_robot_calibration.yaml”

calibration is also correct.

error and warn lines running ros2 launch ur_robot_driver ur_control.launch.py ur_type:=ur5e robot_ip:=192.168.1.102

[rviz2-6] QStandardPaths: wrong permissions on runtime directory /run/user/1000/, 0755 instead of 0700 [controller_stopper_node-3] [INFO] [1723653744.736396677] [Controller stopper]: Waiting for switch controller service to come up on controller_manager/switch_controller [ur_ros2_control_node-1] [WARN] [1723653744.739793032] [controller_manager]: [Deprecated] Passing the robot description parameter directly to the control_manager node is deprecated. Use ‘~/robot_description’ topic from ‘robot_state_publisher’ instead.

[ros2_control_node-1] [WARN] [1723721237.025706627] [UR_Client_Library:]: Your system/user seems not to be setup for FIFO scheduling. We recommend using a lowlatency kernel with FIFO scheduling. See https://github.com/UniversalRobots/Universal_Robots_ROS_Driver/blob/master/ur_robot_driver/doc/real_time.md for details.

[ros2_control_node-1] [ERROR] [1723721237.025435415] [URPositionHardwareInterface]: The calibration parameters of the connected robot don't match the ones from the given kinematics config file. Please be aware that this can lead to critical inaccuracies of tcp positions. Use the ur_calibration tool to extract the correct calibration from the robot and pass that into the description. See [https://github.com/UniversalRobots/Universal_Robots_ROS2_Driver/blob/main/ur_calibration/README.md] for details.

[INFO] [spawner-7]: process has finished cleanly [pid 1053] [rviz2-6] [INFO] [1723653502.837449816] [rviz2]: Stereo is NOT SUPPORTED [rviz2-6] [INFO] [1723653502.837638218] [rviz2]: OpenGl version: 4.1 (GLSL 4.1) [rviz2-6] [INFO] [1723653502.905336838] [rviz2]: Stereo is NOT SUPPORTED [INFO] [spawner-8]: process has finished cleanly [pid 1055] [ur_ros2_control_node-1] [WARN] [1723653520.345612323] [UR_Client_Library:]: Failed to read from stream, reconnecting in 1 seconds… [ur_ros2_control_node-1] [WARN] [1723653522.365852625] [UR_Client_Library:]: Failed to read from stream, reconnecting in 2 seconds… [ur_ros2_control_node-1] [WARN] [1723653525.369592692] [UR_Client_Library:]: Failed to read from stream, reconnecting in 4 seconds…

The problem seems to be connected with the Kernel, but with Ubuntu pro 24.04 Noble and ros2 rolling, it has the same issue.

If I run

ros2 launch ur_robot_driver ur_control.launch.py ur_type:=ur5e robot_ip:=192.168.1.102 kinematics_config:=“${HOME}/my_robot_calibration.yaml”

I get additional errors: [ur_ros2_control_node-1] [ERROR] [1723653502.158118606] [URPositionHardwareInterface]: The calibration parameters of the connected robot don’t match the ones from the given kinematics config file. Please be aware that this can lead to critical inaccuracies of tcp positions. Use the ur_calibration tool to extract the correct calibration from the robot and pass that into the description. See [Universal_Robots_ROS2_Driver/ur_calibration/README.md at main · UniversalRobots/Universal_Robots_ROS2_Driver · GitHub] for details.

[ur_ros2_control_node-1] [WARN] [1723653502.179572416] [controller_manager]: Could not enable FIFO RT scheduling policy [ur_ros2_control_node-1] [WARN] [1723653502.179921928] [UR_Client_Library:]: Your system/user seems not to be setup for FIFO scheduling. We recommend using a lowlatency kernel with FIFO scheduling. See Universal_Robots_ROS_Driver/ur_robot_driver/doc/real_time.md at master · UniversalRobots/Universal_Robots_ROS_Driver · GitHub for details. [ur_ros2_control_node-1] [INFO] [1723653502.421657172] [controller_manager]: Loading controller ‘scaled_joint_trajectory_controller’ [ur_ros2_control_node-1] [WARN] [1723653502.480625431] [scaled_joint_trajectory_controller]: [Deprecated]: “allow_nonzero_velocity_at_trajectory_end” is set to true. The default behavior will change to false.

Updated to the latest kernel version.

202408141037 SMP PREEMPT_DYNAMIC Wed

What is the problem?