[BUG] - Unable to connect to the physical robot KUKA lbr iisy 3

[prasuchit]

Description

Hi @Svastits ,

I do most of my development on fake_hardware since its easier than connecting to the real robot. Recently I wanted to test out my code on the real robot and I tried connecting to it and started having trouble connecting.

To connect to the real robot, I run moveit_planning_example launch file with use_fake_hardware:=false, client_ip:="host computer ip", controller_ip:="KR_C5 micro ip"

I used to be able to connect just fine with the robot until recently when I discovered the controller_manager event_broadcaster dying every time for some reason and you helped me fix that by providing the command to run it after the launch file starts using:

ros2 run controller_manager spawner event_broadcaster -c controller_manager --inactive

Which fixed the problem and I was able to talk to the robot just fine. Now I don't know why, but after I launch the launch file and the event_broadcaster, this is what I see:

[control_node-1] [INFO] [1716838756.222327711] [controller_manager]: Loading controller 'event_broadcaster'
[control_node-1] [INFO] [1716838756.252226713] [controller_manager]: Configuring controller 'event_broadcaster'

But it never returns from this with a status update on whether the configure was successful or not. I'm not sure where it gets stuck or why because I'm able to ping the robot from the host machine and I was able to install all available updates through the software tab on the SmartPad by connecting to myKUKA account.

I have made sure all the hardware connections, software versions, external API, etc are accurate and up-to-date.

Since this problem started abruptly and very recently, I'm not sure what caused it. I have attached the complete log for your reference. Kindly help.

KUKA_log.txt

KUKA robot OS

No response

KUKA robot OS version

iiQKA.OS 1.2.9

KUKA external interface version

1.0.3

Affected robot model(s)

lbr iisy 3 r760

Version or commit hash of the driver

No response

Setup

N/A

Reproduction steps

N/A

Logs

No response

[Svastits]

Hi @prasuchit,

The logs seem perfectly fine, not all controllers log a message about successful configuration, so I do not think anything is stuck. You can check all controller states with ros2 control list_controllers, you should see that it is inactive. If configuration or activation fails, please attach those logs too, from these I cannot see any issue

[prasuchit]

@Svastits when I run configure after this in another terminal, that command never returns. That's what I meant by stuck.

I don't see any reflection of the configure command being executed on the launch file terminal either.

[Svastits]

Ok, now I undertand :) Unfortunately I cannot provide a fix, as this does not seem to be a bug directly related to the driver, rather a ros2 core/environment issue, which I cannot debug.

Based on your previous issue with the event_broadcaster I would assume, that the controller_manager is blocked for some reason and does not get the request for the state transition. I would suggest to put in some additional logs in the robot_manager's on_configure method to see exactly where things get stuck or if the method is called at all. Based on the result I might be able to give some more ideas.

A few additional things would provide more context for me:

• what hardware do you use for the client? How much CPU cores does it have? (On a VM I had slightly similiar issues when not enough cores were available)
• Could you check wheter the issue happens if you use the simple driver bringup (ros2 launch kuka_iiqka_eac_driver startup.launch.py) and configure?

[prasuchit]

@Svastits sorry for the late response. Your suggestions gave me an idea as to what might be causing it and I needed some time to test it out.

I was running an AI model training on the host machine and this was taking up most of the CPU and RAM. After the model finished training, I ran the robot commands and it worked very smoothly!

I didn't realize the robot needed so much of the host machine's CPU power to even connect!! But either way, the problem is solved now, thanks for your help. :)

[Svastits]

Great, I am happy that your issues are solved :) Actually the driver does not need much CPU power for activation, only some calculation capacity for very short times, as both pub-sub and req-rep communication needs a few threads to run, but actual CPU load is quite low.

But this is really valuable feedback, we will look at the current structure and try to optimize the number of necessary threads and processes. I have created an issue (#168) to be able to track this.

[prasuchit]

@Svastits is there any update on when the Jazzy branch is going to be released? I'd like to switch back to developing in Python. Also, if you'd like some feedback on the bugs/feature-requests I have encountered so far, I can either summarize it here or raise a new issue where people can track them. What do you suggest? Please let me know...

[Svastits]

I am not sure about the jazzy timeline, we have not started the migration yet but it is certainly planned.

We would appreciate any feedback, I think the easiest is if you just write it here :)

[prasuchit]

Hi @Svastits,

Here are my bug reports/feature requests:

1. MoveIt interactive marker in Rviz: This is a very useful feature that helps move the robot freely to any valid pose, move linearly along any axis, and rotate the end-effector in RPY. Since ROS packages primarily serve researchers who want to test and debug different things, this feature would be very helpful.

2. Event handler controller dying: As you already know, I have been facing this issue of the event handler dying repeatedly every time I start the launch file. Despite my ensuring that all the requirements and dependencies were satisfied, we couldn't figure out how or why this node was dying. It may be beneficial to rewrite this node such that it is more robust and integrated into the whole system using better exception handling.

3. Creating a Debian version of the external control API: So that rosdep install command can automatically find and install the external control API along with all other dependencies. This way updating the API can also be done simply using sudo apt update or upgrade.

4. Automatically performing configure and activate actions within the launch file: So that all the user has to do is run the launch file to get started. Since ROS2 launch files are in Python, waiting to ensure some event/node/service is active before performing configure and activate within the launch.py file should be possible now.

5. Gazebo support: I noticed that it is part of your planned updates, so I just want to mention it again as part of my feedback to reinforce its importance.

6. End effector signaling control through external API and ROS: This is a very valuable feature and I think you already know how important it is to signal the end effector through the robot using ROS.

7. Switch between cobot mode and industrial robot mode: If through a launch file argument or a change in setting, the robot can be switched between cobot and industrial robot mode to go either slow and safe or fast and agile, that would help researchers in human-robot interaction to test different variations of working as a team.

8. Dependency on the host machine: Currently, from my experience, the robot doesn't seem to connect to the host machine properly without a realtime kernel or if the computer's RAM or CPU are being partly used by other processes. Reducing this level of dependency on the host machine and on a realtime kernel will allow users to set up the robot quickly and multitask while working with the robot.

I hope these points are useful and will be considered in future updates to improve the robot and its ROS interface's performance. Thanks for considering. @VX792 @altex111

[Svastits]

Thanks for the detailed list, we will consider all feature requests and try to integrate these into our backlog.

In point 6 (end effector signaling), do you mean controlling I/O-s of the robot controller through ROS? (By the way point 3 is already in progress, the robot descriptions are avaialable via apt, the external-control-sdk will be in the next humble release and the drivers will follow in the subsequent one)

[prasuchit]

Hi @Svastits ,

Yes, I meant end effector I/O signaling through ROS.

That's great to hear. Thanks for the update. I'll keep an eye out for the new versions. :)

[prasuchit]

@Svastits good news!! :)

I was able to get the MoveIt interactive marker working!

Apparently, MoveIt2 doesn't automatically load the necessary parameters into the Rviz node when you load the moveit_config files using MoveItConfigsBuilder. This has been extensively discussed in several issues, including this one and that is why we also see the warning: "[moveit_ros.robot_model_loader]: No kinematics plugins defined. Fill and load kinematics.yaml!".

To get the MoveIt interactive marker, you just have to load the necessary parameters correctly into the rviz node while launching, as shown here. I have attached my modified moveit_planning_fake_hardware.launch.py and my fake_hardware_planning_template.launch.py with this comment for you and anyone looking to fix this issue.

Additionally, to access the kinematics solver or OMPL pipeline functions independent of the move_group_interface within your code, one workaround is to load them into your current node by doing something similar to this.

Now I can see the blue sphere and move it around as usual:

Hope this feature will be added to future releases.

fake_hardware_planning.zip

@VX792 @altex111

[Svastits]

Thanks for looking into this :)

However your launch files does not quite work for me, I still do not see the sphere, but the following logs:

[rviz2-3] Warning: class_loader.impl: SEVERE WARNING!!! A namespace collision has occurred with plugin factory for class rviz_default_plugins::displays::InteractiveMarkerDisplay. New factory will OVERWRITE existing one. This situation occurs when libraries containing plugins are directly linked against an executable (the one running right now generating this message). Please separate plugins out into their own library or just don't link against the library and use either class_loader::ClassLoader/MultiLibraryClassLoader to open.

[rviz2-3]          at line 253 in /opt/ros/humble/include/class_loader/class_loader/class_loader_core.hpp

[rviz2-3] [ERROR] [1722408695.176709566] [moveit_ros_visualization.motion_planning_frame]: Action server: /recognize_objects not available

[rviz2-3] [ERROR] [1722408695.920595173] [moveit_background_processing.background_processing]: Exception caught while processing action 'loadRobotModel': parameter 'robot_description_kinematics.manipulator.kinematics_solver_timeout' has invalid type: Wrong parameter type, parameter {robot_description_kinematics.manipulator.kinematics_solver_timeout} is of type {double}, setting it to {string} is not allowed.

[rviz2-3] [ERROR] [1722408695.981194447] [moveit_background_processing.background_processing]: Exception caught while processing action 'loadRobotModel': parameter 'robot_description_kinematics.manipulator.kinematics_solver_timeout' has invalid type: Wrong parameter type, parameter {robot_description_kinematics.manipulator.kinematics_solver_timeout} is of type {double}, setting it to {string} is not allowed.

Did you modify anything else aside from the two launch files. Do you also have similiar errors?

[Svastits]

okay, I am not sure why the double parameters were not correctly parsed, but I could fix this PR is ready :) Thanks again

[prasuchit]

Hi @Svastits, happy to help! :)

I see that in the pull request only the fake hardware template launch file has been modified. I'm guessing the same will be applied to the real robot driver launch file as well?

[Svastits]

Thanks for the reminder, the plan is to also add these lines there :)