valid Moveit velocity and acceleration scaling settings break motor controller

[JHeuverRiwo]

Description Setting a higher velocity (0.9+) and acceleration (0.7+) scaling when using Moveit motion planners for the path and then executing a path that's close to a half circle on joint 1 results in a LAG error, crashing the driver, requiring a reinitialization of the CAN modules.

Your environment

ROS Distro: Humble
OS Version:  Ubuntu 22.04 (not realtime)
Branch/Commit hash: #b32613bc2141d03a89bf986993003e2ce6243076
Protocol: CPRCANv2

To Reproduce

1. Start launchfile via 'ros2 launch irc_ros_moveit_config rebel.launch.py rebel_version:=01 gripper:=none use_rviz:=true launch_dio_controller:=false'
2. set robot to zero state
3. set velocity scaling to >=0.9, and acceleration scaling to >=0.7
4. plan goal state [180deg, 0, 0, ...]
5. execute motion plan

Expected behavior robot moves to goal state in controlled fashion.

Actual behavior robot stops moving abruptly before it reaches 90 degrees on joint 1, with debug output in terminal:

[ros2_control_node-2] [INFO] [1683551174.300466625] [iRC_ROS]: Module 0x10: Errors MNE LAG detected, motorstate set to disabled [ros2_control_node-2] [INFO] [1683551174.300470273] [iRC_ROS]: Module 0x10: Reset required [ros2_control_node-2] [ERROR] [1683551174.300479929] [iRC_ROS]: 0x10: Error detected with no recovery option, stopping entire hardware interface for safety reasons

This bug seems to come up when max joint velocities scalings are either: (vel,acc) >= (1, 0.6) or (vel, acc) >= (0.9, 0.7) or (vel, acc) >= (0.8, 0.8). I have not seen the issue so far with other joints than joint 1.

some questions regarding this bug:

• Is this a position lag, or a velocity lag? The docs for the CAN protocol the rebel uses is not clear on that, while the older doc does seem to make this distinction.
• is it strictly necessary to run a realtime kernel when using CAN? Right now we're on a normal linux kernel, and would prefer to keep it this way. There is no extreme jitter when I look at the CAN timing, but i'm not sure whether this could be the source of this issue.

[cpr-fer]

Im not sure if there is a distinction between position and velocity lag in the CPRCANv2 protocol. You could get some additional information about present errors via the ModuleCtrl software.

Regarding the RT kernel I switched to one on the development machine and did not notice any difference. Of course it might be that I missconfigured something, but at least for me it seems to not be strictly required to run a realtime kernel

The limits set in the urdfs are copy and pasted from the TruPhysics ROS driver for the Rebel, I'll check if they match the settings for the igus robot control software.

Edit: On the first look the velocity limits appears to be higher in irc_ros than in irc, while the opposite is true for the acceleration. I'll try to reproduce the issue and check if the new values fix the issue

Edit2: I cant replicate this error with the first joint even with the limits both set to 1.0

Edit3: I created a branch linked to this issue with the limits set to the irc config file. Can you test and see if the error occurs at the same settings?

[JHeuverRiwo]

Interesting, I went back and forwards between the branches a few times. It does seem to fix the issue for me! With the code in the linked branch I can also go to 1 for both velocity and acceleration scaling and the robot still happily does it, so that is great. It seems to move quite a bit slower actually. I wonder why we have this issue on the humble branch, and you cannot replicate it. Is it possible there is a difference in firmware? Thanks for thoroughly checking it out. It is also helpful to know we do not need to move to a realtime system.

[cpr-fer]

Great to hear that it fixed the issue! I'll merge the branch since these settings work with the irc software as well and i suppose they have been tested on more rebels than the previous settings in this package It is definitly slower, the max velocity is reduced by a factor of three. For people who need a faster robot I'll add a few lines of readme/comments before merging.

Regarding the firmware, I am using a 1 or 2 month old firmware for the rebel axes, I'll check the exact firmware version the next time Im using the robot

[JHeuverRiwo]

Great! I will also add the specific firmware version to my (CAN) issues in the future, whenever it may be relevant.