Robot runs into fault state due to RDK Timeliness Monitor missed message

[boshenniu]

Robot runs into fault state frequently.

Using the moveit: $ roslaunch flexiv_bringup rizon_control.launch robot_ip:=[robot_ip] local_ip:=[local_ip]

Connecting to the tablet to see the error, and it is due to singularity.

Run this RDK script can clear the fault temporarily. But after launching moveit, the robot will go into fault state again soon (<1 minutes).

Any insights on how to mitigate this? Thanks!

[munseng-flexiv]

@boshenniu

as you mentioned, the robot's starting position is in singularity. You could change to the Manual mode in Elements (tablet) and run the internal plans to move the robot to the Home position first.

[boshenniu]

Hi @munseng-flexiv , thanks. Just for clarity, I am able to clear the fault and move the robot to the Home position. But the issue here is: when doing my next experiment, the robot will get fault state again soon. FYI, (1) sometimes the robot enters fault state even though it is stationary without any motion command; (2) this issue did not happen such frequently on my device when using moveit few weeks ago... Do you have any insights on this? Thanks!

[kaibiao-flexiv]

Hi @boshenniu, If you could share the robot logs with us here or from emailing to me, that'd be great! Thanks!

[munseng-flexiv]

Hi @munseng-flexiv , thanks. Just for clarity, I am able to clear the fault and move the robot to the Home position. But the issue here is: when doing my next experiment, the robot will get fault state again soon. FYI, (1) sometimes the robot enters fault state even though it is stationary without any motion command; (2) this issue did not happen such frequently on my device when using moveit few weeks ago... Do you have any insights on this? Thanks!

Hi @boshenniu

Which flexiv_ros version do you use and the Robot Control App (RCA) version of the robot? (You can check the version info in Elements)

That might not be due to the flexiv_ros. Can you send us the fault info? It might be due to a bug in the previous version of RDK (https://github.com/flexivrobotics/flexiv_rdk/issues/30) that has been fixed in version 0.9

[boshenniu]

Thanks for the updates. @munseng-flexiv My rdk version is 0.9.1.

@munseng-flexiv @kaibiao-flexiv Here is what I did to repro this: $ roslaunch flexiv_moveit_config moveit_control.launch robot_ip:=[robot_ip] local_ip:=[local_ip] Then executed the robot to move between its home location and another location. At this moment, the robot was successfully executed to the home location. And then the robot reported the fault. ROS log:

[ INFO] [1718485005.240798687]: Combined planning and execution request received for MoveGroup action. Forwarding to planning and execution pipeline.
[ INFO] [1718485005.240836316]: Planning attempt 1 of at most 1
[ INFO] [1718485005.240856100]: Using planning pipeline 'ompl'
[ INFO] [1718485005.241576710]: Planner configuration 'rizon_arm' will use planner 'geometric::RRTConnect'. Additional configuration parameters will be set when the planner is constructed.
[ INFO] [1718485005.241775993]: rizon_arm/rizon_arm: Starting planning with 1 states already in datastructure
[ INFO] [1718485005.241804586]: rizon_arm/rizon_arm: Starting planning with 1 states already in datastructure
[ INFO] [1718485005.241834591]: rizon_arm/rizon_arm: Starting planning with 1 states already in datastructure
[ INFO] [1718485005.241863764]: rizon_arm/rizon_arm: Starting planning with 1 states already in datastructure
[ INFO] [1718485005.252375950]: rizon_arm/rizon_arm: Created 4 states (2 start + 2 goal)
[ INFO] [1718485005.252447177]: rizon_arm/rizon_arm: Created 4 states (2 start + 2 goal)
[ INFO] [1718485005.252527236]: rizon_arm/rizon_arm: Created 5 states (2 start + 3 goal)
[ INFO] [1718485005.252566109]: rizon_arm/rizon_arm: Created 5 states (2 start + 3 goal)
[ INFO] [1718485005.252665554]: ParallelPlan::solve(): Solution found by one or more threads in 0.010977 seconds
[ INFO] [1718485005.254032277]: SimpleSetup: Path simplification took 0.001322 seconds and changed from 3 to 2 states
[ INFO] [1718485007.356266139]: Controller 'position_joint_trajectory_controller' successfully finished
[ INFO] [1718485007.359608626]: Completed trajectory execution with status SUCCEEDED ...
[2024-06-15 13:56:51.685] [warning] [Robot Log] RDK Timeliness Monitor: missed 10/20 real-time command messages
[2024-06-15 13:56:51.689] [warning] [flexiv::Robot] isOperational: Robot is in fault state
[2024-06-15 13:56:51.690] [warning] [flexiv::Robot] isOperational: Robot is in fault state
[2024-06-15 13:56:51.691] [warning] [flexiv::Robot] isOperational: Robot is in fault state
[2024-06-15 13:56:51.692] [warning] [flexiv::Robot] isOperational: Robot is in fault state
[2024-06-15 13:56:51.693] [warning] [flexiv::Robot] isOperational: Robot is in fault state
[2024-06-15 13:56:51.694] [warning] [flexiv::Robot] isOperational: Robot is in fault state
[2024-06-15 13:56:51.695] [warning] [flexiv::Robot] isOperational: Robot is in fault state
[2024-06-15 13:56:51.696] [error] [Robot Log] RDK Timeliness Monitor: missed 20/20 real-time command messages
[2024-06-15 13:56:51.696] [warning] [flexiv::Robot] isOperational: Robot is in fault state
[2024-06-15 13:56:51.697] [warning] [flexiv::Robot] isOperational: Robot is in fault state
[2024-06-15 13:56:51.698] [warning] [flexiv::Robot] isOperational: Robot is in fault state
[2024-06-15 13:56:51.699] [warning] [flexiv::Robot] isOperational: Robot is in fault state
[2024-06-15 13:56:51.700] [warning] [flexiv::Robot] isOperational: Robot is in fault state

Collecting log by unplug the robot from laptop and plug into the tablet to run ExtractLogs: logs.zip

[kaibiao-flexiv]

Hi @boshenniu, Looks like this is related to RDK Timeliness Monitor (under the RT mode, the robot will fault if the commands are not received at 1000 Hz). You might be able to find the solutions to your problem by checking out these previous conversations: flexivrobotics/flexiv_ros2#31, flexivrobotics/flexiv_rdk#36, flexivrobotics/flexiv_rdk#38.

[boshenniu]

Hi @kaibiao-flexiv , I actually don't require RT. Do you know how I can use NRT when doing roslaunch flexiv_moveit_config moveit_control.launch ?

[munseng-flexiv]

Hi @boshenniu

flexiv_ros is based on the Flexiv RDK. The driver is set to use RT mode as default. You can modify the driver to NRT mode by changing the RDK API functions in the flexiv_hardware/src/flexiv_hardware_interface.cpp file.

• Change the streamJointPosition() to sendJointPosition()
• Change the control mode from RT_JOINT_POSITION to NRT_JOINT_POSITION

Attached is the updated flexiv_hardware_interface.cpp with the above changes: flexiv_hardware_interface.zip

After the changes, make sure to catkin_make to rebuild the ROS packages.

[boshenniu]

Modifying to NRT following the instructions in last comment mitigates my issue. Thanks!