search

frankaemika / franka_ros

ROS integration for Franka research robots
https://frankaemika.github.io
Apache License 2.0
364 stars 313 forks source link

Time difference between two calls on franka_ros_control. #322

Closed olivier-stasse closed 1 year ago

olivier-stasse commented 1 year ago

Dear Franka Emika Team, First thanks a lot for having your library open-source, and for providing a ros_control interface. This is very much appreciated.

We tested our control architecture in one Panda robot acquired in 2019. We found some important time variation when being called by the update function of the controller_manager in your franka_ros_control node. In average we found a very good 0.001 ms but with numerous times where the time difference was 0.02 ms or 5ms.

Reading your code, i did a fork and set the scheduler to SCHED_FIFO to make sure that franka_control_node was running in real-time with the max priority but it changed nothing.

Could you confirm that this is supposed to be normal and we need to write our controller to take into account this time variation ?

You can find the tests I used here.

It takes a sliding windows of 2000 time diff measurements and average them. In addition every 2000 iterations it detects the min and max over this window. The output is the following:

[ INFO] [1676304961.904771333]: franka_control: 0.000023 0.001004 0.005073
[ INFO] [1676304963.920705591]: franka_control: 0.000022 0.001004 0.005178
[ INFO] [1676304965.931665732]: franka_control: 0.000022 0.001004 0.004637
[ INFO] [1676304967.940738049]: franka_control: 0.000024 0.001005 0.004999
[ INFO] [1676304969.944624115]: franka_control: 0.000021 0.001004 0.004144
[ INFO] [1676304971.949516294]: franka_control: 0.000023 0.001004 0.004005
[ INFO] [1676304973.968455524]: franka_control: 0.000019 0.001004 0.005745
[ INFO] [1676304975.981437831]: franka_control: 0.000022 0.001004 0.005090
[ INFO] [1676304977.988400271]: franka_control: 0.000023 0.001004 0.004166
[ INFO] [1676304979.996327975]: franka_control: 0.000027 0.001004 0.005015
[ INFO] [1676304982.005296231]: franka_control: 0.000025 0.001004 0.005172
[ INFO] [1676304984.021287036]: franka_control: 0.000022 0.001004 0.005102
[ INFO] [1676304986.031307725]: franka_control: 0.000022 0.001005 0.004076
[ INFO] [1676304988.047156812]: franka_control: 0.000021 0.001004 0.005750
[ INFO] [1676304990.050114736]: franka_control: 0.000022 0.001004 0.004412
[ INFO] [1676304992.057078162]: franka_control: 0.000032 0.001004 0.005276
[ INFO] [1676304994.075079216]: franka_control: 0.000019 0.001004 0.005080
[ INFO] [1676304996.082006002]: franka_control: 0.000021 0.001004 0.004327
[ INFO] [1676304998.088944916]: franka_control: 0.000019 0.001004 0.004154
[ INFO] [1676305000.101991178]: franka_control: 0.000019 0.001004 0.005404
[ INFO] [1676305002.107921281]: franka_control: 0.000019 0.001004 0.004094
[ INFO] [1676305004.110821548]: franka_control: 0.000024 0.001004 0.003635
[ INFO] [1676305006.118746446]: franka_control: 0.000018 0.001004 0.004134
[ INFO] [1676305008.122717050]: franka_control: 0.000022 0.001004 0.003250
[ INFO] [1676305010.125688098]: franka_control: 0.000022 0.001004 0.003347

To avoid any issue the test was done using moveit and the JTC controller to avoid mixing problems with our control architecture.

gavanderhoorn commented 1 year ago

Reminded me of https://github.com/frankaemika/libfranka/issues/120.

olivier-stasse commented 1 year ago

Thanks @gavanderhoorn for pointing out this issue. I got the same graphs than frankaemika/libfranka#120 with another tool, and a student from Prague found the same with yet another test.

I suspect that the underlying protocol with the robot is the root of the problem.

olivier-stasse commented 1 year ago

Follow up: one of our colleague, and this is aligned with some feedback on other issues, said that the network card we are using may not be handling the UDP packets properly. We may need to have a NIC which is compatible with Ethercat.

I will have a look and provide feedback.

gavanderhoorn commented 1 year ago

Unless FCI communication is based on ethercat, I'm unsure how "ethercat compatible" NICs (not sure what that means exactly) would help.

IIRC and IIUC, FCI uses a custom UDP protocol.

olivier-stasse commented 1 year ago

As far as I understood it this link gives a possible explanation: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/performance_tuning_guide/network-nic-offloads "Offloads should be used on high speed systems that transmit or receive large amounts of data and favor throughput over latency. Because using offloads greatly increases the capacity of the driver queue, latency can become an issue. An example of this would be a system transferring large amounts of data using large packet sizes, but is also running lots of interactive applications. Because interactive applications send small packets at timed intervals there is a very real risk that those packets may become 'trapped' in the buffer while larger packets in front of them are processed, causing unacceptable latency. "

gavanderhoorn commented 1 year ago

And the implication is "NICs compatible with ethercat" don't use off-loading?

Tuning your network stack could certainly help, provided the other side (ie: the control box) uses a similarly tuned setup of course. That I wouldn't know.

But the general assertion that "ethercat compatible NICs" would be better is not necessarily one you can make, I believe.

marcbone commented 1 year ago

Ok guys before you start buying a new networks card: For panda it can be expected to have some missed cycles. This should usually not be a huge problem. But we fixed this issue for our new FR3 robot. Basically if you get 98% success rate in the communication test on a Panda, everything is correct on your side. On the FR3 you should get 100%.

gavanderhoorn commented 1 year ago

Isn't this more about the fact the timing of network packets doesn't appear to be deterministic (or at least: show a larger jitter than expected)?

Would the FR3 improvements be "back-portable" to the Panda?

olivier-stasse commented 1 year ago

Some feedback:

marcbone commented 1 year ago

@gavanderhoorn I am sorry, these improvements will probably never be backported. Even though from a customers view there is not much difference between Panda and FR3, there are lots of changes under the hood, so we cannot easily offer an update.

@olivier-stasse I am glad that you were still able to make it work. I would also be interested in what your application is (if it is not confidential), that this was a problem. It helps us to better understand what you guys need. Also I am always interested on what you work on, especially if you push the robot to its limits :sweat_smile:

olivier-stasse commented 1 year ago

@marcbone In this case we want to deploy a framework that plan and execute tasks by exploring a graph of constraints Mirabel, TRO, 2022. We applied it to the humanoid robot TALOS (https://www.youtube.com/watch?v=lWKJoFPZVLY) at 1KHz, the mobile manipulator Tiago (https://www.youtube.com/watch?v=yew7CZMcsyc) at 250 KHz, and a UR-10 (https://peertube.laas.fr/w/hYitAMe96DP9iPuBjSYCc8) at 250 Hz. ros_control is key to lower the transfer from one platform to another. Each of them is using a RT_PREEMPT patch and we did not run into such time differences.