Low control rate at raspberry Pi 4

[lmendyk]

I’m experiencing very low rate at which for example /odom and /tf are being published when the turtlebot3 driver is based on the ros2_controll. No matter what rate is defined by update_rate (I tested 100, 50, 20 hz but I only get 5Hz)

controller_manager:
  ros__parameters:
    update_rate: 50  # Hz

What I observe when testing by ros2 topic hz /odom is about 5 Hz.

It is raspberry Pi 4 with ubuntu with NO RT kernel, but still it works much worse that the turtlebot3 software version which is NOT based on ros2_control but on “legacy” driver.

Can anybody advise if some kernel parameters tuing can help me? I suspect that https://github.com/ros-controls/ros2_control/blob/6f47d8d913f6b8eea1c491087a4555b9c0537a39/controller_manager/src/ros2_control_node.cpp#L86 May sleep longer than expected on my system causing the issue.

[lmendyk]

The „legacy driver” employs different mechanism for creating the loop accessing the hardware. Instead of the sleep_until:

this->create_wall_timer(
    std::chrono::milliseconds(50),

is used and a callback for this timer reads data from the device and publishes data. Which seems to work on that system as it is what I observed using ros2 topic hz

Does it mean that current ros2_control implementation with the sleep_until in fact MANDATES to have RT kernel? If so this warning which is currently printed is a kind of misleading?

[christophfroehlich]

Can you reproduce this with a simple c++ program, without any invocation of ROS?

[lmendyk]

In the meantime I have finally found the way to have the RT kernel for raspberry pi4 following the ros-realtime/linux-real-time-kernel-builder: build and setup RT kernel for the ROS 2 testing (github.com). I have followed the instructions https://control.ros.org/master/doc/ros2_control/controller_manager/doc/userdoc.html#determinism The effect … I don’t get any warnings at startup related to no RT kernel or inability to turn on FIFO RT scheduling policy – so everything seems fine.

But … no progress – I’m still getting very low hz for /odom and /tf topics.. It evens worse than with generic kernel

Now I’m lost ☹

[christophfroehlich]

Can you test your code on a different platform, for example in a docker container? Just to be sure that the problem is on the RPi and not on the controller code.

[lmendyk]

Not sure If I understand. I'm testing with real hardware turtlebot_manipulation with Raspberry Pi 4 as SBC connected to microContorller OpenOC which controls motors. I can't change the platform and be able to test behavior with real hardware as I can't easily connect hardware to for example laptop.

What I can observe is that process "ros2_control_no" has 95-100% CPU - which from one hand seems OK as it has the highest priority, but on the other hand if it makes the sleeps in the main loop is it normal behavior?

I have inactivated all controllers, but still "ros2_control_no" process has 95-100% which is strange as it should do currently ... nothing?

[christophfroehlich]

Which ros2_controller are you using? diff_drive controller? Use this together with a simulation of the turtlebot or example2 from ros2_control_demos and check if you see the expected update rate of your topics.

[lmendyk]

OK I have tested the ros2_control_demo_example_2 on the raspberry pi system and the performance is good.

Finally I have started commenting out lines of code in the turtlebot3_manipulation_hardware, and it looks that the "write" methods is a problem. Unlike "read" instead of writing all control items by single remote call, each motor is updated in separate call (which goes via USB to OpenCR microcontroller) . As turtlebot3_manipulation has more motors than just turtlebot3, this makes difference.

So it looks like that I will have to correct the source code for turtlebot3_manipulation_hardware package.

Sorry for suspecting that the problem is in ros2_control (I was misguide as "legacy" software worked - I forgot that legacy was just for turtlebot3 and turtlebot3_manipulation ) .

Thank you for your eagerness to help. Once more sorry for troubling your.

[christophfroehlich]

Glad you found it!