Very slow comunication with multiple robots on same ROS_DOMAIN_ID

matija9 commented 1 month ago

Robot Model

Turtlebot4 Standard

ROS distro

Humble

Networking Configuration

Simple Discovery

OS

Ubuntu 22.04

Built from source or installed?

Installed

Package version

ros-humble-turtlebot4-desktop version: 1.0.0-1jammy.20240327.181848

ros-humble-turtlebot4-robot version: 1.0.2-1jammy.20240407.070206

Type of issue

Networking

Expected behaviour

Connect multiple turtlebots to the network using simple discovery and have them operate simultaneously.

Actual behaviour

When 3 or more robots are connected at the same time their connection becomes very slow and there is a lot of package loss, to the point not even ssh into a robot is possible. Ping time is between 700 and 2000 ms and packet loss is about 80% with 4 robots running together.

Using different ROS_DOMAIN_IDs makes connections work again so I don't think it is a problem with the network itself. That does solve communication problem but then I can't have robots interacting with one another or one central node, which is my goal.

I tried using both FastRTPS and CycloneDDS but the results were the same.

Error messages

No error message.

To Reproduce

Connect multiple turtlebots to the same network and ROS_DOMAIN_ID with different namespaces
Try running any commands to them
Very slow response

Other notes

No response

smatarCPR commented 1 month ago

Hello @matija9 , Thank you for reaching out on the Turtlebot 4 Github page. I am very sorry to hear that you are having issues with one of our products. With regards to this matter: it is expected that having multiple robots operating on the same ROS domain will increase the load on the network, as more data needs to be broadcast across the network.

A few potential solutions are:

Use seperate ROS Domain ID, and use ROS Domain ID bridging to bridge the necessary data to the same Domain ID https://github.com/ros2/domain_bridge
We are planning on releasing multi-robot support for discovery server very soon, using this setup should help reduce the load you are seeing

Let me know if you have any questions or if I can help with anything else. Best Regards, Saif

matija9 commented 1 month ago

Thank you for your response!

Domain bridge seems to solve my problems for now, and I am looking forward to the release of discovery server for multiple robots.

What do you recommend for launching robots on different domains from the same file? I tried just calling all of the launch files from a shell script and that seems to work fine, but I am wondering if there is a better way of doing it.

Best regards, Matija

smatarCPR commented 1 month ago

Hello @matija9 , It is great to hear that domain bridging has worked! With regards to your follow up question, would it be possible to provide some additional details regarding your goals. When you mention launching robots on different domains from the same fil, are you launching the base launch file from a central source or are you launching a series of nodes that will interact with the robots? Looking forward to hearing from you soon! Best Regards, Saif

matija9 commented 1 month ago

I am using separate nodes with the TurtleBot4Navigator package, something similar to this, for each of the robots. I also made a launch file that starts the node, localization and nav2 for each of the robots. I am trying to use 6 of them for now in a way that there are always two or three driving, while the rest charge. Then I made a master node that monitors the battery state of each of the robots and issues orders to undock a fresh robot once one of the robots discharges below a certain threshold.

For now, I start all of them something like this:

export ROS_DOMAIN_ID=1
ros2 launch launch_robots launch_robots.launch.py namespace:=/robot1 &
export ROS_DOMAIN_ID=2
ros2 launch launch_robots launch_robots.launch.py namespace:=/robot2 &
.
.
.
ros2 run domain_bridge domain_bridge bridge_config.yaml &
ros2 run launch_robots master_node

This starts up all the robots without issue, but when I try to shut it down from the terminal with ctrl+C it does not shut down all the nodes and it gives me trouble when I try to run them all again.

I tried to find a way to do this from ros launch file but didn't find a solution that worked for me.

smatarCPR commented 3 weeks ago

Hello @matija9 , Apologies for the late reply on this. The way you are launching from a script is generally a good way to do so. One note on this method however, it appears that all the processes are being launched in the background, which is likely contributing to them not shutting down properly. You can try adding this line to the top of their script and see if it helps:

trap "trap - SIGTERM && kill -- -$$" SIGINT SIGTERM EXIT

This will propagate a kill command to all of the child processes.

Let me know how this recommendation works for you. Looking forward to hearing from you soon. Best Regards, Saif M.

matija9 commented 2 weeks ago

So if i understand it it should be like this

#!/bin/bash
trap "trap - SIGTERM && kill -- -$$" SIGINT SIGTERM EXIT
export ROS_DOMAIN_ID=80
ros2 launch mock_bringup mock_operation.launch.py map:=map_aula.yaml namespace:=/robot8 &
export ROS_DOMAIN_ID=90
ros2 launch mock_bringup mock_operation.launch.py map:=map_aula.yaml namespace:=/robot9 &
export ROS_DOMAIN_ID=60
ros2 launch mock_bringup mock_operation.launch.py map:=map_aula.yaml namespace:=/robot6 &
export ROS_DOMAIN_ID=70
ros2 launch mock_bringup mock_operation.launch.py map:=map_aula.yaml namespace:=/robot7 &
export ROS_DOMAIN_ID=0
sleep 5
#ros2 run nav_proba master_node &
ros2 launch mock_bringup master_launch.launch.py &
ros2 run domain_bridge domain_bridge ~/new_ws/bridge_config.yaml

When I run it like this I get output: trap: SIGINT: bad trap and after that, everything works as before.

turtlebot / turtlebot4