gazebo ignition simulation crashes when trying to simulate multiple turtlebot4

[abdosalem490]

Please provide the following information:

• OS: Ubuntu 22.04
• ROS Distro: Humble
• Built from source or installed: built from source

• Package version: e21bd2c

  Expected behaviour multiple Turtlebot4 spawned in the simulation

  Actual behaviour the simulation crashes

To Reproduce Provide the steps to reproduce:

I am trying to simulate multiple turtlebot4 on Gazebo Fortress using ROS2 humble on my Ubuntu 22.04 but when I tried to run these commands, each in alone terminal:

ros2 launch turtlebot4_ignition_bringup turtlebot4_ignition.launch.py namespace:=salem
ros2 launch turtlebot4_ignition_bringup turtlebot4_spawn.launch.py namespace:=/robot2 x:=1.0 y:=0.0
ros2 launch turtlebot4_ignition_bringup turtlebot4_spawn.launch.py namespace:=/robot3 x:=2.0 y:=0.0

the simulation crashes and I get the following logs:

[ruby $(which ign) gazebo-1] [Err] [SceneManager.cc:201] Visual: [ground_plane] already exists
[ruby $(which ign) gazebo-1] [Err] [SceneManager.cc:201] Visual: [warehouse] already exists
[ruby $(which ign) gazebo-1] [Err] [SceneManager.cc:201] Visual: [pallet_box_0] already exists
[ruby $(which ign) gazebo-1] [Err] [SceneManager.cc:201] Visual: [shelf_big_0] already exists
[ruby $(which ign) gazebo-1] [Err] [SceneManager.cc:201] Visual: [shelf_big_1] already exists
[ruby $(which ign) gazebo-1] [Err] [SceneManager.cc:201] Visual: [shelf_big_2] already exists
[ruby $(which ign) gazebo-1] [Err] [SceneManager.cc:201] Visual: [shelf_3] already exists
[ruby $(which ign) gazebo-1] [Err] [SceneManager.cc:201] Visual: [shelf_4] already exists
[ruby $(which ign) gazebo-1] [Err] [SceneManager.cc:201] Visual: [shelf_5] already exists
[ruby $(which ign) gazebo-1] [Err] [SceneManager.cc:201] Visual: [shelf_6] already exists
[ruby $(which ign) gazebo-1] [Err] [SceneManager.cc:201] Visual: [shelf_7] already exists
[ruby $(which ign) gazebo-1] [Err] [SceneManager.cc:201] Visual: [shelf_big_3] already exists
[ruby $(which ign) gazebo-1] [Err] [SceneManager.cc:201] Visual: [shelf_big_4] already exists
[ruby $(which ign) gazebo-1] [Err] [SceneManager.cc:201] Visual: [shelf_0] already exists
[ruby $(which ign) gazebo-1] [Err] [SceneManager.cc:201] Visual: [shelf_1] already exists
[ruby $(which ign) gazebo-1] [Err] [SceneManager.cc:201] Visual: [shelf_2] already exists
[ruby $(which ign) gazebo-1] [Err] [SceneManager.cc:201] Visual: [barrier_0] already exists
[ruby $(which ign) gazebo-1] [Err] [SceneManager.cc:201] Visual: [barrier_1] already exists
[ruby $(which ign) gazebo-1] [Err] [SceneManager.cc:201] Visual: [barrier_2] already exists
[ruby $(which ign) gazebo-1] [Err] [SceneManager.cc:201] Visual: [barrier_3] already exists
[ruby $(which ign) gazebo-1] [Err] [SceneManager.cc:201] Visual: [chair_0] already exists
[ruby $(which ign) gazebo-1] [Err] [SceneManager.cc:201] Visual: [chair_1] already exists
[ruby $(which ign) gazebo-1] [Err] [SceneManager.cc:201] Visual: [fchair_0] already exists
[ruby $(which ign) gazebo-1] [Err] [SceneManager.cc:201] Visual: [fchair1] already exists
[ruby $(which ign) gazebo-1] [Err] [SceneManager.cc:201] Visual: [table0] already exists
[ruby $(which ign) gazebo-1] [Err] [SceneManager.cc:201] Visual: [Person 0 - Sitting] already exists
[ruby $(which ign) gazebo-1] [Err] [SceneManager.cc:187] Parent entity with Id: [137] not found. Not adding model visual with ID[141]  and name [artifact_proximity_detector] to the rendering scene.
[ruby $(which ign) gazebo-1] [Err] [SceneManager.cc:201] Visual: [Person 1 - Standing] already exists
[ruby $(which ign) gazebo-1] [Err] [SceneManager.cc:201] Visual: [Person 2 - Walking] already exists
[ruby $(which ign) gazebo-1] [Err] [SceneManager.cc:201] Visual: [salem/standard_dock] already exists
[ruby $(which ign) gazebo-1] [Err] [SceneManager.cc:201] Visual: [salem/turtlebot4] already exists
[ruby $(which ign) gazebo-1] [Err] [SceneManager.cc:201] Visual: [/robot2/standard_dock] already exists
[ruby $(which ign) gazebo-1] [Err] [SceneManager.cc:201] Visual: [/robot2/turtlebot4] already exists
[ruby $(which ign) gazebo-1] [Err] [RenderUtil.cc:1277] Failed to create sensor with name[salem/turtlebot4::oakd_rgb_camera_frame::rgbd_camera] for entity [224]. Parent not found with ID[223].
[ruby $(which ign) gazebo-1] [Err] [RenderUtil.cc:1277] Failed to create sensor with name[/robot2/turtlebot4::oakd_rgb_camera_frame::rgbd_camera] for entity [358]. Parent not found with ID[357].
[ruby $(which ign) gazebo-1] [Err] [RenderUtil.cc:1277] Failed to create sensor with name[salem/standard_dock::green_buoy_link::green_buoy] for entity [155]. Parent not found with ID[154].
[ruby $(which ign) gazebo-1] [Err] [RenderUtil.cc:1277] Failed to create sensor with name[salem/standard_dock::halo_link::halo] for entity [157]. Parent not found with ID[156].
[ruby $(which ign) gazebo-1] [Err] [RenderUtil.cc:1277] Failed to create sensor with name[salem/standard_dock::red_buoy_link::red_buoy] for entity [159]. Parent not found with ID[158].
[ruby $(which ign) gazebo-1] [Err] [RenderUtil.cc:1277] Failed to create sensor with name[salem/standard_dock::yellow_buoy_link::yellow_buoy] for entity [161]. Parent not found with ID[160].
[ruby $(which ign) gazebo-1] [Err] [RenderUtil.cc:1277] Failed to create sensor with name[salem/turtlebot4::base_link::cliff_front_left] for entity [172]. Parent not found with ID[167].
[ruby $(which ign) gazebo-1] [Err] [RenderUtil.cc:1277] Failed to create sensor with name[salem/turtlebot4::base_link::cliff_front_right] for entity [173]. Parent not found with ID[167].
[ruby $(which ign) gazebo-1] [Err] [RenderUtil.cc:1277] Failed to create sensor with name[salem/turtlebot4::base_link::cliff_side_left] for entity [174]. Parent not found with ID[167].
[ruby $(which ign) gazebo-1] [Err] [RenderUtil.cc:1277] Failed to create sensor with name[salem/turtlebot4::base_link::cliff_side_right] for entity [175]. Parent not found with ID[167].
[ruby $(which ign) gazebo-1] [Err] [RenderUtil.cc:1277] Failed to create sensor with name[salem/turtlebot4::ir_intensity_front_center_left::ir_intensity_front_center_left] for entity [194]. Parent not found with ID[193].
[ruby $(which ign) gazebo-1] [Err] [RenderUtil.cc:1277] Failed to create sensor with name[salem/turtlebot4::ir_intensity_front_center_right::ir_intensity_front_center_right] for entity [196]. Parent not found with ID[195].
[ruby $(which ign) gazebo-1] [Err] [RenderUtil.cc:1277] Failed to create sensor with name[salem/turtlebot4::ir_intensity_front_left::ir_intensity_front_left] for entity [198]. Parent not found with ID[197].
[ruby $(which ign) gazebo-1] [Err] [RenderUtil.cc:1277] Failed to create sensor with name[salem/turtlebot4::ir_intensity_front_right::ir_intensity_front_right] for entity [200]. Parent not found with ID[199].
[ruby $(which ign) gazebo-1] [Err] [RenderUtil.cc:1277] Failed to create sensor with name[salem/turtlebot4::ir_intensity_left::ir_intensity_left] for entity [202]. Parent not found with ID[201].
[ruby $(which ign) gazebo-1] [Err] [RenderUtil.cc:1277] Failed to create sensor with name[salem/turtlebot4::ir_intensity_right::ir_intensity_right] for entity [204]. Parent not found with ID[203].
[ruby $(which ign) gazebo-1] [Err] [RenderUtil.cc:1277] Failed to create sensor with name[salem/turtlebot4::ir_intensity_side_left::ir_intensity_side_left] for entity [206]. Parent not found with ID[205].
[ruby $(which ign) gazebo-1] [Err] [RenderUtil.cc:1277] Failed to create sensor with name[salem/turtlebot4::rplidar_link::rplidar] for entity [237]. Parent not found with ID[234].
[ruby $(which ign) gazebo-1] [Err] [RenderUtil.cc:1277] Failed to create sensor with name[/robot2/standard_dock::green_buoy_link::green_buoy] for entity [289]. Parent not found with ID[288].
[ruby $(which ign) gazebo-1] [Err] [RenderUtil.cc:1277] Failed to create sensor with name[/robot2/standard_dock::halo_link::halo] for entity [291]. Parent not found with ID[290].
[ruby $(which ign) gazebo-1] [Err] [RenderUtil.cc:1277] Failed to create sensor with name[/robot2/standard_dock::red_buoy_link::red_buoy] for entity [293]. Parent not found with ID[292].
[ruby $(which ign) gazebo-1] [Err] [RenderUtil.cc:1277] Failed to create sensor with name[/robot2/standard_dock::yellow_buoy_link::yellow_buoy] for entity [295]. Parent not found with ID[294].
[ruby $(which ign) gazebo-1] [Err] [RenderUtil.cc:1277] Failed to create sensor with name[/robot2/turtlebot4::base_link::cliff_front_left] for entity [306]. Parent not found with ID[301].
[ruby $(which ign) gazebo-1] [Err] [RenderUtil.cc:1277] Failed to create sensor with name[/robot2/turtlebot4::base_link::cliff_front_right] for entity [307]. Parent not found with ID[301].
[ruby $(which ign) gazebo-1] [Err] [RenderUtil.cc:1277] Failed to create sensor with name[/robot2/turtlebot4::base_link::cliff_side_left] for entity [308]. Parent not found with ID[301].
[ruby $(which ign) gazebo-1] [Err] [RenderUtil.cc:1277] Failed to create sensor with name[/robot2/turtlebot4::base_link::cliff_side_right] for entity [309]. Parent not found with ID[301].
[ruby $(which ign) gazebo-1] [Err] [RenderUtil.cc:1277] Failed to create sensor with name[/robot2/turtlebot4::ir_intensity_front_center_left::ir_intensity_front_center_left] for entity [328]. Parent not found with ID[327].
[ruby $(which ign) gazebo-1] [Err] [RenderUtil.cc:1277] Failed to create sensor with name[/robot2/turtlebot4::ir_intensity_front_center_right::ir_intensity_front_center_right] for entity [330]. Parent not found with ID[329].
[ruby $(which ign) gazebo-1] [Err] [RenderUtil.cc:1277] Failed to create sensor with name[/robot2/turtlebot4::ir_intensity_front_left::ir_intensity_front_left] for entity [332]. Parent not found with ID[331].
[ruby $(which ign) gazebo-1] [Err] [RenderUtil.cc:1277] Failed to create sensor with name[/robot2/turtlebot4::ir_intensity_front_right::ir_intensity_front_right] for entity [334]. Parent not found with ID[333].
[ruby $(which ign) gazebo-1] [Err] [RenderUtil.cc:1277] Failed to create sensor with name[/robot2/turtlebot4::ir_intensity_left::ir_intensity_left] for entity [336]. Parent not found with ID[335].
[ruby $(which ign) gazebo-1] [Err] [RenderUtil.cc:1277] Failed to create sensor with name[/robot2/turtlebot4::ir_intensity_right::ir_intensity_right] for entity [338]. Parent not found with ID[337].
[ruby $(which ign) gazebo-1] [Err] [RenderUtil.cc:1277] Failed to create sensor with name[/robot2/turtlebot4::ir_intensity_side_left::ir_intensity_side_left] for entity [340]. Parent not found with ID[339].
[ruby $(which ign) gazebo-1] [Err] [RenderUtil.cc:1277] Failed to create sensor with name[/robot2/turtlebot4::rplidar_link::rplidar] for entity [371]. Parent not found with ID[368].

Other notes Add anything else you thing is important.

[hilary-luo]

One thing you can do to make two robots work better is pass the namespace in without a slash prepended. For example:

ros2 launch turtlebot4_ignition_bringup turtlebot4_ignition.launch.py namespace:=salem
ros2 launch turtlebot4_ignition_bringup turtlebot4_spawn.launch.py namespace:=robot2 x:=1.0 y:=0.0

Does this make the system functional for your use case?

When I do this I am able to have the topics show up seemingly properly (I haven't gone through all of the topics or done that much in terms of testing. That being said, it does still seem to be trying to respawn elements from the world, I am not sure why that is. I will add this to my list of things to look into but anyone else feel free to look into this and contribute as well.

I am unable to spawn 3 robots although it is hard to say right now if this is a limitation because of the issue aforementioned or if it is just about the pure number of nodes / topics. You could try disabling sensors that you aren't using and see if that allows you to spawn 3 robots.

[abdosalem490]

The problem is that I needed at least 4 robots to perform navigation and mapping of an area for a school project and the turtlebot4 simulator doesn't support that feature yet without producing errors in the console. would it work with turtlebot3 or shall I search for another robot simulator?

[msminirobot]

The same error occurs for me, having the Debian package installed.

[msminirobot]

The simulation also crashes when adding two robots and one object with inertia. It does not crash when adding one robot and two objects (boxes) with inertia. Adding several boxes with inertia and one robot is working fine though.

[hilary-luo]

@abdosalem490 I cannot speak to the capabilities of Turtlebot3 as I have not used it before. You could inquire on the Turtlebot3 repo or robotics stack exchange.

@msminirobot Are you able to simulate two robots and with no extra objects? As you work through your troubleshooting feel free to make contributions back about this issue.

[msminirobot]

Yes, I am able to simulate two robots with no extra objects. It will show the error messages shown above though. I am trying to spawn a third robot will result in a segmentation fault that crashes the whole simulation. I also tested removing several topics I will not use, which did not help. In addition, trying to spawn more than two Create 3 (without the TurtleBot4 add ons) does not work either.

[msminirobot]

@abdosalem490 It is possible to simulate several TurtleBot3s in Gazebo. We used it with up to 9 robots so far.

[msminirobot]

The problem seems to be connected with the plugins and especially the sensors plugin. When including the turtlebot model in the SDF world file, I can spawn more than two robots when I remove the sensors-systems.so plugin but the simulation crashes if I include the sensors plugin again.

<plugin name='ignition::gazebo::systems::Sensors' filename='libignition-gazebo-sensors-system.so'>
      <render_engine>ogre</render_engine>
 </plugin>

[msminirobot]

Excluding the sensors-system plugin from the robot model and including it only once in the world file fixes the problem. For this purpose, the create3 description (create3_sim/irobot_create_common/irobot_create_description/urdf/create3.urdf.xacro) has to be updated by removing the sensors-system plugin (in addition, I removed the contact-system plugin). Both plugins should be included in the world file, e.g. just after the other plugins. From my tests, the second robot is then spawned without the error messages reported above and I also was able to include a third robot without crashing the simulation.

When using all sensors, the performance of the simulation will significantly drop when adding several robots. Removing unused sensor nodes from the create3_ros_ignition_bridge.launch.py can improve this.

[Kazimbalti]

Yes I have tried many times, for two turtlebot4 it works fine but when you want to add more than two turtlebot4, then the gazebo ignition crashes.

But I developed swarm of turtlebot3 in classic gazebo and implemented NAV2 and it works perfectly fine.

Yes the turtlebot4 simulation and hardware implementation for multiple turtlebot4 is much difficult. I have two turtlebot4, and I am trying to implement some navigation but still trying my best.

[msminirobot]

@Kazimbalti Please see me comment above on how to fix the issue. I hope that there will be an official bug fix, but for now the workaround is well enough to get the simulation running.

[msminirobot]

@Kazimbalti I am talking about the TB4 since this issue and repository is about the TB4. I know that the TB3 works fine (see my previous comment: https://github.com/turtlebot/turtlebot4_simulator/issues/60#issuecomment-1834621579)

[Kazimbalti]

@msminirobot Thank you for your prompt response, Sorry at that time I missed the your comments, thank you.

Please can you explain a bit more, how you spawn more TB4, where is the sensors plugin?

As in turtlebot4_description, there is only one launch file and in urdf there is are folders of standard and sensor, So which thing I have to remove or exclude. thank you

[msminirobot]

@Kazimbalti The TB4 packages depend on the create3 packages (https://github.com/iRobotEducation/create3_sim/tree/main) since the TB4 is based on the create3. The modifications are to be done in the create3 packages.

[Kazimbalti]

@msminirobot Thank you for your help, now the gazebo ignition is not crashing.

After comments out the following:

I got this in the third TB4 terminal:

[lifecycle_manager-56] [INFO] [1702465625.421946453] [robot3.lifecycle_manager_navigation]: Waiting for service controller_server/get_state...

While fourth TB4 is not spawn on the gazebo and I got the following message in terminal 4:

@msminirobot Please guide me, how to resolve these issues and spawn more robots in Gazebo. thank you

[Kazimbalti]

@msminirobot @hilary-luo Secondly, I have two TB4 hardware, can you guide me please how to spawn both TB4 on a single map, and then how to perform and localization and navigation? thank you

[msminirobot]

@Kazimbalti Regarding the simulation, it is likely a local problem I cannot help you with.

Regarding your second point: this issue is meant to solve errors in the code base regarding the simulation crash when spawning multiple turtlebots. For general support on using the ROS ecosystem and simulations, please use https://robotics.stackexchange.com

[msminirobot]

The number of robots that can be spawned also depends heavily on the nodes created by the robots. There seems to be a hart limit of ~120 nodes. Preventing the launch of unused nodes from the create3 and the Turtlebot4 repos allows spawning more robots. Without excluding standard nodes, up to three robots can be spawned.

[abdosalem490]

@msminirobot, I really really like to thank you for posting the solution in this thread, I have been struggling with TB4 for a while and I gave up. so I started implementing my own robot but at some point, I realized it would require me a tedious amount of work to get it working with other packages like slam_toobox and nav2. I tried your solution and it worked like a charm. and again, I really appreciate that when you found the solution, you posted it in this thread. keep it up 👍️.

[msminirobot]

Also, it seems like spawning the robots by one launch script does not work (diff drive controller could not be configured then). Spawning them using separate launch files and individual terminals works though; each robot seems to require an own thread.