Multi-Vehicle simulation with Gazebo ((say)100-10000 drones)

[ishanSharma07]

Hi,

I wanted to simulate a large number of drones ( (say) around 100 to 10000) using Gazebo. I followed the steps mentioned here. I was able to run the multi_uav_mavros_sitl.launch and upload the flight plans via QGC.

Queries:

1. What port ranges could I use for fcu_url, mavlink_udp_port and mavlink_tcp_port? Are there any restrictions? It would affect the number of drones I can deploy.
2. The Multi-vehicle sim page seems outdated since the multi_uav_mavros_sitl.launch does not use any user-configured rcS file (no rcS parameter in the launch file). However, the single_vehicle uses a script located at ROMFS/px4fmu_common/init.d-posix/rcS.
   • I tried replacing it with a custom rcS file as per this file. But encountered the following: 80: /home/user/catkin_ws/src/Firmware/posix-configs/SITL/ekf2/iris_1: mavlink: not found 81: /home/user/catkin_ws/src/Firmware/posix-configs/SITL/ekf2/iris_1: replay: not found

[hamishwillee]

Almost certainly this is restricted to 10 vehicles because of overlap of allocation of the ports. The docs may be out of date, that happens (please check the "master" version and if that is out of date, would love PR / specific issues posted).

FYI will also note in that master also offers a simulation that does not use ROS - https://dev.px4.io/master/en/simulation/multi-vehicle-simulation.html#no_ros

I question whether most computers would have the resource to simulate so many vehicles.

@julianoes Can you advise?

[ishanSharma07]

I used an entirely different range of ports which were modified in multi_uav_mavros_sitl.launch and the rcS script.

From what I understoods, parameters flow from multi_uav_mavros_sitl.launch > single_vehicle_spawn.launch > rcS script. And the rcS script only uses the node ID for calculating the values for the rest of the ports. (i.e. base port value + ID). So, I changed the values to the following:

offboard remote: 22000 offboard local: 27000 simulator tcp (i guess it is the same as mavlink_tcp_port): 32000 mavlink_udp: 37000 gcs_local: 42000

I also changed the corresponding values in multi drone launch file also. Now, theoretically, I can support 5000 drones before the ports start overlapping again.

System configuration: Processor: Intel(R) Core(TM) i7-8700K CPU @ 3.70GHz RAM: 16 GB DDR4 @2400MHz GPU: GeForce® GTX 1080 Ti; 11GB GDDR5 Linux 16.04 LTS (ROS Kinetic + Gazebo7)

Experiments(The drones are stationary; no flight plans loaded in this experiment):

• Wrote a script to deploy drones in a (nxn) grid layout.
• I was able to consistantly deploy 36 drones.
• I managed 49 drones after few reruns (without encountering an error).
• In one rare ocassion I managed to deploy 64 drones.

Anything above an 8x8 layout took too long to execute before my system crashed. Once deployed the 12 threads had ~55% utilization each, RAM at 8GB (idle 1.3 GB) and gazebo utilized round 150MB of VRAM.

I will later be programming these drones with some basic flight plans in order to view further resource utilization.

The error I was earlier referring to :

(base) root@dreamlab-Z370M-D3H:~/.ros/log/1ce13280-518a-11ea-917c-e0d55eb2c4cb# cat uav21-iris_21_spawn-68.log
[rospy.client][INFO] 2020-02-17 19:04:06,498: init_node, name[/uav21/iris_21_spawn], pid[9465]
[xmlrpc][INFO] 2020-02-17 19:04:06,504: XML-RPC server binding to 0.0.0.0:0
[rospy.init][INFO] 2020-02-17 19:04:06,505: ROS Slave URI: [http://dreamlab-Z370M-D3H:32999/]
[xmlrpc][INFO] 2020-02-17 19:04:06,505: Started XML-RPC server [http://dreamlab-Z370M-D3H:32999/]
[rospy.impl.masterslave][INFO] 2020-02-17 19:04:06,506: _ready: http://dreamlab-Z370M-D3H:32999/
[xmlrpc][INFO] 2020-02-17 19:04:06,509: xml rpc node: starting XML-RPC server
[rospy.registration][INFO] 2020-02-17 19:04:06,511: Registering with master node http://localhost:11311
[rospy.init][INFO] 2020-02-17 19:04:06,606: registered with master
[rospy.rosout][INFO] 2020-02-17 19:04:06,608: initializing /rosout core topic
[rospy.rosout][INFO] 2020-02-17 19:04:07,214: connected to core topic /rosout
[rospy.simtime][INFO] 2020-02-17 19:04:08,281: initializing /clock core topic
[rospy.internal][INFO] 2020-02-17 19:04:08,284: topic[/rosout] adding connection to [/rosout], count 0
[rospy.simtime][INFO] 2020-02-17 19:04:11,694: connected to core topic /clock
[rosout][INFO] 2020-02-17 19:04:14,659: Loading model XML from ros parameter
[rosout][INFO] 2020-02-17 19:04:16,312: Waiting for service /gazebo/spawn_urdf_model
[rosout][INFO] 2020-02-17 19:04:35,992: Calling service /gazebo/spawn_urdf_model
[rosout][INFO] 2020-02-17 19:05:02,622: Spawn status: SpawnModel: Successfully spawned entity
[rospy.core][INFO] 2020-02-17 19:05:02,623: signal_shutdown [atexit]
[rospy.internal][INFO] 2020-02-17 19:05:02,797: topic[/rosout] removing connection to /rosout
[rospy.impl.masterslave][INFO] 2020-02-17 19:05:02,798: atexit
(base) root@dreamlab-Z370M-D3H:~/.ros/log/1ce13280-518a-11ea-917c-e0d55eb2c4cb# cat uav11-iris_11_spawn-38.log
[rospy.client][INFO] 2020-02-17 19:04:06,074: init_node, name[/uav11/iris_11_spawn], pid[7792]
[xmlrpc][INFO] 2020-02-17 19:04:06,081: XML-RPC server binding to 0.0.0.0:0
[xmlrpc][INFO] 2020-02-17 19:04:06,082: Started XML-RPC server [http://dreamlab-Z370M-D3H:42867/]
[rospy.impl.masterslave][INFO] 2020-02-17 19:04:06,083: _ready: http://dreamlab-Z370M-D3H:42867/
[rospy.init][INFO] 2020-02-17 19:04:06,085: ROS Slave URI: [http://dreamlab-Z370M-D3H:42867/]
[xmlrpc][INFO] 2020-02-17 19:04:06,091: xml rpc node: starting XML-RPC server
[rospy.registration][INFO] 2020-02-17 19:04:06,092: Registering with master node http://localhost:11311
[rospy.init][INFO] 2020-02-17 19:04:06,195: registered with master
[rospy.rosout][INFO] 2020-02-17 19:04:06,197: initializing /rosout core topic
[rospy.rosout][INFO] 2020-02-17 19:04:06,491: connected to core topic /rosout
[rospy.internal][INFO] 2020-02-17 19:04:06,726: topic[/rosout] adding connection to [/rosout], count 0
[rospy.simtime][INFO] 2020-02-17 19:04:07,012: initializing /clock core topic
[rospy.simtime][INFO] 2020-02-17 19:04:09,393: connected to core topic /clock
[rosout][INFO] 2020-02-17 19:04:12,565: Loading model XML from ros parameter
[rosout][INFO] 2020-02-17 19:04:14,034: Waiting for service /gazebo/spawn_urdf_model
[rosout][INFO] 2020-02-17 19:04:37,360: Calling service /gazebo/spawn_urdf_model
[rospy.internal][INFO] 2020-02-17 19:05:13,160: topic[/clock] adding connection to [http://dreamlab-Z370M-D3H:37675/], count 0
[rosout][INFO] 2020-02-17 19:05:14,241: Spawn status: SpawnModel: Successfully spawned entity
[rospy.core][INFO] 2020-02-17 19:05:14,241: signal_shutdown [atexit]
[rospy.internal][INFO] 2020-02-17 19:05:14,268: topic[/rosout] removing connection to /rosout
[rospy.internal][INFO] 2020-02-17 19:05:14,268: topic[/clock] removing connection to http://dreamlab-Z370M-D3H:37675/
[rospy.internal][WARNING] 2020-02-17 19:05:14,269: Unknown error initiating TCP/IP socket to dreamlab-Z370M-D3H:37409 (http://dreamlab-Z370M-D3H:37675/): Traceback (most recent call last):
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/impl/tcpros_base.py", line 557, in connect
    self.read_header()
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/impl/tcpros_base.py", line 650, in read_header
    self._validate_header(read_ros_handshake_header(sock, self.read_buff, self.protocol.buff_size))
AttributeError: 'NoneType' object has no attribute 'buff_size'

[rospy.topics][ERROR] 2020-02-17 19:05:14,269: Traceback (most recent call last):
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/topics.py", line 326, in close
    c.close()
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/impl/tcpros_base.py", line 838, in close
    self.socket.close()
AttributeError: 'NoneType' object has no attribute 'close'

Some Screenshots of the 8x8 layout:

Queries:

1. Is simulator_tcp_port the same as mavlink_tcp_port?
2. (I am new to this) Are the files in master upstreamed automaticaly in the next release version of the guide or do we have to update each version/branch of the guide seperately.
3. The guide in master for multi-vehicle sim is upto date. Just needs some minor fixes. In reference to [2] should I submit a PR for the master branch or v1.9.0 branch of the dev guide?
4. Would it be helpful to update the "base" values of the ports to the ones I have used (or any recommended range so that it could support the possibility of 1000 drones)? The alternative is that if anyone wants to simulate > 10 drones, he/she could manually set the port ranges.

[hamishwillee]

See https://github.com/PX4/Firmware/pull/14126 - this might do some of what you want.

[julianoes]

1. Yes that's the corresponding TCP port. They need to match.
2. Changes are usually made to master and then a branch is forked off when a release happens, I think. If you really need/want to fix something for an older release, you could of course backport changes.
3. I'd suggest master and then you or @hamishwillee can backport it.
4. Let's follow @Jaeyoung-Lim's proposal in PX4/Firmware#14126. The problem is that there is currently a fixed limit at 255 because the MAVLink sysid is a uint8_t.

[ishanSharma07]

I missed that. Could naively changing the datatype of MAVLink's sysid be innocuous in case if I later want to deploy more than 255 drones?

[hamishwillee]

I missed that. Could naively changing the datatype of MAVLink's sysid be innocuous in case if I later want to deploy more than 255 drones?

No. Changing the packet definition will break everything - tools etc. This would be a major release change: https://mavlink.io/en/guide/serialization.html

To manage more than 255 vehicles you need to have separate mavlink networks. This is of course possible, but simulation and the mavlink project provide anything to help you.

[ishanSharma07]

One last question, when I used the above port range in rcS and executed 'make px4_sitl gazebo_iris' (which also uses the rcS script), the console was stuck at "waiting for the simulator to connect". What am I missing here?

[Jaeyoung-Lim]

@ishanSharma07 your gazebo is not connecting properly with the sitl instance. Please try with the script as documented in https://dev.px4.io/master/en/simulation/multi-vehicle-simulation.html#no_ros

[ishanSharma07]

The ports that I have modified in rcS work fine with multi-vehicle sim. But when executing make px4_sitl gazebo the output is the following:

(gazebo) root@dreamlab-Z370M-D3H:/home/dreamlab/catkin_ws/src/Firmware# sudo make px4_sitl gazebo
[1/4] Performing build step for 'sitl_gazebo'
ninja: no work to do.
[4/4] cd /home/dreamlab/catkin_ws/src/Firmware/build/px4_sitl_default/tmp && /home/dreamlab/catkin_w...e /home/dreamlab/catkin_ws/src/Firmware /home/dreamlab/catkin_ws/src/Firmware/build/px4_sitl_default
SITL ARGS
sitl_bin: /home/dreamlab/catkin_ws/src/Firmware/build/px4_sitl_default/bin/px4
debugger: none
program: gazebo
model: none
src_path: /home/dreamlab/catkin_ws/src/Firmware
build_path: /home/dreamlab/catkin_ws/src/Firmware/build/px4_sitl_default
empty model, setting iris as default
GAZEBO_PLUGIN_PATH :/home/dreamlab/catkin_ws/src/Firmware/build/px4_sitl_default/build_gazebo
GAZEBO_MODEL_PATH :/home/dreamlab/catkin_ws/src/Firmware/Tools/sitl_gazebo/models
LD_LIBRARY_PATH :/home/dreamlab/catkin_ws/src/Firmware/build/px4_sitl_default/build_gazebo
Gazebo multi-robot simulator, version 7.0.0
Copyright (C) 2012-2016 Open Source Robotics Foundation.
Released under the Apache 2 License.
http://gazebosim.org

[Msg] Waiting for master.
[Msg] Connected to gazebo master @ http://127.0.0.1:11345
[Msg] Publicized address: 10.24.24.222
[Wrn] [gazebo_barometer_plugin.cpp:66] [gazebo_barometer_plugin] Using default home altitude of 488 m
[Dbg] [gazebo_mavlink_interface.cpp:224] <joint_name> not found for channel[0] no joint control will be performed for this channel.
[Dbg] [gazebo_mavlink_interface.cpp:224] <joint_name> not found for channel[1] no joint control will be performed for this channel.
[Dbg] [gazebo_mavlink_interface.cpp:224] <joint_name> not found for channel[2] no joint control will be performed for this channel.
[Dbg] [gazebo_mavlink_interface.cpp:224] <joint_name> not found for channel[3] no joint control will be performed for this channel.
[Wrn] [gazebo_mavlink_interface.cpp:213] joint [zephyr_delta_wing::propeller_joint] not found for channel[4] no joint control for this channel.
[Wrn] [gazebo_mavlink_interface.cpp:213] joint [zephyr_delta_wing::flap_left_joint] not found for channel[5] no joint control for this channel.
[Wrn] [gazebo_mavlink_interface.cpp:213] joint [zephyr_delta_wing::flap_right_joint] not found for channel[6] no joint control for this channel.
[Dbg] [gazebo_mavlink_interface.cpp:224] <joint_name> not found for channel[7] no joint control will be performed for this channel.
[Msg] Connecting to PX4 SITL using TCP
[Msg] Lockstep is enabled
[Msg] Speed factor set to: 1
[Msg] Using MAVLink protocol v2.0
SITL COMMAND: "/home/dreamlab/catkin_ws/src/Firmware/build/px4_sitl_default/bin/px4" "/home/dreamlab/catkin_ws/src/Firmware"/ROMFS/px4fmu_common -s etc/init.d-posix/rcS -t "/home/dreamlab/catkin_ws/src/Firmware"/test_data
INFO  [px4] Creating symlink /home/dreamlab/catkin_ws/src/Firmware/ROMFS/px4fmu_common -> /home/dreamlab/catkin_ws/src/Firmware/build/px4_sitl_default/tmp/rootfs/etc

______  __   __    ___ 
| ___ \ \ \ / /   /   |
| |_/ /  \ V /   / /| |
|  __/   /   \  / /_| |
| |     / /^\ \ \___  |
\_|     \/   \/     |_/

px4 starting.

INFO  [px4] Calling startup script: /bin/sh etc/init.d-posix/rcS 0
INFO  [param] selected parameter default file eeprom/parameters_10016
[param] Loaded: eeprom/parameters_10016
INFO  [dataman] Unknown restart, data manager file './dataman' size is 11798680 bytes
INFO  [simulator] Waiting for simulator to connect on TCP port 32000
Gazebo multi-robot simulator, version 7.0.0
Copyright (C) 2012-2016 Open Source Robotics Foundation.
Released under the Apache 2 License.
http://gazebosim.org

[Msg] Waiting for master.
[Msg] Connected to gazebo master @ http://127.0.0.1:11345
[Msg] Publicized address: 10.24.24.222
[Wrn] [Event.cc:87] Warning: Deleteing a connection right after creation. Make sure to save the ConnectionPtr from a Connect call

Gazebo launches with the drone in the simulation, but as you mentioned there are some connection issues. When I revert back to the default rcS script it works fine. If the rcS script with custom port values works with roslaunch then it should also work with the examples mentioned in the SITL documentation. Is it that for the examples, the port values been hard coded somewhere else other than the rcS script?