Error while Simulator is starting

[ISH2YU]

I have followed all the steps but in the end , I am getting this error where simulation process has died and it just continues in loop forever how do i fix it help me

[nayaneshgudla]

I am also facing the same issue!!

[nicholasaleks]

Can you drop into the container and provide the ros/gazebo error logs. Also can you share your specs?

[nicholasaleks]

I just built it and I can not seem to reproduce your error

[nicholasaleks]

@ISH2YU - have you tried to deploy this with sudo ./start.sh --no-wifi or just using docker compose up --build?

[ISH2YU]

@nicholasaleks I actually Increased my allocated RAM from 8gb to 10gb and now its working better but now there is another problem that simulator interface is not loading fully like its still grey and drone dont seem to appear while other tabs are opening just fine. Can you tell me anyway to fix the simulator interface ?

Host Machine Specs Windows 11 Ram : 16gb GPU : Iris XE i71165G7

VM Specs:- Kali Linux 10gb Ram 100gb Disk Space allocated

I used sudo ./start.sh --no-wifi this method

[Rud3m]

Can you run: sudo ./stop.sh git pull sudo ./start.sh

[nicholasaleks]

@ISH2YU - can you confirm that you are using the latest version of the master branch and that by giving your VM additional RAM that you have solved the original problem?

[nicholasaleks]

Reopening the issue with two possible fixes for folks to use: Fix 1: @pljoel shared this issue with me as well via our Slack Channel and mentioned that when he increase his swap to 10GB, the master branch build works.

Fix 2: I built a new branch defcon which has all ROS support removed from the project. This should fix the random ros errors some people are seeing, however this does remove the onboard drone camera RTSP stream support.

If folks can confirm that either of these two fixes work - that would be greatly appreciated!

[pljoel]

@nicholasaleks

Although the defcon branch does work, I found the issue with ROS in the simulator container.

Problem

Just as the OP, the gazebo node was dying with code 137 and rosout kept restarting over and over again:

simulator               | process[rosout-1]: started with pid [52]
simulator               | [rosout-1] process has died [pid 52, exit code -9, cmd /opt/ros/noetic/lib/rosout/rosout __name:=rosout __log:=/root/.ros/log/34e69fc2-5ded-11ef-b07e-02420a0d0005/rosout-1.log].
simulator               | log file: /root/.ros/log/34e69fc2-5ded-11ef-b07e-02420a0d0005/rosout-1*.log                                                                                                                                       
simulator               | [rosout-1] restarting process
simulator               | process[rosout-1]: started with pid [100]
simulator               | Killed
simulator               | [gazebo-2] process has died [pid 55, exit code 137, cmd /opt/ros/noetic/lib/gazebo_ros/gzserver --verbose -e ode /Simulator/ardupilot_gazebo/worlds/damn-vulnerable-drone.world __name:=gazebo __log:=/root/.ros/log/34e69fc2-5ded-11ef-b07e-02420a0d0005/gazebo-2.log].                                                                                                                                                                                      
simulator               | log file: /root/.ros/log/34e69fc2-5ded-11ef-b07e-02420a0d0005/gazebo-2*.log
simulator               | [rosout-1] process has died [pid 100, exit code -9, cmd /opt/ros/noetic/lib/rosout/rosout __name:=rosout __log:=/root/.ros/log/34e69fc2-5ded-11ef-b07e-02420a0d0005/rosout-1.log].
simulator               | log file: /root/.ros/log/34e69fc2-5ded-11ef-b07e-02420a0d0005/rosout-1*.log                                                                                                                                       
simulator               | [rosout-1] restarting process
simulator               | process[rosout-1]: started with pid [101]
simulator               | [rosout-1] process has died [pid 101, exit code -9, cmd /opt/ros/noetic/lib/rosout/rosout __name:=rosout __log:=/root/.ros/log/34e69fc2-5ded-11ef-b07e-02420a0d0005/rosout-1.log].
simulator               | log file: /root/.ros/log/34e69fc2-5ded-11ef-b07e-02420a0d0005/rosout-1*.log                                                                                                                                       
simulator               | [rosout-1] restarting process
simulator               | process[rosout-1]: started with pid [105]
...

The issue is the same as explained here. XmlRpcServer queries all file descriptors until it reaches its limit (ulimit -Hn). In my Kali 2024.2 environment, the maximum number of opened file descriptors is 1073741816. Considering FD size of 4 bytes, that's around 4GB of consumed memory for every ROS nodes. And since we have 4 nodes, that's 16 GB total memory just for ROS. ROS will crash if (# ROS nodes max FD FD size) < (available memory). So considering Kali uses 1.5GB memory and the containers use 18.5 GB, if your working environment is below 20 GB mem+swap, DVD won't work properly.

root@9c9e29820b21:/Simulator/gzweb# rosnode list
/camera_display_node_121_1724040883964
/camera_display_node_7_1724040876681
/camera_rtsp_streamer_42_1724040876695
/gazebo
/rosout

See the simulator container memory usage in a working environment:

┌──(drone㉿dvd)-[~/git/dvd]
└─$ docker stats
CONTAINER ID   NAME                     CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O        PIDS
b384ef84da0f   simulator                218.51%   17.84GiB / 54.93GiB   32.47%    134MB / 6.33GB    4.19GB / 283kB   96
322c4619fcdb   companion-computer       15.48%    393.5MiB / 54.93GiB   0.70%     6.33GB / 134MB    211MB / 877kB    38
3881a0321744   ground-control-station   1.84%     66.36MiB / 54.93GiB   0.12%     9.34MB / 72.8kB   41.7MB / 791kB   9
881c3fa435df   flight-controller        0.00%     10.46MiB / 54.93GiB   0.02%     5.03MB / 0B       9.58MB / 0B      6
787c107168f8   qgc-container            0.00%     2.953MiB / 54.93GiB   0.01%     5.03MB / 0B       3.72MB / 0B      1

Fix

In docker-compose.yaml, if we reduce the limit of the simulator container its max number of opened FD, ROS will play nicely and DVD will work properly.

# File: docker-compose.yaml
  simulator:
    ulimits:
      nofile:
        soft: 1024
        hard: 524288

With 524288 as a hard limit, that's 2 MB memory consumed per node or 8 MB total. That's much better than the previous 16 GB. See the simulator container memory usage in a working environment:

┌──(drone㉿dvd)-[~/git/dvd]
└─$ docker stats
CONTAINER ID   NAME                     CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O        PIDS
d24f8d3bf3c6   simulator                229.79%   1.459GiB / 54.93GiB   2.66%     13.5MB / 626MB    378MB / 266kB    95
61d097d09f27   companion-computer       16.31%    393.4MiB / 54.93GiB   0.70%     626MB / 13.4MB    211MB / 868kB    38
c4c356a6d470   ground-control-station   2.07%     66.07MiB / 54.93GiB   0.12%     4.81MB / 72.2kB   41.7MB / 791kB   9
bd8bd559bd7a   flight-controller        0.00%     12.62MiB / 54.93GiB   0.02%     506kB / 0B        12.3MB / 0B      6
c9392a54de8a   qgc-container            0.01%     784KiB / 54.93GiB     0.00%     506kB / 0B        975kB / 0B       1

[pljoel]

PR #15 fixes the issue.