Performance issue

[nicolaje]

Hi @srmainwaring ,

first thank you a lot for this plugin, it helped significantly.

I am running your "develop" branch with Gazebo 11, and with a simple Wam-V simulation the real-time factor of Gazebo is 0.14 with only one CPU running 100%.

I have a quite decent computer (NVIDIA Quadro T2000, Corei9 with 8 cores, 32GB of RAM).

To reproduce I just clone your repository, checkout development branch, build then:

roslaunch asv_wave_sim_gazebo ocean_world.launch

Then from Gazebo insert the Wam-V model.

Is there something I am missing ?

Thank you !

[srmainwaring]

Hi @nicolaje, I'm glad the plugin has been useful.

If you're using the Wam-V model from this repo I'll need to look into why the performance is poor. I recall it running at better RTF than 0.14. The model has modified collision meshes that are supposed to reduce the triangle count and remove degenerate triangles to improve performance - clearly something is not working as intended.

[nicolaje]

Hi, thank you for the prompt answer !

I am indeed using the model of this repository.

Could you maybe point me in a direction if you need help troubleshooting?

Best regads

PS: Also, be aware that on the fft_waves & gazebo11 branches, all builds fine but when you add by hand a dock / Wam-V etc... they dont' float. I am just using it successfully with the develop branch.

[srmainwaring]

PS: Also, be aware that on the fft_waves & gazebo11 branches, all builds fine but when you add by hand a dock / Wam-V etc... they dont' float. I am just using it successfully with the develop branch.

I suspect that may be because I've changed the name of the ocean model in those branches - so the boat model is not subject to any buoyancy forces. That should be an easy fix if that's the case. If you run gazebo with the --verbose flag there should be messages describing what is going wrong.

I'm in the process of developing support for Ignition Gazebo at the moment which has broken a lot of my old ROS / Gazebo11 environment because of updated dependencies. I'll see if I can run up another Ubuntu VM with Noetic to investigate what is going on.

[srmainwaring]

I'm afraid I can't replicate the performance issue you are seeing (or the WAM-V not floating). My testing was on an Ubuntu 20.04 VM hosted on macOS, so there will be environment differences, but I'll outline the steps taken and results which may help.

System:

As you can see I'm running an Ubuntu 20.04.3 LTS VM. I've allocated 4 cores to the VM and a reasonable amount of RAM. Hardware acceleration is enabled for the machine.

Install ROS Noetic:

$ sudo sh -c 'echo "deb http://packages.ros.org/ros/ubuntu $(lsb_release -sc) main" > /etc/apt/sources.list.d/ros-latest.list'
$ sudo apt install curl
$ curl -s https://raw.githubusercontent.com/ros/rosdistro/master/ros.asc | sudo apt-key add -
$ sudo apt update
$ sudo apt upgrade
$ sudo apt install ros-noetic-desktop-full

Check Gazebo version:

$ gazebo --version
Gazebo multi-robot simulator, version 11.8.1
Copyright (C) 2012 Open Source Robotics Foundation.
Released under the Apache 2 License.
http://gazebosim.org

Install dependencies:

$ sudo apt-get install libcgal-dev
$ sudo apt install fftw-dev
$ sudo apt install libfftw3-dev
$ sudo apt install libclfft-dev
$ sudo apt-get install python3-catkin-tools

Build branch feature/fft_waves:

$ mkdir -p ocean_sim_ws/src && cd ocean_sim_ws/src
$ git clone https://github.com/srmainwaring/asv_wave_sim.git -b feature/fft_waves
$ cd ..
$ source /opt/ros/noetic/setup.bash 
$ catkin config --cmake-args -DCMAKE_BUILD_TYPE=RelWithDebInfo

Test:

$ catkin build --catkin-make-args run_tests

$ catkin_test_results build
Summary: 98 tests, 0 errors, 0 failures, 0 skipped

Configure for hardware acceleration:

I have used some custom OpenGL settings by tweaking the MESA environment variables when using hardware acceleration with Gazebo. These are required because the host is running macOS which only supports OpenGL 4.1 and some of the extensions that the latest Mesa driver is attempting to use are not compatible. The Gallium config is for a HUD report on the OpenGL state:

# Configure the mesa environment
# https://docs.mesa3d.org/envvars.html

export LIBGL_DEBUG=verbose
export LIBGL_ALWAYS_INDIRECT=false
export LIBGL_ALWAYS_SOFTWARE=false
export LIBGL_NO_DRAWARRAYS=false
export LIBGL_SHOW_FPS=false
export LIBGL_DRI2_DISABLE=false
export LIBGL_DRI3_DISABLE=false

# Core Mesa environment variables

export MESA_DEBUG=1
export MESA_GL_VERSION_OVERRIDE=4.1
export MESA_GLSL_VERSION_OVERRIDE=410
export MESA_EXTENSION_OVERRIDE=" \
-GL_ARB_buffer_storage \
-GL_ARB_multi_draw_indirect \
-GL_ARB_texture_buffer_range \
-GL_ARB_compute_shader \
-GL_ARB_ES3_compatibility \
"

# Gallium environment variables

export GALLIUM_HUD="simple,\
fps\
+frametime\
+cpu\
+cpu0\
+cpu1\
+cpu2\
+cpu3\
+samples-passed\
+primitives-generated\
+num-draw-calls\
+nic-rx-ens33\
+nic-tx-ens33\
+num-fallbacks\
+num-flushes\
+num-validations\
+map-buffer-time\
+num-buffers-mapped\
+num-textures-mapped\
+num-bytes-uploaded\
+num-command-buffers\
+command-buffer-size\
+flush-time\
+surface-write-flushes\
+num-readbacks\
+num-resource-updates\
+num-buffer-uploads\
+num-const-buf-updates\
+num-const-updates\
+num-shader-relocations\
+num-surface-relocations\
+memory-used\
+num-shaders\
+num-resources\
+num-state-objects\
+num-surface-views\
+num-generate-mipmap\
+num-failed-allocations\
+num-commands-per-draw\
+shader-mem-used\
"

Run:

$ roslaunch asv_wave_sim_gazebo ocean_world.launch verbose:=true

... logging to /home/rhys/.ros/log/63a9c7d0-0a53-11ec-904f-5579dfb2e5fa/roslaunch-ubuntu-5843.log
Checking log directory for disk usage. This may take a while.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://ubuntu:40737/

SUMMARY
========

PARAMETERS
 * /gazebo/enable_ros_network: True
 * /rosdistro: noetic
 * /rosversion: 1.15.11
 * /use_sim_time: True

NODES
  /
    gazebo (gazebo_ros/gzserver)
    gazebo_gui (gazebo_ros/gzclient)

auto-starting new master
process[master]: started with pid [5851]
ROS_MASTER_URI=http://localhost:11311

setting /run_id to 63a9c7d0-0a53-11ec-904f-5579dfb2e5fa
process[rosout-1]: started with pid [5861]
started core service [/rosout]
process[gazebo-2]: started with pid [5864]
process[gazebo_gui-3]: started with pid [5871]
Gazebo multi-robot simulator, version 11.8.1
Copyright (C) 2012 Open Source Robotics Foundation.
Released under the Apache 2 License.
http://gazebosim.org

[Wrn] [GuiIface.cc:200] g/gui-plugin is really loading a SystemPlugin. To load a GUI plugin please use --gui-client-plugin 
Gazebo multi-robot simulator, version 11.8.1
Copyright (C) 2012 Open Source Robotics Foundation.
Released under the Apache 2 License.
http://gazebosim.org

[ INFO] [1630411382.325008848]: Finished loading Gazebo ROS API Plugin.
[Msg] Waiting for master.
[ INFO] [1630411382.326814769]: waitForService: Service [/gazebo/set_physics_properties] has not been advertised, waiting...
[Msg] Connected to gazebo master @ http://127.0.0.1:11345
[Msg] Publicized address: 192.168.1.171
libGL: Can't open configuration file /etc/drirc: No such file or directory.
libGL: Can't open configuration file /home/rhys/.drirc: No such file or directory.
libGL: using driver vmwgfx for 21
libGL: Can't open configuration file /etc/drirc: No such file or directory.
libGL: Can't open configuration file /home/rhys/.drirc: No such file or directory.
libGL: pci id for fd 21: 15ad:0405, driver vmwgfx
libGL: MESA-LOADER: dlopen(/usr/lib/x86_64-linux-gnu/dri/vmwgfx_dri.so)
libGL: Can't open configuration file /etc/drirc: No such file or directory.
libGL: Can't open configuration file /home/rhys/.drirc: No such file or directory.
libGL: Can't open configuration file /etc/drirc: No such file or directory.
libGL: Can't open configuration file /home/rhys/.drirc: No such file or directory.
libGL: Can't open configuration file /etc/drirc: No such file or directory.
libGL: Can't open configuration file /home/rhys/.drirc: No such file or directory.
libGL: Using DRI3 for screen 0
    nic-rx-ens33
    nic-tx-ens33
[ INFO] [1630411382.419237330]: Finished loading Gazebo ROS API Plugin.
[ INFO] [1630411382.420396594]: waitForService: Service [/gazebo_gui/set_physics_properties] has not been advertised, waiting...
[Msg] Waiting for master.
[Msg] Connected to gazebo master @ http://127.0.0.1:11345
[Msg] Publicized address: 192.168.1.171
libGL: Can't open configuration file /etc/drirc: No such file or directory.
libGL: Can't open configuration file /home/rhys/.drirc: No such file or directory.
libGL: using driver vmwgfx for 38
libGL: Can't open configuration file /etc/drirc: No such file or directory.
libGL: Can't open configuration file /home/rhys/.drirc: No such file or directory.
libGL: pci id for fd 38: 15ad:0405, driver vmwgfx
libGL: MESA-LOADER: dlopen(/usr/lib/x86_64-linux-gnu/dri/vmwgfx_dri.so)
libGL: Can't open configuration file /etc/drirc: No such file or directory.
libGL: Can't open configuration file /home/rhys/.drirc: No such file or directory.
libGL: Can't open configuration file /etc/drirc: No such file or directory.
libGL: Can't open configuration file /home/rhys/.drirc: No such file or directory.
libGL: Can't open configuration file /etc/drirc: No such file or directory.
libGL: Can't open configuration file /home/rhys/.drirc: No such file or directory.
libGL: Using DRI3 for screen 0
    nic-rx-ens33
    nic-tx-ens33
[Msg] Loading world file [/home/rhys/Code/robotics/ocean_sim_ws/src/asv_wave_sim/asv_wave_sim_gazebo/worlds/ocean.world]
[ INFO] [1630411382.645481280]: waitForService: Service [/gazebo/set_physics_properties] is now available.
[ INFO] [1630411382.667069044]: Physics dynamic reconfigure ready.
[Msg] Parameter found - setting <static> to <0>.
[Msg] Parameter found - setting <update_rate> to <30>.
[Msg] Parameter found - setting <wave_patch> to <0>.
[Msg] Parameter found - setting <wave_patch_size> to <4 4>.
[Msg] Parameter found - setting <size> to <1000 1000>.
[Msg] Parameter found - setting <cell_count> to <50 50>.
[Msg] Parameter found - setting <number> to <3>.
[Msg] Parameter found - setting <amplitude> to <1>.
[Msg] Parameter found - setting <period> to <8>.
[Msg] Parameter <phase> not found: Using default value of <0>.
[Msg] Parameter found - setting <direction> to <1 1>.
[Msg] Parameter found - setting <scale> to <2.5>.
[Msg] Parameter found - setting <angle> to <0.3>.
[Msg] Parameter found - setting <steepness> to <1>.
[Msg] Constructing WavefieldOceanTile...
[Msg] Creating WaveParameters.
[Msg] Creating OceanTile.
[Msg] mVertices.size(): 16641
[Msg] Creating grid.
[Msg] Creating triangulated grid.
[Msg] Done constructing WavefieldOceanTile.
libGL: Can't open configuration file /etc/drirc: No such file or directory.
libGL: Can't open configuration file /home/rhys/.drirc: No such file or directory.
libGL: using driver vmwgfx for 61
libGL: Can't open configuration file /etc/drirc: No such file or directory.
libGL: Can't open configuration file /home/rhys/.drirc: No such file or directory.
libGL: pci id for fd 61: 15ad:0405, driver vmwgfx
libGL: MESA-LOADER: dlopen(/usr/lib/x86_64-linux-gnu/dri/vmwgfx_dri.so)
libGL: Can't open configuration file /etc/drirc: No such file or directory.
libGL: Can't open configuration file /home/rhys/.drirc: No such file or directory.
libGL: Can't open configuration file /etc/drirc: No such file or directory.
libGL: Can't open configuration file /home/rhys/.drirc: No such file or directory.
libGL: Can't open configuration file /etc/drirc: No such file or directory.
libGL: Can't open configuration file /home/rhys/.drirc: No such file or directory.
libGL: Using DRI3 for screen 0
[Err] [msgs.cc:2873] Unrecognized geometry type
[Msg] Loading OceanVisualPlugin...
[Msg] Done loading OceanVisualPlugin.
[Msg] Initializing OceanVisualPlugin...
[Msg] Constructing OceanVisual...
[Msg] mVertices.size(): 16641
[Msg] Creating OceanTile mesh...
[Msg] OceanTile mesh created.
[Msg] Creating OceanTile mesh...
[Msg] OceanTile mesh created.
[Msg] Done initializing OceanVisualPlugin.
[Msg] Initializing OceanVisualPlugin...
[Msg] Done initializing OceanVisualPlugin.
[Msg] Parameter found - setting <wave_model> to <ocean>.
[Msg] Parameter found - setting <damping_on> to <1>.
[Msg] Parameter found - setting <viscous_drag_on> to <1>.
[Msg] Parameter found - setting <pressure_drag_on> to <1>.
[Msg] Parameter <cDampL1> not found: Using default value of <1e-06>.
[Msg] Parameter <cDampL2> not found: Using default value of <1e-06>.
[Msg] Parameter <cDampR1> not found: Using default value of <1e-06>.
[Msg] Parameter <cDampR2> not found: Using default value of <1e-06>.
[Msg] Parameter <cPDrag1> not found: Using default value of <100>.
[Msg] Parameter <cPDrag2> not found: Using default value of <100>.
[Msg] Parameter <fPDrag> not found: Using default value of <0.4>.
[Msg] Parameter <cSDrag1> not found: Using default value of <100>.
[Msg] Parameter <cSDrag2> not found: Using default value of <100>.
[Msg] Parameter <fSDrag> not found: Using default value of <0.4>.
[Msg] Parameter <vRDrag> not found: Using default value of <1>.
[Msg] Parameter found - setting <update_rate> to <30>.
[Msg] Parameter found - setting <water_patch> to <0>.
[Msg] Parameter found - setting <waterline> to <0>.
[Msg] Parameter found - setting <underwater_surface> to <0>.
[Msg] Shape:      shape
[Msg] Scale:      1 1 1
[Msg] Type:       2010000
[Msg] Type:       MESH_SHAPE
[Msg] MeshURI:    model://wam-v/meshes/wam-v-base_collision_406.stl
[Msg] MeshStr:    /home/rhys/Code/robotics/ocean_sim_ws/src/asv_wave_sim/asv_wave_sim_gazebo/models/wam-v/meshes/wam-v-base_collision_406.stl
[Msg] Mesh:       /home/rhys/Code/robotics/ocean_sim_ws/src/asv_wave_sim/asv_wave_sim_gazebo/models/wam-v/meshes/wam-v-base_collision_406.stl
[Msg] Vertex:     1218
[Msg] links:  1
[Msg] meshes: 1
[Msg] Water patch size: 12.6209

The hardware accelerated version is still unstable for me on the Ubuntu 20.04 VM - and will lock after a few minutes. This is because I don't have the Mesa environment quite right for the Ogre1 render engine and the macOS host , and is not to do with the wave library.

I repeated the run with hardware acceleration disabled:

export LIBGL_ALWAYS_SOFTWARE=true

The RTF is the same but the framerate drops to around 3-4 FPS.

[nicolaje]

Hi @srmainwaring ,

... --cmake-args -DCMAKE_BUILD_TYPE=RelWithDebInfo

that was it. It is the default Debug build which is slow as hell.

So develop is working great on my desktop when built in release node.

On the other hand, I can't get feature/fft_waves to work. I run the simulation, then if I drop a Duck or a Wam-V it sinks and I have this error message :

Is there a Paypal or something like that to thank you for your plugin ? I would love to buy you a pack of beers.

Cheers

[srmainwaring]

Great! Glad it's running well now.

The issue with the model not floating has to do with the name given to the ocean model. The message is essentially saying that the hydrodynamics plugin associated with the wam-v model cannot find a wave model with the name ocean. I could probably improve that message!

Check the name of the ocean model in the Gazebo sidebar - I suspect it is ocean_waves rather than ocean (i.e. the sim is picking up a duplicated version of the ocean model somewhere with the wrong name). You may have to search for old / installed versions from the master build and remove them.

[ricardohalfeld]

Hello. First let me take a second to congratulate on the result of your efforts. Truly beautiful plugin. This makes Gazebo a better-rounded simulator when it comes to selling a project.

Second, let me just get a little trick out there: On Gazebo's GUI, go to Physics and reduce the Real Time Update Rate. Also adjust Max Step Size to (Real Time Update Rate)^(-1). This will improve your real time simulation at the increased risk of making it less representative. Use with caution and make sure you understand the implications.

Cheers, folks!

[srmainwaring]

I'm going to close this issue as the main problem was resolved by using the release build settings. There is of course ongoing work to improve performance generally which will be discussed in separate issues.