Gazebo RFT dropping when any model is close to tall walls

[SimoneMic]

Hello

I am having an issue when spawning any robot close to a wall (or moving there after navigation) the RTF drops drastically (lower than 0.1). The contacts are present on the whole world. I have tried to remove the textures, but the effect is the same as in the following pictures: Warehouse example:

All the following situations have RFT smaller than 0.1

Spawning iCub next to wall:

Spawning R1 next to wall:

Spawning ergoCubGazeboV1 next to wall:

When spawning the robot in the middle of the scene instead, we have an higher RTF

R1 in center of map RTF=0.99

ergoCubGazeboV1 in center of map RTF=0.6

Does anybody have any suggestion on what could be the problem? To me seems like a collision check threshold, that when the robot moves closer to the wall it starts checking for contacts.

I am also sharing my world file: worlds.zip

Thanks

cc @traversaro @randaz81

[traversaro]

Interesting. Just to isolate the problem, if you comment out the laser the problem does not appear?

[traversaro]

(To double check, which version of Gazebo are you using?)

[traversaro]

Interesting. Just to isolate the problem, if you comment out the laser the problem does not appear?

Actually, some models do not have a laser, so probably this is not related to that. Can you try if this happens also with other physics engine, in particular with dart (i.e. launch gazebo with -e dart)?

[randaz81]

Interesting. Just to isolate the problem, if you comment out the laser the problem does not appear?

I'm also somehow convinced that is somehow connected to collision checking / lidar ray casting. The same issue was reported by kostantinos some time ago and never investigated in detail... ps: to prove that, i think that if you remove all collisions in the walls, the lidar rays will pass through the walls and maybe the simulation will run properly. Also, have gpu_ray plugin been tested?

[xEnVrE]

i think that if you remove all collisions in the walls, the lidar rays will pass through the walls and maybe the simulation will run properly.

I just tried that and indeed the ray pass through walls and RTF was back to the standard one. I am trying to simplify the collision meshes just to see what happens

[xEnVrE]

Can you try if this happens also with other physics engine, in particular with dart (i.e. launch gazebo with -e dart)?

I think we tried once with @randaz81 and with DART the lidar plugin was not working as the ray casting routines are missing in DART If I recall correctly

[traversaro]

Can you try if this happens also with other physics engine, in particular with dart (i.e. launch gazebo with -e dart)?

I think we tried once with @randaz81 and with DART the lidar plugin was not working as the ray casting routines are missing in DART If I recall correctly

Good call, related upstream issue: https://github.com/gazebosim/gazebo-classic/issues/911 . So the issue is related to ray, right? Then do you have any idea why "Spawning iCub next to wall:" also has a low RTF?

[traversaro]

So some pointers:

• What happens if you use gpu_ray as a sensor instead of ray? (see https://answers.gazebosim.org//question/19529/what-is-the-different-between-sdfs-gpu_ray-sensor-and-ray-sensor/ for more details)
• If you want to profile the code to understand what is going on, the possible way to proceed is :
  • Profile what is going on with the Gazebo profiler ( https://classic.gazebosim.org/tutorials?tut=profiler&cat=tools_utilities ). Unfortunatly, this requires rebuilding Gazebo
  • Use an unmodified Gazebo binary with a profiler such as https://github.com/janestreet/magic-trace . However, I do not know if this will work fine as I do not know how much dbg information is available in the default Gazebo binary build.

[SimoneMic]

I did a separate test having ergoCubGazeboV1 spawn outside the warehouse in a confined space using the wall builder inside Gazebo. The RTF is 0.98. The lidar rays are colliding with the walls.

[SimoneMic]

Meanwhile, stickBot doesn't present this behavior. RTF 0.88

The difference is that stickbot doesn't have collisions and meshes on its links. The only difference is that stickBot is on a previous docker without the newest gazebo yarp plugins (that are needed for `

I have tried to remove the collisions on ergoCubGazeboV1 before, but I monitored the behavior only in the middle of the scene and there were not noticeable differences. Maybe in this extreme case will help, so I will try to redo it.

[SimoneMic]

I have removed the collisions on the urdf of ergoCubGazeboV1 except for the _sole links (that don't use a mesh but a normal box from the sdf format) RTF 0.75

Meanwhile, spawning the robot in the middle of the scenario, the improvements from the collisions are minimal. RTF 0.65 from 0.6

[SimoneMic]

I think that the lession learned is to avoid using mesh files as collision surfaces, but only elemental geometrical volumes (like box, cylinder etc.). Since the many mash element that are generated (I don't know how the segmentation of the finite element works in gazebo or by the tools used to generate the meshes and urdf in the repo) are checked by the collision monitor inside gazebo.

This explains the good behavior of stickBot that uses only elementary shapes. In fact I never had issues in simulation before. I'll try to add some collision boxes on ergoCubGazeboV1 (for the hands and torso, maybe) and see how it performs.

[traversaro]

@SimoneMic did you tried to see what happens if gpu_ray is used?

[SimoneMic]

I haven't tried using gpu_ray yet since it needs a bit of cherry-picking from the code that @morpheus1820 forked gazebo-yarp-plugins and update it to the newest release. I am planning to do it tomorrow, since today I have managed to run a long and good simulation.

[SimoneMic]

I have tried using the gpu_ray but with no luck so far. I have applyied the changes from f0bc131 to gazebo-yarp-plugins on my docker, and installed them. In the urdf of the robot ergoCubGazeboV1 I switched the type of sensor from ray to gpu_ray here and here

However, when I spawn the robot, the readings from the sensor are Inf. like the sensor is not finding any object. Here the errors on gazebo: [ERROR] |gazebo-yarp-plugins.plugins.GazeboYarpLaserSensor| Size error, expected: 1600 , got: 0

And a picture with the laser visualized:

I am launching my docker using this script:

#!/bin/bash
NAME=simonemiche/ergocub_ros2
TAG=ecub_first_year_demo
sudo xhost +
sudo docker run \
    --network=host --privileged \
    -it \
    --rm \
    -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=unix${DISPLAY} --device /dev/dri/card0:/dev/dri/card0 --gpus=all \
    -e NVIDIA_DRIVER_CAPABILITIES=all --runtime=nvidia\
    ${NAME}:${TAG} bash

I don't know if there are any other files that need to be modified

[traversaro]

Thanks @SimoneMic ! It is indeed possible that there is a problem upstream in Gazebo.

[SimoneMic]

Closing since the "fix" will be discussed in #83