Trouble with RTF of REEMC in Gazebo

[JJJKon]

Hi,

I've installed the REEM-C simulation according to this tutorial on my laptop running Ubuntu 16.04 with the following specs that meet the requirements as given on this tutorial page:

The graphic card is set up with the proper drivers:

However, when I launch an empty world with REEM-C according to roslaunch reemc_gazebo reemc_empty_world.launch, the RTF of the simulation quickly (after a minute) drops to 0.1, even without enabling the controllers:

I've tried playing around with the solver settings to increase this simulation speed, but the performance increase can only be pushed till 0.15 before the controllers become unstable for these solver settings.

It should also be noted that the MSc. student that worked with REEM-C on a door opening environment before me also suffered from the same slow simulation speed.

Lastly, I've attached the terminal output of the launch file below such that you can inspect it for possible problems if necessary. Based on the warnings about not being able to disable collisions (line 1070 onwards), could it be possible that the engine is overloaded calculating contact dynamics resulting in the slow speed? Also, I was wondering if the fatal error regarding the library is relevant (line 1237)? Finally, could scanning for a joystick slow down the simulation every second, taking up too much resources, as the simulation seems to be blocking periodically?

Thanks in advance!

Output.txt

[v-lopez]

Hello,

There's a pluginlib error in your output:

[ [FATAL] [1568883678.237463598, 0.001000000]: Failed to create robot simulation interface loader: Failed to load library /home/johan/reemc_public_ws/devel/lib//libpal_hardware_gazebo.so. Make sure that you are calling the PLUGINLIB_EXPORT_CLASS macro in the library code, and that names are consistent between this macro and your XML. Error string: Could not load library (Poco exception = /home/johan/reemc_public_ws/devel/lib//libpal_hardware_gazebo.so: undefined symbol: _ZN18gazebo_ros_control17DefaultRobotHWSim13prepareSwitchERKNSt7__cxx114listIN18hardware_interface14ControllerInfoESaIS4EEES8)

Can you please make sure that you have our version of ros control in your workspace, and you have uninstalled all other ros control version from your system?

Maybe the RTF is caused by the robot being on the ground and having lots of contacts like you mentioned. But the robot should not fall to the ground when starting, I believe that these library loading issues are the cause of it.

[JJJKon]

The library is present in the given path:

I am new to ros, so I don't know how to uninstall all other ros control versions and where to find those? I've not installed anything besides the packages mentioned in your tutorials, so I guess there's no other ros control versions? Could the double / in the library path /home/johan/reemc_public_ws/devel/lib//libpal_hardware_gazebo.so of the error message be the cause of the failure?

[v-lopez]

Check if you have other ros control versions with dpkg -l | grep ros-control. Uninstall anything that starts with ros-.

Ros control is released on the official ROS repositories, we're running a modified ABI incompatible version, that can conflict and lead to situations like this.

[JJJKon]

This indeed resolved the library error. Now the robot indeed stays upright after initialization, but the RTF dropped to 0.07:

Maybe the RTF is caused by the robot being on the ground and having lots of contacts like you mentioned.

I was not referring to the contacts with the ground, but self-contacts, as it seems that the launch file outputs that it was not able to disable contact between multiple joints of the REEMC.

[v-lopez]

Can you post the output of the console like you did before?

Also run a top and post it here please.

[JJJKon]

Console output (collision warnings from 1070 onwards): Output2.txt

Top while running the processes:

[v-lopez]

It looks like your CPU is saturated.

Can you print cat /proc/cpuinfo? Also are you sure your nvidia is properly configured? gzclient is taking also too much cpu.

[JJJKon]

cat /proc/cpuinfo produces: cpuinfo.txt

Before I configured nvidia, Ubuntu was using the Nouveau driver with Intel Haswell listed as graphics in the first two images posted. With this configuration, I was not able to run the simulation at all: it would drop to RTF 0.01 and eventually straight-up stop simulating anything. The current situation is after configuring nvidia. Is there any other way I can test if it is properly configured?

[v-lopez]

Run glxgears (install mesa-utils if you don't have it).

It should give an output similar to this:

And if you run top, glxgears should use minimal cpu. If it's using cpu to render, glxgears will use 100% cpu.

[JJJKon]

It is in fact using the cpu to render (edit2: 100% usage, not for rendering), so I'll have a look at the card configuration again.

Edit: glxgears is also an active process on the GPU as indicated by nvidia-smi while also using 100% CPU.

This is the same for gzserver and gzclient

I suppose that if the card was not properly configured, I would not have these processes running on it?

Edit2: glxgears also indicates that it is using the GPU for rendering:

Edit3: profiling the cpu (sudo operf glxgears followed by opreport) shows that another process takes up the biggest chunk. Investigating and solving this is however far beyond my knowledge, so before I continue, I would first like to hear your thoughts on if it is indeed a problem with the video card configuration.

[v-lopez]

Sorry I did not get any notification regarding the edits you made to your comment.

I believe your issues are not related to reemc simulation, but I will try my best to help you.

Tthe glxgears should be running at 50-60fps and take minimal cpu, if not, maybe still some part of your graphic settings or openGL is not configured properly.

Also, your cpu specs may be limiting you for these kind of simulations. My machine has a i7-7700k CPU (which is around 50% faster than your cpu according to benchmarks) , with it we're running it with RTF of 0.6 launching roslaunch reemc_gazebo reemc_empty_world.launch.

And glxgears is using around 2% cpu.

Still I believe you should be able to reach 0.3 or 0.4 RTF on the empty world.