ubi-agni / mujoco_ros_pkgs

Wrappers, tools and additional API's for using ROS with MuJoCo
54 stars 11 forks source link

Clean shutdown #12

Closed rhaschke closed 1 year ago

rhaschke commented 1 year ago

When closing MuJoCo's viewer window, I get the following warnings:

[ WARN] [1665937745.266475546, 340.436999999] [ros.rosconsole_bridge.class_loader.ClassLoader] [/mujoco_server]: SEVERE WARNING!!!
Attempting to unload /vol/s4dx/devel/lib//libmujoco_contact_surfaces.so
while objects created by this library still exist in the heap!
You should delete your objects before destroying the ClassLoader. The library will NOT be unloaded.
[ WARN] [1665937745.266574505, 340.436999999] [ros.rosconsole_bridge.class_loader.ClassLoader] [/mujoco_server]: SEVERE WARNING!!!
Attempting to unload /vol/s4dx/devel/lib//libmujoco_ros_control.so
while objects created by this library still exist in the heap!
You should delete your objects before destroying the ClassLoader. The library will NOT be unloaded.
[ WARN] [1665937745.266619038, 340.436999999] [ros.rosconsole_bridge.class_loader.ClassLoader] [/mujoco_server]: SEVERE WARNING!!!
Attempting to unload /vol/s4dx/devel/lib//libmujoco_ros_sensors.so
while objects created by this library still exist in the heap!
You should delete your objects before destroying the ClassLoader. The library will NOT be unloaded.

The reason is that the plugin libs are being unloaded although object instances created from those libs are still in use. You first need to finish controllers and unregister/delete those instances.

DavidPL1 commented 1 year ago

The error related to pluginlib failing to unload the libraries is fixed by 98c58d415041d37267252e15b396cd943bd1f0cd. However, I still get the following exception on shutdown:

terminate called after throwing an instance of 'boost::wrapexcept<boost::lock_error>'
  what():  boost: mutex lock failed in pthread_mutex_lock: Invalid argument

I'm not quite sure about its origin and debugging this error is not straight forward. We'll have to dig a little deeper for an actually graceful exit.

rhaschke commented 1 year ago

Did you already tried to debug this? You need to figure out which mutex is affected. The error typically is related to the mutex being destroyed already (in some other thread).

DavidPL1 commented 1 year ago

I checked and cleaned up the usages of sim and render mutexes in 6603225092072d1206e5d6585fb8cacf0ccf93cc. But I already suspected this is somehow related to ROS threads, because of the boost wrapped exception.

With gdb I found that the unique pointer to the simulation step action still needed to be reset and would otherwise cause deallocation problems. If the simulation uses the sensors plugin and is shutdown before unpausing, the waitForMessage call in the sensors deferred init function will throw

terminate called after throwing an instance of 'boost::wrapexcept<boost::bad_weak_ptr>'
  what():  tr1::bad_weak_ptr

I'm not sure yet how to handle this. We might have to change how the sensor plugin ensures transforms are available.

Other than this last issue, the simulation should exit gracefully.

rhaschke commented 1 year ago

Could you please call me for this - either by phone or zoom.

rhaschke commented 1 year ago

Looks good!

DavidPL1 commented 1 year ago

I merged the feature branch. For now we fixed all errors on shutdown.