Flaky seg-fault when detaching HarnessPlugin

Original report (archived issue) by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).

The HarnessPlugin was added in pull request #2346 released in gazebo 7.3.0. When detaching the harness joint, it can seg-fault due to a race condition. The race condition can be exacerbated by unthrottling the real-time update rate, and can also be avoided by pausing before detaching.

Here is a backtrace of the relevant threads:

World::RunLoop thread that received the seg-fault

Thread 1 (Thread 0x7f584ebfc700 (LWP 19220)):
#0  0x00007f58b3476c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007f58b347a028 in __GI_abort () at abort.c:89
#2  0x00007f58b346fbf6 in __assert_fail_base (fmt=0x7f58b35c03b8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x7f584d9f1379 "px != 0", file=file@entry=0x7f584d9f0ef0 "/usr/include/boost/smart_ptr/shared_ptr.hpp", line=line@entry=653, function=function@entry=0x7f584d9f2900 <boost::shared_ptr<gazebo::physics::Joint>::operator->() const::__PRETTY_FUNCTION__> "typename boost::detail::sp_member_access<T>::type boost::shared_ptr<T>::operator->() const [with T = gazebo::physics::Joint; typename boost::detail::sp_member_access<T>::type = gazebo::physics::Joint*"...) at assert.c:92
#3  0x00007f58b346fca2 in __GI___assert_fail (assertion=0x7f584d9f1379 "px != 0", file=0x7f584d9f0ef0 "/usr/include/boost/smart_ptr/shared_ptr.hpp", line=653, function=0x7f584d9f2900 <boost::shared_ptr<gazebo::physics::Joint>::operator->() const::__PRETTY_FUNCTION__> "typename boost::detail::sp_member_access<T>::type boost::shared_ptr<T>::operator->() const [with T = gazebo::physics::Joint; typename boost::detail::sp_member_access<T>::type = gazebo::physics::Joint*"...) at assert.c:101
#4  0x00007f584d9e5a43 in boost::shared_ptr<gazebo::physics::Joint>::operator-> (this=<optimized out>) at /usr/include/boost/smart_ptr/shared_ptr.hpp:653
#5  0x00007f584d9e85ae in operator-> (this=<optimized out>) at /var/lib/jenkins/workspace/gazebo7-debbuilder/build/gazebo-7.3.1/plugins/HarnessPlugin.cc:267
#6  gazebo::HarnessPlugin::OnUpdate (this=0x7f582c008560, _info=...) at /var/lib/jenkins/workspace/gazebo7-debbuilder/build/gazebo-7.3.1/plugins/HarnessPlugin.cc:246
#7  0x00007f58b2eb74b9 in operator() (a0=..., this=<optimized out>) at /usr/include/boost/function/function_template.hpp:767
#8  gazebo::event::EventT<void (gazebo::common::UpdateInfo const&)>::Signal<gazebo::common::UpdateInfo>(gazebo::common::UpdateInfo const&) (this=0x7f58b47498c0 <gazebo::event::Events::worldUpdateBegin>, _p=...) at /var/lib/jenkins/workspace/gazebo7-debbuilder/build/gazebo-7.3.1/gazebo/common/Event.hh:385
#9  0x00007f58b2ea4681 in operator()<gazebo::common::UpdateInfo> (_p=..., this=<optimized out>) at /var/lib/jenkins/workspace/gazebo7-debbuilder/build/gazebo-7.3.1/gazebo/common/Event.hh:221
#10 gazebo::physics::World::Update (this=this@entry=0x1cf7760) at /var/lib/jenkins/workspace/gazebo7-debbuilder/build/gazebo-7.3.1/gazebo/physics/World.cc:725
#11 0x00007f58b2eb246b in gazebo::physics::World::Step (this=this@entry=0x1cf7760) at /var/lib/jenkins/workspace/gazebo7-debbuilder/build/gazebo-7.3.1/gazebo/physics/World.cc:656
#12 0x00007f58b2eb289d in gazebo::physics::World::RunLoop (this=0x1cf7760) at /var/lib/jenkins/workspace/gazebo7-debbuilder/build/gazebo-7.3.1/gazebo/physics/World.cc:465
#13 0x00007f58b0ea6a4a in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
#14 0x00007f58b2181184 in start_thread (arg=0x7f584ebfc700) at pthread_create.c:312
#15 0x00007f58b353a37d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Transport thread that calls HarnessPlugin::OnDetach:

Thread 13 (Thread 0x7f5890b4c700 (LWP 19180)):
#0  boost::variant<bool, char, std::string, int, unsigned long, unsigned int, double, float, sdf::Time, sdf::Color, sdf::Vector3, sdf::Vector2i, sdf::Vector2d, sdf::Quaternion, sdf::Pose, ignition::math::Vector3<double>, ignition::math::Vector2<int>, ignition::math::Vector2<double>, ignition::math::Quaternion<double>, ignition::math::Pose3<double> >::internal_apply_visitor<boost::detail::variant::destroyer> (this=0x7f582c0361b0, visitor=...) at /usr/include/boost/variant/variant.hpp:2337
#1  0x00007f58b1ac5046 in destroy_content (this=0x7f582c0361b0) at /usr/include/boost/variant/variant.hpp:1373
#2  ~variant (this=0x7f582c0361b0, __in_chrg=<optimized out>) at /usr/include/boost/variant/variant.hpp:1380
#3  ~ParamPrivate (this=0x7f582c036120, __in_chrg=<optimized out>) at /var/lib/jenkins/workspace/sdformat4-debbuilder/build/sdformat-4.1.1/include/sdf/Param.hh:204
#4  sdf::Param::~Param (this=0x7f582c035860, __in_chrg=<optimized out>) at /var/lib/jenkins/workspace/sdformat4-debbuilder/build/sdformat-4.1.1/src/Param.cc:166
#5  0x00007f58b1ac50c9 in sdf::Param::~Param (this=0x7f582c035860, __in_chrg=<optimized out>) at /var/lib/jenkins/workspace/sdformat4-debbuilder/build/sdformat-4.1.1/src/Param.cc:168
#6  0x00007f58b1a846a2 in _M_release (this=0x7f582c037460) at /usr/include/c++/4.8/bits/shared_ptr_base.h:144
#7  ~__shared_count (this=0x7f582c036218, __in_chrg=<optimized out>) at /usr/include/c++/4.8/bits/shared_ptr_base.h:546
#8  ~__shared_ptr (this=0x7f582c036210, __in_chrg=<optimized out>) at /usr/include/c++/4.8/bits/shared_ptr_base.h:781
#9  ~shared_ptr (this=0x7f582c036210, __in_chrg=<optimized out>) at /usr/include/c++/4.8/bits/shared_ptr.h:93
#10 _Destroy<std::shared_ptr<sdf::Param> > (__pointer=0x7f582c036210) at /usr/include/c++/4.8/bits/stl_construct.h:93
#11 __destroy<std::shared_ptr<sdf::Param>*> (__last=<optimized out>, __first=0x7f582c036210) at /usr/include/c++/4.8/bits/stl_construct.h:103
#12 _Destroy<std::shared_ptr<sdf::Param>*> (__last=<optimized out>, __first=<optimized out>) at /usr/include/c++/4.8/bits/stl_construct.h:126
#13 _Destroy<std::shared_ptr<sdf::Param>*, std::shared_ptr<sdf::Param> > (__last=0x7f582c036220, __first=<optimized out>) at /usr/include/c++/4.8/bits/stl_construct.h:151
#14 ~vector (this=0x7f582c035730, __in_chrg=<optimized out>) at /usr/include/c++/4.8/bits/stl_vector.h:415
#15 ~ElementPrivate (this=0x7f582c035700, __in_chrg=<optimized out>) at /var/lib/jenkins/workspace/sdformat4-debbuilder/build/sdformat-4.1.1/include/sdf/Element.hh:282
#16 sdf::Element::~Element (this=0x7f582c033bb0, __in_chrg=<optimized out>) at /var/lib/jenkins/workspace/sdformat4-debbuilder/build/sdformat-4.1.1/src/Element.cc:34
#17 0x00007f58b1a84a09 in sdf::Element::~Element (this=0x7f582c033bb0, __in_chrg=<optimized out>) at /var/lib/jenkins/workspace/sdformat4-debbuilder/build/sdformat-4.1.1/src/Element.cc:36
#18 0x00007f58b1a844d2 in _M_release (this=0x7f582c0357a0) at /usr/include/c++/4.8/bits/shared_ptr_base.h:144
#19 ~__shared_count (this=0x7f582c035928, __in_chrg=<optimized out>) at /usr/include/c++/4.8/bits/shared_ptr_base.h:546
#20 ~__shared_ptr (this=0x7f582c035920, __in_chrg=<optimized out>) at /usr/include/c++/4.8/bits/shared_ptr_base.h:781
#21 ~shared_ptr (this=0x7f582c035920, __in_chrg=<optimized out>) at /usr/include/c++/4.8/bits/shared_ptr.h:93
#22 _Destroy<std::shared_ptr<sdf::Element> > (__pointer=0x7f582c035920) at /usr/include/c++/4.8/bits/stl_construct.h:93
#23 __destroy<std::shared_ptr<sdf::Element>*> (__last=<optimized out>, __first=0x7f582c035920) at /usr/include/c++/4.8/bits/stl_construct.h:103
#24 _Destroy<std::shared_ptr<sdf::Element>*> (__last=<optimized out>, __first=<optimized out>) at /usr/include/c++/4.8/bits/stl_construct.h:126
#25 _Destroy<std::shared_ptr<sdf::Element>*, std::shared_ptr<sdf::Element> > (__last=0x7f582c035930, __first=<optimized out>) at /usr/include/c++/4.8/bits/stl_construct.h:151
#26 ~vector (this=0x7f582c035670, __in_chrg=<optimized out>) at /usr/include/c++/4.8/bits/stl_vector.h:415
#27 ~ElementPrivate (this=0x7f582c035600, __in_chrg=<optimized out>) at /var/lib/jenkins/workspace/sdformat4-debbuilder/build/sdformat-4.1.1/include/sdf/Element.hh:282
#28 sdf::Element::~Element (this=0x7f582c034110, __in_chrg=<optimized out>) at /var/lib/jenkins/workspace/sdformat4-debbuilder/build/sdformat-4.1.1/src/Element.cc:34
#29 0x00007f58b1a84a09 in sdf::Element::~Element (this=0x7f582c034110, __in_chrg=<optimized out>) at /var/lib/jenkins/workspace/sdformat4-debbuilder/build/sdformat-4.1.1/src/Element.cc:36
#30 0x00007f58b1a844d2 in _M_release (this=0x7f582c0356a0) at /usr/include/c++/4.8/bits/shared_ptr_base.h:144
#31 ~__shared_count (this=0x7f582c034d38, __in_chrg=<optimized out>) at /usr/include/c++/4.8/bits/shared_ptr_base.h:546
#32 ~__shared_ptr (this=0x7f582c034d30, __in_chrg=<optimized out>) at /usr/include/c++/4.8/bits/shared_ptr_base.h:781
#33 ~shared_ptr (this=0x7f582c034d30, __in_chrg=<optimized out>) at /usr/include/c++/4.8/bits/shared_ptr.h:93
#34 _Destroy<std::shared_ptr<sdf::Element> > (__pointer=0x7f582c034d30) at /usr/include/c++/4.8/bits/stl_construct.h:93
#35 __destroy<std::shared_ptr<sdf::Element>*> (__last=<optimized out>, __first=0x7f582c034d30) at /usr/include/c++/4.8/bits/stl_construct.h:103
#36 _Destroy<std::shared_ptr<sdf::Element>*> (__last=<optimized out>, __first=<optimized out>) at /usr/include/c++/4.8/bits/stl_construct.h:126
#37 _Destroy<std::shared_ptr<sdf::Element>*, std::shared_ptr<sdf::Element> > (__last=0x7f582c034d50, __first=<optimized out>) at /usr/include/c++/4.8/bits/stl_construct.h:151
#38 ~vector (this=0x7f582c033ad0, __in_chrg=<optimized out>) at /usr/include/c++/4.8/bits/stl_vector.h:415
#39 ~ElementPrivate (this=0x7f582c033a60, __in_chrg=<optimized out>) at /var/lib/jenkins/workspace/sdformat4-debbuilder/build/sdformat-4.1.1/include/sdf/Element.hh:282
#40 sdf::Element::~Element (this=0x7f582c0338e0, __in_chrg=<optimized out>) at /var/lib/jenkins/workspace/sdformat4-debbuilder/build/sdformat-4.1.1/src/Element.cc:34
#41 0x00007f58b1a84a09 in sdf::Element::~Element (this=0x7f582c0338e0, __in_chrg=<optimized out>) at /var/lib/jenkins/workspace/sdformat4-debbuilder/build/sdformat-4.1.1/src/Element.cc:36
#42 0x00007f58b1a844d2 in _M_release (this=0x7f582c034f90) at /usr/include/c++/4.8/bits/shared_ptr_base.h:144
#43 ~__shared_count (this=0x7f582c062078, __in_chrg=<optimized out>) at /usr/include/c++/4.8/bits/shared_ptr_base.h:546
#44 ~__shared_ptr (this=0x7f582c062070, __in_chrg=<optimized out>) at /usr/include/c++/4.8/bits/shared_ptr_base.h:781
#45 ~shared_ptr (this=0x7f582c062070, __in_chrg=<optimized out>) at /usr/include/c++/4.8/bits/shared_ptr.h:93
#46 _Destroy<std::shared_ptr<sdf::Element> > (__pointer=0x7f582c062070) at /usr/include/c++/4.8/bits/stl_construct.h:93
#47 __destroy<std::shared_ptr<sdf::Element>*> (__last=<optimized out>, __first=0x7f582c062070) at /usr/include/c++/4.8/bits/stl_construct.h:103
#48 _Destroy<std::shared_ptr<sdf::Element>*> (__last=<optimized out>, __first=<optimized out>) at /usr/include/c++/4.8/bits/stl_construct.h:126
#49 _Destroy<std::shared_ptr<sdf::Element>*, std::shared_ptr<sdf::Element> > (__last=0x7f582c062140, __first=<optimized out>) at /usr/include/c++/4.8/bits/stl_construct.h:151
#50 ~vector (this=0x7f582c02d660, __in_chrg=<optimized out>) at /usr/include/c++/4.8/bits/stl_vector.h:415
#51 ~ElementPrivate (this=0x7f582c02d5f0, __in_chrg=<optimized out>) at /var/lib/jenkins/workspace/sdformat4-debbuilder/build/sdformat-4.1.1/include/sdf/Element.hh:282
#52 sdf::Element::~Element (this=0x7f582c02c5c0, __in_chrg=<optimized out>) at /var/lib/jenkins/workspace/sdformat4-debbuilder/build/sdformat-4.1.1/src/Element.cc:34
#53 0x00007f58b1a84a09 in sdf::Element::~Element (this=0x7f582c02c5c0, __in_chrg=<optimized out>) at /var/lib/jenkins/workspace/sdformat4-debbuilder/build/sdformat-4.1.1/src/Element.cc:36
#54 0x00007f58b1a844d2 in _M_release (this=0x7f582c02c520) at /usr/include/c++/4.8/bits/shared_ptr_base.h:144
#55 ~__shared_count (this=0x7f582c02dc48, __in_chrg=<optimized out>) at /usr/include/c++/4.8/bits/shared_ptr_base.h:546
#56 ~__shared_ptr (this=0x7f582c02dc40, __in_chrg=<optimized out>) at /usr/include/c++/4.8/bits/shared_ptr_base.h:781
#57 ~shared_ptr (this=0x7f582c02dc40, __in_chrg=<optimized out>) at /usr/include/c++/4.8/bits/shared_ptr.h:93
#58 _Destroy<std::shared_ptr<sdf::Element> > (__pointer=0x7f582c02dc40) at /usr/include/c++/4.8/bits/stl_construct.h:93
#59 __destroy<std::shared_ptr<sdf::Element>*> (__last=<optimized out>, __first=0x7f582c02dc40) at /usr/include/c++/4.8/bits/stl_construct.h:103
#60 _Destroy<std::shared_ptr<sdf::Element>*> (__last=<optimized out>, __first=<optimized out>) at /usr/include/c++/4.8/bits/stl_construct.h:126
#61 _Destroy<std::shared_ptr<sdf::Element>*, std::shared_ptr<sdf::Element> > (__last=0x7f582c02dc50, __first=<optimized out>) at /usr/include/c++/4.8/bits/stl_construct.h:151
#62 ~vector (this=0x7f582c00a400, __in_chrg=<optimized out>) at /usr/include/c++/4.8/bits/stl_vector.h:415
#63 ~ElementPrivate (this=0x7f582c00a390, __in_chrg=<optimized out>) at /var/lib/jenkins/workspace/sdformat4-debbuilder/build/sdformat-4.1.1/include/sdf/Element.hh:282
#64 sdf::Element::~Element (this=0x7f582c009dc0, __in_chrg=<optimized out>) at /var/lib/jenkins/workspace/sdformat4-debbuilder/build/sdformat-4.1.1/src/Element.cc:34
#65 0x00007f58b1a84a09 in sdf::Element::~Element (this=0x7f582c009dc0, __in_chrg=<optimized out>) at /var/lib/jenkins/workspace/sdformat4-debbuilder/build/sdformat-4.1.1/src/Element.cc:36
#66 0x00007f58b476b8f9 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x7f582c009cb0) at /usr/include/c++/4.8/bits/shared_ptr_base.h:144
#67 0x00007f58b2e40962 in ~__shared_count (this=<optimized out>, __in_chrg=<optimized out>) at /usr/include/c++/4.8/bits/shared_ptr_base.h:546
#68 ~__shared_ptr (this=<optimized out>, __in_chrg=<optimized out>) at /usr/include/c++/4.8/bits/shared_ptr_base.h:781
#69 reset (this=0x7f58b31cd420 <gazebo::physics::Joint::sdfJoint>) at /usr/include/c++/4.8/bits/shared_ptr_base.h:882
#70 gazebo::physics::Joint::Fini (this=this@entry=0x7f582c009e90) at /var/lib/jenkins/workspace/gazebo7-debbuilder/build/gazebo-7.3.1/gazebo/physics/Joint.cc:426
#71 0x00007f58b2d8e658 in gazebo::physics::ODEJoint::Fini (this=this@entry=0x7f582c009e90) at /var/lib/jenkins/workspace/gazebo7-debbuilder/build/gazebo-7.3.1/gazebo/physics/ode/ODEJoint.cc:76
#72 0x00007f58b2d8e677 in gazebo::physics::ODEJoint::~ODEJoint (this=0x7f582c009e90, __in_chrg=<optimized out>) at /var/lib/jenkins/workspace/gazebo7-debbuilder/build/gazebo-7.3.1/gazebo/physics/ode/ODEJoint.cc:58
#73 0x00007f58b2db68d9 in gazebo::physics::ODESliderJoint::~ODESliderJoint (this=0x7f582c009e90, __in_chrg=<optimized out>) at /var/lib/jenkins/workspace/gazebo7-debbuilder/build/gazebo-7.3.1/gazebo/physics/ode/ODESliderJoint.cc:46
#74 0x00007f584d9ea0ae in boost::detail::sp_counted_base::release (this=0x7f582c0461c0) at /usr/include/boost/smart_ptr/detail/sp_counted_base_gcc_x86.hpp:146
#75 0x00007f584d9e5a9a in ~shared_count (this=<optimized out>, __in_chrg=<optimized out>) at /usr/include/boost/smart_ptr/detail/shared_count.hpp:371
#76 ~shared_ptr (this=<optimized out>, __in_chrg=<optimized out>) at /usr/include/boost/smart_ptr/shared_ptr.hpp:328
#77 reset (this=<optimized out>) at /usr/include/boost/smart_ptr/shared_ptr.hpp:619
#78 gazebo::HarnessPlugin::Detach (this=0x7f582c008560) at /var/lib/jenkins/workspace/gazebo7-debbuilder/build/gazebo-7.3.1/plugins/HarnessPlugin.cc:273
#79 0x00007f584d9ec125 in operator() (a0=..., this=0x7f582c009ac0) at /usr/include/boost/function/function_template.hpp:767
#80 gazebo::transport::CallbackHelperT<gazebo::msgs::GzString>::HandleData(std::string const&, boost::function<void (unsigned int)>, unsigned int) (this=0x7f582c009a80, _newdata="\n\004true", _cb=..., _id=0) at /var/lib/jenkins/workspace/gazebo7-debbuilder/build/gazebo-7.3.1/gazebo/transport/CallbackHelper.hh:149
#81 0x00007f58b320f8c6 in gazebo::transport::Node::ProcessIncoming (this=0x7f582c0086e0) at /var/lib/jenkins/workspace/gazebo7-debbuilder/build/gazebo-7.3.1/gazebo/transport/Node.cc:223
#82 0x00007f58b321904b in gazebo::transport::TopicManager::ProcessNodes (this=0x7f58b498bfa0 <SingletonT<gazebo::transport::TopicManager>::GetInstance()::t>, _onlyOut=_onlyOut@entry=false) at /var/lib/jenkins/workspace/gazebo7-debbuilder/build/gazebo-7.3.1/gazebo/transport/TopicManager.cc:197
#83 0x00007f58b32087b6 in gazebo::transport::ConnectionManager::RunUpdate (this=this@entry=0x7f58b498bda0 <SingletonT<gazebo::transport::ConnectionManager>::GetInstance()::t>) at /var/lib/jenkins/workspace/gazebo7-debbuilder/build/gazebo-7.3.1/gazebo/transport/ConnectionManager.cc:283
#84 0x00007f58b3208955 in gazebo::transport::ConnectionManager::Run (this=0x7f58b498bda0 <SingletonT<gazebo::transport::ConnectionManager>::GetInstance()::t>) at /var/lib/jenkins/workspace/gazebo7-debbuilder/build/gazebo-7.3.1/gazebo/transport/ConnectionManager.cc:314
#85 0x00007f58b0ea6a4a in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
#86 0x00007f58b2181184 in start_thread (arg=0x7f5890b4c700) at pthread_create.c:312
#87 0x00007f58b353a37d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

gazebosim / gazebo-classic

Flaky seg-fault when detaching HarnessPlugin #2034