Segfault while running simulations on sourcing dome built from source

[mihirk284]

I am trying to run the simulation using dome built from source, however, it has a segfault and prints out a stack trace.

With the command ign launch -v 4 competition.ign circuit:=cave worldName:=cave_circuit_practice_01 robotName1:=X1 robotConfig1:=X1_SENSOR_CONFIG_1

I get the following segfault when I build the subt repo after sourcing the dome build from source. I am not sure why this is happening. On building the subt repo after sourcing the binary, this error does not show up.

The segfault message is as follows:-

Stack trace (most recent call last) in thread 10049[GUI] [Msg] Loading config [/home/arl/.ignition/launch/gui.config]
:
[GUI] [Dbg] [Application.cc:266] Loading window config
[GUI] [Dbg] [Application.cc:421] Applying config
#[Dbg] [Manager.cc:839] Loading plugin. Name[subt::CommsBrokerPlugin] File[libCommsBrokerPlugin.so]
14   Object "", at 0xffffffffffffffff, in 
[Dbg] [CommsBrokerPlugin.cc:66] Loading visibility_config from SDF: 
RF Configuration (visibility-based)
-- visibilityCostToFadingExponent: 0.2
-- commsCostMax: 10

[Dbg] [CommsBrokerPlugin.cc:98] Loading range_config from SDF: 
RF Configuration (range-based)
-- max_range: 500
-- fading_exponent: 1.5
-- L0: 40
-- sigma: 10
-- scaling_factor: 0.55
-- range_per_hop: 2

[Dbg] [CommsBrokerPlugin.cc:123] Loading radio_config from SDF: 
Radio Configuration
-- capacity: 1e+06
-- default_tx_power: 20
-- noise_floor: -90
-- modulation: QPSK

[GUI] [Dbg] [Application.cc:305] Loading plugin [GzScene3D]
#13   Source "/build/glibc-S9d2JN/glibc-2.27/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S", line 95, in  [0x7fb99ac2f71e]
#12   Source "/build/glibc-S9d2JN/glibc-2.27/nptl/pthread_create.c", line 463, in start_thread [0x7fb99a8f66da]
#11   Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", at 0x7fb997603d7f, in std::error_code::default_error_condition() const
[GUI] [Msg] Video recorder stats topic advertised on [/gui/record_video/stats]
[GUI] [Msg] Transform mode service on [/gui/transform_mode]
[GUI] [Msg] Record video service on [/gui/record_video]
[GUI] [Msg] Move to service on [/gui/move_to]
[GUI] [Msg] Follow service on [/gui/follow]
[GUI] [Msg] View angle service on [/gui/view_angle]
[GUI] [Msg] Move to pose service on [/gui/move_to/pose]
[GUI] [Msg] Camera pose topic advertised on [/gui/camera/pose]
[GUI] [Msg] View collisions service on [/gui/view/collisions]
#10   Source "/home/arl/dome_dev/src/ign-gazebo/src/ServerPrivate.cc", line 165, in ignition::gazebo::v4::ServerPrivate::Run(unsigned long, std::optional<std::condition_variable*>) [0x7fb99617d9d9]
        162:   // simulation runner, and we can avoid using the thread pool.
        163:   if (this->simRunners.size() == 1)
        164:   {
      > 165:     result = this->simRunners[0]->Run(_iterations);
        166:   }
        167:   else
        168:   {
[GUI] [Msg] Added plugin [3D View] to main window
[GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM
[GUI] [Msg] Loaded plugin [GzScene3D] from path [/home/arl/dome_dev/install/lib/ign-gazebo-4/plugins/gui/libGzScene3D.so]
[GUI] [Dbg] [Application.cc:305] Loading plugin [WorldControl]
#9    Source "/home/arl/dome_dev/src/ign-gazebo/src/SimulationRunner.cc", line 740, in ignition::gazebo::v4::SimulationRunner::Run(unsigned long) [0x7fb99618858a]
        737:     }
        738:     else
        739:     {
      > 740:       this->Step(this->currentInfo);
        741:     }
        742: 
        743:     // Handle Server::RunOnce(false) in which a single paused run is executed
#8    Source "/home/arl/dome_dev/src/ign-gazebo/src/SimulationRunner.cc", line 775, in ignition::gazebo::v4::SimulationRunner::Step(ignition::gazebo::v4::UpdateInfo const&) [0x7fb996187e86]
        772:   this->ProcessSystemQueue();
        773: 
        774:   // Update all the systems.
      > 775:   this->UpdateSystems();
        776: 
        777:   if (!this->Paused() && this->pendingSimIterations > 0)
        778:   {
#7    Source "/home/arl/dome_dev/src/ign-gazebo/src/SimulationRunner.cc", line 526, in ignition::gazebo::v4::SimulationRunner::UpdateSystems() [0x7fb996182461]
        523:   {
        524:     IGN_PROFILE("Update");
        525:     for (auto& system : this->systemsUpdate)
      > 526:       system->Update(this->currentInfo, this->entityCompMgr);
        527:   }
        528: 
        529:   {
[GUI] [Msg] Using world control service [/world/cave_circuit_practice_01/control]
[GUI] [Msg] Listening to stats on [/world/cave_circuit_practice_01/stats]
[GUI] [Msg] Added plugin [World control] to main window
[GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM
[GUI] [Msg] Loaded plugin [WorldControl] from path [/home/arl/dome_dev/install/lib/ign-gui-4/plugins/libWorldControl.so]
[GUI] [Dbg] [Application.cc:305] Loading plugin [WorldStats]
[GUI] [Msg] Listening to stats on [/world/cave_circuit_practice_01/stats]
[GUI] [Msg] Added plugin [World stats] to main window
[GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM
[GUI] [Msg] Loaded plugin [WorldStats] from path [/home/arl/dome_dev/install/lib/ign-gui-4/plugins/libWorldStats.so]
[GUI] [Dbg] [GazeboGui.cc:144] Running the GazeboGui plugin.
#6    Source "/home/arl/dome_dev/src/ign-gazebo/src/systems/physics/Physics.cc", line 645, in ignition::gazebo::v4::systems::Physics::Update(ignition::gazebo::v4::UpdateInfo const&, ignition::gazebo::v4::EntityComponentManager&) [0x7fb974517297]
        643:   if (this->dataPtr->engine)
        644:   {
      > 645:     this->dataPtr->CreatePhysicsEntities(_ecm);
        646:     this->dataPtr->UpdatePhysics(_ecm);
        647:     // Only step if not paused.
        648:     if (!_info.paused)
#5    Source "/home/arl/dome_dev/src/ign-gazebo/src/systems/physics/Physics.cc", line 864, in ignition::gazebo::v4::systems::PhysicsPrivate::CreatePhysicsEntities(ignition::gazebo::v4::EntityComponentManager const&) [0x7fb974514a77]
        861:   // collisions
        862:   _ecm.EachNew<components::Collision, components::Name, components::Pose,
        863:             components::Geometry, components::CollisionElement,
      > 864:             components::ParentEntity>(
        865:       [&](const Entity &_entity,
        866:           const components::Collision *,
        867:           const components::Name *_name,
#4  | Source "/home/arl/dome_dev/src/ign-gazebo/include/ignition/gazebo/detail/EntityComponentManager.hh", line 425, in void ignition::gazebo::v4::EntityComponentManager::EachNew<ignition::gazebo::v4::components::Component<std::add_lvalue_reference<void>, ignition::gazebo::v4::components::CollisionTag, ignition::gazebo::v4::serializers::DefaultSerializer<std::add_lvalue_reference<void> > >, ignition::gazebo::v4::components::Component<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ignition::gazebo::v4::components::NameTag, ignition::gazebo::v4::serializers::StringSerializer>, ignition::gazebo::v4::components::Component<ignition::math::v6::Pose3<double>, ignition::gazebo::v4::components::PoseTag, ignition::gazebo::v4::serializers::DefaultSerializer<ignition::math::v6::Pose3<double> > >, ignition::gazebo::v4::components::Component<sdf::v10::Geometry, ignition::gazebo::v4::components::GeometryTag, ignition::gazebo::v4::serializers::ComponentToMsgSerializer<sdf::v10::Geometry, ignition::msgs::Geometry> >, ignition::gazebo::v4::components::Component<sdf::v10::Collision, ignition::gazebo::v4::components::CollisionElementTag, ignition::gazebo::v4::serializers::ComponentToMsgSerializer<sdf::v10::Collision, ignition::msgs::Collision> >, ignition::gazebo::v4::components::Component<unsigned long, ignition::gazebo::v4::components::ParentEntityTag, ignition::gazebo::v4::serializers::DefaultSerializer<unsigned long> > >(ignition::gazebo::v4::EntityComponentManager::identity<std::function<bool (unsigned long const&, ignition::gazebo::v4::components::Component<std::add_lvalue_reference<void>, ignition::gazebo::v4::components::CollisionTag, ignition::gazebo::v4::serializers::DefaultSerializer<std::add_lvalue_reference<void> > > const*, ignition::gazebo::v4::components::Component<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ignition::gazebo::v4::components::NameTag, ignition::gazebo::v4::serializers::StringSerializer> const*, ignition::gazebo::v4::components::Component<ignition::math::v6::Pose3<double>, ignition::gazebo::v4::components::PoseTag, ignition::gazebo::v4::serializers::DefaultSerializer<ignition::math::v6::Pose3<double> > > const*, ignition::gazebo::v4::components::Component<sdf::v10::Geometry, ignition::gazebo::v4::components::GeometryTag, ignition::gazebo::v4::serializers::ComponentToMsgSerializer<sdf::v10::Geometry, ignition::msgs::Geometry> > const*, ignition::gazebo::v4::components::Component<sdf::v10::Collision, ignition::gazebo::v4::components::CollisionElementTag, ignition::gazebo::v4::serializers::ComponentToMsgSerializer<sdf::v10::Collision, ignition::msgs::Collision> > const*, ignition::gazebo::v4::components::Component<unsigned long, ignition::gazebo::v4::components::ParentEntityTag, ignition::gazebo::v4::serializers::DefaultSerializer<unsigned long> > const*)> >::type) const
    |   423:   for (const Entity entity : view.newEntities)
    |   424:   {
    | > 425:     if (!_f(entity, view.Component<ComponentTypeTs>(entity, this)...))
    |   426:     {
    |   427:       break;
      Source "/usr/include/c++/8/bits/std_function.h", line 687, in void ignition::gazebo::v4::EntityComponentManager::EachNew<ignition::gazebo::v4::components::Component<std::add_lvalue_reference<void>, ignition::gazebo::v4::components::CollisionTag, ignition::gazebo::v4::serializers::DefaultSerializer<std::add_lvalue_reference<void> > >, ignition::gazebo::v4::components::Component<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ignition::gazebo::v4::components::NameTag, ignition::gazebo::v4::serializers::StringSerializer>, ignition::gazebo::v4::components::Component<ignition::math::v6::Pose3<double>, ignition::gazebo::v4::components::PoseTag, ignition::gazebo::v4::serializers::DefaultSerializer<ignition::math::v6::Pose3<double> > >, ignition::gazebo::v4::components::Component<sdf::v10::Geometry, ignition::gazebo::v4::components::GeometryTag, ignition::gazebo::v4::serializers::ComponentToMsgSerializer<sdf::v10::Geometry, ignition::msgs::Geometry> >, ignition::gazebo::v4::components::Component<sdf::v10::Collision, ignition::gazebo::v4::components::CollisionElementTag, ignition::gazebo::v4::serializers::ComponentToMsgSerializer<sdf::v10::Collision, ignition::msgs::Collision> >, ignition::gazebo::v4::components::Component<unsigned long, ignition::gazebo::v4::components::ParentEntityTag, ignition::gazebo::v4::serializers::DefaultSerializer<unsigned long> > >(ignition::gazebo::v4::EntityComponentManager::identity<std::function<bool (unsigned long const&, ignition::gazebo::v4::components::Component<std::add_lvalue_reference<void>, ignition::gazebo::v4::components::CollisionTag, ignition::gazebo::v4::serializers::DefaultSerializer<std::add_lvalue_reference<void> > > const*, ignition::gazebo::v4::components::Component<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ignition::gazebo::v4::components::NameTag, ignition::gazebo::v4::serializers::StringSerializer> const*, ignition::gazebo::v4::components::Component<ignition::math::v6::Pose3<double>, ignition::gazebo::v4::components::PoseTag, ignition::gazebo::v4::serializers::DefaultSerializer<ignition::math::v6::Pose3<double> > > const*, ignition::gazebo::v4::components::Component<sdf::v10::Geometry, ignition::gazebo::v4::components::GeometryTag, ignition::gazebo::v4::serializers::ComponentToMsgSerializer<sdf::v10::Geometry, ignition::msgs::Geometry> > const*, ignition::gazebo::v4::components::Component<sdf::v10::Collision, ignition::gazebo::v4::components::CollisionElementTag, ignition::gazebo::v4::serializers::ComponentToMsgSerializer<sdf::v10::Collision, ignition::msgs::Collision> > const*, ignition::gazebo::v4::components::Component<unsigned long, ignition::gazebo::v4::components::ParentEntityTag, ignition::gazebo::v4::serializers::DefaultSerializer<unsigned long> > const*)> >::type) const [0x7fb97455e909]
        684:     {
        685:       if (_M_empty())
        686:    __throw_bad_function_call();
      > 687:       return _M_invoker(_M_functor, std::forward<_ArgTypes>(__args)...);
        688:     }
        689: 
        690: #if __cpp_rtti
#3    Source "/home/arl/dome_dev/src/ign-gazebo/src/systems/physics/Physics.cc", line 912, in ignition::gazebo::v4::systems::PhysicsPrivate::CreatePhysicsEntities(ignition::gazebo::v4::EntityComponentManager const&)::{lambda(unsigned long const&, ignition::gazebo::v4::components::Component<std::add_lvalue_reference<void>, ignition::gazebo::v4::components::CollisionTag, ignition::gazebo::v4::serializers::DefaultSerializer<std::add_lvalue_reference<void> > > const*, ignition::gazebo::v4::components::Component<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ignition::gazebo::v4::components::NameTag, ignition::gazebo::v4::serializers::StringSerializer> const*, ignition::gazebo::v4::components::Component<ignition::math::v6::Pose3<double>, ignition::gazebo::v4::components::PoseTag, ignition::gazebo::v4::serializers::DefaultSerializer<ignition::math::v6::Pose3<double> > > const*, ignition::gazebo::v4::components::Component<sdf::v10::Geometry, ignition::gazebo::v4::components::GeometryTag, ignition::gazebo::v4::serializers::ComponentToMsgSerializer<sdf::v10::Geometry, ignition::msgs::Geometry> > const*, ignition::gazebo::v4::components::Component<sdf::v10::Collision, ignition::gazebo::v4::components::CollisionElementTag, ignition::gazebo::v4::serializers::ComponentToMsgSerializer<sdf::v10::Collision, ignition::msgs::Collision> > const*, ignition::gazebo::v4::components::Component<unsigned long, ignition::gazebo::v4::components::ParentEntityTag, ignition::gazebo::v4::serializers::DefaultSerializer<unsigned long> > const*)#4}::operator()(unsigned long const&, ignition::gazebo::v4::components::Component<std::add_lvalue_reference<void>, ignition::gazebo::v4::components::CollisionTag, ignition::gazebo::v4::serializers::DefaultSerializer<std::add_lvalue_reference<void> > > const*, ignition::gazebo::v4::components::Component<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ignition::gazebo::v4::components::NameTag, ignition::gazebo::v4::serializers::StringSerializer> const*, ignition::gazebo::v4::components::Component<ignition::math::v6::Pose3<double>, ignition::gazebo::v4::components::PoseTag, ignition::gazebo::v4::serializers::DefaultSerializer<ignition::math::v6::Pose3<double> > > const*, ignition::gazebo::v4::components::Component<sdf::v10::Geometry, ignition::gazebo::v4::components::GeometryTag, ignition::gazebo::v4::serializers::ComponentToMsgSerializer<sdf::v10::Geometry, ignition::msgs::Geometry> > const*, ignition::gazebo::v4::components::Component<sdf::v10::Collision, ignition::gazebo::v4::components::CollisionElementTag, ignition::gazebo::v4::serializers::ComponentToMsgSerializer<sdf::v10::Collision, ignition::msgs::Collision> > const*, ignition::gazebo::v4::components::Component<unsigned long, ignition::gazebo::v4::components::ParentEntityTag, ignition::gazebo::v4::serializers::DefaultSerializer<unsigned long> > const*) const [clone .isra.3592] [0x7fb974515bd5]
        909:           }
        910: 
        911:           auto &meshManager = *ignition::common::MeshManager::Instance();
      > 912:           auto fullPath = asFullPath(meshSdf->Uri(), meshSdf->FilePath());
        913:           auto *mesh = meshManager.Load(fullPath);
        914:           if (nullptr == mesh)
        915:           {
#2  | Source "/home/arl/dome_dev/src/ign-gazebo/src/Util.cc", line 267, in ignition::gazebo::v4::asFullPath(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
    |   265:   #endif
    |   266:   if (_uri.find("://") != std::string::npos ||
    | > 267:       !path(_uri).is_relative())
    |   268:   {
    |   269:     return _uri;
    | Source "/usr/include/c++/8/bits/fs_path.h", line 209, in 
    |   207:       { _M_split_cmpts(); }
    |   208: 
    | > 209:     ~path() = default;
    |   210: 
    |   211:     // assignments
    | Source "/usr/include/c++/8/bits/stl_vector.h", line 567, in 
    |   565:       ~vector() _GLIBCXX_NOEXCEPT
    |   566:       {
    | > 567:    std::_Destroy(this->_M_impl._M_start, this->_M_impl._M_finish,
    |   568:              _M_get_Tp_allocator());
    |   569:    _GLIBCXX_ASAN_ANNOTATE_BEFORE_DEALLOC;
    | Source "/usr/include/c++/8/bits/stl_construct.h", line 206, in 
    |   204:         allocator<_Tp>&)
    |   205:     {
    | > 206:       _Destroy(__first, __last);
    |   207:     }
    | Source "/usr/include/c++/8/bits/stl_construct.h", line 137, in 
    |   135: #endif
    |   136:       std::_Destroy_aux<__has_trivial_destructor(_Value_type)>::
    | > 137:    __destroy(__first, __last);
    |   138:     }
    | Source "/usr/include/c++/8/bits/stl_construct.h", line 108, in 
    |   106:    {
    |   107:      for (; __first != __last; ++__first)
    | > 108:        std::_Destroy(std::__addressof(*__first));
    |   109:    }
    |   110:     };
    | Source "/usr/include/c++/8/bits/stl_construct.h", line 98, in 
    |    96:     inline void
    |    97:     _Destroy(_Tp* __pointer)
    | >  98:     { __pointer->~_Tp(); }
    |    99: 
    |   100:   template<bool>
    | Source "/usr/include/c++/8/bits/fs_path.h", line 644, in 
    |   642:   };
    |   643: 
    | > 644:   struct path::_Cmpt : path
    |   645:   {
    |   646:     _Cmpt(string_type __s, _Type __t, size_t __pos)
      Source "/usr/include/c++/8/bits/fs_path.h", line 209, in ignition::gazebo::v4::asFullPath(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) [0x7fb996195f78]
        206:       : _M_pathname(_S_convert_loc(__first, __last, __loc))
        207:       { _M_split_cmpts(); }
        208: 
      > 209:     ~path() = default;
        210: 
        211:     // assignments
#1  | Source "/usr/include/c++/8/bits/stl_vector.h", line 567, in std::vector<std::filesystem::__cxx11::path::_Cmpt, std::allocator<std::filesystem::__cxx11::path::_Cmpt> >::~vector()
    |   565:       ~vector() _GLIBCXX_NOEXCEPT
    |   566:       {
    | > 567:    std::_Destroy(this->_M_impl._M_start, this->_M_impl._M_finish,
    |   568:              _M_get_Tp_allocator());
    |   569:    _GLIBCXX_ASAN_ANNOTATE_BEFORE_DEALLOC;
    | Source "/usr/include/c++/8/bits/stl_construct.h", line 206, in 
    |   204:         allocator<_Tp>&)
    |   205:     {
    | > 206:       _Destroy(__first, __last);
    |   207:     }
    | Source "/usr/include/c++/8/bits/stl_construct.h", line 137, in 
    |   135: #endif
    |   136:       std::_Destroy_aux<__has_trivial_destructor(_Value_type)>::
    | > 137:    __destroy(__first, __last);
    |   138:     }
    | Source "/usr/include/c++/8/bits/stl_construct.h", line 108, in 
    |   106:    {
    |   107:      for (; __first != __last; ++__first)
    | > 108:        std::_Destroy(std::__addressof(*__first));
    |   109:    }
    |   110:     };
    | Source "/usr/include/c++/8/bits/stl_construct.h", line 98, in 
    |    96:     inline void
    |    97:     _Destroy(_Tp* __pointer)
    | >  98:     { __pointer->~_Tp(); }
    |    99: 
    |   100:   template<bool>
    | Source "/usr/include/c++/8/bits/fs_path.h", line 644, in 
    |   642:   };
    |   643: 
    | > 644:   struct path::_Cmpt : path
    |   645:   {
    |   646:     _Cmpt(string_type __s, _Type __t, size_t __pos)
      Source "/usr/include/c++/8/bits/fs_path.h", line 209, in std::vector<std::filesystem::__cxx11::path::_Cmpt, std::allocator<std::filesystem::__cxx11::path::_Cmpt> >::~vector() [0x7fb996196ba0]
        206:       : _M_pathname(_S_convert_loc(__first, __last, __loc))
        207:       { _M_split_cmpts(); }
        208: 
      > 209:     ~path() = default;
        210: 
        211:     // assignments
#0    Source "/usr/include/c++/8/bits/stl_vector.h", line 567, in std::vector<std::filesystem::__cxx11::path::_Cmpt, std::allocator<std::filesystem::__cxx11::path::_Cmpt> >::~vector() [0x7fb996196b84]
        564:        */
        565:       ~vector() _GLIBCXX_NOEXCEPT
        566:       {
      > 567:    std::_Destroy(this->_M_impl._M_start, this->_M_impl._M_finish,
        568:              _M_get_Tp_allocator());
        569:    _GLIBCXX_ASAN_ANNOTATE_BEFORE_DEALLOC;
        570:       }
Segmentation fault (Signal sent by the kernel [(nil)])

[1]    9664 segmentation fault (core dumped)  ign launch -v 4 competition.ign circuit:=cave  robotName1:=X1

[peci1]

You should probably document how you setup the sources of Dome. Mainly, what versions you are using, and how do you "instruct" them to override the system-installed libraries (if you have them). There are cross-repo dependencies for some new features, and you have to have all ignition libraries in correct versions/commits for them to work together. I'd suggest listing the versions of all libraries at a given time on the Ubuntu APT repo, and locking your local checkouts to tags corresponding to these versions.

Also, is this stack trace from a Debug build or from a RelWithDebInfo of Dome?

[mihirk284]

Hi @peci1, Sorry for the delayed response.

You should probably document how you setup the sources of Dome. Mainly, what versions you are using, and how do you "instruct" them to override the system-installed libraries (if you have them).

I followed the wiki instructions for setting up Dome. I installed Dome from source referring to this link, meanwhile parallelly, I installed dome binaries from this link.

On cloning the osrf/subt repo, sourced Dome using source ./install/setup.zsh from inside the dome_ws folder. This works, and on entering which ign, I get the path as /home/mihir/dome/install/bin/ign, which is expected.

After this, when the subt_ws is built using catkin_make install and sourced using source ./install/setup.zsh (from inside the SubT workspace folder). On doing this, I get the exception as detailed above.

Alternatively, when the dome binaries from this link are used (verified using which ign -> /usr/bin/ign), and the subt_ws is built using catkin_make install similarly as above, this error does not appear.

I have tried this on two different machines, and both are having a similar error.

There are cross-repo dependencies for some new features, and you have to have all ignition libraries in correct versions/commits for them to work together. I'd suggest listing the versions of all libraries at a given time on the Ubuntu APT repo, and locking your local checkouts to tags corresponding to these versions.

I checked with this:-

I think the tags match. I am not sure how to check for commits or for the tags or versions of the packages such as ign-rendering that are not exposed to check for versions.

Also, is this stack trace from a Debug build or from a RelWithDebInfo of Dome?

This was printed as the code exited due to errors on using ign launch -v 4. I did not do anything specific or different for this.

[peci1]

And can you run at least some normal non-subt worlds? like ign gazebo -v4 sensors_demo.sdf?

Also, if you have a source build, it should be easy to modify the source code and add some debug prints to know why does the simulator crash. Especially, it would be useful to know what are the values used in

"/home/arl/dome_dev/src/ign-gazebo/src/systems/physics/Physics.cc", line 912

Because this is probably the place where the crash somehow begins (loading a collision element). Even better would be attaching a debugger and stepping the code, but I don't know about any easy way to attach a debugger to ign gazebo. Maybe somebody from Open Robotics would know?

One quick thing you can try is calling export LC_ALL=C before launching the simulator. There are various locale-induced issues in Ignition and this should at least rule out this kind of problems.

[mihirk284]

Update:-

On updating the source to the following versions (all of these are updated with the binary that I updated today):-

ign msg --versions
6.4.0
ign fuel --versions
5.1.1
ign topic --versions
9.1.0
ign service --versions
9.1.0
ign gazebo --versions
4.6.0
ign log --versions
9.1.0
ign launch --versions
3.1.1
ign gui --versions
4.2.0
ign sdf --versions
8.9.1
10.3.0

I get a different segfault error. However, this is similar in nature and occurs with respect to std::vectors (same as above). It would be great if anyone could point out why this might be happening and how I could go about solving this.

The simulation runs without any errors when I build the subt_ws when I have the Dome binaries sourced.

The errors message is as follows:-

[libprotobuf ERROR google/protobuf/descriptor_database.cc:57] File already exists in database: ignition/msgs/time.proto
[libprotobuf FATAL google/protobuf/descriptor.cc:1164] CHECK failed: generated_database_->Add(encoded_file_descriptor, size): 
terminate called after throwing an instance of 'google::protobuf::FatalException'
  what():  CHECK failed: generated_database_->Add(encoded_file_descriptor, size): 
Stack trace (most recent call last):
#31   Object "/home/mihir/dome/install/lib/ign-launch-3/plugins/libignition-launch-gazebo.so", at 0x7f4592b2c57c, in ignition::launch::GazeboServer::Load(tinyxml2::XMLElement const*)
#30   Source "/home/mihir/dome/src/ign-gazebo/src/Server.cc", line 191, in ignition::gazebo::v4::Server::Server(ignition::gazebo::v4::ServerConfig const&) [0x7f45927f6b6b]
        188:     this->dataPtr->AddRecordPlugin(_config);
        189:   }
        190: 
      > 191:   this->dataPtr->CreateEntities();
        192: 
        193:   // Set the desired update period, this will override the desired RTF given in
        194:   // the world file which was parsed by CreateEntities.
#29 | Source "/home/mihir/dome/src/ign-gazebo/src/ServerPrivate.cc", line 313, in ignition::gazebo::v4::ServerPrivate::CreateEntities()
    |   311:     }
    |   312:     auto runner = std::make_unique<SimulationRunner>(
    | > 313:         world, this->systemLoader, this->config);
    |   314:     runner->SetFuelUriMap(this->fuelUriMap);
    |   315:     this->simRunners.push_back(std::move(runner));
      Source "/usr/include/c++/8/bits/unique_ptr.h", line 835, in ignition::gazebo::v4::ServerPrivate::CreateEntities() [0x7f45927fc826]
        832:   template<typename _Tp, typename... _Args>
        833:     inline typename _MakeUniq<_Tp>::__single_object
        834:     make_unique(_Args&&... __args)
      > 835:     { return unique_ptr<_Tp>(new _Tp(std::forward<_Args>(__args)...)); }
        836: 
        837:   /// std::make_unique for arrays of unknown bound
        838:   template<typename _Tp>
#28   Source "/home/mihir/dome/src/ign-gazebo/src/SimulationRunner.cc", line 163, in ignition::gazebo::v4::SimulationRunner::SimulationRunner(sdf::v10::World const*, std::shared_ptr<ignition::gazebo::v4::SystemLoader> const&, ignition::gazebo::v4::ServerConfig const&) [0x7f45928064b8]
        160:   this->levelMgr->UpdateLevelsState();
        161: 
        162:   // Load any additional plugins from the Server Configuration
      > 163:   this->LoadServerPlugins(this->serverConfig.Plugins());
        164: 
        165:   // If we have reached this point and no systems have been loaded, then load
        166:   // a default set of systems.
#27   Source "/home/mihir/dome/src/ign-gazebo/src/SimulationRunner.cc", line 912, in ignition::gazebo::v4::SimulationRunner::LoadServerPlugins(std::__cxx11::list<ignition::gazebo::v4::ServerConfig::PluginInfo, std::allocator<ignition::gazebo::v4::ServerConfig::PluginInfo> > const&) [0x7f4592804868]
        910:     if (kNullEntity != entity)
        911:     {
      > 912:       this->LoadPlugin(entity, plugin.Filename(), plugin.Name(), plugin.Sdf());
        913:     }
        914:   }
        915: }
#26   Source "/home/mihir/dome/src/ign-gazebo/src/SimulationRunner.cc", line 815, in ignition::gazebo::v4::SimulationRunner::LoadPlugin(unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<sdf::v10::Element> const&) [0x7f4592802d59]
        812:   std::optional<SystemPluginPtr> system;
        813:   {
        814:     std::lock_guard<std::mutex> lock(this->systemLoaderMutex);
      > 815:     system = this->systemLoader->LoadPlugin(_fname, _name, _sdf);
        816:   }
        817: 
        818:   // System correctly loaded from library, try to configure
#25   Source "/home/mihir/dome/src/ign-gazebo/src/SystemLoader.cc", line 156, in ignition::gazebo::v4::SystemLoader::LoadPlugin(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<sdf::v10::Element> const&) [0x7f45928130c8]
        153:     return {};
        154:   }
        155: 
      > 156:   auto ret = this->dataPtr->InstantiateSystemPlugin(_filename,
        157:                                                     _name,
        158:                                                     _sdf, plugin);
        159:   if (ret && plugin)
#24   Source "/home/mihir/dome/src/ign-gazebo/src/SystemLoader.cc", line 72, in ignition::gazebo::v4::SystemLoaderPrivate::InstantiateSystemPlugin(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<sdf::v10::Element> const&, ignition::plugin::TemplatePluginPtr<ignition::plugin::Plugin>&) [0x7f4592813984]
         69:       return false;
         70:     }
         71: 
      >  72:     auto pluginNames = this->loader.LoadLib(pathToLib);
         73:     if (pluginNames.empty())
         74:     {
         75:       ignerr << "Failed to load system plugin [" << _filename <<
#23   Source "/home/mihir/dome/src/ign-plugin/loader/src/Loader.cc", line 212, in ignition::plugin::Loader::LoadLib(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) [0x7f459419d11c]
        210:       // Attempt to load the library at this path
        211:       const std::shared_ptr<void> &dlHandle =
      > 212:           this->dataPtr->LoadLib(_pathToLibrary);
        213: 
        214:       // Quit early and return an empty set of plugin names if we did not
        215:       // actually get a valid dlHandle.
#22   Source "/home/mihir/dome/src/ign-plugin/loader/src/Loader.cc", line 457, in ignition::plugin::Loader::Implementation::LoadLib(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) [0x7f459419c579]
        455:       // NOTE: We open using RTLD_LOCAL instead of RTLD_GLOBAL to prevent the
        456:       // symbols of different libraries from writing over each other.
      > 457:       void *dlHandle = dlopen(_full_path.c_str(), RTLD_LAZY | RTLD_LOCAL);
        458: 
        459:       const char *loadError = dlerror();
        460:       if (nullptr == dlHandle || nullptr != loadError)
#21   Source "/build/glibc-S9d2JN/glibc-2.27/dlfcn/dlopen.c", line 87, in __dlopen [0x7f4596aeb050]
#20   Source "/build/glibc-S9d2JN/glibc-2.27/dlfcn/dlerror.c", line 162, in _dlerror_run [0x7f4596aeb744]
#19   Source "/build/glibc-S9d2JN/glibc-2.27/elf/dl-error-skeleton.c", line 215, in __GI__dl_catch_error [0x7f45972f527e]
#18   Source "/build/glibc-S9d2JN/glibc-2.27/elf/dl-error-skeleton.c", line 196, in __GI__dl_catch_exception [0x7f45972f51ee]
#17   Source "/build/glibc-S9d2JN/glibc-2.27/dlfcn/dlopen.c", line 66, in dlopen_doit [0x7f4596aeaf95]
#16   Source "/build/glibc-S9d2JN/glibc-2.27/elf/dl-open.c", line 605, in _dl_open [0x7f4597a44969]
#15   Source "/build/glibc-S9d2JN/glibc-2.27/elf/dl-error-skeleton.c", line 196, in __GI__dl_catch_exception [0x7f45972f51ee]
#14   Source "/build/glibc-S9d2JN/glibc-2.27/elf/dl-open.c", line 522, in dl_open_worker [0x7f4597a4539e]
#13 | Source "/build/glibc-S9d2JN/glibc-2.27/elf/dl-init.c", line 119, in _dl_init
      Source "/build/glibc-S9d2JN/glibc-2.27/elf/dl-init.c", line 72, in call_init [0x7f4597a408d2]
#12   Object "/usr/lib/x86_64-linux-gnu/libignition-msgs4.so.4", at 0x7f45575f06d9, in ignition::msgs::protobuf_AddDesc_ignition_2fmsgs_2factor_2eproto()
#11   Object "/usr/lib/x86_64-linux-gnu/libignition-msgs4.so.4", at 0x7f45576278f7, in ignition::msgs::protobuf_AddDesc_ignition_2fmsgs_2fentity_2eproto()
#10   Object "/usr/lib/x86_64-linux-gnu/libignition-msgs4.so.4", at 0x7f4557641155, in ignition::msgs::protobuf_AddDesc_ignition_2fmsgs_2fheader_2eproto()
#9    Object "/usr/lib/x86_64-linux-gnu/libignition-msgs4.so.4", at 0x7f4557715d73, in ignition::msgs::protobuf_AddDesc_ignition_2fmsgs_2ftime_2eproto()
#8    Object "/usr/lib/x86_64-linux-gnu/libprotobuf.so.10", at 0x7f4591c0a3db, in google::protobuf::DescriptorPool::InternalAddGeneratedFile(void const*, int)
#7    Object "/lib/x86_64-linux-gnu/libgcc_s.so.1", at 0x7f45939ab8a5, in _Unwind_Resume
#6    Object "/lib/x86_64-linux-gnu/libgcc_s.so.1", at 0x7f45939aaefe, in _Unwind_GetTextRelBase
#5    Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", at 0x7f4593c57ff0, in __gxx_personality_v0
#4    Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", at 0x7f4593c57648, in __cxa_free_dependent_exception
#3    Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", at 0x7f4593c58605, in std::rethrow_exception(std::__exception_ptr::exception_ptr)
#2    Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", at 0x7f4593c4d256, in __cxa_throw_bad_array_new_length
#1    Source "/build/glibc-S9d2JN/glibc-2.27/stdlib/abort.c", line 79, in __GI_abort [0x7f45971ce920]
#0    Source "/build/glibc-S9d2JN/glibc-2.27/signal/../sysdeps/unix/sysv/linux/raise.c", line 51, in __GI_raise [0x7f45971ccfb7]
Aborted (Signal sent by tkill() 21789 1000)
[1]    21789 abort (core dumped)  ign launch -v 4 competition.ign worldName:=tunnel_circuit_practice_01 

[mihirk284]

Hi @peci1 , Thank you for the response. Thank you very much for taking out the time to help me with this.

And can you run at least some normal non-subt worlds? like ign gazebo -v4 sensors_demo.sdf?

Yes, I can run normal ign gazebo worlds without any issue. All other standalone aspects of Ignition work as expected. It is only when used with subt simulator, that it crashes like this.

Also, if you have a source build, it should be easy to modify the source code and add some debug prints to know why does the simulator crash. Especially, it would be useful to know what are the values used in

"/home/arl/dome_dev/src/ign-gazebo/src/systems/physics/Physics.cc", line 912

Because this is probably the place where the crash somehow begins (loading a collision element). Even better would be attaching a debugger and stepping the code, but I don't know about any easy way to attach a debugger to ign gazebo. Maybe somebody from Open Robotics would know?

I will try that. On the update comment above, it seems like it may be more of an issue related to the C++ libraries, more specifically for the destruction of vectors? I am not very sure. Some guidance on this may be really helpful.

One quick thing you can try is calling export LC_ALL=C before launching the simulator. There are various locale-induced issues in Ignition and this should at least rule out this kind of problems.

I will try this out and revert back soon :)

[peci1]

No, the vector destruction is a false lead.

The important line in your latest update is

[libprotobuf ERROR google/protobuf/descriptor_database.cc:57] File already exists in database: ignition/msgs/time.proto

This happens when you pull two different versions of libignition-msgs. E.g. one used by the simulator and a different one used by a plugin loaded by a model. You should always check that your subt_ws is built exactly with the ignition libraries you run it with (e.g. either binary or source build, do not mix them). Do a full rebuild (or keep a separate workspace) after switching between binary and source builds.