osrf-migration
commented
11 years ago Original comment by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).
While testing pull request #584, I was able to reproduce this seg-fault using branch out_of_memory (commit e2c758db4ee312a3041de706fa1957622c87fe2d). It took 213 runs to generate the failure.
Core was generated by `test/performance/transport_stress'.
Program terminated with signal 11, Segmentation fault.
#0 operator++ (this=<optimized out>) at /usr/include/c++/4.6/bits/stl_list.h:234
234 _M_node = _M_node->_M_next;
(gdb) bt
#0 operator++ (this=<optimized out>) at /usr/include/c++/4.6/bits/stl_list.h:234
#1 __distance<std::_List_const_iterator<boost::shared_ptr<gazebo::transport::Node> > > (__last=..., __first=...)
at /usr/include/c++/4.6/bits/stl_iterator_base_funcs.h:82
#2 distance<std::_List_const_iterator<boost::shared_ptr<gazebo::transport::Node> > > (__last=..., __first=...)
at /usr/include/c++/4.6/bits/stl_iterator_base_funcs.h:117
#3 size (this=0x2009850) at /usr/include/c++/4.6/bits/stl_list.h:846
#4 gazebo::transport::Publication::RemoveSubscription (this=0x2009820, _host=..., _port=38642)
at /home/scpeters/osrf/gazebo3/gazebo/transport/Publication.cc:220
#5 0x00007f323f958114 in gazebo::transport::TopicManager::DisconnectPubFromSub (this=<optimized out>, topic=..., host=..., port=38642)
at /home/scpeters/osrf/gazebo3/gazebo/transport/TopicManager.cc:275
#6 0x00007f323f9410ea in gazebo::transport::ConnectionManager::ProcessMessage (this=<optimized out>, _data=...)
at /home/scpeters/osrf/gazebo3/gazebo/transport/ConnectionManager.cc:430
#7 0x00007f323f941ecc in gazebo::transport::ConnectionManager::RunUpdate (this=0x6a32e0)
at /home/scpeters/osrf/gazebo3/gazebo/transport/ConnectionManager.cc:263
#8 0x00007f323f9431b2 in gazebo::transport::ConnectionManager::Run (this=0x6a32e0)
at /home/scpeters/osrf/gazebo3/gazebo/transport/ConnectionManager.cc:310
#9 0x00007f323ed51ce9 in thread_proxy () from /usr/lib/libboost_thread.so.1.46.1
#10 0x00007f323f6d6e9a in start_thread (arg=0x7f322a04f700) at pthread_create.c:308
#11 0x00007f323decdcbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#12 0x0000000000000000 in ?? ()
(gdb) Full backtrace:
(gdb) thread apply all bt
Thread 17 (Thread 0x7f320affb700 (LWP 23141)):
#0 0x00007f323dea4267 in sched_yield () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f323e6cc9d5 in ?? () from /usr/lib/libtbb.so.2
#2 0x00007f323e6cd221 in ?? () from /usr/lib/libtbb.so.2
#3 0x00007f323e6c909d in ?? () from /usr/lib/libtbb.so.2
#4 0x00007f323e6c854b in ?? () from /usr/lib/libtbb.so.2
#5 0x00007f323e6c608f in ?? () from /usr/lib/libtbb.so.2
#6 0x00007f323e6c62c9 in ?? () from /usr/lib/libtbb.so.2
#7 0x00007f323f6d6e9a in start_thread (arg=0x7f320affb700) at pthread_create.c:308
#8 0x00007f323decdcbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#9 0x0000000000000000 in ?? ()
Thread 16 (Thread 0x7f322b852700 (LWP 23135)):
#0 0x00007f323f6dbd6f in __pthread_getspecific (key=6) at pthread_getspecific.c:33
#1 0x00007f323e6ccf31 in ?? () from /usr/lib/libtbb.so.2
#2 0x000000000045eeb6 in enqueue (t=...) at /usr/include/tbb/task.h:720
#3 gazebo::transport::Connection::OnReadData<boost::_bi::bind_t<void, boost::_mfi::mf2<void, gazebo::Master, unsigned int, std::string const&>, boost::_bi::list3<boost::_bi::value<gazebo::Master*>, boost::_bi::value<unsigned int>, boost::arg<1> > > > (this=<optimized out>, _e=..., _handler=...)
at /home/scpeters/osrf/gazebo3/gazebo/transport/Connection.hh:318
#4 0x0000000000462394 in operator() (a2=..., p=<optimized out>, this=0x7f322b851aa0, a1=...) at /usr/include/boost/bind/mem_fn_template.hpp:280
#5 operator()<boost::_mfi::mf2<void, gazebo::transport::Connection, const boost::system::error_code&, boost::tuples::tuple<boost::_bi::bind_t<void, boost::_mfi::mf2<void, gazebo::Master, unsigned int, const std::basic_string<char>&>, boost::_bi::list3<boost::_bi::value<gazebo::Master*>, boost::_bi::value<unsigned int>, boost::arg<1> > >, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type> >, boost::_bi::list2<const boost::system::error_code&, const long unsigned int&> > (a=<synthetic pointer>, f=..., this=0x7f322b851ab0) at /usr/include/boost/bind/bind.hpp:392
#6 operator()<boost::system::error_code, long unsigned int> (a2=@0x7f322b851a98: 104, this=0x7f322b851aa0, a1=...)
at /usr/include/boost/bind/bind_template.hpp:102
#7 operator() (ec=..., this=0x7f322b851a80, bytes_transferred=<optimized out>, start=<optimized out>) at /usr/include/boost/asio/impl/read.hpp:215
#8 operator() (start=0, bytes_transferred=<optimized out>, ec=..., this=0x7f322b851a80)
at /usr/include/boost/asio/detail/reactive_socket_recv_op.hpp:84
#9 operator() (this=0x7f322b851a80) at /usr/include/boost/asio/detail/bind_handler.hpp:97
#10 asio_handler_invoke<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp>, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::_bi::bind_t<void, boost::_mfi::mf2<void, gazebo::transport::Connection, boost::system::error_code const&, boost::tuples::tuple<boost::_bi::bind_t<void, boost::_mfi::mf2<void, gazebo::Master, unsigned int, std::basic_string<char> const&>, boost::_bi::list3<boost::_bi::value<gazebo::Master*>, boost::_bi::value<unsigned int>, boost::arg<1> > >, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type> >, boost::_bi::list3<boost::_bi::value<gazebo::transport::Connection*>, boost::arg<1> (*)(), boost::_bi::value<boost::tuples::tuple<boost::_bi::bind_t<void, boost::_mfi::mf2<void, gazebo::Master, unsigned int, std::basic_string<char> const&>, boost::_bi::list3<boost::_bi::value<gazebo::Master*>, boost::_bi::value<unsigned int>, boost::arg<1> > >, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type> > > > >, boost::system::error_code, unsigned long> > (function=...) at /usr/include/boost/asio/handler_invoke_hook.hpp:64
#11 invoke<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp>, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::_bi::bind_t<void, boost::_mfi::mf2<void, gazebo::transport::Connection, boost::system::error_code const&, boost::tuples::tuple<boost::_bi::bind_t<void, boost::_mfi::mf2<void, gazebo::Master, unsigned int, std::basic_string<char> const&>, boost::_bi::list3<boost::_bi::value<gazebo::Master*>, boost::_bi::value<unsigned int>, boost::arg<1> > >, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type> >, boost::_bi::list3<boost::_bi::value<gazebo::transport::Connection*>, boost::arg<1> (*)(), boost::_bi::value<boost::tuples::tuple<boost::_bi::bind_t<void, boost::_mfi::mf2<void, gazebo::Master, unsigned int, std::basic_string<char> const&>, boost::_bi::list3<boost::_bi::value<gazebo::Master*>, boost::_bi::value<unsigned int>, boost::arg<1> > >, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type> > > > >, boost::system::error_code, unsigned long>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, gazebo::transport::Connection, boost::system::error_code const&, boost::tuples::tuple<boost::_bi::bind_t<void, boost::_mfi::mf2<void, gazebo::Master, unsigned int, std::basic_string<char> const&>, boost::_bi::list3<boost::_bi::value<gazebo::Master*>, boost::_bi::value<unsigned int>, boost::arg<1> > >, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type> >, boost::_bi::list3<boost::_bi::value<gazebo::transport::Connection*>, boost::arg<1> (*)(), boost::_bi::value<boost::tuples::tuple<boost::_bi::bind_t<void, boost::_mfi::mf2<void, gazebo::Master, unsigned int, std::basic_string<char> const&>, boost::_bi::list3<boost::_bi::value<gazebo::Master*>, boost::_bi::value<unsigned int>, boost::arg<1> > >, boost::tuples::null_type, boost::tuples::null_type, ---Type <return> to continue, or q <return> to quit---
boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type> > > > > (function=..., context=...) at /usr/include/boost/asio/detail/handler_invoke_helpers.hpp:39
#12 asio_handler_invoke<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp>, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::_bi::bind_t<void, boost::_mfi::mf2<void, gazebo::transport::Connection, boost::system::error_code const&, boost::tuples::tuple<boost::_bi::bind_t<void, boost::_mfi::mf2<void, gazebo::Master, unsigned int, std::basic_string<char> const&>, boost::_bi::list3<boost::_bi::value<gazebo::Master*>, boost::_bi::value<unsigned int>, boost::arg<1> > >, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type> >, boost::_bi::list3<boost::_bi::value<gazebo::transport::Connection*>, boost::arg<1> (*)(), boost::_bi::value<boost::tuples::tuple<boost::_bi::bind_t<void, boost::_mfi::mf2<void, gazebo::Master, unsigned int, std::basic_string<char> const&>, boost::_bi::list3<boost::_bi::value<gazebo::Master*>, boost::_bi::value<unsigned int>, boost::arg<1> > >, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type> > > > >, boost::system::error_code, unsigned long>, boost::asio::basic_stream_socket<boost::asio::ip::tcp>, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::_bi::bind_t<void, boost::_mfi::mf2<void, gazebo::transport::Connection, boost::system::error_code const&, boost::tuples::tuple<boost::_bi::bind_t<void, boost::_mfi::mf2<void, gazebo::Master, unsigned int, std::basic_string<char> const&>, boost::_bi::list3<boost::_bi::value<gazebo::Master*>, boost::_bi::value<unsigned int>, boost::arg<1> > >, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type> >, boost::_bi::list3<boost::_bi::value<gazebo::transport::Connection*>, boost::arg<1> (*)(), boost::_bi::value<boost::tuples::tuple<boost::_bi::bind_t<void, boost::_mfi::mf2<void, gazebo::Master, unsigned int, std::basic_string<char> const&>, boost::_bi::list3<boost::_bi::value<gazebo::Master*>, boost::_bi::value<unsigned int>, boost::arg<1> > >, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type> > > > > (function=..., this_handler=<optimized out>) at /usr/include/boost/asio/impl/read.hpp:253
#13 invoke<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp>, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::_bi::bind_t<void, boost::_mfi::mf2<void, gazebo::transport::Connection, boost::system::error_code const&, boost::tuples::tuple<boost::_bi::bind_t<void, boost::_mfi::mf2<void, gazebo::Master, unsigned int, std::basic_string<char> const&>, boost::_bi::list3<boost::_bi::value<gazebo::Master*>, boost::_bi::value<unsigned int>, boost::arg<1> > >, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type> >, boost::_bi::list3<boost::_bi::value<gazebo::transport::Connection*>, boost::arg<1> (*)(), boost::_bi::value<boost::tuples::tuple<boost::_bi::bind_t<void, boost::_mfi::mf2<void, gazebo::Master, unsigned int, std::basic_string<char> const&>, boost::_bi::list3<boost::_bi::value<gazebo::Master*>, boost::_bi::value<unsigned int>, boost::arg<1> > >, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type> > > > >, boost::system::error_code, unsigned long>, boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp>, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::_bi::bind_t<void, boost::_mfi::mf2<void, gazebo::transport::Connection, boost::system::error_code const&, boost::tuples::tuple<boost::_bi::bind_t<void, boost::_mfi::mf2<void, gazebo::Master, unsigned int, std::basic_string<char> const&>, boost::_bi::list3<boost::_bi::value<gazebo::Master*>, boost::_bi::value<unsigned int>, boost::arg<1> > >, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type> >, boost::_bi::list3<boost::_bi::value<gazebo::transport::Connection*>, boost::arg<1> (*)(), boost::_bi::value<boost::tuples::tuple<boost::_bi::bind_t<void, boost::_mfi::mf2<void, gazebo::Master, unsigned int, std::basic_string<char> const&>, boost::_bi::list3<boost::_bi::value<gazebo::Master*>, boost::_bi::value<unsigned int>, boost::arg<1> > >, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type> > > > > > (function=..., context=...) at /usr/include/boost/asio/detail/handler_invoke_helpers.hpp:39
#14 boost::asio::detail::reactive_socket_recv_op<boost::asio::mutable_buffers_1, boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::_bi::bind_t<void, boost::_mfi::mf2<void, gazebo::transport::Connection, boost::system::error_code const&, boost::tuples::tuple<boost::_bi::bind_t<void, boost::_mfi::mf2<void, gazebo::Master, unsigned int, std::string const&>, boost::_bi::list3<boost::_bi::value<gazebo::Master*>, boost::_bi::value<unsigned int>, boost::arg<1> > >, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type> >, boost::_bi::list3<boost::_bi::value<gazebo::transport::Connection*>, boost::arg<1> (*)(), boost::_bi::value<boost::tuples::tuple<boost::_bi::bind_t<void, boost::_mfi::mf2<void, gazebo::Master, unsigned int, std::string const&>, boost::_bi::list3<boost::_bi::value<gazebo::Master*>, boost::_bi::value<unsigned int>, boost::arg<1> > >, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type> > > > > >::do_complete (owner=0x7f321c522af0,
base=<optimized out>) at /usr/include/boost/asio/detail/reactive_socket_recv_op.hpp:106
#15 0x00007f323f94b9c7 in complete (owner=..., this=0x7f32240078d0) at /usr/include/boost/asio/detail/task_io_service_operation.hpp:35
#16 do_one (this_idle_thread=<optimized out>, lock=..., this=<optimized out>) at /usr/include/boost/asio/detail/impl/task_io_service.ipp:278
#17 boost::asio::detail::task_io_service::run (this=0x7f321c522af0, ec=...) at /usr/include/boost/asio/detail/impl/task_io_service.ipp:130
#18 0x00007f323f94bca5 in boost::asio::io_service::run (this=0x7f321c472b10) at /usr/include/boost/asio/impl/io_service.ipp:57
#19 0x00007f323ed51ce9 in thread_proxy () from /usr/lib/libboost_thread.so.1.46.1
#20 0x00007f323f6d6e9a in start_thread (arg=0x7f322b852700) at pthread_create.c:308
#21 0x00007f323decdcbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
---Type <return> to continue, or q <return> to quit---
#22 0x0000000000000000 in ?? ()
Thread 15 (Thread 0x7f3209bf8700 (LWP 23147)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1 0x00007f3240ae42a3 in boost::condition_variable::wait (this=0x7f321c214c40, m=...) at /usr/include/boost/thread/pthread/condition_variable.hpp:53
#2 0x00007f32402ac879 in gazebo::sensors::SensorManager::SensorContainer::RunLoop (this=0x7f321c214ba0)
at /home/scpeters/osrf/gazebo3/gazebo/sensors/SensorManager.cc:463
#3 0x00007f323ed51ce9 in thread_proxy () from /usr/lib/libboost_thread.so.1.46.1
#4 0x00007f323f6d6e9a in start_thread (arg=0x7f3209bf8700) at pthread_create.c:308
#5 0x00007f323decdcbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6 0x0000000000000000 in ?? ()
Thread 14 (Thread 0x7f32093f7700 (LWP 23148)):
#0 0x00007f323f6de52d in nanosleep () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f3240af830d in gazebo::common::Time::Sleep (_time=...) at /home/scpeters/osrf/gazebo3/gazebo/common/Time.cc:171
#2 0x00007f32405d4adf in gazebo::physics::World::Step (this=0x7f321c4977f0) at /home/scpeters/osrf/gazebo3/gazebo/physics/World.cc:483
#3 0x00007f32405d787c in gazebo::physics::World::RunLoop (this=0x7f321c4977f0) at /home/scpeters/osrf/gazebo3/gazebo/physics/World.cc:365
#4 0x00007f323ed51ce9 in thread_proxy () from /usr/lib/libboost_thread.so.1.46.1
#5 0x00007f323f6d6e9a in start_thread (arg=0x7f32093f7700) at pthread_create.c:308
#6 0x00007f323decdcbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7 0x0000000000000000 in ?? ()
Thread 13 (Thread 0x7f31e3fff700 (LWP 23149)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1 0x00007f32405d628b in wait (m=..., this=0x7f321c497f40) at /usr/include/boost/thread/pthread/condition_variable.hpp:53
#2 gazebo::physics::World::LogWorker (this=0x7f321c4977f0) at /home/scpeters/osrf/gazebo3/gazebo/physics/World.cc:1889
#3 0x00007f323ed51ce9 in thread_proxy () from /usr/lib/libboost_thread.so.1.46.1
#4 0x00007f323f6d6e9a in start_thread (arg=0x7f31e3fff700) at pthread_create.c:308
#5 0x00007f323decdcbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6 0x0000000000000000 in ?? ()
Thread 12 (Thread 0x7f322b051700 (LWP 23134)):
#0 0x00007f323f6de52d in nanosleep () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f3240af830d in gazebo::common::Time::Sleep (_time=...) at /home/scpeters/osrf/gazebo3/gazebo/common/Time.cc:171
#2 0x00007f3240af8853 in gazebo::common::Time::MSleep (_ms=<optimized out>) at /home/scpeters/osrf/gazebo3/gazebo/common/Time.cc:189
#3 0x000000000044d9bf in gazebo::Server::Run (this=0x7f321c18a730) at /home/scpeters/osrf/gazebo3/gazebo/Server.cc:470
#4 0x0000000000444378 in ServerFixture::RunServer (this=0x1fd9a00, _worldFilename=..., _paused=false, _physics=...)
at /home/scpeters/osrf/gazebo3/test/ServerFixture.hh:222
#5 0x00007f323ed51ce9 in thread_proxy () from /usr/lib/libboost_thread.so.1.46.1
#6 0x00007f323f6d6e9a in start_thread (arg=0x7f322b051700) at pthread_create.c:308
#7 0x00007f323decdcbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#8 0x0000000000000000 in ?? ()
Thread 11 (Thread 0x7f3229641700 (LWP 23143)):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:39
#1 0x00007f323e6c6293 in ?? () from /usr/lib/libtbb.so.2
#2 0x00007f323e6c62c9 in ?? () from /usr/lib/libtbb.so.2
#3 0x00007f323f6d6e9a in start_thread (arg=0x7f3229641700) at pthread_create.c:308
#4 0x00007f323decdcbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 10 (Thread 0x7f322a850700 (LWP 23136)):
#0 0x00007f323f6de52d in nanosleep () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f3240af830d in gazebo::common::Time::Sleep (_time=...) at /home/scpeters/osrf/gazebo3/gazebo/common/Time.cc:171
#2 0x00007f3240af8853 in gazebo::common::Time::MSleep (_ms=<optimized out>) at /home/scpeters/osrf/gazebo3/gazebo/common/Time.cc:189
#3 0x000000000045c78d in gazebo::Master::Run (this=0x7f321c534150) at /home/scpeters/osrf/gazebo3/gazebo/Master.cc:300
#4 0x00007f323ed51ce9 in thread_proxy () from /usr/lib/libboost_thread.so.1.46.1
#5 0x00007f323f6d6e9a in start_thread (arg=0x7f322a850700) at pthread_create.c:308
---Type <return> to continue, or q <return> to quit---
#6 0x00007f323decdcbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7 0x0000000000000000 in ?? ()
Thread 9 (Thread 0x7f320bfff700 (LWP 23140)):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:39
#1 0x00007f323e6c6293 in ?? () from /usr/lib/libtbb.so.2
#2 0x00007f323e6c62c9 in ?? () from /usr/lib/libtbb.so.2
#3 0x00007f323f6d6e9a in start_thread (arg=0x7f320bfff700) at pthread_create.c:308
#4 0x00007f323decdcbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 8 (Thread 0x7f322861d700 (LWP 23144)):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:39
#1 0x00007f323e6c6293 in ?? () from /usr/lib/libtbb.so.2
#2 0x00007f323e6c62c9 in ?? () from /usr/lib/libtbb.so.2
#3 0x00007f323f6d6e9a in start_thread (arg=0x7f322861d700) at pthread_create.c:308
#4 0x00007f323decdcbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 7 (Thread 0x7f320abfa700 (LWP 23145)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1 0x00007f3240ae42a3 in boost::condition_variable::wait (this=0x6a3a80, m=...) at /usr/include/boost/thread/pthread/condition_variable.hpp:53
#2 0x00007f3240adebf7 in gazebo::common::ModelDatabase::UpdateModelCache (this=0x6a39a0, _fetchImmediately=false)
at /home/scpeters/osrf/gazebo3/gazebo/common/ModelDatabase.cc:274
#3 0x00007f323ed51ce9 in thread_proxy () from /usr/lib/libboost_thread.so.1.46.1
#4 0x00007f323f6d6e9a in start_thread (arg=0x7f320abfa700) at pthread_create.c:308
#5 0x00007f323decdcbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6 0x0000000000000000 in ?? ()
Thread 6 (Thread 0x7f320a3f9700 (LWP 23146)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1 0x00007f3240ae42a3 in boost::condition_variable::wait (this=0x7f321c213df0, m=...) at /usr/include/boost/thread/pthread/condition_variable.hpp:53
#2 0x00007f32402ac879 in gazebo::sensors::SensorManager::SensorContainer::RunLoop (this=0x7f321c213d50)
at /home/scpeters/osrf/gazebo3/gazebo/sensors/SensorManager.cc:463
#3 0x00007f323ed51ce9 in thread_proxy () from /usr/lib/libboost_thread.so.1.46.1
#4 0x00007f323f6d6e9a in start_thread (arg=0x7f320a3f9700) at pthread_create.c:308
#5 0x00007f323decdcbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6 0x0000000000000000 in ?? ()
Thread 5 (Thread 0x7f3241194840 (LWP 23117)):
#0 0x00007f323f6d918b in __pthread_mutex_trylock (mutex=0x20098b0) at pthread_mutex_trylock.c:65
#1 0x00007f323f951e07 in try_lock (this=<optimized out>) at /usr/include/boost/thread/pthread/mutex.hpp:66
#2 try_lock (this=0x7fff48fe8560) at /usr/include/boost/thread/locks.hpp:421
#3 unique_lock (this=0x7fff48fe8560, m_=...) at /usr/include/boost/thread/locks.hpp:301
#4 try_lock_wrapper (m=..., this=0x7fff48fe8560) at /usr/include/boost/thread/locks.hpp:995
#5 gazebo::transport::Publication::RemoveSubscription (this=0x2009820, _node=...) at /home/scpeters/osrf/gazebo3/gazebo/transport/Publication.cc:162
#6 0x00007f323f959589 in gazebo::transport::TopicManager::Unsubscribe (this=0x6a3140, _topic=..., _node=...)
at /home/scpeters/osrf/gazebo3/gazebo/transport/TopicManager.cc:254
#7 0x00007f323f9576d7 in gazebo::transport::Subscriber::Unsubscribe (this=0x20be5f0)
at /home/scpeters/osrf/gazebo3/gazebo/transport/Subscriber.cc:52
#8 0x00007f323f9577ac in gazebo::transport::Subscriber::~Subscriber (this=0x20be5f0, __in_chrg=<optimized out>)
at /home/scpeters/osrf/gazebo3/gazebo/transport/Subscriber.cc:37
#9 0x00007f323f9578b9 in gazebo::transport::Subscriber::~Subscriber (this=0x20be5f0, __in_chrg=<optimized out>)
at /home/scpeters/osrf/gazebo3/gazebo/transport/Subscriber.cc:39
#10 0x00000000004433b9 in release (this=0x20be210) at /usr/include/boost/smart_ptr/detail/sp_counted_base_gcc_x86.hpp:145
#11 release (this=0x20be210) at /home/scpeters/osrf/gazebo3/test/performance/transport_stress.cc:243
#12 boost::detail::shared_count::~shared_count (this=<optimized out>, __in_chrg=<optimized out>)
at /usr/include/boost/smart_ptr/detail/shared_count.hpp:217
---Type <return> to continue, or q <return> to quit---
#13 0x000000000043f56f in ~shared_ptr (this=0x20be6c0, __in_chrg=<optimized out>) at /usr/include/boost/smart_ptr/shared_ptr.hpp:168
#14 destroy (__p=0x20be6c0, this=<optimized out>) at /usr/include/c++/4.6/ext/new_allocator.h:118
#15 _M_clear (this=0x7fff48fe87a0) at /usr/include/c++/4.6/bits/list.tcc:78
#16 ~_List_base (this=0x7fff48fe87a0, __in_chrg=<optimized out>) at /usr/include/c++/4.6/bits/stl_list.h:372
#17 ~list (this=0x7fff48fe87a0, __in_chrg=<optimized out>) at /usr/include/c++/4.6/bits/stl_list.h:429
#18 TransportStressTest_ManyNodes_Test::TestBody (this=<optimized out>) at /home/scpeters/osrf/gazebo3/test/performance/transport_stress.cc:234
#19 0x0000000000480c6d in HandleSehExceptionsInMethodIfSupported<testing::Test, void> (method=<optimized out>, object=<optimized out>,
location=<optimized out>) at /home/scpeters/osrf/gazebo3/test/gtest/src/gtest.cc:2090
#20 testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=0x1fd9a00, method=&virtual testing::Test::TestBody(),
location=0x48692e "the test body") at /home/scpeters/osrf/gazebo3/test/gtest/src/gtest.cc:2126
#21 0x0000000000478ba1 in testing::Test::Run (this=0x1fd9a00) at /home/scpeters/osrf/gazebo3/test/gtest/src/gtest.cc:2162
#22 0x0000000000478c86 in testing::TestInfo::Run (this=0x2002a40) at /home/scpeters/osrf/gazebo3/test/gtest/src/gtest.cc:2338
#23 0x0000000000478dc7 in testing::TestCase::Run (this=0x2002960) at /home/scpeters/osrf/gazebo3/test/gtest/src/gtest.cc:2445
#24 0x000000000047911e in RunAllTests (this=0x2002680) at /home/scpeters/osrf/gazebo3/test/gtest/src/gtest.cc:4237
#25 testing::internal::UnitTestImpl::RunAllTests (this=0x2002680) at /home/scpeters/osrf/gazebo3/test/gtest/src/gtest.cc:4145
#26 0x00000000004807ed in HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (method=<optimized out>,
object=<optimized out>, location=<optimized out>) at /home/scpeters/osrf/gazebo3/test/gtest/src/gtest.cc:2090
#27 testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x2002680, method=
(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x478e40 <testing::internal::UnitTestImpl::RunAllTests()>,
location=0x487b08 "auxiliary test code (environments or event listeners)") at /home/scpeters/osrf/gazebo3/test/gtest/src/gtest.cc:2126
#28 0x00000000004781fe in testing::UnitTest::Run (this=<optimized out>) at /home/scpeters/osrf/gazebo3/test/gtest/src/gtest.cc:3874
#29 0x000000000043af2f in main (argc=1, argv=<optimized out>) at /home/scpeters/osrf/gazebo3/test/performance/transport_stress.cc:242
Thread 4 (Thread 0x7f3228a1e700 (LWP 23142)):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:39
#1 0x00007f323e6c6293 in ?? () from /usr/lib/libtbb.so.2
#2 0x00007f323e6c62c9 in ?? () from /usr/lib/libtbb.so.2
#3 0x00007f323f6d6e9a in start_thread (arg=0x7f3228a1e700) at pthread_create.c:308
#4 0x00007f323decdcbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 3 (Thread 0x7f320b3fc700 (LWP 23138)):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:39
#1 0x00007f323e6c6293 in ?? () from /usr/lib/libtbb.so.2
#2 0x00007f323e6c62c9 in ?? () from /usr/lib/libtbb.so.2
#3 0x00007f323f6d6e9a in start_thread (arg=0x7f320b3fc700) at pthread_create.c:308
#4 0x00007f323decdcbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 2 (Thread 0x7f320bbfe700 (LWP 23139)):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:39
#1 0x00007f323e6c6293 in ?? () from /usr/lib/libtbb.so.2
#2 0x00007f323e6c62c9 in ?? () from /usr/lib/libtbb.so.2
#3 0x00007f323f6d6e9a in start_thread (arg=0x7f320bbfe700) at pthread_create.c:308
#4 0x00007f323decdcbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 1 (Thread 0x7f322a04f700 (LWP 23137)):
#0 operator++ (this=<optimized out>) at /usr/include/c++/4.6/bits/stl_list.h:234
#1 __distance<std::_List_const_iterator<boost::shared_ptr<gazebo::transport::Node> > > (__last=..., __first=...)
at /usr/include/c++/4.6/bits/stl_iterator_base_funcs.h:82
#2 distance<std::_List_const_iterator<boost::shared_ptr<gazebo::transport::Node> > > (__last=..., __first=...)
at /usr/include/c++/4.6/bits/stl_iterator_base_funcs.h:117
#3 size (this=0x2009850) at /usr/include/c++/4.6/bits/stl_list.h:846
#4 gazebo::transport::Publication::RemoveSubscription (this=0x2009820, _host=..., _port=38642)
at /home/scpeters/osrf/gazebo3/gazebo/transport/Publication.cc:220
#5 0x00007f323f958114 in gazebo::transport::TopicManager::DisconnectPubFromSub (this=<optimized out>, topic=..., host=..., port=38642)
at /home/scpeters/osrf/gazebo3/gazebo/transport/TopicManager.cc:275
---Type <return> to continue, or q <return> to quit---
#6 0x00007f323f9410ea in gazebo::transport::ConnectionManager::ProcessMessage (this=<optimized out>, _data=...)
at /home/scpeters/osrf/gazebo3/gazebo/transport/ConnectionManager.cc:430
#7 0x00007f323f941ecc in gazebo::transport::ConnectionManager::RunUpdate (this=0x6a32e0)
at /home/scpeters/osrf/gazebo3/gazebo/transport/ConnectionManager.cc:263
#8 0x00007f323f9431b2 in gazebo::transport::ConnectionManager::Run (this=0x6a32e0)
at /home/scpeters/osrf/gazebo3/gazebo/transport/ConnectionManager.cc:310
#9 0x00007f323ed51ce9 in thread_proxy () from /usr/lib/libboost_thread.so.1.46.1
#10 0x00007f323f6d6e9a in start_thread (arg=0x7f322a04f700) at pthread_create.c:308
#11 0x00007f323decdcbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#12 0x0000000000000000 in ?? ()
(gdb)
Original report (archived issue) by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).
I have seen infrequent seg-faults of the transport_stress test in my overnight builds for a while (about 1 in 50). It occurs during teardown after an otherwise successful test.
Console output:
I've just gotten a backtrace from a source build of gazebo_1.8.5 (9793bc8cc2ee4548b221d35e8c64489b366268ff). The complaint seems to be an increment operator in Publication::RemoveSubscription
Backtrace of all threads: