search

gazebosim / gz-launch

Run and manage programs and plugins.
https://gazebosim.org
Apache License 2.0
10 stars 14 forks source link

Deadlock at startup due to SIGCHILD #6

Open osrf-migration opened 4 years ago

osrf-migration commented 4 years ago

Original report (archived issue) by Addisu Z. Taddese (Bitbucket: azeey, GitHub: azeey).

The original report had attachments: test_launch.ign

I have encountered deadlocks multiple times when launching competition in SubT. I've traced it down to the fact that ManagerPrivate::OnSigChild uses non async-signal-safe functions which could cause deadlocks at startup when child processes exit. I've attached a simple .ign file that reproduces the issue for me. Basically, it starts gazebo and a bunch of sleep commands with random amounts of sleep. The deadlock usually occurs deep inside malloc.

#0  __lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95
#1  0x00007fe0f55d72db in __GI___libc_malloc (bytes=bytes@entry=513) at malloc.c:3063
#2  0x00007fe0f1f26258 in operator new (sz=513) at ../../../../src/libstdc++-v3/libsupc++/new_op.cc:50
#3  0x00007fe0f1fb817d in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::reserve (this=this@entry=0x7ffcfe19afb0, __res=<optimized out>)
    at /build/gcc-8-IFl8q6/gcc-8-8.3.0/build/x86_64-linux-gnu/libstdc++-v3/include/bits/basic_string.h:167
#4  0x00007fe0f1fabdd8 in std::__cxx11::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >::overflow (this=0x7ffcfe19b148, __c=97) at /build/gcc-8-IFl8q6/gcc-8-8.3.0/build/x86_64-linux-gnu/libstdc++-v3/include/ext/new_allocator.h:86
#5  0x00007fe0f1fb65ab in std::basic_streambuf<char, std::char_traits<char> >::xsputn (this=0x7ffcfe19b148, __s=0x563654769b78 "anager.cc:295] ", __n=23) at /build/gcc-8-IFl8q6/gcc-8-8.3.0/build/x86_64-linux-gnu/libstdc++-v3/include/bits/char_traits.h:371
#6  0x00007fe0f1fa6cb4 in std::basic_streambuf<char, std::char_traits<char> >::sputn (__n=23, __s=0x563654769b70 "[Dbg] [Manager.cc:295] ", this=<optimized out>) at /build/gcc-8-IFl8q6/gcc-8-8.3.0/build/x86_64-linux-gnu/libstdc++-v3/include/streambuf:457
#7  std::__ostream_write<char, std::char_traits<char> > (__n=23, __s=0x563654769b70 "[Dbg] [Manager.cc:295] ", __out=...) at /build/gcc-8-IFl8q6/gcc-8-8.3.0/build/x86_64-linux-gnu/libstdc++-v3/include/bits/ostream_insert.h:50
#8  std::__ostream_insert<char, std::char_traits<char> > (__out=..., __s=0x563654769b70 "[Dbg] [Manager.cc:295] ", __n=23) at /build/gcc-8-IFl8q6/gcc-8-8.3.0/build/x86_64-linux-gnu/libstdc++-v3/include/bits/ostream_insert.h:101
#9  0x00007fe0f2656ed7 in std::operator<< <char, std::char_traits<char>, std::allocator<char> > (__str="[Dbg] [Manager.cc:295] ", __os=...) at /usr/include/c++/8/bits/basic_string.h:6323
#10 ignition::common::Logger::Buffer::sync() () at /home/addisu/ws/subt_ign/src/ign-common/src/Console.cc:148
#11 0x00007fe0f1fa65b8 in std::basic_streambuf<char, std::char_traits<char> >::pubsync (this=<optimized out>) at /build/gcc-8-IFl8q6/gcc-8-8.3.0/build/x86_64-linux-gnu/libstdc++-v3/include/streambuf:278
#12 std::ostream::sentry::~sentry (this=0x7ffcfe19b340, __in_chrg=<optimized out>) at /build/gcc-8-IFl8q6/gcc-8-8.3.0/build/x86_64-linux-gnu/libstdc++-v3/include/ostream:460
#13 0x00007fe0f1fa6c7a in std::__ostream_insert<char, std::char_traits<char> > (__out=..., __s=0x5636548e55b0 "[Dbg] [Manager.cc:295] ", __n=<optimized out>)
    at /build/gcc-8-IFl8q6/gcc-8-8.3.0/build/x86_64-linux-gnu/libstdc++-v3/include/bits/ios_base.h:727
#14 0x00007fe0f26577b8 in std::operator<< <char, std::char_traits<char>, std::allocator<char> > (__str="[Dbg] [Manager.cc:295] ", __os=...) at /usr/include/c++/8/bits/basic_string.h:6323
#15 ignition::common::Logger::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int) () at /home/addisu/ws/subt_ign/src/ign-common/src/Console.cc:111
#16 0x00007fe0f2a9d4bc in ignition::launch::IGNITION_LAUNCH_VERSION_NAMESPACE::ManagerPrivate::OnSigChild(int) () at /usr/include/c++/8/bits/basic_string.h:252
#17 <signal handler called>
#18 0x00007fe0f5624b1a in __libc_fork () at ../sysdeps/nptl/fork.c:135
#19 0x00007fe0f2a9c5fb in ignition::launch::IGNITION_LAUNCH_VERSION_NAMESPACE::ManagerPrivate::RunExecutable(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, bool, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) () at /home/addisu/ws/subt_ign/src/ign-launch/src/Manager.cc:401
#20 0x00007fe0f2a9dd97 in ignition::launch::IGNITION_LAUNCH_VERSION_NAMESPACE::ManagerPrivate::ParseExecutables(tinyxml2::XMLElement const*) () at /home/addisu/ws/subt_ign/src/ign-launch/src/Manager.cc:541
#21 0x00007fe0f2a9f36d in ignition::launch::IGNITION_LAUNCH_VERSION_NAMESPACE::ManagerPrivate::ParseConfig(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
    at /home/addisu/ws/subt_ign/src/ign-launch/src/Manager.cc:356
#22 0x00007fe0f2a9f60f in ignition::launch::IGNITION_LAUNCH_VERSION_NAMESPACE::Manager::RunConfig(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () at /usr/include/c++/8/bits/unique_ptr.h:342
#23 0x00007fe0f2a9b6db in run () at /usr/include/c++/8/bits/char_traits.h:287
#24 0x00007fe0f2eaedae in ffi_call_unix64 () from /usr/lib/x86_64-linux-gnu/libffi.so.6
#25 0x00007fe0f2eae71f in ffi_call () from /usr/lib/x86_64-linux-gnu/libffi.so.6
#26 0x00007fe0f30b58f8 in ?? () from /usr/lib/x86_64-linux-gnu/ruby/2.5.0/fiddle.so
#27 0x00007fe0f5ab0b9b in ?? () from /usr/lib/x86_64-linux-gnu/libruby-2.5.so.2.5
#28 0x00007fe0f30b568d in ?? () from /usr/lib/x86_64-linux-gnu/ruby/2.5.0/fiddle.so
#29 0x00007fe0f5ad7289 in ?? () from /usr/lib/x86_64-linux-gnu/libruby-2.5.so.2.5
#30 0x00007fe0f5ae57b3 in ?? () from /usr/lib/x86_64-linux-gnu/libruby-2.5.so.2.5
#31 0x00007fe0f5adbc45 in ?? () from /usr/lib/x86_64-linux-gnu/libruby-2.5.so.2.5
#32 0x00007fe0f5ae1a24 in ?? () from /usr/lib/x86_64-linux-gnu/libruby-2.5.so.2.5
#33 0x00007fe0f59bc0c4 in ?? () from /usr/lib/x86_64-linux-gnu/libruby-2.5.so.2.5
#34 0x00007fe0f59bdf4d in ruby_exec_node () from /usr/lib/x86_64-linux-gnu/libruby-2.5.so.2.5
#35 0x00007fe0f59c042e in ruby_run_node () from /usr/lib/x86_64-linux-gnu/libruby-2.5.so.2.5
#36 0x00005636536798cb in ?? ()
#37 0x00007fe0f5561b97 in __libc_start_main (main=0x563653679880, argc=5, argv=0x7ffcfe19d108, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffcfe19d0f8) at ../csu/libc-start.c:310
#38 0x00005636536798fa in _start ()
/home/addisu/ws/subt_ign/src/ign-sensors - __lll_lock_wait_private
osrf-migration commented 4 years ago

Original comment by Michael Carroll (Bitbucket: Michael Carroll, GitHub: mjcarroll).

osrf-migration commented 4 years ago

Original comment by Michael Carroll (Bitbucket: Michael Carroll, GitHub: mjcarroll).

I have the same behavior with that launch file on my system.