Frequent SEGV when running dE solutions

[gijzelaerr]

at 2011-11-05 13:58:33 Oleg Smirnov reported:

Frequent SEGV when running dE solutions

[gijzelaerr]

Original comment thread migrated from bugzilla

at 2011-11-05 13:58:33 Oleg Smirnov replied:

When running dE solutions with a frequency tile of None (but not 16, funnily) on the A773 data, meqserver dumps core seemingly every other run. Seems like a doubly-free'd memory block problem, again.

0 0x00002b74eff34165 in ___GI_raise (sig=) at

../nptl/sysdeps/unix/sysv/linux/raise.c:64

1 0x00002b74eff36f70 in *__GI_abort () at abort.c:92

2 0x00002b74eff6a27b in __libc_message (do_abort=, fmt=

) at ../sysdeps/unix/sysv/linux/libc_fatal.c:189 #3 0x00002b74eff73ad6 in malloc_printerr (action=3, str=0x2b74f0027b75 "free(): invalid pointer", ptr=) at malloc.c:6267 #4 0x00002b74eff7884c in *__GI___libc_free (mem=) at malloc.c:3739 #5 0x00002b74ede5efb8 in DMI::SmartBlock::destroy (this=0x2b75787a7370) at /home/oms/Timba/DMI/src/SmartBlock.cc:147 #6 0x00002b74ede5effb in ~SmartBlock (this=0x7df1, __in_chrg=) at /home/oms/Timba/DMI/src/SmartBlock.cc:76 #7 0x00002b74eddd3477 in DMI::CountedRefBase::detach (this=0x2b757b475898) at /home/oms/Timba/DMI/src/CountedRefBase.cc:391 #8 0x00002b74edde817a in DMI::NumArray::clear (this=0x2b757b475770) at /home/oms/Timba/DMI/src/NumArray.cc:409 #9 0x00002b74edde9630 in ~NumArray (this=0x7df1, __in_chrg=) at /home/oms/Timba/DMI/src/NumArray.cc:206 #10 0x00002b74ece83e5e in ~Vells (this=0x2b757b475770, __in_chrg=) at /home/oms/Timba/MEQ/src/Vells.cc:163 #11 0x00002b74eddd3477 in DMI::CountedRefBase::detach (this=0x2b7590f9b270) at /home/oms/Timba/DMI/src/CountedRefBase.cc:391 #12 0x00002b74ede6e175 in ~CountedRefBase (this=0x2b757e1c6570) at /home/oms/Timba/DMI/src/CountedRefBase.h:434 #13 ~CountedRef (this=0x2b757e1c6570) at /home/oms/Timba/DMI/src/CountedRef.h:45 #14 ~Element (this=0x2b757e1c6570) at /home/oms/Timba/DMI/src/Vec.h:49 #15 __gnu_cxx::__mt_alloc_baseDMI::Vec::Element::destroy (this=0x2b757e1c6570) at /usr/include/c++/4.4/ext/mt_allocator.h:603 #16 _Destroy > > (this=0x2b757e1c6570) at /usr/include/c++/4.4/bits/stl_construct.h:134 #17 std::vector > >::_M_erase_at_end (this=0x2b757e1c6570) at /usr/include/c++/4.4/bits/stl_vector.h:1150 #18 std::vector > >::clear (this=0x2b757e1c6570) at /usr/include/c++/4.4/bits/stl_vector.h:951 #19 DMI::Vec::clear (this=0x2b757e1c6570) at /home/oms/Timba/DMI/src/Vec.cc:258 #20 0x00002b74ede72936 in ~Vec (this=0x7df1, __in_chrg=) at /home/oms/Timba/DMI/src/Vec.cc:62 #21 0x00002b74eddd3477 in DMI::CountedRefBase::detach (this=0x2b757e917b80) at /home/oms/Timba/DMI/src/CountedRefBase.cc:391 #22 0x0000000000687113 in ~CountedRefBase (this=0x2b757e917b80, __in_chrg=) at /home/oms/Timba/DMI/src/CountedRefBase.h:434 #23 0x00002b74ede58751 in ~CountedRef (this=) at /home/oms/Timba/DMI/src/CountedRef.h:45 #24 ~Field (this=) at /home/oms/Timba/DMI/src/Record.h:71 #25 ~pair (this=) at /usr/include/c++/4.4/bits/stl_pair.h:68 #26 __gnu_cxx::__mt_alloc_base > ::destroy (this=) > at /usr/include/c++/4.4/ext/mt_allocator.h:603 > #27 __gnu_cxx::hashtable, > DMI::HIID, __gnu_cxx::hashDMI::HIID, std::_Select1st const, DMI::Record::Field> >, std::equal_toDMI::HIID, > __gnu_cxx::__mt_alloc __gnu_cxx::__common_pool_policy<__gnu_cxx::__pool, true> > >::_M_delete_node > (this=) > at /usr/include/c++/4.4/backward/hashtable.h:616 > #28 __gnu_cxx::hashtable, > DMI::HIID, __gnu_cxx::hashDMI::HIID, std::_Select1st const, DMI::Record::Field> >, std::equal_toDMI::HIID, > __gnu_cxx::__mt_alloc __gnu_cxx::__common_pool_policy<__gnu_cxx::__pool, true> > >::clear (this= optimized out>) at /usr/include/c++/4.4/backward/hashtable.h:1080 > #29 __gnu_cxx::hash_map __gnu_cxx::hash, std::equal_toDMI::HIID, __gnu_cxx::__mt_alloc Type to continue, or q to quit--- > MI::Record::Field, __gnu_cxx::__common_pool_policy<__gnu_cxx::__pool, true> > > ::clear (this=) > at /usr/include/c++/4.4/ext/hash_map:245 > #30 DMI::Record::clear (this=) at > /home/oms/Timba/DMI/src/Record.cc:62 > #31 0x00002b74ece97592 in Meq::VellSet::clear (this=0x2b757f2aac00) at > /home/oms/Timba/MEQ/src/VellSet.cc:375 > #32 0x00002b74ece9763b in ~VellSet (this=0x7df1, __in_chrg= out>) at /home/oms/Timba/MEQ/src/VellSet.cc:125 > #33 0x00002b74eddd3477 in DMI::CountedRefBase::detach (this=0x2b7580e008c0) at > /home/oms/Timba/DMI/src/CountedRefBase.cc:391 > #34 0x00002b74ede6e175 in ~CountedRefBase (this=0x2b755dfcec80) at > /home/oms/Timba/DMI/src/CountedRefBase.h:434 > #35 ~CountedRef (this=0x2b755dfcec80) at /home/oms/Timba/DMI/src/CountedRef.h:45 > #36 ~Element (this=0x2b755dfcec80) at /home/oms/Timba/DMI/src/Vec.h:49 > #37 __gnu_cxx::__mt_alloc_baseDMI::Vec::Element::destroy (this=0x2b755dfcec80) > at /usr/include/c++/4.4/ext/mt_allocator.h:603 > #38 _Destroy __gnu_cxx::__common_pool_policy<__gnu_cxx::__pool, true> > > > (this=0x2b755dfcec80) at /usr/include/c++/4.4/bits/stl_construct.h:134 > #39 std::vector __gnu_cxx::__common_pool_policy<__gnu_cxx::__pool, true> > >::_M_erase_at_end > (this=0x2b755dfcec80) at /usr/include/c++/4.4/bits/stl_vector.h:1150 > #40 std::vector __gnu_cxx::__common_pool_policy<__gnu_cxx::__pool, true> > >::clear > (this=0x2b755dfcec80) at /usr/include/c++/4.4/bits/stl_vector.h:951 > #41 DMI::Vec::clear (this=0x2b755dfcec80) at /home/oms/Timba/DMI/src/Vec.cc:258 > #42 0x00002b74ede72936 in ~Vec (this=0x7df1, __in_chrg=) at > /home/oms/Timba/DMI/src/Vec.cc:62 > #43 0x00002b74eddd3477 in DMI::CountedRefBase::detach (this=0x2b74fc20b210) at > /home/oms/Timba/DMI/src/CountedRefBase.cc:391 > #44 0x0000000000687113 in ~CountedRefBase (this=0x2b74fc20b210, __in_chrg= optimized out>) > at /home/oms/Timba/DMI/src/CountedRefBase.h:434 > #45 0x00002b74ede5e251 in std::pair DMI::Record::Field>::~pair() () > from /home/oms/Timba/build/release/DMI/libdmi.so > #46 0x00002b74ede57d62 in __gnu_cxx::__mt_alloc_base DMI::Record::Field> >::destroy ( > this=0x2b74fc2db160, __in_chrg=) at > /usr/include/c++/4.4/ext/mt_allocator.h:603 > #47 __gnu_cxx::hashtable, > DMI::HIID, __gnu_cxx::hashDMI::HIID, std::_Select1st const, DMI::Record::Field> >, std::equal_toDMI::HIID, > __gnu_cxx::__mt_alloc __gnu_cxx::__common_pool_policy<__gnu_cxx::__pool, true> > >::_M_delete_node > (this=0x2b74fc2db160, __in_chrg=) > at /usr/include/c++/4.4/backward/hashtable.h:616 > #48 __gnu_cxx::hashtable, > DMI::HIID, __gnu_cxx::hashDMI::HIID, std::_Select1st const, DMI::Record::Field> >, std::equal_toDMI::HIID, > __gnu_cxx::__mt_alloc __gnu_cxx::__common_pool_policy<__gnu_cxx::__pool, true> > >::clear > (this=0x2b74fc2db160, __in_chrg=) > at /usr/include/c++/4.4/backward/hashtable.h:1080 > #49 ~hashtable (this=0x2b74fc2db160, __in_chrg=) at > /usr/include/c++/4.4/backward/hashtable.h:355 > #50 ~hash_map (this=0x2b74fc2db160, __in_chrg=) at > /usr/include/c++/4.4/ext/hash_map:80 > #51 ~Record (this=0x2b74fc2db160, __in_chrg=) at > /home/oms/Timba/DMI/src/Record.cc:57 > #52 0x00002b74ece61368 in ~Result (this=0x2b74fc2db160, __in_chrg= optimized out>) at /home/oms/Timba/MEQ/src/Result.cc:60 > #53 0x00002b74eddd3477 in DMI::CountedRefBase::detach (this=0x2b7508fe9a48) at > /home/oms/Timba/DMI/src/CountedRefBase.cc:391 > #54 0x00002b74ece20f14 in Meq::Node::Cache::clear (this=0x2b7508fe93d0, > recursive=false) at /home/oms/Timba/MEQ/src/Node.h:945 > #55 Meq::Node::clearCache (this=0x2b7508fe93d0, recursive=false) at > /home/oms/Timba/MEQ/src/Node.cc:484 > #56 0x00002b74ece203bf in Meq::Node::getCachedResult (this=0x2b7508fe93d0, > retcode=@0x2b74f5ad86e0, ref=..., req=...) gdb> p this->name_ ... "visibility:4:B" I've left a copy of the cores on cedar:~oms/A773/cores. ##### at 2011-11-05 14:04:16 Oleg Smirnov replied: Another backtrace. This one is assigning a result ref in MatrixMultiply. (gdb) bt #0 *__GI___libc_free (mem=0x2ab800000000) at malloc.c:3710 #1 0x00002ab87dda1fb8 in DMI::SmartBlock::destroy (this=0x2ab8ee0f5a20) at /home/oms/Timba/DMI/src/SmartBlock.cc:147 #2 0x00002ab87dda1ffb in ~SmartBlock (this=0x2ab800000000, __in_chrg=) at /home/oms/Timba/DMI/src/SmartBlock.cc:76 #3 0x00002ab87dd16477 in DMI::CountedRefBase::detach (this=0x2ab8ec2357c8) at /home/oms/Timba/DMI/src/CountedRefBase.cc:391 #4 0x00002ab87dd2b17a in DMI::NumArray::clear (this=0x2ab8ec2356a0) at /home/oms/Timba/DMI/src/NumArray.cc:409 #5 0x00002ab87dd2c630 in ~NumArray (this=0x2ab800000000, __in_chrg=) at /home/oms/Timba/DMI/src/NumArray.cc:206 #6 0x00002ab87cdc6e5e in ~Vells (this=0x2ab8ec2356a0, __in_chrg=) at /home/oms/Timba/MEQ/src/Vells.cc:163 #7 0x00002ab87dd16477 in DMI::CountedRefBase::detach (this=0x2ab900393a70) at /home/oms/Timba/DMI/src/CountedRefBase.cc:391 #8 0x0000000000687113 in ~CountedRefBase (this=0x2ab900393a70, __in_chrg=) at /home/oms/Timba/DMI/src/CountedRefBase.h:434 #9 0x00002ab87dd9b751 in ~CountedRef (this=) at /home/oms/Timba/DMI/src/CountedRef.h:45 #10 ~Field (this=) at /home/oms/Timba/DMI/src/Record.h:71 #11 ~pair (this=) at /usr/include/c++/4.4/bits/stl_pair.h:68 #12 __gnu_cxx::__mt_alloc_base > ::destroy (this=) > at /usr/include/c++/4.4/ext/mt_allocator.h:603 > #13 __gnu_cxx::hashtable, > DMI::HIID, __gnu_cxx::hashDMI::HIID, std::_Select1st const, DMI::Record::Field> >, std::equal_toDMI::HIID, > __gnu_cxx::__mt_alloc __gnu_cxx::__common_pool_policy<__gnu_cxx::__pool, true> > >::_M_delete_node > (this=) > at /usr/include/c++/4.4/backward/hashtable.h:616 > #14 __gnu_cxx::hashtable, > DMI::HIID, __gnu_cxx::hashDMI::HIID, std::_Select1st const, DMI::Record::Field> >, std::equal_toDMI::HIID, > __gnu_cxx::__mt_alloc __gnu_cxx::__common_pool_policy<__gnu_cxx::__pool, true> > >::clear (this= optimized out>) at /usr/include/c++/4.4/backward/hashtable.h:1080 > #15 __gnu_cxx::hash_map __gnu_cxx::hash, std::equal_toDMI::HIID, > __gnu_cxx::__mt_alloc __gnu_cxx::__common_pool_policy<__gnu_cxx::__pool, true> > >::clear (this= optimized out>) > at /usr/include/c++/4.4/ext/hash_map:245 > #16 DMI::Record::clear (this=) at > /home/oms/Timba/DMI/src/Record.cc:62 > #17 0x00002ab87cdda592 in Meq::VellSet::clear (this=0x2ab8efb923d0) at > /home/oms/Timba/MEQ/src/VellSet.cc:375 > #18 0x00002ab87cdda63b in ~VellSet (this=0x2ab800000000, __in_chrg= optimized out>) at /home/oms/Timba/MEQ/src/VellSet.cc:125 > #19 0x00002ab87dd16477 in DMI::CountedRefBase::detach (this=0x2ab8ed184cb0) at > /home/oms/Timba/DMI/src/CountedRefBase.cc:391 > #20 0x00002ab87ddb1175 in ~CountedRefBase (this=0x2ab8ed330760) at > /home/oms/Timba/DMI/src/CountedRefBase.h:434 > #21 ~CountedRef (this=0x2ab8ed330760) at /home/oms/Timba/DMI/src/CountedRef.h:45 > #22 ~Element (this=0x2ab8ed330760) at /home/oms/Timba/DMI/src/Vec.h:49 > #23 __gnu_cxx::__mt_alloc_baseDMI::Vec::Element::destroy (this=0x2ab8ed330760) > at /usr/include/c++/4.4/ext/mt_allocator.h:603 > #24 _Destroy __gnu_cxx::__common_pool_policy<__gnu_cxx::__pool, true> > > > (this=0x2ab8ed330760) at /usr/include/c++/4.4/bits/stl_construct.h:134 > #25 std::vector __gnu_cxx::__common_pool_policy<__gnu_cxx::__pool, true> > >::_M_erase_at_end > (this=0x2ab8ed330760) at /usr/include/c++/4.4/bits/stl_vector.h:1150 > #26 std::vector __gnu_cxx::__common_pool_policy<__gnu_cxx::__pool, true> > >::clear > (this=0x2ab8ed330760) at /usr/include/c++/4.4/bits/stl_vector.h:951 > #27 DMI::Vec::clear (this=0x2ab8ed330760) at /home/oms/Timba/DMI/src/Vec.cc:258 > ---Type to continue, or q to quit--- > #28 0x00002ab87ddb5936 in ~Vec (this=0x2ab800000000, __in_chrg= out>) at /home/oms/Timba/DMI/src/Vec.cc:62 > #29 0x00002ab87dd16477 in DMI::CountedRefBase::detach (this=0x2ab90c30f9d0) at > /home/oms/Timba/DMI/src/CountedRefBase.cc:391 > #30 0x0000000000687113 in ~CountedRefBase (this=0x2ab90c30f9d0, __in_chrg= optimized out>) > at /home/oms/Timba/DMI/src/CountedRefBase.h:434 > #31 0x00002ab87dda1251 in std::pair DMI::Record::Field>::~pair() () > from /home/oms/Timba/build/release/DMI/libdmi.so > #32 0x00002ab87dd9ad62 in __gnu_cxx::__mt_alloc_base DMI::Record::Field> >::destroy ( > this=0x2ab8edb9bca0, __in_chrg=) at > /usr/include/c++/4.4/ext/mt_allocator.h:603 > #33 __gnu_cxx::hashtable, > DMI::HIID, __gnu_cxx::hashDMI::HIID, std::_Select1st const, DMI::Record::Field> >, std::equal_toDMI::HIID, > __gnu_cxx::__mt_alloc __gnu_cxx::__common_pool_policy<__gnu_cxx::__pool, true> > >::_M_delete_node > (this=0x2ab8edb9bca0, __in_chrg=) > at /usr/include/c++/4.4/backward/hashtable.h:616 > #34 __gnu_cxx::hashtable, > DMI::HIID, __gnu_cxx::hashDMI::HIID, std::_Select1st const, DMI::Record::Field> >, std::equal_toDMI::HIID, > __gnu_cxx::__mt_alloc __gnu_cxx::__common_pool_policy<__gnu_cxx::__pool, true> > >::clear > (this=0x2ab8edb9bca0, __in_chrg=) > at /usr/include/c++/4.4/backward/hashtable.h:1080 > #35 ~hashtable (this=0x2ab8edb9bca0, __in_chrg=) at > /usr/include/c++/4.4/backward/hashtable.h:355 > #36 ~hash_map (this=0x2ab8edb9bca0, __in_chrg=) at > /usr/include/c++/4.4/ext/hash_map:80 > #37 ~Record (this=0x2ab8edb9bca0, __in_chrg=) at > /home/oms/Timba/DMI/src/Record.cc:57 > #38 0x00002ab87cda4368 in ~Result (this=0x2ab8edb9bca0, __in_chrg= optimized out>) at /home/oms/Timba/MEQ/src/Result.cc:60 > #39 0x00002ab87dd16477 in DMI::CountedRefBase::detach (this=0x2ab8ec516408) at > /home/oms/Timba/DMI/src/CountedRefBase.cc:391 > #40 0x00002ab87dd16db7 in DMI::CountedRefBase::copy (this=0x2ab800000000, > other=..., flags=0, depth=-1) > at /home/oms/Timba/DMI/src/CountedRefBase.cc:130 > #41 0x00002ab87be07e39 in DMI::CountedRefBase::operator= (this= out>, resref=, > childres=, request=) at > /home/oms/Timba/DMI/src/CountedRefBase.h:443 > #42 operator= (this=, resref=, > childres=, > request=) at /home/oms/Timba/DMI/src/CountedRef.h:45 > #43 Meq::MatrixMultiply::getResult (this=, resref= optimized out>, childres=, > request=) at > /home/oms/Timba/MeqNodes/src/MatrixMultiply.cc:409 > #44 0x00002ab87cd71858 in Meq::Node::execute (this=0x63f5810, ref= optimized out>, req=..., depth=) > at /home/oms/Timba/MEQ/src/Node.cc:1073 > #45 0x00002ab87cd863dd in Meq::NodeNursery::syncPoll (this=0x41abfa8, resref= > , childres=..., > req=, depth=) at > /home/oms/Timba/MEQ/src/NodeNursery.cc:555 > #46 0x00002ab87cd6392b in Meq::Node::pollChildren (this=0x41abe50, resref=..., > childres=..., req=...) > at /home/oms/Timba/MEQ/src/Node.cc:899 > #47 0x00002ab87cd71719 in Meq::Node::execute (this=0x41abe50, ref= optimized out>, req=..., depth=) > at /home/oms/Timba/MEQ/src/Node.cc:1025 > #48 0x00002ab87cd863dd in Meq::NodeNursery::syncPoll (this=0x3f79ba8, resref= > , childres=..., > req=, depth=) at > /home/oms/Timba/MEQ/src/NodeNursery.cc:555 > #49 0x00002ab87cd6392b in Meq::Node::pollChildren (this=0x3f79a50, resref=..., > childres=..., req=...) > at /home/oms/Timba/MEQ/src/Node.cc:899 > #50 0x00002ab87cd71719 in Meq::Node::execute (this=0x3f79a50, ref= optimized out>, req=..., depth=) > ---Type to continue, or q to quit---q > at /hQuit > (gdb) up 44 > #44 0x00002ab87cd71858 in Meq::Node::execute (this=0x63f5810, ref= optimized out>, req=..., depth=) > at /home/oms/Timba/MEQ/src/Node.cc:1073 > 1073 int code = getResult(ref,child_results_,req,new_request_); > (gdb) p this->name_ > $1 = {static npos = 18446744073709551615, > _M_dataplus = {std::allocator = {<__gnu_cxx::new_allocator> = > {}, }, > _M_p = 0x63f5608 "sky:E-cc:0:8"}} > (gdb) ##### at 2011-11-05 14:07:18 Oleg Smirnov replied: And another. Looks like it falls over at pretty random spots, so definitely someone somewhere is doing a bit of memory hanky-panky. Grrrr! (gdb) bt #0 ~_Vector_base (this=0x2ac8cc13d2f0, __in_chrg=) at /usr/include/c++/4.4/bits/stl_vector.h:132 #1 ~vector (this=0x2ac8cc13d2f0, __in_chrg=) at /usr/include/c++/4.4/bits/stl_vector.h:313 #2 ~HIID (this=0x2ac8cc13d2f0, __in_chrg=) at /home/oms/Timba/DMI/src/HIID.h:46 #3 ~Subscription (this=0x2ac8cc13d2f0, __in_chrg=) at /home/oms/Timba/DMI/src/Container.h:566 #4 __gnu_cxx::new_allocatorDMI::Container::Subscription::destroy (this=0x2ac8cc13d2f0, __in_chrg=) at /usr/include/c++/4.4/ext/new_allocator.h:115 #5 std::_List_base >::_M_clear (this=0x2ac8cc13d2f0, __in_chrg=) at /usr/include/c++/4.4/bits/list.tcc:76 #6 ~_List_base (this=0x2ac8cc13d2f0, __in_chrg=) at /usr/include/c++/4.4/bits/stl_list.h:360 #7 ~list (this=0x2ac8cc13d2f0, __in_chrg=) at /usr/include/c++/4.4/bits/stl_list.h:418 #8 ~Container (this=0x2ac8cc13d2f0, __in_chrg=) at /home/oms/Timba/DMI/src/Container.h:115 #9 ~Record (this=0x2ac8cc13d2f0, __in_chrg=) at /home/oms/Timba/DMI/src/Record.cc:57 #10 0x00002ac83e2cc477 in DMI::CountedRefBase::detach (this=0x2ac8cd588f30) at /home/oms/Timba/DMI/src/CountedRefBase.cc:391 #11 0x0000000000687113 in ~CountedRefBase (this=0x2ac8cd588f30, __in_chrg=) at /home/oms/Timba/DMI/src/CountedRefBase.h:434 #12 0x00002ac83e357251 in std::pair::~pair() () from /home/oms/Timba/build/release/DMI/libdmi.so #13 0x00002ac83e350d62 in __gnu_cxx::__mt_alloc_base >::destroy ( this=0x2ac8cce9eca0, __in_chrg=) at /usr/include/c++/4.4/ext/mt_allocator.h:603 #14 __gnu_cxx::hashtable, DMI::HIID, __gnu_cxx::hashDMI::HIID, std::_Select1st >, std::equal_toDMI::HIID, __gnu_cxx::__mt_alloc > >::_M_delete_node (this=0x2ac8cce9eca0, __in_chrg=) at /usr/include/c++/4.4/backward/hashtable.h:616 #15 __gnu_cxx::hashtable, DMI::HIID, __gnu_cxx::hashDMI::HIID, std::_Select1st >, std::equal_toDMI::HIID, __gnu_cxx::__mt_alloc > >::clear (this=0x2ac8cce9eca0, __in_chrg=) at /usr/include/c++/4.4/backward/hashtable.h:1080 #16 ~hashtable (this=0x2ac8cce9eca0, __in_chrg=) at /usr/include/c++/4.4/backward/hashtable.h:355 #17 ~hash_map (this=0x2ac8cce9eca0, __in_chrg=) at /usr/include/c++/4.4/ext/hash_map:80 #18 ~Record (this=0x2ac8cce9eca0, __in_chrg=) at /home/oms/Timba/DMI/src/Record.cc:57 #19 0x00002ac83d2b8e27 in ~Cells (this=0x2ac8cce9eca0, __in_chrg=) at /home/oms/Timba/MEQ/src/Cells.cc:44 #20 0x00002ac83e2cc477 in DMI::CountedRefBase::detach (this=0x2ac8cca383c0) at /home/oms/Timba/DMI/src/CountedRefBase.cc:391 #21 0x0000000000687113 in ~CountedRefBase (this=0x2ac8cca383c0, __in_chrg=) at /home/oms/Timba/DMI/src/CountedRefBase.h:434 #22 0x00002ac83e357251 in std::pair::~pair() () from /home/oms/Timba/build/release/DMI/libdmi.so #23 0x00002ac83e350d62 in __gnu_cxx::__mt_alloc_base >::destroy ( this=0x2ac8cd5713a0, __in_chrg=) at /usr/include/c++/4.4/ext/mt_allocator.h:603 #24 __gnu_cxx::hashtable, DMI::HIID, __gnu_cxx::hashDMI::HIID, std::_Select1st >, std::equal_toDMI::HIID, __gnu_cxx::__mt_alloc > >::_M_delete_node (this=0x2ac8cd5713a0, __in_chrg=) at /usr/include/c++/4.4/backward/hashtable.h:616 ---Type to continue, or q to quit--- #25 __gnu_cxx::hashtable, DMI::HIID, __gnu_cxx::hashDMI::HIID, std::_Select1st >, std::equal_toDMI::HIID, __gnu_cxx::__mt_alloc > >::clear (this=0x2ac8cd5713a0, __in_chrg=) at /usr/include/c++/4.4/backward/hashtable.h:1080 #26 ~hashtable (this=0x2ac8cd5713a0, __in_chrg=) at /usr/include/c++/4.4/backward/hashtable.h:355 #27 ~hash_map (this=0x2ac8cd5713a0, __in_chrg=) at /usr/include/c++/4.4/ext/hash_map:80 #28 ~Record (this=0x2ac8cd5713a0, __in_chrg=) at /home/oms/Timba/DMI/src/Record.cc:57 #29 0x00002ac83d35a368 in ~Result (this=0x2ac8cd5713a0, __in_chrg=) at /home/oms/Timba/MEQ/src/Result.cc:60 #30 0x00002ac83e2cc477 in DMI::CountedRefBase::detach (this=0x2ac852386fb8) at /home/oms/Timba/DMI/src/CountedRefBase.cc:391 #31 0x00002ac83d319f14 in Meq::Node::Cache::clear (this=0x2ac852386940, recursive=false) at /home/oms/Timba/MEQ/src/Node.h:945 #32 Meq::Node::clearCache (this=0x2ac852386940, recursive=false) at /home/oms/Timba/MEQ/src/Node.cc:484 #33 0x00002ac83d3193bf in Meq::Node::getCachedResult (this=0x2ac852386940, retcode=@0x2ac844fcd9a0, ref=..., req=...) at /home/oms/Timba/MEQ/src/Node.cc:594 #34 0x00002ac83d327230 in Meq::Node::execute (this=0x2ac852386940, ref=, req=..., depth=) at /home/oms/Timba/MEQ/src/Node.cc:971 #35 0x00002ac83d33c3dd in Meq::NodeNursery::syncPoll (this=0x2ac85e982548, resref=, childres=..., req=, depth=) at /home/oms/Timba/MEQ/src/NodeNursery.cc:555 #36 0x00002ac83d31992b in Meq::Node::pollChildren (this=0x2ac85e9823f0, resref=..., childres=..., req=...) at /home/oms/Timba/MEQ/src/Node.cc:899 #37 0x00002ac83d327719 in Meq::Node::execute (this=0x2ac85e9823f0, ref=, req=..., depth=) at /home/oms/Timba/MEQ/src/Node.cc:1025 #38 0x00002ac83d33c3dd in Meq::NodeNursery::syncPoll (this=0x2ac85312fab8, resref=, childres=..., req=, depth=) at /home/oms/Timba/MEQ/src/NodeNursery.cc:555 #39 0x00002ac83d31992b in Meq::Node::pollChildren (this=0x2ac85312f960, resref=..., childres=..., req=...) at /home/oms/Timba/MEQ/src/Node.cc:899 #40 0x00002ac83d327719 in Meq::Node::execute (this=0x2ac85312f960, ref=, req=..., depth=) at /home/oms/Timba/MEQ/src/Node.cc:1025 #41 0x00002ac83d33c3dd in Meq::NodeNursery::syncPoll (this=0x2ac85c5bbea8, resref=, childres=..., req=, depth=) at /home/oms/Timba/MEQ/src/NodeNursery.cc:555 #42 0x00002ac83d31992b in Meq::Node::pollChildren (this=0x2ac85c5bbd50, resref=..., childres=..., req=...) at /home/oms/Timba/MEQ/src/Node.cc:899 #43 0x00002ac83d327719 in Meq::Node::execute (this=0x2ac85c5bbd50, ref=, req=..., depth=) at /home/oms/Timba/MEQ/src/Node.cc:1025 #44 0x00002ac83d33c3dd in Meq::NodeNursery::syncPoll (this=0x2ac85e109c68, resref=, childres=..., req=, depth=) at /home/oms/Timba/MEQ/src/NodeNursery.cc:555 #45 0x00002ac83d31992b in Meq::Node::pollChildren (this=0x2ac85e109b10, resref=..., childres=..., req=...) at /home/oms/Timba/MEQ/src/Node.cc:899 #46 0x00002ac83d327719 in Meq::Node::execute (this=0x2ac85e109b10, ref=, req=..., depth=) at /home/oms/Timba/MEQ/src/Node.cc:1025 #47 0x00002ac83d33c3dd in Meq::NodeNursery::syncPoll (this=0x2ac85e9f6f08, resref=, childres=..., req=, depth=) at /home/oms/Timba/MEQ/src/NodeNursery.cc:555 #48 0x00002ac83d31992b in Meq::Node::pollChildren (this=0x2ac85e9f6db0, resref=..., childres=..., req=...) ---Type to continue, or q to quit---q at /home/oms/Timba/MEQ/srQuit (gdb) up 33 #33 0x00002ac83d3193bf in Meq::Node::getCachedResult (this=0x2ac852386940, retcode=@0x2ac844fcd9a0, ref=..., req=...) at /home/oms/Timba/MEQ/src/Node.cc:594 594 clearCache(false); (gdb) p this->name_ $1 = {static npos = 18446744073709551615, _M_dataplus = {std::allocator = {<__gnu_cxx::new_allocator> = {}, }, _M_p = 0x2ac852386738 "coh:E-cc:3:9"}} (gdb) ##### at 2011-11-05 17:15:00 Oleg Smirnov replied: SEGV also shows up when changing the time tile to None. This was preceded by an error message: "no perturbed set 0 for spid xxxxxx". Very mysterious. Will try to change time tile to 30 and overall tile to 60. Does this only happen when one subtile is None and the other isn't? ##### at 2011-11-05 21:10:06 Oleg Smirnov replied: Also falls over on dE solution with no subtiling at all (tile size 10). It is definitely significant that it's only dE solutions that crash, not G -- and also that it never happens when dE is subtiled along BOTH axes. I just can't figure out right now what it is that's so different about this scenario! ##### at 2011-11-07 15:30:06 Oleg Smirnov replied: On a hunch, I added mutex locking (for each VellSet and each perturbed_values vector) in TensorFunctionPert, but that didn't help, and only caused the occasional deadlock (see r8559). My next suspect is the ComposedPolc, since this behaviour is so tightly coupled to the subtiling setting.