search

cms-sw / cmssw

CMS Offline Software
http://cms-sw.github.io/
Apache License 2.0
1.08k stars 4.3k forks source link

Fatal system signal has occurred during exit #32045

Closed makortel closed 3 years ago

makortel commented 3 years ago

Occasionally random jobs in IB RelVals fail with

Fatal system signal has occurred during exit

at the very end of the log without any further hint of what is causing the problem. This purpose of this issue is to collect their occurrences, and to discuss how to debug it and what could be causing the problem.

makortel commented 3 years ago

assign core

cmsbuild commented 3 years ago

New categories assigned: core

@Dr15Jones,@smuzaffar,@makortel you have been requested to review this Pull request/Issue and eventually sign? Thanks

cmsbuild commented 3 years ago

A new Issue was created by @makortel Matti Kortelainen.

@Dr15Jones, @dpiparo, @silviodonato, @smuzaffar, @makortel, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

makortel commented 3 years ago

Occurred in 250407.17 step 2 in slc7_amd64_gcc10 IB CMSSW_11_2_X_2020-11-05-2300

dropped waiting message count 0
06-Nov-2020 07:35:58 CETClosed file

Fatal system signal has occurred during exit

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc10/CMSSW_11_2_X_2020-11-05-2300/pyRelValMatrixLogs/run/250407.17_NuGun_UP17+FS_NuGun_UP17_PRMXUP17_PU50+HARVESTUP17FS+MINIAODMCUP17FS/step2_NuGun_UP17+FS_NuGun_UP17_PRMXUP17_PU50+HARVESTUP17FS+MINIAODMCUP17FS.log#/

makortel commented 3 years ago

@hufnagel reported "high rate of job failures" with the same error at T3_US_TACC in https://hypernews.cern.ch/HyperNews/CMS/get/edmFramework/3902.html with CMSSW_9_3_18 and CMSSW_9_4_7 (slc6_amd64_gcc630). The HN thread did not end up in any conclusion (yet). Some of the failed jobs did give stack traces:

Thread 1 (Thread 0x2b03cb67af80 (LWP 9154)):
#0  0x00002b03c9ba0403 in poll () from /lib64/libc.so.6
#1  0x00002b03d2633554 in full_read.constprop () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#2  0x00002b03d2633d2a in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  0x00002b03d2634d85 in sig_dostack_then_abort () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00002b03c8572814 in arena_run_heap_remove (ph=<optimized out>, phn=0x2b061ed2cd88) at src/arena.c:114
#6  0x00002b03c8574742 in arena_avail_remove (npages=0, pageind=1909, chunk=0x2b061ed00000, arena=0x2b03cb7e7580) at src/arena.c:211
#7  arena_run_split_remove (arena=arena@entry=0x2b03cb7e7580, chunk=chunk@entry=0x2b061ed00000, run_ind=run_ind@entry=1909, flag_dirty=flag_dirty@entry=0, flag_decommitted=0, need_pages=need_pages@entry=0) at src/arena.c:398
#8  0x00002b03c8575f8c in arena_run_split_large_helper (zero=false, remove=true, size=0, run=0x2b061ed2cdb0, arena=<optimized out>) at src/arena.c:447
#9  arena_run_split_large (zero=false, size=0, run=0x2b061ed2cdb0, arena=<optimized out>) at src/arena.c:502
#10 arena_stash_dirty (purge_chunks_sentinel=0x7ffe3ccf67d0, purge_runs_sentinel=0x7ffe3ccf6780, ndirty_limit=1553, chunk_hooks=0x7ffe3ccf6790, arena=0x2b03cb7e7580, tsdn=0x2b03cb675b40) at src/arena.c:1660
#11 arena_purge_to_limit (tsdn=tsdn@entry=0x2b03cb675b40, arena=arena@entry=0x2b03cb7e7580, ndirty_limit=1553) at src/arena.c:1859
#12 0x00002b03c857533b in arena_maybe_purge_ratio (arena=0x2b03cb7e7580, tsdn=0x2b03cb675b40) at src/arena.c:1492
#13 je_arena_maybe_purge (tsdn=0x2b03cb675b40, arena=0x2b03cb7e7580) at src/arena.c:1549
#14 0x00002b03c857b034 in arena_dalloc_bin_run (chunk=<optimized out>, bin=0x2b03cb7e8248, run=<optimized out>, arena=0x2b03cb7e7580, tsdn=0x2b03cb675b40) at src/arena.c:2912
#15 arena_dalloc_bin_locked_impl (bitselm=<optimized out>, junked=true, ptr=<optimized out>, chunk=<optimized out>, arena=0x2b03cb7e7580, tsdn=0x2b03cb675b40) at src/arena.c:2966
#16 je_arena_dalloc_bin_junked_locked (tsdn=tsdn@entry=0x2b03cb675b40, arena=arena@entry=0x2b03cb7e7580, chunk=<optimized out>, ptr=<optimized out>, bitselm=<optimized out>) at src/arena.c:2981
#17 0x00002b03c85ac784 in je_tcache_bin_flush_small (tsd=0x2b03cb675b40, tcache=<optimized out>, tbin=0x2b03cb7020c8, binind=<optimized out>, rem=23) at src/tcache.c:137
#18 0x00002b03c85acc0c in je_tcache_event_hard (tsd=<optimized out>, tcache=0x2b03cb702000) at src/tcache.c:48
#19 0x00002b03c75d1f18 in std::_Rb_tree<int, std::pair<int const, int>, std::_Select1st<std::pair<int const, int> >, std::less<int>, std::allocator<std::pair<int const, int> > >::_M_erase(std::_Rb_tree_node<std::pair<int const, int> >*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/libDataFormatsProvenance.so
#20 0x00002b03c75d1f0c in std::_Rb_tree<int, std::pair<int const, int>, std::_Select1st<std::pair<int const, int> >, std::less<int>, std::allocator<std::pair<int const, int> > >::_M_erase(std::_Rb_tree_node<std::pair<int const, int> >*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/libDataFormatsProvenance.so
#21 0x00002b040050c0bd in std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > > >, std::_Select1st<std::pair<unsigned int const, std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > > > >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > > > > >::_M_erase(std::_Rb_tree_node<std::pair<unsigned int const, std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > > > >*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/pluginSimGeneralDataMixingModulePlugins.so
#22 0x00002b040050c0ac in std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > > >, std::_Select1st<std::pair<unsigned int const, std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > > > >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > > > > >::_M_erase(std::_Rb_tree_node<std::pair<unsigned int const, std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > > > >*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/pluginSimGeneralDataMixingModulePlugins.so
#23 0x00002b040050c0ac in std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > > >, std::_Select1st<std::pair<unsigned int const, std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > > > >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > > > > >::_M_erase(std::_Rb_tree_node<std::pair<unsigned int const, std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > > > >*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/pluginSimGeneralDataMixingModulePlugins.so
#24 0x00002b040050c0ac in std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > > >, std::_Select1st<std::pair<unsigned int const, std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > > > >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > > > > >::_M_erase(std::_Rb_tree_node<std::pair<unsigned int const, std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > > > >*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/pluginSimGeneralDataMixingModulePlugins.so
#25 0x00002b040050c0ac in std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > > >, std::_Select1st<std::pair<unsigned int const, std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > > > >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > > > > >::_M_erase(std::_Rb_tree_node<std::pair<unsigned int const, std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > > > >*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/pluginSimGeneralDataMixingModulePlugins.so
#26 0x00002b040050c0ac in std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > > >, std::_Select1st<std::pair<unsigned int const, std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > > > >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > > > > >::_M_erase(std::_Rb_tree_node<std::pair<unsigned int const, std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > > > >*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/pluginSimGeneralDataMixingModulePlugins.so
#27 0x00002b040050c0ac in std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > > >, std::_Select1st<std::pair<unsigned int const, std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > > > >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > > > > >::_M_erase(std::_Rb_tree_node<std::pair<unsigned int const, std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > > > >*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/pluginSimGeneralDataMixingModulePlugins.so
#28 0x00002b040050c0ac in std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > > >, std::_Select1st<std::pair<unsigned int const, std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > > > >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > > > > >::_M_erase(std::_Rb_tree_node<std::pair<unsigned int const, std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > > > >*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/pluginSimGeneralDataMixingModulePlugins.so
#29 0x00002b040050c0ac in std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > > >, std::_Select1st<std::pair<unsigned int const, std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > > > >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > > > > >::_M_erase(std::_Rb_tree_node<std::pair<unsigned int const, std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > > > >*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/pluginSimGeneralDataMixingModulePlugins.so
#30 0x00002b04005012b9 in edm::DataMixingSiPixelMCDigiWorker::~DataMixingSiPixelMCDigiWorker() () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/pluginSimGeneralDataMixingModulePlugins.so
#31 0x00002b04005013d9 in edm::DataMixingSiPixelMCDigiWorker::~DataMixingSiPixelMCDigiWorker() () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/pluginSimGeneralDataMixingModulePlugins.so
#32 0x00002b04004ede0d in edm::DataMixingModule::~DataMixingModule() () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/pluginSimGeneralDataMixingModulePlugins.so
#33 0x00002b04004ee279 in edm::DataMixingModule::~DataMixingModule() () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/pluginSimGeneralDataMixingModulePlugins.so
#34 0x00002b03c74beb36 in edm::stream::ProducingModuleAdaptorBase<edm::stream::EDProducerBase>::~ProducingModuleAdaptorBase() () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#35 0x00002b040054ee58 in edm::stream::ProducingModuleAdaptor<edm::DataMixingModule, edm::stream::EDProducerBase, edm::stream::EDProducerAdaptorBase>::~ProducingModuleAdaptor() () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/pluginSimGeneralDataMixingModulePlugins.so
#36 0x00002b03d7fe74d1 in std::_Sp_counted_ptr_inplace<edm::maker::ModuleHolderT<edm::stream::EDProducerAdaptorBase>, std::allocator<edm::maker::ModuleHolderT<edm::stream::EDProducerAdaptorBase> >, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/pluginRecoLocalMuonRPCRecHit.so
#37 0x00002b03c741eb4a in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#38 0x00002b03c741eaed in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#39 0x00002b03c741eaed in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#40 0x00002b03c741eaed in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#41 0x00002b03c741eaed in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#42 0x00002b03c741eaed in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#43 0x00002b03c741eaed in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#44 0x00002b03c741eaed in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#45 0x00002b03c741eaed in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#46 0x00002b03c741eaed in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#47 0x00002b03c741eb96 in std::_Sp_counted_ptr<edm::ModuleRegistry*, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#48 0x0000000000411d26 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() ()
#49 0x00002b03c743d171 in std::default_delete<edm::Schedule>::operator()(edm::Schedule*) const [clone .isra.816] () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#50 0x00002b03c7440b97 in edm::EventProcessor::~EventProcessor() () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#51 0x000000000040d953 in (anonymous namespace)::EventProcessorWithSentry::~EventProcessorWithSentry() ()
#52 0x000000000040d1b7 in main ()
Thread 1 (Thread 0x2aaaaf0c4f80 (LWP 66)):
#0  0x00002aaad56d219a in clang::CXXDestructorDecl::Create(clang::ASTContext&, clang::CXXRecordDecl*, clang::SourceLocation, clang::DeclarationNameInfo const&, clang::QualType, clang::TypeSourceInfo*, bool, bool) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#1  0x00002aaad55857fe in clang::TemplateDeclInstantiator::VisitCXXMethodDecl(clang::CXXMethodDecl*, clang::TemplateParameterList*, bool) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#2  0x00002aaad555919e in clang::Sema::InstantiateClass(clang::SourceLocation, clang::CXXRecordDecl*, clang::CXXRecordDecl*, clang::MultiLevelTemplateArgumentList const&, clang::TemplateSpecializationKind, bool) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#3  0x00002aaad55704a0 in clang::Sema::InstantiateClassTemplateSpecialization(clang::SourceLocation, clang::ClassTemplateSpecializationDecl*, clang::TemplateSpecializationKind, bool) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#4  0x00002aaad559dd04 in clang::Sema::RequireCompleteTypeImpl(clang::SourceLocation, clang::QualType, clang::Sema::TypeDiagnoser*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#5  0x00002aaad559df55 in clang::Sema::RequireCompleteType(clang::SourceLocation, clang::QualType, clang::Sema::TypeDiagnoser&) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#6  0x00002aaad52dfcb3 in clang::Sema::CheckBaseSpecifier(clang::CXXRecordDecl*, clang::SourceRange, bool, clang::AccessSpecifier, clang::TypeSourceInfo*, clang::SourceLocation) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#7  0x00002aaad5558930 in clang::Sema::SubstBaseSpecifiers(clang::CXXRecordDecl*, clang::CXXRecordDecl*, clang::MultiLevelTemplateArgumentList const&) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#8  0x00002aaad5559009 in clang::Sema::InstantiateClass(clang::SourceLocation, clang::CXXRecordDecl*, clang::CXXRecordDecl*, clang::MultiLevelTemplateArgumentList const&, clang::TemplateSpecializationKind, bool) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#9  0x00002aaad55704a0 in clang::Sema::InstantiateClassTemplateSpecialization(clang::SourceLocation, clang::ClassTemplateSpecializationDecl*, clang::TemplateSpecializationKind, bool) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#10 0x00002aaad559dd04 in clang::Sema::RequireCompleteTypeImpl(clang::SourceLocation, clang::QualType, clang::Sema::TypeDiagnoser*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#11 0x00002aaad559df55 in clang::Sema::RequireCompleteType(clang::SourceLocation, clang::QualType, clang::Sema::TypeDiagnoser&) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#12 0x00002aaad52dfcb3 in clang::Sema::CheckBaseSpecifier(clang::CXXRecordDecl*, clang::SourceRange, bool, clang::AccessSpecifier, clang::TypeSourceInfo*, clang::SourceLocation) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#13 0x00002aaad5558930 in clang::Sema::SubstBaseSpecifiers(clang::CXXRecordDecl*, clang::CXXRecordDecl*, clang::MultiLevelTemplateArgumentList const&) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#14 0x00002aaad5559009 in clang::Sema::InstantiateClass(clang::SourceLocation, clang::CXXRecordDecl*, clang::CXXRecordDecl*, clang::MultiLevelTemplateArgumentList const&, clang::TemplateSpecializationKind, bool) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#15 0x00002aaad559dc5e in clang::Sema::RequireCompleteTypeImpl(clang::SourceLocation, clang::QualType, clang::Sema::TypeDiagnoser*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#16 0x00002aaad559df55 in clang::Sema::RequireCompleteType(clang::SourceLocation, clang::QualType, clang::Sema::TypeDiagnoser&) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#17 0x00002aaad559e07b in clang::Sema::RequireCompleteType(clang::SourceLocation, clang::QualType, unsigned int) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#18 0x00002aaad525ff8e in clang::Sema::CheckFieldDecl(clang::DeclarationName, clang::QualType, clang::TypeSourceInfo*, clang::RecordDecl*, clang::SourceLocation, bool, clang::Expr*, clang::InClassInitStyle, clang::SourceLocation, clang::AccessSpecifier, clang::NamedDecl*, clang::Declarator*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#19 0x00002aaad5577a1b in clang::TemplateDeclInstantiator::VisitFieldDecl(clang::FieldDecl*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#20 0x00002aaad555919e in clang::Sema::InstantiateClass(clang::SourceLocation, clang::CXXRecordDecl*, clang::CXXRecordDecl*, clang::MultiLevelTemplateArgumentList const&, clang::TemplateSpecializationKind, bool) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#21 0x00002aaad55704a0 in clang::Sema::InstantiateClassTemplateSpecialization(clang::SourceLocation, clang::ClassTemplateSpecializationDecl*, clang::TemplateSpecializationKind, bool) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#22 0x00002aaad559dd04 in clang::Sema::RequireCompleteTypeImpl(clang::SourceLocation, clang::QualType, clang::Sema::TypeDiagnoser*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#23 0x00002aaad559df55 in clang::Sema::RequireCompleteType(clang::SourceLocation, clang::QualType, clang::Sema::TypeDiagnoser&) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#24 0x00002aaad52dfcb3 in clang::Sema::CheckBaseSpecifier(clang::CXXRecordDecl*, clang::SourceRange, bool, clang::AccessSpecifier, clang::TypeSourceInfo*, clang::SourceLocation) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#25 0x00002aaad5558930 in clang::Sema::SubstBaseSpecifiers(clang::CXXRecordDecl*, clang::CXXRecordDecl*, clang::MultiLevelTemplateArgumentList const&) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#26 0x00002aaad5559009 in clang::Sema::InstantiateClass(clang::SourceLocation, clang::CXXRecordDecl*, clang::CXXRecordDecl*, clang::MultiLevelTemplateArgumentList const&, clang::TemplateSpecializationKind, bool) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#27 0x00002aaad55704a0 in clang::Sema::InstantiateClassTemplateSpecialization(clang::SourceLocation, clang::ClassTemplateSpecializationDecl*, clang::TemplateSpecializationKind, bool) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#28 0x00002aaad559dd04 in clang::Sema::RequireCompleteTypeImpl(clang::SourceLocation, clang::QualType, clang::Sema::TypeDiagnoser*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#29 0x00002aaad559df55 in clang::Sema::RequireCompleteType(clang::SourceLocation, clang::QualType, clang::Sema::TypeDiagnoser&) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#30 0x00002aaad51c85a1 in clang::Sema::RequireCompleteDeclContext(clang::CXXScopeSpec&, clang::DeclContext*&) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#31 0x00002aaad411a983 in cling::LookupHelper::findScope(llvm::StringRef, cling::LookupHelper::DiagSetting, clang::Type const**, bool) const () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#32 0x00002aaad40e342a in TClingClassInfo::TClingClassInfo(cling::Interpreter*, char const*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#33 0x00002aaad40a8f74 in TCling::SetClassInfo(TClass*, bool) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#34 0x00002aaaabbbbebf in TClass::SetUnloaded() () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCore.so
#35 0x00002aaaabb85ec4 in ROOT::RemoveClass(char const*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCore.so
#36 0x00002aaaabbd6b1e in ROOT::TGenericClassInfo::~TGenericClassInfo() () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCore.so
#37 0x00002aaaad540a02 in exit () from /lib64/libc.so.6
#38 0x00002aaaad529d27 in __libc_start_main () from /lib64/libc.so.6
#39 0x000000000040d499 in _start ()
Thread 1 (Thread 0x2aaaaf0c4f80 (LWP 66)):
#0  0x00002aaad56d219a in clang::CXXDestructorDecl::Create(clang::ASTContext&, clang::CXXRecordDecl*, clang::SourceLocation, clang::DeclarationNameInfo const&, clang::QualType, clang::TypeSourceInfo*, bool, bool) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#1  0x00002aaad55857fe in clang::TemplateDeclInstantiator::VisitCXXMethodDecl(clang::CXXMethodDecl*, clang::TemplateParameterList*, bool) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#2  0x00002aaad555919e in clang::Sema::InstantiateClass(clang::SourceLocation, clang::CXXRecordDecl*, clang::CXXRecordDecl*, clang::MultiLevelTemplateArgumentList const&, clang::TemplateSpecializationKind, bool) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#3  0x00002aaad55704a0 in clang::Sema::InstantiateClassTemplateSpecialization(clang::SourceLocation, clang::ClassTemplateSpecializationDecl*, clang::TemplateSpecializationKind, bool) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#4  0x00002aaad559dd04 in clang::Sema::RequireCompleteTypeImpl(clang::SourceLocation, clang::QualType, clang::Sema::TypeDiagnoser*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#5  0x00002aaad559df55 in clang::Sema::RequireCompleteType(clang::SourceLocation, clang::QualType, clang::Sema::TypeDiagnoser&) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#6  0x00002aaad52dfcb3 in clang::Sema::CheckBaseSpecifier(clang::CXXRecordDecl*, clang::SourceRange, bool, clang::AccessSpecifier, clang::TypeSourceInfo*, clang::SourceLocation) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#7  0x00002aaad5558930 in clang::Sema::SubstBaseSpecifiers(clang::CXXRecordDecl*, clang::CXXRecordDecl*, clang::MultiLevelTemplateArgumentList const&) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#8  0x00002aaad5559009 in clang::Sema::InstantiateClass(clang::SourceLocation, clang::CXXRecordDecl*, clang::CXXRecordDecl*, clang::MultiLevelTemplateArgumentList const&, clang::TemplateSpecializationKind, bool) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#9  0x00002aaad55704a0 in clang::Sema::InstantiateClassTemplateSpecialization(clang::SourceLocation, clang::ClassTemplateSpecializationDecl*, clang::TemplateSpecializationKind, bool) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#10 0x00002aaad559dd04 in clang::Sema::RequireCompleteTypeImpl(clang::SourceLocation, clang::QualType, clang::Sema::TypeDiagnoser*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#11 0x00002aaad559df55 in clang::Sema::RequireCompleteType(clang::SourceLocation, clang::QualType, clang::Sema::TypeDiagnoser&) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#12 0x00002aaad52dfcb3 in clang::Sema::CheckBaseSpecifier(clang::CXXRecordDecl*, clang::SourceRange, bool, clang::AccessSpecifier, clang::TypeSourceInfo*, clang::SourceLocation) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#13 0x00002aaad5558930 in clang::Sema::SubstBaseSpecifiers(clang::CXXRecordDecl*, clang::CXXRecordDecl*, clang::MultiLevelTemplateArgumentList const&) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#14 0x00002aaad5559009 in clang::Sema::InstantiateClass(clang::SourceLocation, clang::CXXRecordDecl*, clang::CXXRecordDecl*, clang::MultiLevelTemplateArgumentList const&, clang::TemplateSpecializationKind, bool) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#15 0x00002aaad559dc5e in clang::Sema::RequireCompleteTypeImpl(clang::SourceLocation, clang::QualType, clang::Sema::TypeDiagnoser*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#16 0x00002aaad559df55 in clang::Sema::RequireCompleteType(clang::SourceLocation, clang::QualType, clang::Sema::TypeDiagnoser&) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#17 0x00002aaad559e07b in clang::Sema::RequireCompleteType(clang::SourceLocation, clang::QualType, unsigned int) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#18 0x00002aaad525ff8e in clang::Sema::CheckFieldDecl(clang::DeclarationName, clang::QualType, clang::TypeSourceInfo*, clang::RecordDecl*, clang::SourceLocation, bool, clang::Expr*, clang::InClassInitStyle, clang::SourceLocation, clang::AccessSpecifier, clang::NamedDecl*, clang::Declarator*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#19 0x00002aaad5577a1b in clang::TemplateDeclInstantiator::VisitFieldDecl(clang::FieldDecl*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#20 0x00002aaad555919e in clang::Sema::InstantiateClass(clang::SourceLocation, clang::CXXRecordDecl*, clang::CXXRecordDecl*, clang::MultiLevelTemplateArgumentList const&, clang::TemplateSpecializationKind, bool) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#21 0x00002aaad55704a0 in clang::Sema::InstantiateClassTemplateSpecialization(clang::SourceLocation, clang::ClassTemplateSpecializationDecl*, clang::TemplateSpecializationKind, bool) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#22 0x00002aaad559dd04 in clang::Sema::RequireCompleteTypeImpl(clang::SourceLocation, clang::QualType, clang::Sema::TypeDiagnoser*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#23 0x00002aaad559df55 in clang::Sema::RequireCompleteType(clang::SourceLocation, clang::QualType, clang::Sema::TypeDiagnoser&) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#24 0x00002aaad52dfcb3 in clang::Sema::CheckBaseSpecifier(clang::CXXRecordDecl*, clang::SourceRange, bool, clang::AccessSpecifier, clang::TypeSourceInfo*, clang::SourceLocation) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#25 0x00002aaad5558930 in clang::Sema::SubstBaseSpecifiers(clang::CXXRecordDecl*, clang::CXXRecordDecl*, clang::MultiLevelTemplateArgumentList const&) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#26 0x00002aaad5559009 in clang::Sema::InstantiateClass(clang::SourceLocation, clang::CXXRecordDecl*, clang::CXXRecordDecl*, clang::MultiLevelTemplateArgumentList const&, clang::TemplateSpecializationKind, bool) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#27 0x00002aaad55704a0 in clang::Sema::InstantiateClassTemplateSpecialization(clang::SourceLocation, clang::ClassTemplateSpecializationDecl*, clang::TemplateSpecializationKind, bool) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#28 0x00002aaad559dd04 in clang::Sema::RequireCompleteTypeImpl(clang::SourceLocation, clang::QualType, clang::Sema::TypeDiagnoser*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#29 0x00002aaad559df55 in clang::Sema::RequireCompleteType(clang::SourceLocation, clang::QualType, clang::Sema::TypeDiagnoser&) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#30 0x00002aaad51c85a1 in clang::Sema::RequireCompleteDeclContext(clang::CXXScopeSpec&, clang::DeclContext*&) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#31 0x00002aaad411a983 in cling::LookupHelper::findScope(llvm::StringRef, cling::LookupHelper::DiagSetting, clang::Type const**, bool) const () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#32 0x00002aaad40e342a in TClingClassInfo::TClingClassInfo(cling::Interpreter*, char const*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#33 0x00002aaad40a8f74 in TCling::SetClassInfo(TClass*, bool) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCling.so
#34 0x00002aaaabbbbebf in TClass::SetUnloaded() () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCore.so
#35 0x00002aaaabb85ec4 in ROOT::RemoveClass(char const*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCore.so
#36 0x00002aaaabbd6b1e in ROOT::TGenericClassInfo::~TGenericClassInfo() () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/external/slc6_amd64_gcc630/lib/libCore.so
#37 0x00002aaaad540a02 in exit () from /lib64/libc.so.6
#38 0x00002aaaad529d27 in __libc_start_main () from /lib64/libc.so.6
#39 0x000000000040d499 in _start ()
Thread 1 (Thread 0x2aaaaf0c4f80 (LWP 66)):
#0  0x00002aaaabfbc814 in arena_run_heap_remove (ph=<optimized out>, phn=0x2aad0baa9788) at src/arena.c:114
#1  0x00002aaaabfbe742 in arena_avail_remove (npages=5727902784, pageind=1765, chunk=0x2aad0ba80000, arena=0x2aaaaf223b00) at src/arena.c:211
#2  arena_run_split_remove (arena=arena@entry=0x2aaaaf223b00, chunk=chunk@entry=0x2aad0ba80000, run_ind=run_ind@entry=1765, flag_dirty=flag_dirty@entry=0, flag_decommitted=0, need_pages=need_pages@entry=5727902784) at src/arena.c:398
#3  0x00002aaaabfbff8c in arena_run_split_large_helper (zero=false, remove=true, size=23461489803264, run=0x2aad0baa97b0, arena=<optimized out>) at src/arena.c:447
#4  arena_run_split_large (zero=false, size=23461489803264, run=0x2aad0baa97b0, arena=<optimized out>) at src/arena.c:502
#5  arena_stash_dirty (purge_chunks_sentinel=0x7fffffff09c0, purge_runs_sentinel=0x7fffffff0970, ndirty_limit=1170, chunk_hooks=0x7fffffff0980, arena=0x2aaaaf223b00, tsdn=0x2aaaaf0bfb40) at src/arena.c:1660
#6  arena_purge_to_limit (tsdn=tsdn@entry=0x2aaaaf0bfb40, arena=arena@entry=0x2aaaaf223b00, ndirty_limit=1170) at src/arena.c:1859
#7  0x00002aaaabfbf33b in arena_maybe_purge_ratio (arena=0x2aaaaf223b00, tsdn=0x2aaaaf0bfb40) at src/arena.c:1492
#8  je_arena_maybe_purge (tsdn=0x2aaaaf0bfb40, arena=0x2aaaaf223b00) at src/arena.c:1549
#9  0x00002aaaabfc5034 in arena_dalloc_bin_run (chunk=<optimized out>, bin=0x2aaaaf2247c8, run=<optimized out>, arena=0x2aaaaf223b00, tsdn=0x2aaaaf0bfb40) at src/arena.c:2912
#10 arena_dalloc_bin_locked_impl (bitselm=<optimized out>, junked=true, ptr=<optimized out>, chunk=<optimized out>, arena=0x2aaaaf223b00, tsdn=0x2aaaaf0bfb40) at src/arena.c:2966
#11 je_arena_dalloc_bin_junked_locked (tsdn=tsdn@entry=0x2aaaaf0bfb40, arena=arena@entry=0x2aaaaf223b00, chunk=<optimized out>, ptr=<optimized out>, bitselm=<optimized out>) at src/arena.c:2981
#12 0x00002aaaabff6784 in je_tcache_bin_flush_small (tsd=0x2aaaaf0bfb40, tcache=tcache@entry=0x2aaaaf142000, tbin=tbin@entry=0x2aaaaf1420c8, binind=binind@entry=5, rem=rem@entry=100) at src/tcache.c:137
#13 0x00002aaaabfb0a6b in je_tcache_dalloc_small (slow_path=false, binind=<optimized out>, ptr=0x2aabd4878c40, tcache=0x2aaaaf142000, tsd=<optimized out>) at include/jemalloc/internal/tcache.h:419
#14 je_arena_dalloc (slow_path=false, tcache=0x2aaaaf142000, ptr=0x2aabd4878c40, tsdn=<optimized out>) at include/jemalloc/internal/arena.h:1441
#15 je_idalloctm (slow_path=false, is_metadata=false, tcache=0x2aaaaf142000, ptr=0x2aabd4878c40, tsdn=<optimized out>) at include/jemalloc/internal/jemalloc_internal.h:1170
#16 je_iqalloc (slow_path=false, tcache=0x2aaaaf142000, ptr=0x2aabd4878c40, tsd=<optimized out>) at include/jemalloc/internal/jemalloc_internal.h:1187
#17 ifree (tsd=<optimized out>, slow_path=false, tcache=0x2aaaaf142000, ptr=0x2aabd4878c40) at src/jemalloc.c:1896
#18 free (ptr=0x2aabd4878c40) at src/jemalloc.c:2016
#19 0x00002aaae3ae2de9 in std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::vector<bool, std::allocator<bool> > >, std::_Select1st<std::pair<unsigned int const, std::vector<bool, std::allocator<bool> > > >, std::less<unsigned int>, std::allocator<std::pair<u
nsigned int const, std::vector<bool, std::allocator<bool> > > > >::_M_erase(std::_Rb_tree_node<std::pair<unsigned int const, std::vector<bool, std::allocator<bool> > > >*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/pluginSimT
rackerSiStripDigitizerPlugins.so
#20 0x00002aaae3ae2dcc in std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::vector<bool, std::allocator<bool> > >, std::_Select1st<std::pair<unsigned int const, std::vector<bool, std::allocator<bool> > > >, std::less<unsigned int>, std::allocator<std::pair<u
nsigned int const, std::vector<bool, std::allocator<bool> > > > >::_M_erase(std::_Rb_tree_node<std::pair<unsigned int const, std::vector<bool, std::allocator<bool> > > >*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/pluginSimT
rackerSiStripDigitizerPlugins.so
#21 0x00002aaae3ae2dcc in std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::vector<bool, std::allocator<bool> > >, std::_Select1st<std::pair<unsigned int const, std::vector<bool, std::allocator<bool> > > >, std::less<unsigned int>, std::allocator<std::pair<u
nsigned int const, std::vector<bool, std::allocator<bool> > > > >::_M_erase(std::_Rb_tree_node<std::pair<unsigned int const, std::vector<bool, std::allocator<bool> > > >*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/pluginSimT
rackerSiStripDigitizerPlugins.so
#22 0x00002aaae3ae2dcc in std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::vector<bool, std::allocator<bool> > >, std::_Select1st<std::pair<unsigned int const, std::vector<bool, std::allocator<bool> > > >, std::less<unsigned int>, std::allocator<std::pair<u
nsigned int const, std::vector<bool, std::allocator<bool> > > > >::_M_erase(std::_Rb_tree_node<std::pair<unsigned int const, std::vector<bool, std::allocator<bool> > > >*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/pluginSimT
rackerSiStripDigitizerPlugins.so
#23 0x00002aaae3ae2dcc in std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::vector<bool, std::allocator<bool> > >, std::_Select1st<std::pair<unsigned int const, std::vector<bool, std::allocator<bool> > > >, std::less<unsigned int>, std::allocator<std::pair<u
nsigned int const, std::vector<bool, std::allocator<bool> > > > >::_M_erase(std::_Rb_tree_node<std::pair<unsigned int const, std::vector<bool, std::allocator<bool> > > >*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/pluginSimTrackerSiStripDigitizerPlugins.so
#24 0x00002aaae3ae2dcc in std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::vector<bool, std::allocator<bool> > >, std::_Select1st<std::pair<unsigned int const, std::vector<bool, std::allocator<bool> > > >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::vector<bool, std::allocator<bool> > > > >::_M_erase(std::_Rb_tree_node<std::pair<unsigned int const, std::vector<bool, std::allocator<bool> > > >*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/pluginSimTrackerSiStripDigitizerPlugins.so
#25 0x00002aaae3ae2dcc in std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::vector<bool, std::allocator<bool> > >, std::_Select1st<std::pair<unsigned int const, std::vector<bool, std::allocator<bool> > > >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::vector<bool, std::allocator<bool> > > > >::_M_erase(std::_Rb_tree_node<std::pair<unsigned int const, std::vector<bool, std::allocator<bool> > > >*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/pluginSimTrackerSiStripDigitizerPlugins.so
#26 0x00002aaae3e58b15 in edm::DataMixingSiStripMCDigiWorker::~DataMixingSiStripMCDigiWorker() () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/pluginSimGeneralDataMixingModulePlugins.so
#27 0x00002aaae3e58e39 in edm::DataMixingSiStripMCDigiWorker::~DataMixingSiStripMCDigiWorker() () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/pluginSimGeneralDataMixingModulePlugins.so
#28 0x00002aaae3e041df in edm::DataMixingModule::~DataMixingModule() () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/pluginSimGeneralDataMixingModulePlugins.so
#29 0x00002aaae3e04279 in edm::DataMixingModule::~DataMixingModule() () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/pluginSimGeneralDataMixingModulePlugins.so
#30 0x00002aaaaaf06b36 in edm::stream::ProducingModuleAdaptorBase<edm::stream::EDProducerBase>::~ProducingModuleAdaptorBase() () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#31 0x00002aaae3e64e58 in edm::stream::ProducingModuleAdaptor<edm::DataMixingModule, edm::stream::EDProducerBase, edm::stream::EDProducerAdaptorBase>::~ProducingModuleAdaptor() () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/pluginSimGeneralDataMixingModulePlugins.so
#32 0x00002aaadbdf14d1 in std::_Sp_counted_ptr_inplace<edm::maker::ModuleHolderT<edm::stream::EDProducerAdaptorBase>, std::allocator<edm::maker::ModuleHolderT<edm::stream::EDProducerAdaptorBase> >, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/pluginRecoLocalMuonRPCRecHit.so
#33 0x00002aaaaae66b4a in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#34 0x00002aaaaae66aed in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#35 0x00002aaaaae66aed in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#36 0x00002aaaaae66aed in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#37 0x00002aaaaae66aed in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#38 0x00002aaaaae66aed in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#39 0x00002aaaaae66aed in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#40 0x00002aaaaae66aed in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#41 0x00002aaaaae66aed in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#42 0x00002aaaaae66aed in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#43 0x00002aaaaae66b96 in std::_Sp_counted_ptr<edm::ModuleRegistry*, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#44 0x0000000000411d26 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() ()
#45 0x00002aaaaae85171 in std::default_delete<edm::Schedule>::operator()(edm::Schedule*) const [clone .isra.816] () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#46 0x00002aaaaae88b97 in edm::EventProcessor::~EventProcessor() () from /cvmfs/cms.cern.ch/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_4_7/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#47 0x000000000040d953 in (anonymous namespace)::EventProcessorWithSentry::~EventProcessorWithSentry() ()
#48 0x000000000040d1b7 in main ()
hufnagel commented 3 years ago

One conclusion from the tests at TACC was that the failure is reproducible (although whether or not you end up getting a stack trace seems to be random). Another interesting data point was that the same workflow had almost 100% failure rate at TACC and ran fine elsewhere.

makortel commented 3 years ago

Occurred in

makortel commented 3 years ago

To add here as well, with gdb I managed to get a stack trace of origin of the fatal system signal in https://github.com/cms-sw/cmssw/issues/31896#issuecomment-735025379 and https://github.com/cms-sw/cmssw/pull/32102#issuecomment-738277482

Thread 1 "cmsRun" received signal SIGSEGV, Segmentation fault.
0x00007fffea676ba9 in coral::MessageStream::doOutput() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-25-2300/external/slc7_amd64_gcc820/lib/liblcg_CoralBase.so
(gdb) where
#0  0x00007fffea676ba9 in coral::MessageStream::doOutput() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-25-2300/external/slc7_amd64_gcc820/lib/liblcg_CoralBase.so
#1  0x00007fffd68fcb9b in coral::ConnectionService::ConnectionPool::~ConnectionPool() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-25-2300/external/slc7_amd64_gcc820/lib/liblcg_ConnectionService.so
#2  0x00007fffd68fcd29 in coral::ConnectionService::ConnectionPool::~ConnectionPool() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-25-2300/external/slc7_amd64_gcc820/lib/liblcg_ConnectionService.so
#3  0x00007fffd6903360 in coral::ConnectionService::ConnectionService::~ConnectionService() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-25-2300/external/slc7_amd64_gcc820/lib/liblcg_ConnectionService.so
#4  0x00007fffd69033f9 in coral::ConnectionService::ConnectionService::~ConnectionService() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-25-2300/external/slc7_amd64_gcc820/lib/liblcg_ConnectionService.so
#5  0x00007fffea6a2fdc in coral::Context::~Context() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-25-2300/external/slc7_amd64_gcc820/lib/liblcg_CoralKernel.so
#6  0x00007ffff33e0ce9 in __run_exit_handlers () from /lib64/libc.so.6
#7  0x00007ffff33e0d37 in exit () from /lib64/libc.so.6
#8  0x00007ffff33c955c in __libc_start_main () from /lib64/libc.so.6
#9  0x00000000004115f9 in _start ()
makortel commented 3 years ago

Just to note here as well, the crash in coral::MessageStream is addressed in https://github.com/cms-sw/cmssw/pull/32503.

silviodonato commented 3 years ago

We see again Fatal system signal has occurred during exit starting from CMSSW_11_3_X_2021-03-08-1800. (All relvals were ok in CMSSW_11_3_X_2021-03-08-1100 both in DEFAULT and SCRAMV3_X). The PR merged in CMSSW_11_3_X_2021-03-08-1800 were

32804 from Dr15Jones: Migrate to tbb::task_group

#33095 from davidlange6: correct Buildfiles in SimDataFormats
#33019 from guitargeek: Remove not needed RooFit dependencies from BuildFiles
#33066 from davidlange6: FileInPath: replace boost::filesystem by std::filesystem

In particular #32804 caused several problems (see #33107 Assertion failures in edm::service::Timing) and they were solved by #33125. However, we still see the error Fatal system signal has occurred during exit appears randomly in the IB tests.

Dr15Jones commented 3 years ago

@silviodonato #33126 is meant to fix the crashes at end of job

makortel commented 3 years ago

+1

We have not seen these crashes for a long time, so I think we can close the issue.

cmsbuild commented 3 years ago

This issue is fully signed and ready to be closed.