search

cms-sw / cmssw

CMS Offline Software
http://cms-sw.github.io/
Apache License 2.0
1.07k stars 4.28k forks source link

Crash in egHLT::trigTools::getFiltersPassed CMSSW_8_1_THREADED_X_2016-06-12-2300 #14897

Closed bouril closed 5 years ago

bouril commented 8 years ago

Build CMSSW_8_1_THREADED_X_2016-06-12-2300

1335.0 Wjet_Pt_80_120_13

egHLT::trigTools::getFiltersPassed

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc530/CMSSW_8_1_THREADED_X_2016-06-12-2300/pyRelValMatrixLogs/run/1335.0_Wjet_Pt_80_120_13+Wjet_Pt_80_120_13INPUT+DIGIUP15+RECOUP15+HARVESTUP15/step3_Wjet_Pt_80_120_13+Wjet_Pt_80_120_13INPUT+DIGIUP15+RECOUP15+HARVESTUP15.log

Thread 3 (Thread 0x7f20409fe700 (LWP 8330)):

0 0x0000003fccadf283 in poll () from /lib64/libc.so.6

1 0x00007f2150dd9d74 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-12-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so

2 0x00007f2150dd9fda in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-12-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so

3 0x00007f2150dda15b in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-12-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so

4

5 0x00007f2156bea1f2 in edm::InputTag::InputTag(std::string const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-12-2300/lib/slc6_amd64_gcc530/libFWCoreUtilities.so

6 0x00007f210c80500b in egHLT::trigTools::getFiltersPassed(std::vector<std::pair<std::string, int>, std::allocator<std::pair<std::string, int> > > const&, trigger::TriggerEvent const*, std::string const&, egHLT::TrigCodes const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-12-2300/lib/slc6_amd64_gcc530/pluginDQMOfflineTrigger.so

7 0x00007f210c718dc5 in egHLT::OffHelper::setTrigInfo(edm::Event const&, egHLT::OffEvt&, egHLT::TrigCodes const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-12-2300/lib/slc6_amd64_gcc530/pluginDQMOfflineTrigger.so

8 0x00007f210c71cc19 in egHLT::OffHelper::makeOffEvt(edm::Event const&, edm::EventSetup const&, egHLT::OffEvt&, egHLT::TrigCodes const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-12-2300/lib/slc6_amd64_gcc530/pluginDQMOfflineTrigger.so

9 0x00007f210c80b938 in EgHLTOfflineSource::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-12-2300/lib/slc6_amd64_gcc530/pluginDQMOfflineTrigger.so

10 0x00007f2156fa9e9a in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry, edm::ModuleCallingContext const) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-12-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so

11 0x00007f2156ef078f in edm::WorkerTedm::stream::EDAnalyzerAdaptorBase::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-12-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so

12 0x00007f2156ea450c in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const)::{lambda()#1}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-12-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so

13 0x00007f2156ea46d2 in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-12-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so

14 0x00007f2156ecff5e in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-12-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so

15 0x00007f2156ed0212 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-12-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so

16 0x00007f2156ed0635 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-12-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so

17 0x00007f2156ed092b in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-12-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so

18 0x00007f2156ed0b9f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-12-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so

19 0x00007f2156f47122 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-12-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so

20 0x00007f2156f47857 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-12-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so

21 0x00007f2156f4e104 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-12-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so

22 0x00007f2155c8e9b5 in tbb::internal::custom_schedulertbb::internal::IntelSchedulerTraits::local_wait_for_all (this=0x7f21324a7e00, parent=..., child=) at ../../src/tbb/custom_scheduler.h:463

23 0x00007f2155c881a4 in tbb::internal::arena::process (this=this@entry=0x7f21547df780, s=...) at ../../src/tbb/arena.cpp:156

24 0x00007f2155c87264 in tbb::internal::market::process (this=0x7f2154807c00, j=...) at ../../src/tbb/market.cpp:502

25 0x00007f2155c83736 in tbb::internal::rml::private_worker::run (this=0x7f2154626180) at ../../src/tbb/private_server.cpp:275

26 0x00007f2155c83989 in tbb::internal::rml::private_worker::thread_routine (arg=) at ../../src/tbb/private_server.cpp:228

27 0x0000003fcce07aa1 in start_thread () from /lib64/libpthread.so.0

28 0x0000003fccae8aad in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f203feff700 (LWP 8331)):

cmsbuild commented 8 years ago

A new Issue was created by @bouril Dimitri Bourilkov.

@davidlange6, @smuzaffar, @degano, @davidlt, @Dr15Jones can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

Dr15Jones commented 8 years ago

@dan131riley This appears to have happen again but this time two threads in the same job crashed ! That's a new one to me: https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc530/CMSSW_8_1_THREADED_X_2016-06-28-2300/pyRelValMatrixLogs/run/250205.0_ZTT_13+ZTT_13INPUT+DIGIPRMXUP15_PU25+RECOPRMXUP15_PU25+HARVESTUP15_PU25/step3_ZTT_13+ZTT_13INPUT+DIGIPRMXUP15_PU25+RECOPRMXUP15_PU25+HARVESTUP15_PU25.log

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thread 5 (Thread 0x7f3be5c3f700 (LWP 150519)):
#0  0x0000003e8cc0f37d in waitpid () from /lib64/libpthread.so.0
#1  0x00007f3bf6759697 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007f3bf6759f42 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007f3bfdb2aa50 in std::(anonymous namespace)::execute_native_thread_routine (__p=<optimized out>) at ../../../../../libstdc++-v3/src/c++11/thread.cc:84
#4  0x0000003e8cc07aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003e8c8e8aad in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f3ae333f700 (LWP 154179)):
#0  0x0000003e8c8df283 in poll () from /lib64/libc.so.6
#1  0x00007f3bf6759da4 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007f3bf675a00a in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007f3bf675a18b in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007f3bfeda4412 in edm::InputTag::InputTag(std::string const&) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreUtilities.so
#6  0x00007f3bb316669e in void egHLT::trigTools::setFiltersObjPasses<egHLT::OffPho>(std::vector<egHLT::OffPho, std::allocator<egHLT::OffPho> >&, std::vector<std::string, std::allocator<std::string> > const&, std::vector<std::pair<std::string, std::string>, std::allocator<std::pair<std::string, std::string> > > const&, std::bitset<128ul> const&, egHLT::TrigCodes const&, trigger::TriggerEvent const*, std::string const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-06-28-2300/lib/slc6_amd64_gcc530/pluginDQMOfflineTrigger.so
#7  0x00007f3bb315f649 in egHLT::OffHelper::setTrigInfo(edm::Event const&, egHLT::OffEvt&, egHLT::TrigCodes const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-06-28-2300/lib/slc6_amd64_gcc530/pluginDQMOfflineTrigger.so
#8  0x00007f3bb31633e9 in egHLT::OffHelper::makeOffEvt(edm::Event const&, edm::EventSetup const&, egHLT::OffEvt&, egHLT::TrigCodes const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-06-28-2300/lib/slc6_amd64_gcc530/pluginDQMOfflineTrigger.so
#9  0x00007f3bb324cc18 in EgHLTOfflineSource::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-06-28-2300/lib/slc6_amd64_gcc530/pluginDQMOfflineTrigger.so
#10 0x00007f3bff166b2a in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#11 0x00007f3bff0ac9cf in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#12 0x00007f3bff06366c in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#13 0x00007f3bff063832 in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#14 0x00007f3bff08b41e in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#15 0x00007f3bff08b6d2 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007f3bff08baf5 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007f3bff08bdeb in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007f3bff08c05f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007f3bff10bda2 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007f3bff10c61a in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007f3bff112cb4 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007f3bfde4a9b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f3bd8e27e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#23 0x00007f3bfde441a4 in tbb::internal::arena::process (this=this@entry=0x7f3bfc78b780, s=...) at ../../src/tbb/arena.cpp:156
#24 0x00007f3bfde43264 in tbb::internal::market::process (this=0x7f3bfc7a7900, j=...) at ../../src/tbb/market.cpp:502
#25 0x00007f3bfde3f736 in tbb::internal::rml::private_worker::run (this=0x7f3bfc5c7100) at ../../src/tbb/private_server.cpp:275
#26 0x00007f3bfde3f989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#27 0x0000003e8cc07aa1 in start_thread () from /lib64/libpthread.so.0
#28 0x0000003e8c8e8aad in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f3ae293e700 (LWP 154180)):
#0  0x0000003e8c8e548a in mmap64 () from /lib64/libc.so.6
#1  0x0000003e8c8662ac in _IO_file_doallocate_internal () from /lib64/libc.so.6
#2  0x0000003e8c87376c in _IO_doallocbuf_internal () from /lib64/libc.so.6
#3  0x0000003e8c87168c in _IO_file_xsgetn_internal () from /lib64/libc.so.6
#4  0x0000003e8c867422 in fread () from /lib64/libc.so.6
#5  0x00007f3bf3577164 in xercesc_3_1::PosixFileMgr::fileRead(void*, unsigned long, unsigned char*, xercesc_3_1::MemoryManager*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-06-28-2300/external/slc6_amd64_gcc530/lib/libxerces-c-3.1.so
#6  0x00007f3bf33c802f in xercesc_3_1::XMLPlatformUtils::readFileBuffer(void*, unsigned long, unsigned char*, xercesc_3_1::MemoryManager*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-06-28-2300/external/slc6_amd64_gcc530/lib/libxerces-c-3.1.so
#7  0x00007f3bf33c48a0 in xercesc_3_1::BinFileInputStream::readBytes(unsigned char*, unsigned long) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-06-28-2300/external/slc6_amd64_gcc530/lib/libxerces-c-3.1.so
#8  0x00007f3bf34a7899 in xercesc_3_1::XMLReader::refreshRawBuffer() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-06-28-2300/external/slc6_amd64_gcc530/lib/libxerces-c-3.1.so
#9  0x00007f3bf34a3842 in xercesc_3_1::XMLReader::XMLReader(unsigned short const*, unsigned short const*, xercesc_3_1::BinInputStream*, xercesc_3_1::XMLReader::RefFrom, xercesc_3_1::XMLReader::Types, xercesc_3_1::XMLReader::Sources, bool, bool, unsigned long, xercesc_3_1::XMLReader::XMLVersion, xercesc_3_1::MemoryManager*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-06-28-2300/external/slc6_amd64_gcc530/lib/libxerces-c-3.1.so
#10 0x00007f3bf348a4dc in xercesc_3_1::ReaderMgr::createReader(xercesc_3_1::InputSource const&, bool, xercesc_3_1::XMLReader::RefFrom, xercesc_3_1::XMLReader::Types, xercesc_3_1::XMLReader::Sources, bool, unsigned long) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-06-28-2300/external/slc6_amd64_gcc530/lib/libxerces-c-3.1.so
#11 0x00007f3bf34822dc in xercesc_3_1::IGXMLScanner::scanReset(xercesc_3_1::InputSource const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-06-28-2300/external/slc6_amd64_gcc530/lib/libxerces-c-3.1.so
#12 0x00007f3bf347487b in xercesc_3_1::IGXMLScanner::scanDocument(xercesc_3_1::InputSource const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-06-28-2300/external/slc6_amd64_gcc530/lib/libxerces-c-3.1.so
#13 0x00007f3bf34a904f in xercesc_3_1::XMLScanner::scanDocument(unsigned short const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-06-28-2300/external/slc6_amd64_gcc530/lib/libxerces-c-3.1.so
#14 0x00007f3bf34a9306 in xercesc_3_1::XMLScanner::scanDocument(char const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-06-28-2300/external/slc6_amd64_gcc530/lib/libxerces-c-3.1.so
#15 0x00007f3bf34e04bd in xercesc_3_1::SAX2XMLReaderImpl::parse(char const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-06-28-2300/external/slc6_amd64_gcc530/lib/libxerces-c-3.1.so
#16 0x00007f3be0f3653c in DDLParser::parseFile(int const&) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libDetectorDescriptionParser.so
#17 0x00007f3be0f37374 in DDLParser::parse(DDLDocumentProvider const&) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libDetectorDescriptionParser.so
#18 0x00007f3bdfa893fe in XMLIdealGeometryESSource::produce() () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/pluginGeometryReadersXMLIdealGeometryESSource.so
#19 0x00007f3bdfa89619 in XMLIdealGeometryESSource::produceGeom(IdealGeometryRecord const&) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/pluginGeometryReadersXMLIdealGeometryESSource.so
#20 0x00007f3bdfa8b278 in edm::eventsetup::CallbackProxy<edm::eventsetup::Callback<XMLIdealGeometryESSource, std::unique_ptr<DDCompactView, std::default_delete<DDCompactView> >, IdealGeometryRecord, edm::eventsetup::CallbackSimpleDecorator<IdealGeometryRecord> >, IdealGeometryRecord, std::unique_ptr<DDCompactView, std::default_delete<DDCompactView> > >::getImpl(edm::eventsetup::EventSetupRecord const&, edm::eventsetup::DataKey const&) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/pluginGeometryReadersXMLIdealGeometryESSource.so
#21 0x00007f3bff0ea05f in edm::eventsetup::DataProxy::get(edm::eventsetup::EventSetupRecord const&, edm::eventsetup::DataKey const&, bool) const () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007f3bff0fcb8b in edm::eventsetup::EventSetupRecord::getFromProxy(edm::eventsetup::DataKey const&, edm::eventsetup::ComponentDescription const*&, bool) const () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007f3be1158919 in TotemRPGeometryESModule::produceMeasuredDDCV(VeryForwardMeasuredGeometryRecord const&) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/pluginGeometryTotemRPGeometryBuilderPlugins.so
#24 0x00007f3be115d7e8 in edm::eventsetup::CallbackProxy<edm::eventsetup::Callback<TotemRPGeometryESModule, std::unique_ptr<DDCompactView, std::default_delete<DDCompactView> >, VeryForwardMeasuredGeometryRecord, edm::eventsetup::CallbackSimpleDecorator<VeryForwardMeasuredGeometryRecord> >, VeryForwardMeasuredGeometryRecord, std::unique_ptr<DDCompactView, std::default_delete<DDCompactView> > >::getImpl(edm::eventsetup::EventSetupRecord const&, edm::eventsetup::DataKey const&) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/pluginGeometryTotemRPGeometryBuilderPlugins.so
#25 0x00007f3bff0ea05f in edm::eventsetup::DataProxy::get(edm::eventsetup::EventSetupRecord const&, edm::eventsetup::DataKey const&, bool) const () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#26 0x00007f3bff0fcb8b in edm::eventsetup::EventSetupRecord::getFromProxy(edm::eventsetup::DataKey const&, edm::eventsetup::ComponentDescription const*&, bool) const () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#27 0x00007f3be115861c in TotemRPGeometryESModule::produceMeasuredGD(VeryForwardMeasuredGeometryRecord const&) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/pluginGeometryTotemRPGeometryBuilderPlugins.so
#28 0x00007f3be115d8e3 in edm::eventsetup::CallbackProxy<edm::eventsetup::Callback<TotemRPGeometryESModule, std::unique_ptr<DetGeomDesc, std::default_delete<DetGeomDesc> >, VeryForwardMeasuredGeometryRecord, edm::eventsetup::CallbackSimpleDecorator<VeryForwardMeasuredGeometryRecord> >, VeryForwardMeasuredGeometryRecord, std::unique_ptr<DetGeomDesc, std::default_delete<DetGeomDesc> > >::getImpl(edm::eventsetup::EventSetupRecord const&, edm::eventsetup::DataKey const&) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/pluginGeometryTotemRPGeometryBuilderPlugins.so
#29 0x00007f3bff0ea05f in edm::eventsetup::DataProxy::get(edm::eventsetup::EventSetupRecord const&, edm::eventsetup::DataKey const&, bool) const () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#30 0x00007f3bff0fcb8b in edm::eventsetup::EventSetupRecord::getFromProxy(edm::eventsetup::DataKey const&, edm::eventsetup::ComponentDescription const*&, bool) const () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#31 0x00007f3be116033a in void edm::eventsetup::EventSetupRecord::get<edm::ESHandle<DetGeomDesc> >(edm::ESHandle<DetGeomDesc>&) const () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/pluginGeometryTotemRPGeometryBuilderPlugins.so
#32 0x00007f3be115b8c2 in TotemRPGeometryESModule::produceRealGD(VeryForwardRealGeometryRecord const&) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/pluginGeometryTotemRPGeometryBuilderPlugins.so
#33 0x00007f3be115d9d3 in edm::eventsetup::CallbackProxy<edm::eventsetup::Callback<TotemRPGeometryESModule, std::unique_ptr<DetGeomDesc, std::default_delete<DetGeomDesc> >, VeryForwardRealGeometryRecord, edm::eventsetup::CallbackSimpleDecorator<VeryForwardRealGeometryRecord> >, VeryForwardRealGeometryRecord, std::unique_ptr<DetGeomDesc, std::default_delete<DetGeomDesc> > >::getImpl(edm::eventsetup::EventSetupRecord const&, edm::eventsetup::DataKey const&) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/pluginGeometryTotemRPGeometryBuilderPlugins.so
#34 0x00007f3bff0ea05f in edm::eventsetup::DataProxy::get(edm::eventsetup::EventSetupRecord const&, edm::eventsetup::DataKey const&, bool) const () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#35 0x00007f3bff0fcb8b in edm::eventsetup::EventSetupRecord::getFromProxy(edm::eventsetup::DataKey const&, edm::eventsetup::ComponentDescription const*&, bool) const () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#36 0x00007f3be116033a in void edm::eventsetup::EventSetupRecord::get<edm::ESHandle<DetGeomDesc> >(edm::ESHandle<DetGeomDesc>&) const () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/pluginGeometryTotemRPGeometryBuilderPlugins.so
#37 0x00007f3be115a5e6 in TotemRPGeometryESModule::produceRealTG(VeryForwardRealGeometryRecord const&) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/pluginGeometryTotemRPGeometryBuilderPlugins.so
#38 0x00007f3be1160fac in edm::eventsetup::CallbackProxy<edm::eventsetup::Callback<TotemRPGeometryESModule, std::unique_ptr<TotemRPGeometry, std::default_delete<TotemRPGeometry> >, VeryForwardRealGeometryRecord, edm::eventsetup::CallbackSimpleDecorator<VeryForwardRealGeometryRecord> >, VeryForwardRealGeometryRecord, std::unique_ptr<TotemRPGeometry, std::default_delete<TotemRPGeometry> > >::getImpl(edm::eventsetup::EventSetupRecord const&, edm::eventsetup::DataKey const&) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/pluginGeometryTotemRPGeometryBuilderPlugins.so
#39 0x00007f3bff0ea05f in edm::eventsetup::DataProxy::get(edm::eventsetup::EventSetupRecord const&, edm::eventsetup::DataKey const&, bool) const () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#40 0x00007f3bff0fcb8b in edm::eventsetup::EventSetupRecord::getFromProxy(edm::eventsetup::DataKey const&, edm::eventsetup::ComponentDescription const*&, bool) const () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#41 0x00007f3b94b116f4 in TotemRPUVPatternFinder::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/pluginRecoCTPPSTotemRPLocalPlugins.so
#42 0x00007f3bff163552 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#43 0x00007f3bff0acc2f in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#44 0x00007f3bff06366c in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#45 0x00007f3bff063832 in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#46 0x00007f3bff05fe3a in edm::UnscheduledProductResolver::resolveProduct_(edm::ProductResolverBase::ResolveStatus&, edm::Principal const&, bool, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#47 0x00007f3bff060152 in edm::UnscheduledProductResolver::resolveProduct_(edm::ProductResolverBase::ResolveStatus&, edm::Principal const&, bool, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#48 0x00007f3bff114af1 in edm::Principal::prefetch(unsigned int, bool, edm::ModuleCallingContext const*) const () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#49 0x00007f3bff063647 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#50 0x00007f3bff063832 in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#51 0x00007f3bff08b41e in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#52 0x00007f3bff08b6d2 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#53 0x00007f3bff08baf5 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#54 0x00007f3bff08bdeb in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#55 0x00007f3bff08c05f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#56 0x00007f3bff10bda2 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#57 0x00007f3bff10c4d7 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#58 0x00007f3bff112cb4 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#59 0x00007f3bfde4a9b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f3bd8e2fe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#60 0x00007f3bfde441a4 in tbb::internal::arena::process (this=this@entry=0x7f3bfc78b780, s=...) at ../../src/tbb/arena.cpp:156
#61 0x00007f3bfde43264 in tbb::internal::market::process (this=0x7f3bfc7a7900, j=...) at ../../src/tbb/market.cpp:502
#62 0x00007f3bfde3f736 in tbb::internal::rml::private_worker::run (this=0x7f3bfc5c7180) at ../../src/tbb/private_server.cpp:275
#63 0x00007f3bfde3f989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#64 0x0000003e8cc07aa1 in start_thread () from /lib64/libpthread.so.0
#65 0x0000003e8c8e8aad in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f3ae1e3f700 (LWP 154181)):
#0  0x0000003e8c8df283 in poll () from /lib64/libc.so.6
#1  0x00007f3bf6759da4 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007f3bf675a00a in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007f3bf675a18b in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007f3bfeda4412 in edm::InputTag::InputTag(std::string const&) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreUtilities.so
#6  0x00007f3bb316669e in void egHLT::trigTools::setFiltersObjPasses<egHLT::OffPho>(std::vector<egHLT::OffPho, std::allocator<egHLT::OffPho> >&, std::vector<std::string, std::allocator<std::string> > const&, std::vector<std::pair<std::string, std::string>, std::allocator<std::pair<std::string, std::string> > > const&, std::bitset<128ul> const&, egHLT::TrigCodes const&, trigger::TriggerEvent const*, std::string const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-06-28-2300/lib/slc6_amd64_gcc530/pluginDQMOfflineTrigger.so
#7  0x00007f3bb315f649 in egHLT::OffHelper::setTrigInfo(edm::Event const&, egHLT::OffEvt&, egHLT::TrigCodes const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-06-28-2300/lib/slc6_amd64_gcc530/pluginDQMOfflineTrigger.so
#8  0x00007f3bb31633e9 in egHLT::OffHelper::makeOffEvt(edm::Event const&, edm::EventSetup const&, egHLT::OffEvt&, egHLT::TrigCodes const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-06-28-2300/lib/slc6_amd64_gcc530/pluginDQMOfflineTrigger.so
#9  0x00007f3bb324cc18 in EgHLTOfflineSource::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-06-28-2300/lib/slc6_amd64_gcc530/pluginDQMOfflineTrigger.so
#10 0x00007f3bff166b2a in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#11 0x00007f3bff0ac9cf in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#12 0x00007f3bff06366c in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#13 0x00007f3bff063832 in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#14 0x00007f3bff08b41e in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#15 0x00007f3bff08b6d2 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007f3bff08baf5 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007f3bff08bdeb in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007f3bff08c05f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007f3bff10bda2 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007f3bff10c4d7 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007f3bff112cb4 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007f3bfde4a9b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f3bd8e1fe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#23 0x00007f3bfde441a4 in tbb::internal::arena::process (this=this@entry=0x7f3bfc78b780, s=...) at ../../src/tbb/arena.cpp:156
#24 0x00007f3bfde43264 in tbb::internal::market::process (this=0x7f3bfc7a7900, j=...) at ../../src/tbb/market.cpp:502
#25 0x00007f3bfde3f736 in tbb::internal::rml::private_worker::run (this=0x7f3bfc5c7080) at ../../src/tbb/private_server.cpp:275
#26 0x00007f3bfde3f989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#27 0x0000003e8cc07aa1 in start_thread () from /lib64/libpthread.so.0
#28 0x0000003e8c8e8aad in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f3bfd7c5ba0 (LWP 150160)):
#0  0x0000003e8cc0e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x0000003e8cc095f3 in _L_lock_892 () from /lib64/libpthread.so.0
#2  0x0000003e8cc094d7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007f3bff0ea018 in edm::eventsetup::DataProxy::get(edm::eventsetup::EventSetupRecord const&, edm::eventsetup::DataKey const&, bool) const () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#4  0x00007f3bff0fcb8b in edm::eventsetup::EventSetupRecord::getFromProxy(edm::eventsetup::DataKey const&, edm::eventsetup::ComponentDescription const*&, bool) const () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#5  0x00007f3b94b116f4 in TotemRPUVPatternFinder::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/pluginRecoCTPPSTotemRPLocalPlugins.so
#6  0x00007f3bff163552 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#7  0x00007f3bff0acc2f in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#8  0x00007f3bff06366c in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#9  0x00007f3bff063832 in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#10 0x00007f3bff05fe3a in edm::UnscheduledProductResolver::resolveProduct_(edm::ProductResolverBase::ResolveStatus&, edm::Principal const&, bool, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#11 0x00007f3bff060152 in edm::UnscheduledProductResolver::resolveProduct_(edm::ProductResolverBase::ResolveStatus&, edm::Principal const&, bool, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#12 0x00007f3bff114af1 in edm::Principal::prefetch(unsigned int, bool, edm::ModuleCallingContext const*) const () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#13 0x00007f3bff063647 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#14 0x00007f3bff063832 in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#15 0x00007f3bff08b41e in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007f3bff08b6d2 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007f3bff08baf5 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007f3bff08bdeb in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007f3bff08c05f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007f3bff10bda2 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007f3bff10c4d7 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007f3bff112cb4 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007f3bfde4a9b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f3bfc7a2600, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#24 0x00007f3bff10c32e in edm::EventProcessor::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#25 0x00007f3bff06c54a in statemachine::HandleEvent::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#26 0x00007f3bff06dad8 in statemachine::HandleEvent::HandleEvent(boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::my_context) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#27 0x00007f3bff07585e in boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::shallow_construct(boost::intrusive_ptr<statemachine::HandleLumis> const&, boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>&) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#28 0x00007f3bff075a2a in boost::statechart::simple_state<statemachine::FirstLumi, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#29 0x00007f3bff112768 in boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&) () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#30 0x00007f3bff105a5d in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/2016-27/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-26-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#31 0x000000000040c804 in main::{lambda()#1}::operator()() const ()
#32 0x000000000040ac84 in main ()
Dr15Jones commented 8 years ago

Something I had missed before is the crash happens in edm::InputTag::InputTag(std::string const&) but the only explicit calls to construct an edm::InputTag in that code uses the three argument form of the constructor. Therefore the crash happens in code that the compile added because a string was passed to an function that required an edm::InputTag.

Dr15Jones commented 8 years ago

So I think I figured out why edm::InputTag::InputTag(std::string const&) is being called. I think it is from line 56 http://cmslxr.fnal.gov/lxr/source/DQMOffline/Trigger/interface/EgHLTTrigTools.h?v=CMSSW_8_1_X_2016-06-26-2300#0056 and line 79 http://cmslxr.fnal.gov/lxr/source/DQMOffline/Trigger/interface/EgHLTTrigTools.h?v=CMSSW_8_1_X_2016-06-26-2300#0079

line 56 is 

 size_t filterNrInEvt = trigEvt->filterIndex(edm::InputTag(filters[filterNrInVec],"",hltTag).encode());

What this line does is create a temporary edm::InputTag and then tell it to make a std::string representation of itself. However, filterIndex does not want a std::string but instead wants a edm::InputTag. Therefore the compiler takes the std::string and creates another edm::InputTag out of that string! So we are doing edm::InputTag -> std::string -> edm::InputTag conversion for no good reason!

Although this is just wasting CPU I don't see why it would lead to a crash.

Dr15Jones commented 8 years ago

I got rid of the unnecessary conversions #15051

Dr15Jones commented 8 years ago

assign core

cmsbuild commented 8 years ago

New categories assigned: core

@Dr15Jones,@smuzaffar you have been requested to review this Pull request/Issue and eventually sign? Thanks

Dr15Jones commented 8 years ago

Found another instance of the crash https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc530/CMSSW_8_1_THREADED_X_2016-06-28-1100/pyRelValMatrixLogs/run/1306.0_SingleMuPt1_UP15+SingleMuPt1_UP15INPUT+DIGIUP15+RECOUP15+HARVESTUP15/step3_SingleMuPt1_UP15+SingleMuPt1_UP15INPUT+DIGIUP15+RECOUP15+HARVESTUP15.log

dan131riley commented 8 years ago

I'm having a hard time imagining how edm::InputTag().encode() produces a string that crashes edm::InputTag(). Also odd that in both the latest two crashes another thread is in basically the same spot deep inside TotemRPGeometryESModule. Will see if these reproduce...

Dr15Jones commented 8 years ago

I agree about edm::InputTag().encode() probably being a victim of a 'drive-by' memory hit :).

Dr15Jones commented 8 years ago

I noticed that there are two different crashes, one is egHLT::trigTools::getFiltersPassed the other is egHLT::trigTools::setFiltersObjPasses<egHLT::OffPho> although they both crash in edm::InputTag::InputTag(std::string const&).

dan131riley commented 8 years ago

So far I've completely failed to reproduce this one, even running hundreds of (automated) trials of three different workflows, so it seems like I'm missing some necessary condition.

Both crashes are in calls to TriggerEvent::filterIndex(), which converts the InputTag back to a string:

trigger::size_type filterIndex(const edm::InputTag& filterTag) const {
  const std::string encodedFilterTag(filterTag.encode());

so the net effect (without fixes) is something like

InputTag(InputTag(string, string(""), string).encode()).encode()

and InputTag(string) does a bunch more string manipulations (I'm thinking about a test making InputTag(string) explicit just to see how much we're using it in implicit conversions). There's a lot of memory allocations and frees, and I'm guessing maybe some of those are hitting mutexes in jemalloc. So I've been thinking about how mutexes can open up race windows.

What I mean is that with something like:

gMinuit = this;
gROOT->GetListOfSpecials()->Add(gMinuit);

in practical terms there's no race condition on gMinuit, it'll be in a register so in our use it is only a theoretical race--the practical race is on corrupting the specials list with simultaneous updates. Add a mutex:

gMinuit = this;
R__LOCKGUARD2(gROOTMutex);
gROOT->GetListOfSpecials()->Add(gMinuit);

and that fixes the race updating the specials lists, but it also turns the gMinuit race from theoretical to real with a huge window--the threads pile up at the mutex and it forces a memory read of gMinuit afterward.

My hunch is that there's some pattern of allocations that's hitting a mutex in the allocator, some memory overwrite is happening while threads are sitting on that mutex, the crash happens after return from the allocator, and that synchronization point is how we can end up with two threads crashing at the same spot at the same time.

I've tried inserting nano-sleeps into InputTag(string) to widen the window, but that doesn't do it, will have to try with a mutex and a random sleep so the threads are synced. There might be some utility to an LD_PRELOAD that adds random delays to every mutex acquisition, or some other way of adding artificial jitter. I know of one profiling tool that slows down sections of code for profiling purposes (https://github.com/plasma-umass/coz), but haven't found any debugging tools that systematically introduce synchronization jitter.

Or I could be off in completely the wrong direction...

-dan

Dr15Jones commented 8 years ago

Seems like a reasonable guess to me. The idea of the mutex timing causing the appearance of synchronization like behavior is really interesting.

dan131riley commented 8 years ago

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc530/CMSSW_8_1_X_2016-08-16-1100/pyRelValMatrixLogs/run/1340.0_PhiToMuMu_13+PhiToMuMu_13INPUT+DIGIUP15+RECOUP15+HARVESTUP15/step3_PhiToMuMu_13+PhiToMuMu_13INPUT+DIGIUP15+RECOUP15+HARVESTUP15.log

with the new stack-trace-pause handlers. Two threads are in tokenize, three are in InputTag, all four are in egHLT::trigTools::getFiltersPassed. Could there be an issue with tokenize?

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thread 5 (Thread 0x7f6a593bf700 (LWP 2408)):
#0  0x00000030e7e0f37d in waitpid () from /lib64/libpthread.so.0
#1  0x00007f6a6b313d47 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-16-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007f6a6b3146d2 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-16-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007f6a711c6a50 in std::(anonymous namespace)::execute_native_thread_routine (__p=<optimized out>) at ../../../../../libstdc++-v3/src/c++11/thread.cc:84
#4  0x00000030e7e07aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000030e7ae8aad in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f6a08b7f700 (LWP 2481)):
#0  0x00000030e7aacbdd in nanosleep () from /lib64/libc.so.6
#1  0x00000030e7aaca50 in sleep () from /lib64/libc.so.6
#2  0x00007f6a6b313b7e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-16-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  std::string::_Rep::_M_refcopy (this=0x7f694f292600) at /mnt/build/davidlt/gcc530_final/b/BUILD/slc6_amd64_gcc530/external/gcc/5.3.0/gcc-tags_gcc_5_3_0_release-231255/obj/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/basic_string.h:2659
#5  std::string::_Rep::_M_grab (__alloc2=..., __alloc1=..., this=0x7f694f292600) at /mnt/build/davidlt/gcc530_final/b/BUILD/slc6_amd64_gcc530/external/gcc/5.3.0/gcc-tags_gcc_5_3_0_release-231255/obj/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/basic_string.h:2627
#6  std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string (this=0x7f6a08b79648, __str=...) at /mnt/build/davidlt/gcc530_final/b/BUILD/slc6_amd64_gcc530/external/gcc/5.3.0/gcc-tags_gcc_5_3_0_release-231255/obj/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/basic_string.tcc:617
#7  0x00007f6a728aa310 in edm::tokenize(std::string const&, std::string const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-16-1100/lib/slc6_amd64_gcc530/libFWCoreUtilities.so
#8  0x00007f6a728ef958 in edm::InputTag::InputTag(std::string const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-16-1100/lib/slc6_amd64_gcc530/libFWCoreUtilities.so
#9  0x00007f6a28d1b10b in egHLT::trigTools::getFiltersPassed(std::vector<std::pair<std::string, int>, std::allocator<std::pair<std::string, int> > > const&, trigger::TriggerEvent const*, std::string const&, egHLT::TrigCodes const&) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/pluginDQMOfflineTrigger.so
#10 0x00007f6a28c24f95 in egHLT::OffHelper::setTrigInfo(edm::Event const&, egHLT::OffEvt&, egHLT::TrigCodes const&) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/pluginDQMOfflineTrigger.so
#11 0x00007f6a28c28de9 in egHLT::OffHelper::makeOffEvt(edm::Event const&, edm::EventSetup const&, egHLT::OffEvt&, egHLT::TrigCodes const&) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/pluginDQMOfflineTrigger.so
#12 0x00007f6a28c0a558 in EgHLTOfflineSource::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/pluginDQMOfflineTrigger.so
#13 0x00007f6a72cbc2fa in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#14 0x00007f6a72c97aa2 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#15 0x00007f6a72bcfbe4 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007f6a72bcfe78 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007f6a72bd01ed in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007f6a72bee9ce in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007f6a72beec82 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007f6a72bef0a5 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007f6a72bef39b in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007f6a72bef60f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007f6a72c71f92 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#24 0x00007f6a72c7280a in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#25 0x00007f6a72c78f24 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#26 0x00007f6a719949b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f6a4d83fe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#27 0x00007f6a7198e1a4 in tbb::internal::arena::process (this=this@entry=0x7f6a6fe1f780, s=...) at ../../src/tbb/arena.cpp:156
#28 0x00007f6a7198d264 in tbb::internal::market::process (this=0x7f6a6fe47e00, j=...) at ../../src/tbb/market.cpp:502
#29 0x00007f6a71989736 in tbb::internal::rml::private_worker::run (this=0x7f6a6fe3fa00) at ../../src/tbb/private_server.cpp:275
#30 0x00007f6a71989989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#31 0x00000030e7e07aa1 in start_thread () from /lib64/libpthread.so.0
#32 0x00000030e7ae8aad in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f6a0817e700 (LWP 2482)):
#0  0x00000030e7adf283 in poll () from /lib64/libc.so.6
#1  0x00007f6a6b314534 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-16-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007f6a6b31479a in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-16-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007f6a6b31508e in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-16-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007f6a728efa92 in edm::InputTag::InputTag(std::string const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-16-1100/lib/slc6_amd64_gcc530/libFWCoreUtilities.so
#6  0x00007f6a28d1b10b in egHLT::trigTools::getFiltersPassed(std::vector<std::pair<std::string, int>, std::allocator<std::pair<std::string, int> > > const&, trigger::TriggerEvent const*, std::string const&, egHLT::TrigCodes const&) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/pluginDQMOfflineTrigger.so
#7  0x00007f6a28c24f95 in egHLT::OffHelper::setTrigInfo(edm::Event const&, egHLT::OffEvt&, egHLT::TrigCodes const&) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/pluginDQMOfflineTrigger.so
#8  0x00007f6a28c28de9 in egHLT::OffHelper::makeOffEvt(edm::Event const&, edm::EventSetup const&, egHLT::OffEvt&, egHLT::TrigCodes const&) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/pluginDQMOfflineTrigger.so
#9  0x00007f6a28c0a558 in EgHLTOfflineSource::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/pluginDQMOfflineTrigger.so
#10 0x00007f6a72cbc2fa in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#11 0x00007f6a72c97aa2 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#12 0x00007f6a72bcfbe4 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#13 0x00007f6a72bcfe78 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#14 0x00007f6a72bd01ed in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#15 0x00007f6a72bee9ce in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007f6a72beec82 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007f6a72bef0a5 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007f6a72bef39b in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007f6a72bef60f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007f6a72c71f92 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007f6a72c726c7 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007f6a72c78f24 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007f6a719949b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f6a4d84fe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#24 0x00007f6a7198e1a4 in tbb::internal::arena::process (this=this@entry=0x7f6a6fe1f780, s=...) at ../../src/tbb/arena.cpp:156
#25 0x00007f6a7198d264 in tbb::internal::market::process (this=0x7f6a6fe47e00, j=...) at ../../src/tbb/market.cpp:502
#26 0x00007f6a71989736 in tbb::internal::rml::private_worker::run (this=0x7f6a6fe3fa80) at ../../src/tbb/private_server.cpp:275
#27 0x00007f6a71989989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#28 0x00000030e7e07aa1 in start_thread () from /lib64/libpthread.so.0
#29 0x00000030e7ae8aad in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f6a076ff700 (LWP 2483)):
#0  0x00000030e7aacbdd in nanosleep () from /lib64/libc.so.6
#1  0x00000030e7aaca50 in sleep () from /lib64/libc.so.6
#2  0x00007f6a6b313b7e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-16-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f6a711dcae7 in __gnu_cxx::__atomic_add (__val=1, __mem=0x7f699314ac90) at /mnt/build/davidlt/gcc530_final/b/BUILD/slc6_amd64_gcc530/external/gcc/5.3.0/gcc-tags_gcc_5_3_0_release-231255/obj/x86_64-pc-linux-gnu/libstdc++-v3/include/ext/atomicity.h:53
#5  __gnu_cxx::__atomic_add_dispatch (__val=1, __mem=0x7f699314ac90) at /mnt/build/davidlt/gcc530_final/b/BUILD/slc6_amd64_gcc530/external/gcc/5.3.0/gcc-tags_gcc_5_3_0_release-231255/obj/x86_64-pc-linux-gnu/libstdc++-v3/include/ext/atomicity.h:96
#6  std::string::_Rep::_M_refcopy (this=0x7f699314ac80) at /mnt/build/davidlt/gcc530_final/b/BUILD/slc6_amd64_gcc530/external/gcc/5.3.0/gcc-tags_gcc_5_3_0_release-231255/obj/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/basic_string.h:2661
#7  std::string::_Rep::_M_grab (__alloc2=..., __alloc1=..., this=0x7f699314ac80) at /mnt/build/davidlt/gcc530_final/b/BUILD/slc6_amd64_gcc530/external/gcc/5.3.0/gcc-tags_gcc_5_3_0_release-231255/obj/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/basic_string.h:2627
#8  std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string (this=0x7f6a076f9748, __str=...) at /mnt/build/davidlt/gcc530_final/b/BUILD/slc6_amd64_gcc530/external/gcc/5.3.0/gcc-tags_gcc_5_3_0_release-231255/obj/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/basic_string.tcc:617
#9  0x00007f6a728aa6b6 in edm::tokenize(std::string const&, std::string const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-16-1100/lib/slc6_amd64_gcc530/libFWCoreUtilities.so
#10 0x00007f6a728ef958 in edm::InputTag::InputTag(std::string const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-16-1100/lib/slc6_amd64_gcc530/libFWCoreUtilities.so
#11 0x00007f6a28d1b10b in egHLT::trigTools::getFiltersPassed(std::vector<std::pair<std::string, int>, std::allocator<std::pair<std::string, int> > > const&, trigger::TriggerEvent const*, std::string const&, egHLT::TrigCodes const&) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/pluginDQMOfflineTrigger.so
#12 0x00007f6a28c24f95 in egHLT::OffHelper::setTrigInfo(edm::Event const&, egHLT::OffEvt&, egHLT::TrigCodes const&) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/pluginDQMOfflineTrigger.so
#13 0x00007f6a28c28de9 in egHLT::OffHelper::makeOffEvt(edm::Event const&, edm::EventSetup const&, egHLT::OffEvt&, egHLT::TrigCodes const&) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/pluginDQMOfflineTrigger.so
#14 0x00007f6a28c0a558 in EgHLTOfflineSource::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/pluginDQMOfflineTrigger.so
#15 0x00007f6a72cbc2fa in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007f6a72c97aa2 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007f6a72bcfbe4 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007f6a72bcfe78 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007f6a72bd01ed in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007f6a72bee9ce in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007f6a72beec82 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007f6a72bef0a5 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007f6a72bef39b in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#24 0x00007f6a72bef60f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#25 0x00007f6a72c71f92 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#26 0x00007f6a72c726c7 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#27 0x00007f6a72c78f24 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#28 0x00007f6a719949b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f6a4d82fe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#29 0x00007f6a7198e1a4 in tbb::internal::arena::process (this=this@entry=0x7f6a6fe1f780, s=...) at ../../src/tbb/arena.cpp:156
#30 0x00007f6a7198d264 in tbb::internal::market::process (this=0x7f6a6fe47e00, j=...) at ../../src/tbb/market.cpp:502
#31 0x00007f6a71989736 in tbb::internal::rml::private_worker::run (this=0x7f6a6fe3f980) at ../../src/tbb/private_server.cpp:275
#32 0x00007f6a71989989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#33 0x00000030e7e07aa1 in start_thread () from /lib64/libpthread.so.0
#34 0x00000030e7ae8aad in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f6a70e5fbc0 (LWP 2376)):
#0  0x00000030e7aacbdd in nanosleep () from /lib64/libc.so.6
#1  0x00000030e7aaca50 in sleep () from /lib64/libc.so.6
#2  0x00007f6a6b313b7e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-16-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f6a28d1b170 in egHLT::trigTools::getFiltersPassed(std::vector<std::pair<std::string, int>, std::allocator<std::pair<std::string, int> > > const&, trigger::TriggerEvent const*, std::string const&, egHLT::TrigCodes const&) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/pluginDQMOfflineTrigger.so
#5  0x00007f6a28c24f95 in egHLT::OffHelper::setTrigInfo(edm::Event const&, egHLT::OffEvt&, egHLT::TrigCodes const&) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/pluginDQMOfflineTrigger.so
#6  0x00007f6a28c28de9 in egHLT::OffHelper::makeOffEvt(edm::Event const&, edm::EventSetup const&, egHLT::OffEvt&, egHLT::TrigCodes const&) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/pluginDQMOfflineTrigger.so
#7  0x00007f6a28c0a558 in EgHLTOfflineSource::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/pluginDQMOfflineTrigger.so
#8  0x00007f6a72cbc2fa in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#9  0x00007f6a72c97aa2 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#10 0x00007f6a72bcfbe4 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#11 0x00007f6a72bcfe78 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#12 0x00007f6a72bd01ed in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#13 0x00007f6a72bee9ce in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#14 0x00007f6a72beec82 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#15 0x00007f6a72bef0a5 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007f6a72bef39b in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007f6a72bef60f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007f6a72c71f92 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007f6a72c726c7 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007f6a72c78f24 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007f6a719949b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f6a6fe46400, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#22 0x00007f6a72c7251e in edm::EventProcessor::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007f6a72bd2f0a in statemachine::HandleEvent::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#24 0x00007f6a72bd4498 in statemachine::HandleEvent::HandleEvent(boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::my_context) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#25 0x00007f6a72bdc1de in boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::shallow_construct(boost::intrusive_ptr<statemachine::HandleLumis> const&, boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>&) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#26 0x00007f6a72bdc3aa in boost::statechart::simple_state<statemachine::FirstLumi, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#27 0x00007f6a72c789d8 in boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#28 0x00007f6a72c6bafd in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#29 0x000000000040c844 in main::{lambda()#1}::operator()() const ()
#30 0x000000000040acc4 in main ()

Current Modules:

Module: EgHLTOfflineSource (crashed)
Module: EgHLTOfflineSource
Module: EgHLTOfflineSource
Module: EgHLTOfflineSource

A fatal system signal has occurred: segmentation violation
Dr15Jones commented 8 years ago

This code has a weird behavior. The line http://cmslxr.fnal.gov/source/DQMOffline/Trigger/src/EgHLTTrigTools.cc?v=CMSSW_8_1_X_2016-08-03-2300#0018

size_t filterNrInEvt = trigEvt->filterIndex(edm::InputTag(filters[filterNrInVec].first,"",hltTag).encode());

creates a temporary InputTag, calls encode on that which creates a string but then filterIndex has as an argument an InputTag which means the compiler adds a call to InputTag(const std::string&) where we crash.

I'm going to create a pull request which removes the call to encode() so we only create one temporary InputTag. It probably is not the cause of the crash but it will make the code cleaner and faster :).

Dr15Jones commented 8 years ago

@dan131riley Do you want to see if there is a problem in tokenizer?

Dr15Jones commented 8 years ago

Now reading the whole thread, I see that I had fixed other .encode() calls but missed this one.

dan131riley commented 8 years ago

I'll take another look at it--I already ran through the bizarre chain of construct/tokenize/construct once and didn't spot anything, but could have missed something.

I had also started on tracking down all the implicit uses of the InputTag(string) constructor (by making it explicit), should finish that, there's a few other places where we're calling it without any reason.

Dr15Jones commented 8 years ago

edm::InputTag::encode contains a const static http://cmslxr.fnal.gov/source/FWCore/Utilities/src/InputTag.cc#0168

although that should be safe, maybe there is a problem with the code the compiler generates? Also, there is no reason to have that as a function local static. It could be a file scope static instead.

dan131riley commented 8 years ago

On Aug 17, 2016, at 12:01, Chris Jones notifications@github.com wrote:

edm::InputTag::encode contains a const static http://cmslxr.fnal.gov/source/FWCore/Utilities/src/InputTag.cc#0168

although that should be safe, maybe there is a problem with the code the compiler generates? Also, there is no reason to have that as a function local static. It could be a file scope static instead.

Make it file scope and we can use it here too:

http://cmslxr.fnal.gov/source/FWCore/Utilities/src/InputTag.cc#0054

std::vectorstd::string tokens = tokenize(s, ":");

args to tokenize are (const string& input, const string& separator)

and then it immediately does separator.c_str(), so we're doing

string(":").c_str()...

dan131riley commented 8 years ago

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc530/CMSSW_8_1_X_2016-08-17-1100/pyRelValMatrixLogs/run/1306.0_SingleMuPt1_UP15+SingleMuPt1_UP15INPUT+DIGIUP15+RECOUP15+HARVESTUP15/step3_SingleMuPt1_UP15+SingleMuPt1_UP15INPUT+DIGIUP15+RECOUP15+HARVESTUP15.log

This one is very interesting--two threads in EgHLTOfflineSource, one in TrackingMonitor (our other favorite module, #15145) and one just having left it. The actual crash is in jemalloc, underneath edm::InputTag::encode(). This adds some support to the theory that something else is corrupting the memory allocator (which EgHLTOfflineSource vigorously exercises with its InputTag thrashing), and raises the possibility that the something else could be TrackingMonitor.

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thread 5 (Thread 0x7f3ac6dbf700 (LWP 22854)):
#0  0x0000003de220f37d in waitpid () from /lib64/libpthread.so.0
#1  0x00007f3ad9e53d47 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-17-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007f3ad9e546d2 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-17-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007f3ae00f6a50 in std::(anonymous namespace)::execute_native_thread_routine (__p=<optimized out>) at ../../../../../libstdc++-v3/src/c++11/thread.cc:84
#4  0x0000003de2207aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003de1ee8aad in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f3a7753f700 (LWP 22890)):
#0  0x0000003de1eacbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003de1eaca50 in sleep () from /lib64/libc.so.6
#2  0x00007f3ad9e53b7e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-17-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000003de220e82d in read () from /lib64/libpthread.so.0
#5  0x00007f3ad9e652ca in edm::service::ProcInfoFetcher::fetch() const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-17-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#6  0x00007f3ad9e693dd in edm::service::SimpleMemoryCheck::fetch() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-17-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#7  0x00007f3ad9e69806 in edm::service::SimpleMemoryCheck::update() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-17-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#8  0x00007f3ad9e6bbdb in edm::service::SimpleMemoryCheck::updateAndPrint(std::string const&, std::string const&, std::string const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-17-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#9  0x00007f3ad9e6c695 in edm::service::SimpleMemoryCheck::postModule(edm::StreamContext const&, edm::ModuleCallingContext const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-17-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#10 0x00007f3ae1bec333 in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#11 0x00007f3ae1bc7aa2 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#12 0x00007f3ae1affbe4 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#13 0x00007f3ae1affe78 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#14 0x00007f3ae1b001ed in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#15 0x00007f3ae1b1e9ce in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007f3ae1b1ec82 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007f3ae1b1f0a5 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007f3ae1b1f39b in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007f3ae1b1f60f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007f3ae1ba1f92 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007f3ae1ba26c7 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007f3ae1ba8f24 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007f3ae08c49b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f3abc0ffe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#24 0x00007f3ae08be1a4 in tbb::internal::arena::process (this=this@entry=0x7f3aded3f780, s=...) at ../../src/tbb/arena.cpp:156
#25 0x00007f3ae08bd264 in tbb::internal::market::process (this=0x7f3aded67e00, j=...) at ../../src/tbb/market.cpp:502
#26 0x00007f3ae08b9736 in tbb::internal::rml::private_worker::run (this=0x7f3aded5fa00) at ../../src/tbb/private_server.cpp:275
#27 0x00007f3ae08b9989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#28 0x0000003de2207aa1 in start_thread () from /lib64/libpthread.so.0
#29 0x0000003de1ee8aad in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f3a76b3e700 (LWP 22891)):
#0  0x0000003de1eacbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003de1eaca50 in sleep () from /lib64/libc.so.6
#2  0x00007f3ad9e53b7e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-17-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f3aded92f6c in rml::internal::Block::allocateFromFreeList (this=0x7f3abc2ac000, this=0x7f3abc2ac000) at ../../src/tbbmalloc/frontend.cpp:2054
#5  rml::internal::Block::allocate (this=0x7f3abc2ac000) at ../../src/tbbmalloc/frontend.cpp:2087
#6  rml::internal::internalPoolMalloc (memPool=memPool@entry=0x7f3aded9e440 <rml::internal::defaultMemPool_space>, size=16) at ../../src/tbbmalloc/frontend.cpp:2499
#7  0x00007f3aded93234 in rml::internal::internalMalloc (size=<optimized out>) at ../../src/tbbmalloc/frontend.cpp:2586
#8  0x00007f3aded93b36 in scalable_malloc (size=<optimized out>) at ../../src/tbbmalloc/frontend.cpp:2833
#9  0x00007f3ae08b4c0a in tbb::internal::allocate_via_handler_v3 (n=<optimized out>) at ../../src/tbb/cache_aligned_allocator.cpp:231
#10 0x00007f3ad4096adb in reco::parser::LazyInvoker::invoker(edm::TypeWithDict const&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-17-1100/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#11 0x00007f3ad4097ebb in reco::parser::LazyInvoker::invokeLast(edm::ObjectWithDict const&, std::vector<edm::ObjectWithDict, std::allocator<edm::ObjectWithDict> >&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-17-1100/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#12 0x00007f3ad40bea1f in reco::parser::ExpressionLazyVar::value(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-17-1100/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#13 0x00007f3ad409afb6 in reco::parser::BinarySelector::operator()(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-17-1100/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#14 0x00007f3a9be22c89 in TrackingMonitor::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-17-1100/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#15 0x00007f3ae1bec2fa in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007f3ae1bc7aa2 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007f3ae1affbe4 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007f3ae1affe78 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007f3ae1b001ed in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007f3ae1b1e9ce in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007f3ae1b1ec82 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007f3ae1b1f0a5 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007f3ae1b1f39b in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#24 0x00007f3ae1b1f60f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#25 0x00007f3ae1ba1f92 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#26 0x00007f3ae1ba26c7 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#27 0x00007f3ae1ba8f24 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#28 0x00007f3ae08c49b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f3abc10fe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#29 0x00007f3ae08be1a4 in tbb::internal::arena::process (this=this@entry=0x7f3aded3f780, s=...) at ../../src/tbb/arena.cpp:156
#30 0x00007f3ae08bd264 in tbb::internal::market::process (this=0x7f3aded67e00, j=...) at ../../src/tbb/market.cpp:502
#31 0x00007f3ae08b9736 in tbb::internal::rml::private_worker::run (this=0x7f3aded5fa80) at ../../src/tbb/private_server.cpp:275
#32 0x00007f3ae08b9989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#33 0x0000003de2207aa1 in start_thread () from /lib64/libpthread.so.0
#34 0x0000003de1ee8aad in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f3a7603f700 (LWP 22892)):
#0  0x0000003de1edf283 in poll () from /lib64/libc.so.6
#1  0x00007f3ad9e54534 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-17-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007f3ad9e5479a in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-17-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007f3ad9e5508e in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-17-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  je_tsd_arenas_tdata_get (tsd=<optimized out>) at include/jemalloc/internal/tsd.h:716
#6  je_arena_tdata_get (refresh_if_missing=<optimized out>, ind=<optimized out>, tsd=<optimized out>) at include/jemalloc/internal/jemalloc_internal.h:832
#7  je_decay_ticker_get (ind=<optimized out>, tsd=<optimized out>) at include/jemalloc/internal/jemalloc_internal.h:874
#8  je_arena_decay_ticks (nticks=<optimized out>, arena=<optimized out>, tsdn=<optimized out>) at include/jemalloc/internal/arena.h:1309
#9  je_tcache_bin_flush_small (tsd=0x7f007603b310, tcache=<optimized out>, tbin=0x7f3a7604a0c8, binind=<optimized out>, rem=2) at src/tcache.c:148
#10 0x00007f3ae0b1e9dc in je_tcache_event_hard (tsd=<optimized out>, tcache=tcache@entry=0x7f3a7604a000) at src/tcache.c:45
#11 0x00007f3ae0aeb8cb in je_tcache_event (tcache=<optimized out>, tsd=<optimized out>) at include/jemalloc/internal/tcache.h:249
#12 je_tcache_alloc_large (slow_path=<optimized out>, zero=<optimized out>, binind=<optimized out>, size=<optimized out>, tcache=<optimized out>, arena=<optimized out>, tsd=<optimized out>) at include/jemalloc/internal/tcache.h:397
#13 je_arena_malloc (slow_path=<optimized out>, tcache=<optimized out>, zero=<optimized out>, ind=<optimized out>, size=<optimized out>, arena=<optimized out>, tsdn=<optimized out>) at include/jemalloc/internal/arena.h:1337
#14 je_iallocztm (slow_path=<optimized out>, arena=<optimized out>, is_metadata=<optimized out>, tcache=<optimized out>, zero=<optimized out>, ind=<optimized out>, size=<optimized out>, tsdn=<optimized out>) at include/jemalloc/internal/jemalloc_internal.h:967
#15 je_ialloc (slow_path=<optimized out>, zero=<optimized out>, ind=<optimized out>, size=<optimized out>, tsd=<optimized out>) at include/jemalloc/internal/jemalloc_internal.h:979
#16 ialloc_body (slow_path=<optimized out>, usize=<optimized out>, tsdn=<optimized out>, zero=<optimized out>, size=<optimized out>) at src/jemalloc.c:1520
#17 malloc (size=size@entry=29) at src/jemalloc.c:1562
#18 0x00007f3ae00ccf68 in operator new (sz=29) at ../../../../libstdc++-v3/libsupc++/new_op.cc:50
#19 0x00007f3ae010b639 in __gnu_cxx::new_allocator<char>::allocate (this=<optimized out>, __n=<optimized out>) at /mnt/build/davidlt/gcc530_final/b/BUILD/slc6_amd64_gcc530/external/gcc/5.3.0/gcc-tags_gcc_5_3_0_release-231255/obj/x86_64-pc-linux-gnu/libstdc++-v3/include/ext/new_allocator.h:104
#20 std::string::_Rep::_S_create (__capacity=4, __old_capacity=<optimized out>, __alloc=...) at /mnt/build/davidlt/gcc530_final/b/BUILD/slc6_amd64_gcc530/external/gcc/5.3.0/gcc-tags_gcc_5_3_0_release-231255/obj/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/basic_string.tcc:1051
#21 0x00007f3ae010c41b in std::string::_Rep::_M_clone (this=0x7f3ad745cf80, __alloc=..., __res=<optimized out>) at /mnt/build/davidlt/gcc530_final/b/BUILD/slc6_amd64_gcc530/external/gcc/5.3.0/gcc-tags_gcc_5_3_0_release-231255/obj/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/basic_string.tcc:1073
#22 0x00007f3ae010c4c4 in std::string::reserve (this=this@entry=0x7f3a76039b10, __res=<optimized out>, __res@entry=4) at /mnt/build/davidlt/gcc530_final/b/BUILD/slc6_amd64_gcc530/external/gcc/5.3.0/gcc-tags_gcc_5_3_0_release-231255/obj/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/basic_string.tcc:954
#23 0x00007f3ae010c598 in std::string::append (this=0x7f3a76039b10, __str=...) at /mnt/build/davidlt/gcc530_final/b/BUILD/slc6_amd64_gcc530/external/gcc/5.3.0/gcc-tags_gcc_5_3_0_release-231255/obj/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/basic_string.tcc:776
#24 0x00007f3ae1cfb1cc in std::basic_string<char, std::char_traits<char>, std::allocator<char> > std::operator+<char, std::char_traits<char>, std::allocator<char> >(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreParameterSet.so
#25 0x00007f3ae1820156 in edm::InputTag::encode() const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-17-1100/lib/slc6_amd64_gcc530/libFWCoreUtilities.so
#26 0x00007f3a975caa10 in void egHLT::trigTools::setFiltersObjPasses<egHLT::OffEle>(std::vector<egHLT::OffEle, std::allocator<egHLT::OffEle> >&, std::vector<std::string, std::allocator<std::string> > const&, std::vector<std::pair<std::string, std::string>, std::allocator<std::pair<std::string, std::string> > > const&, std::bitset<128ul> const&, egHLT::TrigCodes const&, trigger::TriggerEvent const*, std::string const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-17-1100/lib/slc6_amd64_gcc530/pluginDQMOfflineTrigger.so
#27 0x00007f3a975c440c in egHLT::OffHelper::setTrigInfo(edm::Event const&, egHLT::OffEvt&, egHLT::TrigCodes const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-17-1100/lib/slc6_amd64_gcc530/pluginDQMOfflineTrigger.so
#28 0x00007f3a975c81d9 in egHLT::OffHelper::makeOffEvt(edm::Event const&, edm::EventSetup const&, egHLT::OffEvt&, egHLT::TrigCodes const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-17-1100/lib/slc6_amd64_gcc530/pluginDQMOfflineTrigger.so
#29 0x00007f3a976bdef8 in EgHLTOfflineSource::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-17-1100/lib/slc6_amd64_gcc530/pluginDQMOfflineTrigger.so
#30 0x00007f3ae1bec2fa in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#31 0x00007f3ae1bc7aa2 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#32 0x00007f3ae1affbe4 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#33 0x00007f3ae1affe78 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#34 0x00007f3ae1b001ed in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#35 0x00007f3ae1b1e9ce in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#36 0x00007f3ae1b1ec82 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#37 0x00007f3ae1b1f0a5 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#38 0x00007f3ae1b1f39b in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#39 0x00007f3ae1b1f60f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#40 0x00007f3ae1ba1f92 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#41 0x00007f3ae1ba26c7 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#42 0x00007f3ae1ba8f24 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#43 0x00007f3ae08c49b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f3abc0efe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#44 0x00007f3ae08be1a4 in tbb::internal::arena::process (this=this@entry=0x7f3aded3f780, s=...) at ../../src/tbb/arena.cpp:156
#45 0x00007f3ae08bd264 in tbb::internal::market::process (this=0x7f3aded67e00, j=...) at ../../src/tbb/market.cpp:502
#46 0x00007f3ae08b9736 in tbb::internal::rml::private_worker::run (this=0x7f3aded5f980) at ../../src/tbb/private_server.cpp:275
#47 0x00007f3ae08b9989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#48 0x0000003de2207aa1 in start_thread () from /lib64/libpthread.so.0
#49 0x0000003de1ee8aad in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f3adfd8fbc0 (LWP 22824)):
#0  0x0000003de1eacbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003de1eaca50 in sleep () from /lib64/libc.so.6
#2  0x00007f3ad9e53b7e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-17-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000003de1f3383b in __strlen_sse42 () from /lib64/libc.so.6
#5  0x00007f3ae010b3b8 in std::char_traits<char>::length (__s=<optimized out>) at /mnt/build/davidlt/gcc530_final/b/BUILD/slc6_amd64_gcc530/external/gcc/5.3.0/gcc-tags_gcc_5_3_0_release-231255/obj/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/char_traits.h:267
#6  std::string::compare (this=<optimized out>, __s=0x7ffe997ca250 "hltEle30WP60Ele8Mass55Filter") at /mnt/build/davidlt/gcc530_final/b/BUILD/slc6_amd64_gcc530/external/gcc/5.3.0/gcc-tags_gcc_5_3_0_release-231255/obj/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/basic_string.tcc:1398
#7  0x00007f3a976a471b in egHLT::TrigCodes::getCode(char const*) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-17-1100/lib/slc6_amd64_gcc530/pluginDQMOfflineTrigger.so
#8  0x00007f3a975cab06 in void egHLT::trigTools::setFiltersObjPasses<egHLT::OffEle>(std::vector<egHLT::OffEle, std::allocator<egHLT::OffEle> >&, std::vector<std::string, std::allocator<std::string> > const&, std::vector<std::pair<std::string, std::string>, std::allocator<std::pair<std::string, std::string> > > const&, std::bitset<128ul> const&, egHLT::TrigCodes const&, trigger::TriggerEvent const*, std::string const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-17-1100/lib/slc6_amd64_gcc530/pluginDQMOfflineTrigger.so
#9  0x00007f3a975c440c in egHLT::OffHelper::setTrigInfo(edm::Event const&, egHLT::OffEvt&, egHLT::TrigCodes const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-17-1100/lib/slc6_amd64_gcc530/pluginDQMOfflineTrigger.so
#10 0x00007f3a975c81d9 in egHLT::OffHelper::makeOffEvt(edm::Event const&, edm::EventSetup const&, egHLT::OffEvt&, egHLT::TrigCodes const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-17-1100/lib/slc6_amd64_gcc530/pluginDQMOfflineTrigger.so
#11 0x00007f3a976bdef8 in EgHLTOfflineSource::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-17-1100/lib/slc6_amd64_gcc530/pluginDQMOfflineTrigger.so
#12 0x00007f3ae1bec2fa in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#13 0x00007f3ae1bc7aa2 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#14 0x00007f3ae1affbe4 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#15 0x00007f3ae1affe78 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007f3ae1b001ed in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007f3ae1b1e9ce in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007f3ae1b1ec82 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007f3ae1b1f0a5 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007f3ae1b1f39b in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007f3ae1b1f60f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007f3ae1ba1f92 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007f3ae1ba26c7 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#24 0x00007f3ae1ba8f24 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#25 0x00007f3ae08c49b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f3aded66400, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#26 0x00007f3ae1ba251e in edm::EventProcessor::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#27 0x00007f3ae1b02f0a in statemachine::HandleEvent::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#28 0x00007f3ae1b04498 in statemachine::HandleEvent::HandleEvent(boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::my_context) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#29 0x00007f3ae1b0c1de in boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::shallow_construct(boost::intrusive_ptr<statemachine::HandleLumis> const&, boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>&) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#30 0x00007f3ae1b0c3aa in boost::statechart::simple_state<statemachine::FirstLumi, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#31 0x00007f3ae1ba89d8 in boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#32 0x00007f3ae1b9bafd in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#33 0x000000000040c844 in main::{lambda()#1}::operator()() const ()
#34 0x000000000040acc4 in main ()

Current Modules:

Module: EgHLTOfflineSource (crashed)
Module: TrackingMonitor
Module: EgHLTOfflineSource
Module: TrackingMonitor

A fatal system signal has occurred: segmentation violation
dan131riley commented 8 years ago

Note that this crash happened on record 8, while #15145 seems to be an initialization bug, so maybe different bugs but possibly both in TrackingMonitor? Re-running memcheck on this wf...

Dr15Jones commented 5 years ago

I think we have another piece of the puzzle here. The Address Santitizer found the following:

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc700/CMSSW_10_5_ASAN_X_2019-01-11-2300/pyRelValMatrixLogs/run/10802.0_SingleElectronPt35+SingleElectronPt35_pythia8_2018_GenSimFullINPUT+DigiFull_2018+RecoFull_2018+HARVESTFull_2018+ALCAFull_2018+NanoFull_2018/step3_SingleElectronPt35+SingleElectronPt35_pythia8_2018_GenSimFullINPUT+DigiFull_2018+RecoFull_2018+HARVESTFull_2018+ALCAFull_2018+NanoFull_2018.log#/

    #0 0x7f1968462833 in __interceptor_strlen ../../../../libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:225
    #1 0x7f1967cf61c1 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::compare(char const*) const (/cvmfs/cms-ib.cern.ch/nweek-02558/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_5_ASAN_X_2019-01-11-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so+0x19f1c1)
    #2 0x7f190ea66134 in egHLT::TrigCodes::getCode(char const*) const (/cvmfs/cms-ib.cern.ch/nweek-02558/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_5_ASAN_X_2019-01-11-2300/lib/slc7_amd64_gcc700/libDQMOfflineTrigger.so+0x21a134)
    #3 0x7f190ea43023 in egHLT::trigTools::getFiltersPassed(std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int> > > const&amp, trigger::TriggerEvent const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&amp, egHLT::TrigCodes const&amp) (/cvmfs/cms-ib.cern.ch/nweek-02558/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_5_ASAN_X_2019-01-11-2300/lib/slc7_amd64_gcc700/libDQMOfflineTrigger.so+0x1f7023)
    #4 0x7f190e92337d in egHLT::OffHelper::setTrigInfo(edm::Event const&amp, egHLT::OffEvt&amp, egHLT::TrigCodes const&amp) (/cvmfs/cms-ib.cern.ch/nweek-02558/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_5_ASAN_X_2019-01-11-2300/lib/slc7_amd64_gcc700/libDQMOfflineTrigger.so+0xd737d)
    #5 0x7f190e93ae78 in egHLT::OffHelper::makeOffEvt(edm::Event const&amp, edm::EventSetup const&amp, egHLT::OffEvt&amp, egHLT::TrigCodes const&amp) (/cvmfs/cms-ib.cern.ch/nweek-02558/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_5_ASAN_X_2019-01-11-2300/lib/slc7_amd64_gcc700/libDQMOfflineTrigger.so+0xeee78)
    #6 0x7f190d84b220 in EgHLTOfflineSource::analyze(edm::Event const&amp, edm::EventSetup const&amp) (/cvmfs/cms-ib.cern.ch/nweek-02558/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_5_ASAN_X_2019-01-11-2300/lib/slc7_amd64_gcc700/pluginDQMOfflineTriggerPlugins.so+0x8d0220)
    #7 0x7f196829ce34 in edm::one::EDProducerBase::doEvent(edm::EventPrincipal const&amp, edm::EventSetup const&amp, edm::ActivityRegistry*, edm::ModuleCallingContext const*) (/cvmfs/cms-ib.cern.ch/nweek-02558/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_5_ASAN_X_2019-01-11-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so+0x745e34)
[cut irrelevant part of stack trace]

Address 0x7f192ae05198 is located in stack of thread T2 at offset 200 in frame
    #0 0x7f190e99c38f in egHLT::EleHLTFilterMon::fill(egHLT::OffEvt const&amp, float) (/cvmfs/cms-ib.cern.ch/nweek-02558/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_5_ASAN_X_2019-01-11-2300/lib/slc7_amd64_gcc700/libDQMOfflineTrigger.so+0x15038f)

  This frame has 36 object(s):
    [32, 40) '__for_begin'
    [96, 104) '__for_end'
    [160, 168) '__for_begin'
    [224, 232) '__for_end' <== Memory access at offset 200 underflows this variable
    [288, 296) '__for_begin'
    [352, 360) '__for_end'
    [416, 424) '<unknown>'
    [480, 488) '<unknown>'
    [544, 552) '<unknown>'
    [608, 616) '<unknown>'
    [672, 680) '<unknown>'
    [736, 744) '<unknown>'
    [800, 816) '<unknown>'
    [864, 880) '<unknown>'
    [928, 944) '<unknown>'
    [992, 1008) '<unknown>'
    [1056, 1072) '<unknown>'
    [1120, 1136) '<unknown>'
    [1184, 1200) '<unknown>'
    [1248, 1264) '<unknown>'
    [1312, 1328) '<unknown>'
    [1376, 1392) '<unknown>'
    [1440, 1456) '<unknown>'
    [1504, 1520) '<unknown>'
    [1568, 1584) '<unknown>'
    [1632, 1648) '<unknown>'
    [1696, 1712) '<unknown>'
    [1760, 1776) '<unknown>'
    [1824, 1840) '<unknown>'
    [1888, 1904) '<unknown>'
    [1952, 1968) '__result'
    [2016, 2032) '__result'
    [2080, 2096) '__result'
    [2144, 2160) '__result'
    [2208, 2224) '__result'
    [2272, 2288) '__result'
HINT: this may be a false positive if your program uses some custom stack unwind mechanism or swapcontext
      (longjmp and C++ exceptions *are* supported)

SUMMARY: AddressSanitizer: stack-buffer-overflow ../../../../libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:225 in __interceptor_strlen
Shadow bytes around the buggy address:
  0x0fe3a55b89e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0fe3a55b89f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0fe3a55b8a00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0fe3a55b8a10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0fe3a55b8a20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0fe3a55b8a30: 00 00 00[00]00 00 00 00 00 00 00 00 00 00 00 00
  0x0fe3a55b8a40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0fe3a55b8a50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0fe3a55b8a60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0fe3a55b8a70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0fe3a55b8a80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==11767==ABORTING

The point of code is here https://github.com/cms-sw/cmssw/blob/02d4198c0b6615287fd88e9a8ff650aea994412e/DQMOffline/Trigger/src/EgHLTTrigCodes.cc#L47

Notice the fixed sized array without any bounds checking. If we wrote passed the end we'd be doing bad stuff to the stack which could cause all kinds of chaos.

dan131riley commented 5 years ago

Good find. I would bet on strtok() being the cause of the crash. It writes null characters into the saved string, and the corruption looks like thread 0 writing a null into a localDescript that was on the stack of thread 2. Good to get both problems fixed, I'll bet it finally resolves this issue.

smuzaffar commented 5 years ago

@Dr15Jones , should we close this now or you would like to keep it open?

Dr15Jones commented 5 years ago

+1

cmsbuild commented 5 years ago

This issue is fully signed and ready to be closed.

Dr15Jones commented 5 years ago

Let's go ahead and close