Seg Fault at beginning of event loop in threaded job

[Dr15Jones]

This job crashed with a segmentation fault: https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc530/CMSSW_8_1_THREADED_X_2016-07-06-2300/pyRelValMatrixLogs/run/140.2_PhotonJets_Pt_10_13_HI+PhotonJets_Pt_10_13_HIINPUT+DIGIHI+RECOHI+HARVESTHI/step3_PhotonJets_Pt_10_13_HI+PhotonJets_Pt_10_13_HIINPUT+DIGIHI+RECOHI+HARVESTHI.log

What is interesting is only two of the threads had begun processing events before the crash (two of the threads are waiting in TBB probably because they have yet to get their task to run). So although the thread with the crash has no useable traceback, we know we are out of the Source and somewhere processing data for one of the events .

The relevant parts of the log are below:

%MSG-i ThreadSetup:  (NoModuleName) 07-Jul-2016 06:56:22 CEST pre-events
setting # threads 4
%MSG
DQMStore: histogram collation is enabled
07-Jul-2016 06:56:44 CEST  Initiating request to open file file:step2.root
07-Jul-2016 06:56:47 CEST  Successfully opened file file:step2.root
Begin processing the 1st record. Run 1, Event 1351, LumiSection 10 at 07-Jul-2016 06:58:44.164 CEST
Begin processing the 2nd record. Run 1, Event 1352, LumiSection 10 at 07-Jul-2016 06:58:44.166 CEST
%MSG-w EcalDQM:  EcalDQMonitorTask:ecalMonitorTask 07-Jul-2016 06:59:02 CEST  Run: 1 Event: 1352
Ecal Monitor Source::runOnCollection: EBReducedRecHit does not exist
%MSG
%MSG-w EcalDQM:  EcalDQMonitorTask:ecalMonitorTask 07-Jul-2016 06:59:02 CEST  Run: 1 Event: 1352
Ecal Monitor Source::runOnCollection: EEReducedRecHit does not exist
%MSG
%MSG-w EcalDQM:  EcalDQMonitorTask:ecalMonitorTask 07-Jul-2016 06:59:02 CEST  Run: 1 Event: 1351
Ecal Monitor Source::runOnCollection: EBReducedRecHit does not exist
%MSG
%MSG-w EcalDQM:  EcalDQMonitorTask:ecalMonitorTask 07-Jul-2016 06:59:02 CEST  Run: 1 Event: 1351
Ecal Monitor Source::runOnCollection: EEReducedRecHit does not exist
%MSG
%MSG-e Bad PilotBlade blade number :   SiPixelHitEfficiencySource:SiPixelHitEfficiencySource_HeavyIons  07-Jul-2016 06:59:03 CEST Run: 1 Event: 1351
5
%MSG
%MSG-e Bad PilotBlade blade number :   SiPixelHitEfficiencySource:SiPixelHitEfficiencySource_HeavyIons  07-Jul-2016 06:59:03 CEST Run: 1 Event: 1352
7
%MSG
%MSG-e Bad PilotBlade blade number :   SiPixelHitEfficiencySource:SiPixelHitEfficiencySource_HeavyIons  07-Jul-2016 06:59:03 CEST Run: 1 Event: 1352
9
%MSG
%MSG-e Bad PilotBlade blade number :   SiPixelHitEfficiencySource:SiPixelHitEfficiencySource_HeavyIons  07-Jul-2016 06:59:03 CEST Run: 1 Event: 1352
10
%MSG
%MSG-e Bad PilotBlade blade number :   SiPixelHitEfficiencySource:SiPixelHitEfficiencySource_HeavyIons  07-Jul-2016 06:59:03 CEST Run: 1 Event: 1351
6
%MSG
%MSG-e Bad PilotBlade blade number :   SiPixelHitEfficiencySource:SiPixelHitEfficiencySource_HeavyIons  07-Jul-2016 06:59:03 CEST Run: 1 Event: 1351
8
%MSG
%MSG-e Bad PilotBlade blade number :   SiPixelHitEfficiencySource:SiPixelHitEfficiencySource_HeavyIons  07-Jul-2016 06:59:03 CEST Run: 1 Event: 1351
9
%MSG
%MSG-e Bad PilotBlade blade number :   SiPixelHitEfficiencySource:SiPixelHitEfficiencySource_HeavyIons  07-Jul-2016 06:59:03 CEST Run: 1 Event: 1351
10
%MSG
%MSG-e Bad PilotBlade blade number :   SiPixelHitEfficiencySource:SiPixelHitEfficiencySource_HeavyIons  07-Jul-2016 06:59:03 CEST Run: 1 Event: 1352
10
%MSG
%MSG-e Bad PilotBlade blade number :   SiPixelHitEfficiencySource:SiPixelHitEfficiencySource_HeavyIons  07-Jul-2016 06:59:03 CEST Run: 1 Event: 1351
5
%MSG
%MSG-e Bad PilotBlade blade number :   SiPixelHitEfficiencySource:SiPixelHitEfficiencySource_HeavyIons  07-Jul-2016 06:59:03 CEST Run: 1 Event: 1351
9
%MSG

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thread 5 (Thread 0x7f813b4bf700 (LWP 16989)):
#0  0x0000003338c0f37d in waitpid () from /lib64/libpthread.so.0
#1  0x00007f814bfd36a7 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/2016-28/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-05-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007f814bfd3f52 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/2016-28/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-05-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007f8151c97a50 in std::(anonymous namespace)::execute_native_thread_routine (__p=<optimized out>) at ../../../../../libstdc++-v3/src/c++11/thread.cc:84
#4  0x0000003338c07aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033388e8aad in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f804a4ff700 (LWP 17156)):
#0  0x00000033388e5309 in syscall () from /lib64/libc.so.6
#1  0x00007f8151fe2962 in tbb::internal::futex_wait (comparand=2, futex=0x7f815091ba2c) at ../../include/tbb/machine/linux_common.h:60
#2  tbb::internal::binary_semaphore::P (this=0x7f815091ba2c) at ../../src/tbb/semaphore.h:206
#3  rml::internal::thread_monitor::commit_wait (c=<synthetic pointer>, this=0x7f815091ba20) at ../../src/rml/include/../server/thread_monitor.h:259
#4  tbb::internal::rml::private_worker::run (this=0x7f815091ba00) at ../../src/tbb/private_server.cpp:282
#5  0x00007f8151fe2989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#6  0x0000003338c07aa1 in start_thread () from /lib64/libpthread.so.0
#7  0x00000033388e8aad in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f8049afe700 (LWP 17157)):
#0  0x00000033388df283 in poll () from /lib64/libc.so.6
#1  0x00007f814bfd3db4 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/2016-28/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-05-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007f814bfd401a in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/2016-28/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-05-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007f814bfd419b in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/2016-28/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-05-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007f814da7d08e in ?? ()
#6  0x00007f8049af84a8 in ?? ()
#7  0x00007f80290978a8 in ?? ()
#8  0x0000000000000000 in ?? ()
Thread 2 (Thread 0x7f804903f700 (LWP 17158)):
#0  0x00000033388e5309 in syscall () from /lib64/libc.so.6
#1  0x00007f8151fe2962 in tbb::internal::futex_wait (comparand=2, futex=0x7f815091b9ac) at ../../include/tbb/machine/linux_common.h:60
#2  tbb::internal::binary_semaphore::P (this=0x7f815091b9ac) at ../../src/tbb/semaphore.h:206
#3  rml::internal::thread_monitor::commit_wait (c=<synthetic pointer>, this=0x7f815091b9a0) at ../../src/rml/include/../server/thread_monitor.h:259
#4  tbb::internal::rml::private_worker::run (this=0x7f815091b980) at ../../src/tbb/private_server.cpp:282
#5  0x00007f8151fe2989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#6  0x0000003338c07aa1 in start_thread () from /lib64/libpthread.so.0
#7  0x00000033388e8aad in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f81533beba0 (LWP 16916)):
#0  0x0000003338c0f06d in open64 () from /lib64/libpthread.so.0
#1  0x00007f81463978f8 in llvm::sys::fs::openFileForRead(llvm::Twine const&, int&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-06-2300/external/slc6_amd64_gcc530/lib/libCling.so
#2  0x00007f8146397955 in llvm::sys::fs::identify_magic(llvm::Twine const&, llvm::sys::fs::file_magic&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-06-2300/external/slc6_amd64_gcc530/lib/libCling.so
#3  0x00007f8144eeda2f in cling::isSharedLib(llvm::StringRef, bool*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-06-2300/external/slc6_amd64_gcc530/lib/libCling.so
#4  0x00007f8144eee9b2 in cling::DynamicLibraryManager::lookupLibrary(llvm::StringRef) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-06-2300/external/slc6_amd64_gcc530/lib/libCling.so
#5  0x00007f8144eefda8 in cling::DynamicLibraryManager::loadLibrary(std::string const&, bool) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-06-2300/external/slc6_amd64_gcc530/lib/libCling.so
#6  0x00007f8144e8d115 in TCling::RegisterLoadedSharedLibrary(char const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-06-2300/external/slc6_amd64_gcc530/lib/libCling.so
#7  0x00007f8144e8d2a7 in TCling::UpdateListOfLoadedSharedLibraries() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-06-2300/external/slc6_amd64_gcc530/lib/libCling.so
#8  0x00007f8144e8d44d in TCling::Load(char const*, bool) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-06-2300/external/slc6_amd64_gcc530/lib/libCling.so
#9  0x00007f81526aa5dc in TSystem::Load(char const*, char const*, bool) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-06-2300/external/slc6_amd64_gcc530/lib/libCore.so
#10 0x00007f815260d687 in TROOT::LoadClass(char const*, char const*, bool) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-06-2300/external/slc6_amd64_gcc530/lib/libCore.so
#11 0x00007f8144e88a9d in TCling::AutoLoad(char const*, bool) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-06-2300/external/slc6_amd64_gcc530/lib/libCling.so
#12 0x00007f8152743c9b in TClass::GetClass(char const*, bool, bool) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-06-2300/external/slc6_amd64_gcc530/lib/libCore.so
#13 0x00007f815275eac9 in TStreamerElement::GetClassPointer() const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-06-2300/external/slc6_amd64_gcc530/lib/libCore.so
#14 0x00007f815275f22d in TStreamerElement::Init(TObject*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-06-2300/external/slc6_amd64_gcc530/lib/libCore.so
#15 0x00007f8152978408 in TStreamerInfo::BuildOld() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-06-2300/external/slc6_amd64_gcc530/lib/libRIO.so
#16 0x00007f815273a0fc in TClass::GetStreamerInfo(int) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-06-2300/external/slc6_amd64_gcc530/lib/libCore.so
#17 0x00007f8152761ee2 in TStreamerBase::InitStreaming() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-06-2300/external/slc6_amd64_gcc530/lib/libCore.so
#18 0x00007f8152979f8d in TStreamerInfo::BuildOld() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-06-2300/external/slc6_amd64_gcc530/lib/libRIO.so
#19 0x00007f8152a2ac66 in TBufferFile::ReadClassBuffer(TClass const*, void*, TClass const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-06-2300/external/slc6_amd64_gcc530/lib/libRIO.so
#20 0x00007f8152ab1b27 in int TStreamerInfo::ReadBuffer<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-06-2300/external/slc6_amd64_gcc530/lib/libRIO.so
#21 0x00007f815299a80d in TStreamerInfoActions::GenericReadAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-06-2300/external/slc6_amd64_gcc530/lib/libRIO.so
#22 0x00007f8152a29d75 in TBufferFile::ApplySequence(TStreamerInfoActions::TActionSequence const&, void*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-06-2300/external/slc6_amd64_gcc530/lib/libRIO.so
#23 0x00007f8152e5e9cd in TBranchElement::ReadLeavesMember(TBuffer&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-06-2300/external/slc6_amd64_gcc530/lib/libTree.so
#24 0x00007f8152ec1082 in TBranch::GetEntry(long long, int) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-06-2300/external/slc6_amd64_gcc530/lib/libTree.so
#25 0x00007f8152e6a320 in TBranchElement::GetEntry(long long, int) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-06-2300/external/slc6_amd64_gcc530/lib/libTree.so
#26 0x00007f8152e6a2c5 in TBranchElement::GetEntry(long long, int) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-06-2300/external/slc6_amd64_gcc530/lib/libTree.so
#27 0x00007f81358c2caf in edm::RootTree::getEntry(TBranch*, long long) const () from /cvmfs/cms-ib.cern.ch/2016-28/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-05-2300/lib/slc6_amd64_gcc530/pluginIOPoolInput.so
#28 0x00007f81358cbb89 in edm::RootDelayedReader::getProduct_(edm::BranchKey const&, edm::EDProductGetter const*) () from /cvmfs/cms-ib.cern.ch/2016-28/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-05-2300/lib/slc6_amd64_gcc530/pluginIOPoolInput.so
#29 0x00007f8153265db2 in edm::DelayedReader::getProduct(edm::BranchKey const&, edm::EDProductGetter const*, edm::signalslot::Signal<void (edm::StreamContext const&, edm::ModuleCallingContext const&)> const*, edm::signalslot::Signal<void (edm::StreamContext const&, edm::ModuleCallingContext const&)> const*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-28/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-05-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#30 0x00007f8153334a54 in edm::EventPrincipal::readFromSource_(edm::ProductResolverBase const&, edm::ModuleCallingContext const*) const () from /cvmfs/cms-ib.cern.ch/2016-28/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-05-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#31 0x00007f81532556f0 in edm::InputProductResolver::resolveProduct_(edm::ProductResolverBase::ResolveStatus&, edm::Principal const&, bool, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const () from /cvmfs/cms-ib.cern.ch/2016-28/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-05-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#32 0x00007f81532e53d1 in edm::Principal::prefetch(unsigned int, bool, edm::ModuleCallingContext const*) const () from /cvmfs/cms-ib.cern.ch/2016-28/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-05-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#33 0x00007f8153259717 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-28/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-05-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#34 0x00007f8153259902 in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-28/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-05-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#35 0x00007f8153275e3e in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-28/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-05-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#36 0x00007f81532760f2 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-28/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-05-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#37 0x00007f8153276515 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-28/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-05-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#38 0x00007f815327680b in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-28/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-05-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#39 0x00007f8153276a7f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-28/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-05-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#40 0x00007f81532dc622 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-28/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-05-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#41 0x00007f81532dcd57 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-28/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-05-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#42 0x00007f81532e3534 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-28/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-05-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#43 0x00007f8151fed9b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f8150922400, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#44 0x00007f81532dcbae in edm::EventProcessor::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/2016-28/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-05-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#45 0x00007f815325bb5a in statemachine::HandleEvent::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/2016-28/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-05-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#46 0x00007f815325d0e8 in statemachine::HandleEvent::HandleEvent(boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::my_context) () from /cvmfs/cms-ib.cern.ch/2016-28/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-05-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#47 0x00007f8153264e6e in boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::shallow_construct(boost::intrusive_ptr<statemachine::HandleLumis> const&, boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>&) () from /cvmfs/cms-ib.cern.ch/2016-28/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-05-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#48 0x00007f815326503a in boost::statechart::simple_state<statemachine::FirstLumi, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*) () from /cvmfs/cms-ib.cern.ch/2016-28/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-05-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#49 0x00007f81532e2fe8 in boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&) () from /cvmfs/cms-ib.cern.ch/2016-28/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-05-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#50 0x00007f81532d62dd in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/2016-28/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-05-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#51 0x000000000040c814 in main::{lambda()#1}::operator()() const ()
#52 0x000000000040ac94 in main ()

A fatal system signal has occurred: segmentation violation

[cmsbuild]

A new Issue was created by @Dr15Jones Chris Jones.

@davidlange6, @smuzaffar, @degano, @davidlt, @Dr15Jones can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[Dr15Jones]

assign core

[cmsbuild]

New categories assigned: core

@Dr15Jones,@smuzaffar you have been requested to review this Pull request/Issue and eventually sign? Thanks

[Dr15Jones]

@dan131riley

[dan131riley]

A somewhat similar no-traceback crash:

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc530/CMSSW_8_1_THREADED_X_2016-07-04-1100/pyRelValMatrixLogs/run/1004.0_RunHI2011+RunHI2011+TIER0EXPHI+ALCAEXPHI+ALCAHARVD1HI+ALCAHARVD2HI+ALCAHARVD3HI+ALCAHARVD5HI/step2_RunHI2011+RunHI2011+TIER0EXPHI+ALCAEXPHI+ALCAHARVD1HI+ALCAHARVD2HI+ALCAHARVD3HI+ALCAHARVD5HI.log

I looked at that one a bit on Tuesday, but my main discovery so far is the uninitialized variable that led to #15131.

-dan

[Dr15Jones]

The no-traceback crashes happen periodically and I've not made an effort of making issues for them since I could normally find ones with full tracebacks to look at :). We should definitely start cataloging these no-traceback ones. My suspicion is they are from code generated by cling.

[dan131riley]

Another one:

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc530/CMSSW_8_1_THREADED_X_2016-07-15-2300/pyRelValMatrixLogs/run/140.53_RunHI2011+RunHI2011+RECOHID11+HARVESTDHI/step2_RunHI2011+RunHI2011+RECOHID11+HARVESTDHI.log

I did figure out that the reason the first one only has two threads processing events is because that workflow only has two events :)

I'm going to start cataloging what the other threads are doing, as that may give some clue where the (assumed) conflict is occurring. In this case, two of the other threads are sitting in edm::SharedResourcesAcquirer::lock(), and the fourth is in edm::Principal::prefetch():

Thread 4 (Thread 0x7fbfa357f700 (LWP 18565)):
#0  0x00007fc073436051 in RPCStripsRing::compressConnections() () from /cvmfs/cms-ib.cern.ch/2016-29/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-13-2300/lib/slc6_amd64_gcc530/libL1TriggerRPCTrigger.so
#1  0x00007fc073464fa1 in RPCConeBuilder::produce(L1RPCConeBuilderRcd const&) () from /cvmfs/cms-ib.cern.ch/2016-29/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-13-2300/lib/slc6_amd64_gcc530/pluginL1TriggerRPCTriggerPlugins.so
#2  0x00007fc073467a2a in edm::eventsetup::CallbackProxy<edm::eventsetup::Callback<RPCConeBuilder, std::shared_ptr<L1RPCConeBuilder>, L1RPCConeBuilderRcd, edm::eventsetup::ESPreFunctorDecorator<L1RPCConeBuilderRcd, edm::eventsetup::DependsOnCaller<RPCConeBuilder, L1RPCConeBuilderRcd, L1RPCConeDefinitionRcd, edm::eventsetup::DependsOnCaller<RPCConeBuilder, L1RPCConeBuilderRcd, MuonGeometryRecord, edm::eventsetup::DependsOnDoNothingCaller<L1RPCConeBuilderRcd> > > > >, L1RPCConeBuilderRcd, std::shared_ptr<L1RPCConeBuilder> >::getImpl(edm::eventsetup::EventSetupRecord const&, edm::eventsetup::DataKey const&) () from /cvmfs/cms-ib.cern.ch/2016-29/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-13-2300/lib/slc6_amd64_gcc530/pluginL1TriggerRPCTriggerPlugins.so
#3  0x00007fc08ee028af in edm::eventsetup::DataProxy::get(edm::eventsetup::EventSetupRecord const&, edm::eventsetup::DataKey const&, bool) const () from /cvmfs/cms-ib.cern.ch/2016-29/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-13-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#4  0x00007fc08ee159ab in edm::eventsetup::EventSetupRecord::getFromProxy(edm::eventsetup::DataKey const&, edm::eventsetup::ComponentDescription const_&, bool) const () from /cvmfs/cms-ib.cern.ch/2016-29/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-13-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#5  0x00007fc073468a5d in RPCTrigger::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/2016-29/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-13-2300/lib/slc6_amd64_gcc530/pluginL1TriggerRPCTriggerPlugins.so
#6  0x00007fc08ee5c791 in edm::EDProducer::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry_, edm::ModuleCallingContext const_) () from /cvmfs/cms-ib.cern.ch/2016-29/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-13-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#7  0x00007fc08edcc0ef in edm::WorkerTedm::EDProducer::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const_) () from /cvmfs/cms-ib.cern.ch/2016-29/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-13-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#8  0x00007fc08ed8648c in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_)::{lambda()#1}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-29/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-13-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#9  0x00007fc08ed86652 in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_) () from /cvmfs/cms-ib.cern.ch/2016-29/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-13-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#10 0x00007fc08ed82c5a in edm::UnscheduledProductResolver::resolveProduct_(edm::ProductResolverBase::ResolveStatus&, edm::Principal const&, bool, edm::SharedResourcesAcquirer_, edm::ModuleCallingContext const_) const::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-29/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-13-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#11 0x00007fc08ed82f72 in edm::UnscheduledProductResolver::resolveProduct_(edm::ProductResolverBase::ResolveStatus&, edm::Principal const&, bool, edm::SharedResourcesAcquirer_, edm::ModuleCallingContext const_) const () from /cvmfs/cms-ib.cern.ch/2016-29/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-13-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#12 0x00007fc08ee2deb1 in edm::Principal::prefetch(unsigned int, bool, edm::ModuleCallingContext const_) const () from /cvmfs/cms-ib.cern.ch/2016-29/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-13-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#13 0x00007fc08ed86467 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_)::{lambda()#1}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-29/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-13-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#14 0x00007fc08ed86652 in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_) () from /cvmfs/cms-ib.cern.ch/2016-29/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-13-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#15 0x00007fc08ed82c5a in edm::UnscheduledProductResolver::resolveProduct_(edm::ProductResolverBase::ResolveStatus&, edm::Principal const&, bool, edm::SharedResourcesAcquirer_, edm::ModuleCallingContext const_) const::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-29/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-13-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007fc08ed82f72 in edm::UnscheduledProductResolver::resolveProduct_(edm::ProductResolverBase::ResolveStatus&, edm::Principal const&, bool, edm::SharedResourcesAcquirer_, edm::ModuleCallingContext const_) const () from /cvmfs/cms-ib.cern.ch/2016-29/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-13-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007fc08ee2deb1 in edm::Principal::prefetch(unsigned int, bool, edm::ModuleCallingContext const_) const () from /cvmfs/cms-ib.cern.ch/2016-29/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-13-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so

[Dr15Jones]

A thread waiting in edm::SharedResourcesAcquirer::lock() usually means the thread wants to run a legacy module but another thread is already running one.

The thread running edm::Principal::prefetch() is the one running the legacy module since it has edm::EDProducer::doEvent in the call stack. That same thread also has the EventSetup mutex lock since it is running an ESProducer.

P.S. I'm somewhere in the middle of the Gulf of Mexico :)

[dan131riley]

I'm not seeing any pattern developing...

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc530/CMSSW_8_1_THREADED_X_2016-07-22-2300/pyRelValMatrixLogs/run/4.22_RunCosmics2011A+RunCosmics2011A+RECOCOSD+ALCACOSD+SKIMCOSD+HARVESTDC/step2_RunCosmics2011A+RunCosmics2011A+RECOCOSD+ALCACOSD+SKIMCOSD+HARVESTDC.log

Something I find surprising...the

A fatal system signal has occurred: segmentation violation The following is the call stack containing the origin of the signal.

message occurs while processing the initial four records, but between that and the stack trace there are 800 lines of normal output, including "begin processing" messages for 8 events. So the stack trace may not be at all representative of what the other threads were doing at the time of the segfault. Odd that it took so long. Is there a way to have the signal handler suspend everything besides the stacktrace thread so that we get more informative stack traces?

Thread 4 (Thread 0x7fd0150bf700 (LWP 29530)):
#0  0x00007fd080b83036 in MagBSector::findVolume(Point3DBase<float, GlobalTag> const&, double) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libMagneticFieldLayers.so
#1  0x00007fd080b82bc3 in MagBLayer::findVolume(Point3DBase<float, GlobalTag> const&, double) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libMagneticFieldLayers.so
#2  0x00007fd080b8b936 in MagGeometry::findVolume(Point3DBase<float, GlobalTag> const&, double) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libMagneticFieldVolumeBasedEngine.so
#3  0x00007fd07f239ced in SteppingHelixPropagator::getNextState(SteppingHelixStateInfo const&, SteppingHelixStateInfo&, double, CLHEP::Hep3Vector const&, CLHEP::Hep3Vector const&, double, double, ROOT::Math::SMatrix<double, 5u, 5u, ROOT::Math::MatRepStd<double, 5u, 5u> > const&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libTrackPropagationSteppingHelixPropagator.so
#4  0x00007fd07f23a483 in SteppingHelixPropagator::makeAtomStep(SteppingHelixStateInfo&, SteppingHelixStateInfo&, double, PropagationDirection, SteppingHelixPropagator::Fancy) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libTrackPropagationSteppingHelixPropagator.so
#5  0x00007fd07f24023b in SteppingHelixPropagator::propagate(SteppingHelixStateInfo (&) [8], int&, SteppingHelixPropagator::DestType, double const_, double) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libTrackPropagationSteppingHelixPropagator.so
#6  0x00007fd07f24563c in SteppingHelixPropagator::propagate(SteppingHelixStateInfo const&, Plane const&, SteppingHelixStateInfo&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libTrackPropagationSteppingHelixPropagator.so
#7  0x00007fd07f280301 in CachedTrajectory::propagate(SteppingHelixStateInfo&, Plane const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libTrackingToolsTrackAssociator.so
#8  0x00007fd07f280c62 in CachedTrajectory::propagateForward(SteppingHelixStateInfo&, float) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libTrackingToolsTrackAssociator.so
#9  0x00007fd07f282fa3 in CachedTrajectory::propagateAll(SteppingHelixStateInfo const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libTrackingToolsTrackAssociator.so
#10 0x00007fd07f26e312 in TrackDetectorAssociator::associate(edm::Event const&, edm::EventSetup const&, TrackAssociatorParameters const&, FreeTrajectoryState const_, FreeTrajectoryState const_) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libTrackingToolsTrackAssociator.so
#11 0x00007fd07f26ea6a in TrackDetectorAssociator::associate(edm::Event const&, edm::EventSetup const&, reco::Track const&, TrackAssociatorParameters const&, TrackDetectorAssociator::Direction) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libTrackingToolsTrackAssociator.so
#12 0x00007fd05612c474 in MuonIdProducer::fillMuonId(edm::Event&, edm::EventSetup const&, reco::Muon&, TrackDetectorAssociator::Direction) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/pluginRecoMuonMuonIdentificationPlugins.so
#13 0x00007fd056131be4 in MuonIdProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/pluginRecoMuonMuonIdentificationPlugins.so
#14 0x00007fd098192822 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry_, edm::ModuleCallingContext const_) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#15 0x00007fd0980e510f in edm::WorkerTedm::stream::EDProducerAdaptorBase::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const_) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007fd09809c6dc in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_)::{lambda()#1}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007fd09809c8a2 in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007fd098098eaa in edm::UnscheduledProductResolver::resolveProduct_(edm::ProductResolverBase::ResolveStatus&, edm::Principal const&, bool, edm::SharedResourcesAcquirer_, edm::ModuleCallingContext const_) const::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007fd0980991c2 in edm::UnscheduledProductResolver::resolveProduct_(edm::ProductResolverBase::ResolveStatus&, edm::Principal const&, bool, edm::SharedResourcesAcquirer_, edm::ModuleCallingContext const_) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007fd09813f4a1 in edm::Principal::prefetch(unsigned int, bool, edm::ModuleCallingContext const_) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007fd09809c6b7 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_)::{lambda()#1}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007fd09809c8a2 in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007fd0980d85de in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#24 0x00007fd0980d8892 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#25 0x00007fd0980d8dcc in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#26 0x00007fd0980d8fab in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#27 0x00007fd0980d921f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#28 0x00007fd098136622 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#29 0x00007fd098136d57 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>_) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#30 0x00007fd09813d604 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#31 0x00007fd096e3d9b5 in tbb::internal::custom_schedulertbb::internal::IntelSchedulerTraits::local_wait_for_all (this=0x7fd0130b7e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#32 0x00007fd096e371a4 in tbb::internal::arena::process (this=this@entry=0x7fd09577b780, s=...) at ../../src/tbb/arena.cpp:156
#33 0x00007fd096e36264 in tbb::internal::market::process (this=0x7fd0957a3e00, j=...) at ../../src/tbb/market.cpp:502
#34 0x00007fd096e32736 in tbb::internal::rml::private_worker::run (this=0x7fd09579ba00) at ../../src/tbb/private_server.cpp:275
#35 0x00007fd096e32989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#36 0x00000036a8007aa1 in start_thread () from /lib64/libpthread.so.0
#37 0x00000036a7ce8aad in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fd0146be700 (LWP 29531)):
#0  0x00000036a9026470 in sincos () from /lib64/libm.so.6
#1  0x00007fd07f23bbde in SteppingHelixPropagator::makeAtomStep(SteppingHelixStateInfo&, SteppingHelixStateInfo&, double, PropagationDirection, SteppingHelixPropagator::Fancy) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libTrackPropagationSteppingHelixPropagator.so
#2  0x00007fd07f24023b in SteppingHelixPropagator::propagate(SteppingHelixStateInfo (&) [8], int&, SteppingHelixPropagator::DestType, double const_, double) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libTrackPropagationSteppingHelixPropagator.so
#3  0x00007fd07f244734 in SteppingHelixPropagator::propagate(SteppingHelixStateInfo const&, Cylinder const&, SteppingHelixStateInfo&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libTrackPropagationSteppingHelixPropagator.so
#4  0x00007fd07f244f97 in SteppingHelixPropagator::propagateWithPath(FreeTrajectoryState const&, Cylinder const&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libTrackPropagationSteppingHelixPropagator.so
#5  0x00007fd07f246ced in Propagator::propagateWithPath(TrajectoryStateOnSurface const&, Cylinder const&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libTrackPropagationSteppingHelixPropagator.so
#6  0x00007fd07fbf985f in BarrelDetLayer::compatible(TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libTrackingToolsDetLayers.so
#7  0x00007fd07f1ca2a3 in MuRodBarrelLayer::compatibleDets(TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libRecoMuonDetLayers.so
#8  0x00007fd05b90e989 in CosmicMuonTrajectoryBuilder::trajectories(TrajectorySeed const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/pluginRecoMuonCosmicMuonProducer.so
#9  0x00007fd07e6d3d4e in MuonTrackFinder::reconstruct(edm::Handleedm::View<TrajectorySeed > const&, edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libRecoMuonTrackingTools.so
#10 0x00007fd05b91e90e in CosmicMuonProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/pluginRecoMuonCosmicMuonProducer.so
#11 0x00007fd098192822 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry_, edm::ModuleCallingContext const_) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#12 0x00007fd0980e510f in edm::WorkerTedm::stream::EDProducerAdaptorBase::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const_) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#13 0x00007fd09809c6dc in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_)::{lambda()#1}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#14 0x00007fd09809c8a2 in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#15 0x00007fd098098eaa in edm::UnscheduledProductResolver::resolveProduct_(edm::ProductResolverBase::ResolveStatus&, edm::Principal const&, bool, edm::SharedResourcesAcquirer_, edm::ModuleCallingContext const_) const::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007fd0980991c2 in edm::UnscheduledProductResolver::resolveProduct_(edm::ProductResolverBase::ResolveStatus&, edm::Principal const&, bool, edm::SharedResourcesAcquirer_, edm::ModuleCallingContext const_) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007fd09813f4a1 in edm::Principal::prefetch(unsigned int, bool, edm::ModuleCallingContext const_) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007fd09809c6b7 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_)::{lambda()#1}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007fd09809c8a2 in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007fd0980d85de in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007fd0980d8892 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007fd0980d8dcc in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007fd0980d8fab in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#24 0x00007fd0980d921f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#25 0x00007fd098136622 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#26 0x00007fd098136d57 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>_) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#27 0x00007fd09813d604 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#28 0x00007fd096e3d9b5 in tbb::internal::custom_schedulertbb::internal::IntelSchedulerTraits::local_wait_for_all (this=0x7fd0131b7e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#29 0x00007fd096e371a4 in tbb::internal::arena::process (this=this@entry=0x7fd09577b780, s=...) at ../../src/tbb/arena.cpp:156
#30 0x00007fd096e36264 in tbb::internal::market::process (this=0x7fd0957a3e00, j=...) at ../../src/tbb/market.cpp:502
#31 0x00007fd096e32736 in tbb::internal::rml::private_worker::run (this=0x7fd09579ba80) at ../../src/tbb/private_server.cpp:275
#32 0x00007fd096e32989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#33 0x00000036a8007aa1 in start_thread () from /lib64/libpthread.so.0
#34 0x00000036a7ce8aad in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fd013c3f700 (LWP 29532)):
#0  0x00000036a7cdf283 in poll () from /lib64/libc.so.6
#1  0x00007fd092f53d94 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007fd092f53ffa in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007fd092f5417b in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007fd07ff0d08e in ?? ()
#6  0x00007fd013c394a8 in ?? ()
#7  0x00007fcfef2bf5d0 in ?? ()
#8  0x0000000000000000 in ?? ()
Thread 1 (Thread 0x7fd09730eba0 (LWP 29386)):
#0  0x00007fd0970762cf in arena_run_tree_remove (rbtree=rbtree@entry=0x7fd096701070, node=node@entry=0x7fd0128c10c8) at src/arena.c:76
#1  0x00007fd09707cca9 in arena_bin_runs_remove (run=0x7fd0128c10d8, bin=<optimized out>) at src/arena.c:2090
#2  arena_dissociate_bin_run (bin=<optimized out>, run=0x7fd0128c10d8, chunk=<optimized out>) at src/arena.c:2662
#3  arena_dalloc_bin_locked_impl (junked=true, bitselm=<optimized out>, ptr=<optimized out>, chunk=<optimized out>, arena=0x7fd096700600) at src/arena.c:2730
#4  je_arena_dalloc_bin_junked_locked (arena=0x7fd096700600, chunk=<optimized out>, ptr=<optimized out>, bitselm=<optimized out>) at src/arena.c:2746
#5  0x00007fd0970a2696 in je_tcache_bin_flush_small (tsd=0x7fd09730a7c8, tcache=tcache@entry=0x7fd0966c2000, tbin=tbin@entry=0x7fd0966c2088, binind=<optimized out>, rem=100) at src/tcache.c:132
#6  0x00007fd09706c7db in je_tcache_dalloc_small (slow_path=false, binind=<optimized out>, ptr=0x7fd00a669830, tcache=0x7fd0966c2000, tsd=<optimized out>) at include/jemalloc/internal/tcache.h:417
#7  je_arena_dalloc (slow_path=false, tcache=0x7fd0966c2000, ptr=0x7fd00a669830, tsd=<optimized out>) at include/jemalloc/internal/arena.h:1370
#8  je_idalloctm (slow_path=false, is_metadata=false, tcache=0x7fd0966c2000, ptr=0x7fd00a669830, tsd=<optimized out>) at include/jemalloc/internal/jemalloc_internal.h:1056
#9  je_iqalloc (slow_path=false, tcache=0x7fd0966c2000, ptr=0x7fd00a669830, tsd=<optimized out>) at include/jemalloc/internal/jemalloc_internal.h:1080
#10 ifree (tsd=<optimized out>, slow_path=false, tcache=0x7fd0966c2000, ptr=0x7fd00a669830) at src/jemalloc.c:1819
#11 free (ptr=0x7fd00a669830) at src/jemalloc.c:1922
#12 0x00007fd080923ed8 in std::_Rb_tree<DetId, DetId, std::_Identity<DetId>, std::less<DetId>, std::allocator<DetId> >::_M_erase(std::_Rb_tree_node<DetId>_) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libGeometryEcalAlgo.so
#13 0x00007fd080923ecc in std::_Rb_tree<DetId, DetId, std::_Identity<DetId>, std::less<DetId>, std::allocator<DetId> >::_M_erase(std::_Rb_tree_node<DetId>_) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libGeometryEcalAlgo.so
#14 0x00007fd080923ecc in std::_Rb_tree<DetId, DetId, std::_Identity<DetId>, std::less<DetId>, std::allocator<DetId> >::_M_erase(std::_Rb_tree_node<DetId>_) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libGeometryEcalAlgo.so
#15 0x00007fd080923ecc in std::_Rb_tree<DetId, DetId, std::_Identity<DetId>, std::less<DetId>, std::allocator<DetId> >::_M_erase(std::_Rb_tree_node<DetId>_) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libGeometryEcalAlgo.so
#16 0x00007fd07f26bc18 in TrackDetectorAssociator::fillCaloTowers(edm::Event const&, TrackDetMatchInfo&, TrackAssociatorParameters const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libTrackingToolsTrackAssociator.so
#17 0x00007fd07f26e813 in TrackDetectorAssociator::associate(edm::Event const&, edm::EventSetup const&, TrackAssociatorParameters const&, FreeTrajectoryState const_, FreeTrajectoryState const_) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libTrackingToolsTrackAssociator.so
#18 0x00007fd07f26f028 in TrackDetectorAssociator::associate(edm::Event const&, edm::EventSetup const&, FreeTrajectoryState const&, TrackAssociatorParameters const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libTrackingToolsTrackAssociator.so
#19 0x00007fd055f9dc5a in muonisolation::CaloExtractorByAssociator::deposits(edm::Event const&, edm::EventSetup const&, reco::Track const&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/pluginRecoMuonMuonIsolationPlugins.so
#20 0x00007fd05612bfbd in MuonIdProducer::fillMuonIsolation(edm::Event&, edm::EventSetup const&, reco::Muon&, reco::IsoDeposit&, reco::IsoDeposit&, reco::IsoDeposit&, reco::IsoDeposit&, reco::IsoDeposit&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/pluginRecoMuonMuonIdentificationPlugins.so
#21 0x00007fd056131c92 in MuonIdProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/pluginRecoMuonMuonIdentificationPlugins.so
#22 0x00007fd098192822 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry_, edm::ModuleCallingContext const_) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007fd0980e510f in edm::WorkerTedm::stream::EDProducerAdaptorBase::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const_) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#24 0x00007fd09809c6dc in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_)::{lambda()#1}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#25 0x00007fd09809c8a2 in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#26 0x00007fd098098eaa in edm::UnscheduledProductResolver::resolveProduct_(edm::ProductResolverBase::ResolveStatus&, edm::Principal const&, bool, edm::SharedResourcesAcquirer_, edm::ModuleCallingContext const_) const::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#27 0x00007fd0980991c2 in edm::UnscheduledProductResolver::resolveProduct_(edm::ProductResolverBase::ResolveStatus&, edm::Principal const&, bool, edm::SharedResourcesAcquirer_, edm::ModuleCallingContext const_) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#28 0x00007fd09813f4a1 in edm::Principal::prefetch(unsigned int, bool, edm::ModuleCallingContext const_) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#29 0x00007fd09809c6b7 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_)::{lambda()#1}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#30 0x00007fd09809c8a2 in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#31 0x00007fd0980d85de in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#32 0x00007fd0980d8892 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#33 0x00007fd0980d8dcc in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#34 0x00007fd0980d8fab in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#35 0x00007fd0980d921f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipBegin processing the 13th record. Run 160960, Event 10002522, LumiSection 277 at 23-Jul-2016 15:23:13.987 CEST
al, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#36 0x00007fd098136622 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#37 0x00007fd098136d57 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>_) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#38 0x00007fd09813d604 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#39 0x00007fd096e3d9b5 in tbb::internal::custom_schedulertbb::internal::IntelSchedulerTraits::local_wait_for_all (this=0x7fd0957a2400, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#40 0x00007fd098136bae in edm::EventProcessor::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#41 0x00007fd0980a8eda in statemachine::HandleEvent::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#42 0x00007fd0980aa468 in statemachine::HandleEvent::HandleEvent(boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::my_context) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#43 0x00007fd0980b21ee in boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::shallow_construct(boost::intrusive_ptrstatemachine::HandleLumis const&, boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#44 0x00007fd0980b23ba in boost::statechart::simple_state<statemachine::FirstLumi, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#45 0x00007fd09813d0b8 in boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#46 0x00007fd0981302dd in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-22-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#47 0x000000000040c814 in main::{lambda()#1}::operator()() const ()
#48 0x000000000040ac94 in main ()

[dan131riley]

Same thing, 6 events processed after segfault:

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc530/CMSSW_8_1_THREADED_X_2016-07-21-1100/pyRelValMatrixLogs/run/1307.0_CosmicsSPLoose_UP15+CosmicsSPLoose_UP15INPUT+DIGICOS_UP15+RECOCOS_UP15+ALCACOS_UP15+HARVESTCOS_UP15/step3_CosmicsSPLoose_UP15+CosmicsSPLoose_UP15INPUT+DIGICOS_UP15+RECOCOS_UP15+ALCACOS_UP15+HARVESTCOS_UP15.log

most threads are idle because, while the segfault happened in the first 4 events, the job had started on all 10 events requested and some threads had run out of work.

Thread 5 (Thread 0x7fc9eea3f700 (LWP 9285)):
#0  0x0000003bec80f37d in waitpid () from /lib64/libpthread.so.0
#1  0x00007fca007136b7 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-21-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007fca00713f62 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-21-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007fca05047a50 in std::(anonymous namespace)::execute_native_thread_routine (__p=<optimized out>) at ../../../../../libstdc++-v3/src/c++11/thread.cc:84
#4  0x0000003bec807aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003bec4e8aad in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fc9794ff700 (LWP 9506)):
#0  0x00007fc9ec315691 in SteppingHelixStateInfo::getStateOnSurface(Surface const&, bool) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-21-1100/lib/slc6_amd64_gcc530/libTrackPropagationSteppingHelixPropagator.so
#1  0x00007fc9ec3264b6 in SteppingHelixPropagator::propagateWithPath(FreeTrajectoryState const&, Plane const&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-21-1100/lib/slc6_amd64_gcc530/libTrackPropagationSteppingHelixPropagator.so
#2  0x00007fc9ec326d6d in Propagator::propagateWithPath(TrajectoryStateOnSurface const&, Plane const&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-21-1100/lib/slc6_amd64_gcc530/libTrackPropagationSteppingHelixPropagator.so
#3  0x00007fc9d3357426 in CosmicMuonSmoother::fit(TrajectorySeed const&, std::vector<std::shared_ptr<TrackingRecHit const>, std::allocator<std::shared_ptr<TrackingRecHit const> > > const&, TrajectoryStateOnSurface const&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-21-1100/lib/slc6_amd64_gcc530/pluginRecoMuonCosmicMuonProducer.so
#4  0x00007fc9d335b45b in CosmicMuonSmoother::fit(Trajectory const&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-21-1100/lib/slc6_amd64_gcc530/pluginRecoMuonCosmicMuonProducer.so
#5  0x00007fc9d335c391 in CosmicMuonSmoother::trajectory(Trajectory const&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-21-1100/lib/slc6_amd64_gcc530/pluginRecoMuonCosmicMuonProducer.so
#6  0x00007fc9d33693d4 in CosmicMuonTrajectoryBuilder::trajectories(TrajectorySeed const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-21-1100/lib/slc6_amd64_gcc530/pluginRecoMuonCosmicMuonProducer.so
#7  0x00007fc9eb7c684e in MuonTrackFinder::reconstruct(edm::Handleedm::View<TrajectorySeed > const&, edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-21-1100/lib/slc6_amd64_gcc530/libRecoMuonTrackingTools.so
#8  0x00007fc9d334b63e in CosmicMuonProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-21-1100/lib/slc6_amd64_gcc530/pluginRecoMuonCosmicMuonProducer.so
#9  0x00007fca066f28c2 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry_, edm::ModuleCallingContext const_) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-21-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#10 0x00007fca066de1ef in edm::WorkerTedm::stream::EDProducerAdaptorBase::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const_) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-21-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#11 0x00007fca065f751c in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_)::{lambda()#1}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-21-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#12 0x00007fca065f76e2 in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-21-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#13 0x00007fca066303ae in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-21-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#14 0x00007fca06630662 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-21-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#15 0x00007fca06630a85 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-21-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007fca06630d7b in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-21-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007fca06630fef in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-21-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007fca06683d72 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-21-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007fca066844a7 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>_) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-21-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007fca0668af04 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-21-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007fca0539d9b5 in tbb::internal::custom_schedulertbb::internal::IntelSchedulerTraits::local_wait_for_all (this=0x7fc9dd8bfe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#22 0x00007fca053971a4 in tbb::internal::arena::process (this=this@entry=0x7fca03cbb780, s=...) at ../../src/tbb/arena.cpp:156
#23 0x00007fca05396264 in tbb::internal::market::process (this=0x7fca03ce3e00, j=...) at ../../src/tbb/market.cpp:502
#24 0x00007fca05392736 in tbb::internal::rml::private_worker::run (this=0x7fca03cdba00) at ../../src/tbb/private_server.cpp:275
#25 0x00007fca05392989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#26 0x0000003bec807aa1 in start_thread () from /lib64/libpthread.so.0
#27 0x0000003bec4e8aad in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fc978afe700 (LWP 9507)):
#0  0x0000003bec4df283 in poll () from /lib64/libc.so.6
#1  0x00007fca00713dc4 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-21-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007fca0071402a in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-21-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007fca007141ab in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-21-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007fc9ecceb08e in ?? ()
#6  0x00007fc978af84a8 in ?? ()
#7  0x00007fc96f118760 in ?? ()
#8  0x0000000000000000 in ?? ()
Thread 2 (Thread 0x7fc97803f700 (LWP 9508)):
#0  0x0000003bec4e5309 in syscall () from /lib64/libc.so.6
#1  0x00007fca05392962 in tbb::internal::futex_wait (comparand=2, futex=0x7fca03cdb9ac) at ../../include/tbb/machine/linux_common.h:60
#2  tbb::internal::binary_semaphore::P (this=0x7fca03cdb9ac) at ../../src/tbb/semaphore.h:206
#3  rml::internal::thread_monitor::commit_wait (c=<synthetic pointer>, this=0x7fca03cdb9a0) at ../../src/rml/include/../server/thread_monitor.h:259
#4  tbb::internal::rml::private_worker::run (this=0x7fca03cdb980) at ../../src/tbb/private_server.cpp:282
#5  0x00007fca05392989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#6  0x0000003bec807aa1 in start_thread () from /lib64/libpthread.so.0
#7  0x0000003bec4e8aad in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fca060b6ba0 (LWP 9229)):
#0  0x0000003bec4cf437 in sched_yield () from /lib64/libc.so.6
#1  0x00007fca0539bc97 in tbb::internal::custom_schedulertbb::internal::IntelSchedulerTraits::receive_or_steal_task (this=0x7fca03ce2400, completion_ref_count=@0x7fca03cd7228: 3) at ../../src/tbb/custom_scheduler.h:273
#2  0x00007fca0539d80d in tbb::internal::custom_schedulertbb::internal::IntelSchedulerTraits::local_wait_for_all (this=0x7fca03ce2400, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:581
#3  0x00007fca066842fe in edm::EventProcessor::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-21-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#4  0x00007fca065ff19a in statemachine::HandleEvent::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-21-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#5  0x00007fca06600728 in statemachine::HandleEvent::HandleEvent(boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::my_context) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-21-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#6  0x00007fca066084ae in boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::shallow_construct(boost::intrusive_ptrstatemachine::HandleLumis const&, boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-21-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#7  0x00007fca0660867a in boost::statechart::simple_state<statemachine::FirstLumi, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-21-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#8  0x00007fca0668a9b8 in boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-21-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#9  0x00007fca0667da2d in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_THREADED_X_2016-07-21-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#10 0x000000000040c814 in main::{lambda()#1}::operator()() const ()
#11 0x000000000040ac94 in main ()

[dan131riley]

Yet another workflow

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc530/CMSSW_8_1_THREADED_X_2016-07-26-2300/pyRelValMatrixLogs/run/1325.0_TTbar_13+TTbar_13INPUT+DIGIUP15+RECOUP15+HARVESTUP15+ALCATTUP15/step5_TTbar_13+TTbar_13INPUT+DIGIUP15+RECOUP15+HARVESTUP15+ALCATTUP15.log

Thread 5 (Thread 0x7f43d40bf700 (LWP 3803)):
#0  0x0000003fcce0f37d in waitpid () from /lib64/libpthread.so.0
#1  0x00007f43e08596b7 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007f43e0859f62 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007f43e3cdca50 in std::(anonymous namespace)::execute_native_thread_routine (__p=<optimized out>) at ../../../../../libstdc++-v3/src/c++11/thread.cc:84
#4  0x0000003fcce07aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003fccae8aad in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f43a7dff700 (LWP 3973)):
#0  0x0000003fcce0e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x0000003fcce095f3 in _L_lock_892 () from /lib64/libpthread.so.0
#2  0x0000003fcce094d7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007f43e50f9448 in edm::eventsetup::DataProxy::get(edm::eventsetup::EventSetupRecord const&, edm::eventsetup::DataKey const&, bool) const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#4  0x00007f43e508b60b in edm::eventsetup::EventSetupRecord::getFromProxy(edm::eventsetup::DataKey const&, edm::eventsetup::ComponentDescription const_&, bool) const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#5  0x00007f43b46e6ffd in TkAlCaRecoMonitor::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/pluginDQMOfflineAlignment.so
#6  0x00007f43e511904a in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry_, edm::ModuleCallingContext const_) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#7  0x00007f43e50599df in edm::WorkerTedm::stream::EDAnalyzerAdaptorBase::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const_) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#8  0x00007f43e501981c in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_)::{lambda()#1}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#9  0x00007f43e50199e2 in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#10 0x00007f43e504b6ee in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#11 0x00007f43e504b9a2 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#12 0x00007f43e504bdc5 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#13 0x00007f43e504c0bb in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#14 0x00007f43e504c32f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#15 0x00007f43e50b39a2 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007f43e50b40d7 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>_) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007f43e50ba7e4 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007f43e3e119b5 in tbb::internal::custom_schedulertbb::internal::IntelSchedulerTraits::local_wait_for_all (this=0x7f43c0927e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#19 0x00007f43e3e0b1a4 in tbb::internal::arena::process (this=this@entry=0x7f43e293f780, s=...) at ../../src/tbb/arena.cpp:156
#20 0x00007f43e3e0a264 in tbb::internal::market::process (this=0x7f43e2967c00, j=...) at ../../src/tbb/market.cpp:502
#21 0x00007f43e3e06736 in tbb::internal::rml::private_worker::run (this=0x7f43e2786100) at ../../src/tbb/private_server.cpp:275
#22 0x00007f43e3e06989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#23 0x0000003fcce07aa1 in start_thread () from /lib64/libpthread.so.0
#24 0x0000003fccae8aad in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f43a73fe700 (LWP 3974)):
#0  0x00007f43df6861fc in xercesc_3_1::IGXMLScanner::scanAttValue(xercesc_3_1::XMLAttDef const_, unsigned short const_, xercesc_3_1::XMLBuffer&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/external/slc6_amd64_gcc530/lib/libxerces-c-3.1.so
#1  0x00007f43df678adf in xercesc_3_1::IGXMLScanner::scanStartTag(bool&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/external/slc6_amd64_gcc530/lib/libxerces-c-3.1.so
#2  0x00007f43df676325 in xercesc_3_1::IGXMLScanner::scanContent() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/external/slc6_amd64_gcc530/lib/libxerces-c-3.1.so
#3  0x00007f43df6748f1 in xercesc_3_1::IGXMLScanner::scanDocument(xercesc_3_1::InputSource const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/external/slc6_amd64_gcc530/lib/libxerces-c-3.1.so
#4  0x00007f43df6e026d in xercesc_3_1::SAX2XMLReaderImpl::parse(xercesc_3_1::InputSource const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/external/slc6_amd64_gcc530/lib/libxerces-c-3.1.so
#5  0x00007f43cd31a825 in DDLParser::parse(std::vector<unsigned char, std::allocator<unsigned char> > const&, unsigned int) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libDetectorDescriptionParser.so
#6  0x00007f43cd3ecad4 in magneticfield::VolumeBasedMagneticFieldESProducerFromDB::produce(IdealMagneticFieldRecord const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/pluginMagneticFieldGeomBuilderPlugins.so
#7  0x00007f43cd3ef652 in edm::eventsetup::CallbackProxy<edm::eventsetup::Callback<magneticfield::VolumeBasedMagneticFieldESProducerFromDB, std::unique_ptr<MagneticField, std::default_delete<MagneticField> >, IdealMagneticFieldRecord, edm::eventsetup::CallbackSimpleDecorator<IdealMagneticFieldRecord> >, IdealMagneticFieldRecord, std::unique_ptr<MagneticField, std::default_delete<MagneticField> > >::getImpl(edm::eventsetup::EventSetupRecord const&, edm::eventsetup::DataKey const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/pluginMagneticFieldGeomBuilderPlugins.so
#8  0x00007f43e50f948f in edm::eventsetup::DataProxy::get(edm::eventsetup::EventSetupRecord const&, edm::eventsetup::DataKey const&, bool) const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#9  0x00007f43e508b60b in edm::eventsetup::EventSetupRecord::getFromProxy(edm::eventsetup::DataKey const&, edm::eventsetup::ComponentDescription const_&, bool) const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#10 0x00007f43b46e6ffd in TkAlCaRecoMonitor::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/pluginDQMOfflineAlignment.so
#11 0x00007f43e511904a in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry_, edm::ModuleCallingContext const_) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#12 0x00007f43e50599df in edm::WorkerTedm::stream::EDAnalyzerAdaptorBase::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const_) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#13 0x00007f43e501981c in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_)::{lambda()#1}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#14 0x00007f43e50199e2 in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#15 0x00007f43e504b6ee in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007f43e504b9a2 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007f43e504bdc5 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007f43e504c0bb in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007f43e504c32f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007f43e50b39a2 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007f43e50b421a in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>_) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007f43e50ba7e4 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007f43e3e119b5 in tbb::internal::custom_schedulertbb::internal::IntelSchedulerTraits::local_wait_for_all (this=0x7f43c092fe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#24 0x00007f43e3e0b1a4 in tbb::internal::arena::process (this=this@entry=0x7f43e293f780, s=...) at ../../src/tbb/arena.cpp:156
#25 0x00007f43e3e0a264 in tbb::internal::market::process (this=0x7f43e2967c00, j=...) at ../../src/tbb/market.cpp:502
#26 0x00007f43e3e06736 in tbb::internal::rml::private_worker::run (this=0x7f43e2786180) at ../../src/tbb/private_server.cpp:275
#27 0x00007f43e3e06989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#28 0x0000003fcce07aa1 in start_thread () from /lib64/libpthread.so.0
#29 0x0000003fccae8aad in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f43a68bf700 (LWP 3975)):
#0  0x0000003fccadf283 in poll () from /lib64/libc.so.6
#1  0x00007f43e0859dc4 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007f43e085a02a in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007f43e085a1ab in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007f43a5e9f08e in ?? ()
#6  0x00007f43a68b9488 in ?? ()
#7  0x00007f43a59bd150 in ?? ()
#8  0x0000000000000000 in ?? ()
Thread 1 (Thread 0x7f43e3975bc0 (LWP 3731)):
#0  0x0000003fcce0e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x0000003fcce095f3 in _L_lock_892 () from /lib64/libpthread.so.0
#2  0x0000003fcce094d7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007f43e50f9448 in edm::eventsetup::DataProxy::get(edm::eventsetup::EventSetupRecord const&, edm::eventsetup::DataKey const&, bool) const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#4  0x00007f43e508b60b in edm::eventsetup::EventSetupRecord::getFromProxy(edm::eventsetup::DataKey const&, edm::eventsetup::ComponentDescription const_&, bool) const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#5  0x00007f43b46e6ffd in TkAlCaRecoMonitor::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/pluginDQMOfflineAlignment.so
#6  0x00007f43e511904a in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry_, edm::ModuleCallingContext const_) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#7  0x00007f43e50599df in edm::WorkerTedm::stream::EDAnalyzerAdaptorBase::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const_) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#8  0x00007f43e501981c in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_)::{lambda()#1}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#9  0x00007f43e50199e2 in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#10 0x00007f43e504b6ee in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#11 0x00007f43e504b9a2 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#12 0x00007f43e504bdc5 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#13 0x00007f43e504c0bb in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#14 0x00007f43e504c32f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#15 0x00007f43e50b39a2 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007f43e50b40d7 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007f43e50ba7e4 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007f43e3e119b5 in tbb::internal::custom_schedulertbb::internal::IntelSchedulerTraits::local_wait_for_all (this=0x7f43e2956600, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#19 0x00007f43e50b3f2e in edm::EventProcessor::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007f43e50216ca in statemachine::HandleEvent::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007f43e5022c58 in statemachine::HandleEvent::HandleEvent(boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::my_context) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007f43e502a9de in boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::shallow_construct(boost::intrusive_ptrstatemachine::HandleLumis const&, boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007f43e502abaa in boost::statechart::simple_state<statemachine::FirstLumi, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#24 0x00007f43e50ba298 in boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#25 0x00007f43e50ad65d in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-07-26-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#26 0x000000000040c814 in main::{lambda()#1}::operator()() const ()
#27 0x000000000040ac94 in main ()

[dan131riley]

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc530/CMSSW_8_1_X_2016-08-15-1100/pyRelValMatrixLogs/run/4.22_RunCosmics2011A+RunCosmics2011A+RECOCOSD+ALCACOSD+SKIMCOSD+HARVESTDC/step2_RunCosmics2011A+RunCosmics2011A+RECOCOSD+ALCACOSD+SKIMCOSD+HARVESTDC.log

First stack trace with the pause feature inspired by this issue!

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thread 10 (Thread 0x7f49b64ff700 (LWP 19972)):
#0  0x00000032b6c0f37d in waitpid () from /lib64/libpthread.so.0
#1  0x00007f49c7753d47 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-15-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007f49c77546d2 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-15-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007f49cb1aaa50 in std::(anonymous namespace)::execute_native_thread_routine (__p=<optimized out>) at ../../../../../libstdc++-v3/src/c++11/thread.cc:84
#4  0x00000032b6c07aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000032b68e8aad in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f49aff3c700 (LWP 20023)):
#0  0x00000032b68e90a3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f49b00093ba in XrdSys::IOEvents::PollE::Begin (this=0x7f49b2cd3b00, syncsem=<optimized out>, retcode=<optimized out>, eTxt=<optimized out>) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/./XrdSys/XrdSysIOEventsPollE.icc:213
#2  0x00007f49b0005a8d in XrdSys::IOEvents::BootStrap::Start (parg=0x7ffe0f149cf0) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdSys/XrdSysIOEvents.cc:131
#3  0x00007f49b0004647 in XrdSysThread_Xeq (myargs=0x7f49b0719940) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdSys/XrdSysPthread.cc:86
#4  0x00000032b6c07aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000032b68e8aad in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f49af53b700 (LWP 20024)):
#0  0x00000032b6c0f00d in nanosleep () from /lib64/libpthread.so.0
#1  0x00007f49b0004deb in XrdSysTimer::Wait (mills=<optimized out>) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdSys/XrdSysTimer.cc:239
#2  0x00007f49b010fcaa in XrdCl::TaskManager::RunTasks (this=0x7f49b17c41c0) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdCl/XrdClTaskManager.cc:244
#3  0x00007f49b010fe29 in RunRunnerThread (arg=<optimized out>) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdCl/XrdClTaskManager.cc:37
#4  0x00000032b6c07aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000032b68e8aad in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f49aeb3a700 (LWP 20025)):
#0  0x00000032b68e5309 in syscall () from /lib64/libc.so.6
#1  0x00007f49b016b4e1 in XrdSys::LinuxSemaphore::Wait (this=0x7f49b4c7d388) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/./XrdSys/XrdSysLinuxSemaphore.hh:161
#2  XrdCl::SyncQueue<XrdCl::JobManager::JobHelper>::Get (this=0x7f49b2cd3a38) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/./XrdCl/XrdClSyncQueue.hh:67
#3  XrdCl::JobManager::RunJobs (this=0x7f49b2cd3a20) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdCl/XrdClJobManager.cc:146
#4  0x00007f49b016b679 in RunRunnerThread (arg=<optimized out>) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdCl/XrdClJobManager.cc:33
#5  0x00000032b6c07aa1 in start_thread () from /lib64/libpthread.so.0
#6  0x00000032b68e8aad in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f49ae139700 (LWP 20026)):
#0  0x00000032b68e5309 in syscall () from /lib64/libc.so.6
#1  0x00007f49b016b4e1 in XrdSys::LinuxSemaphore::Wait (this=0x7f49b4c7d388) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/./XrdSys/XrdSysLinuxSemaphore.hh:161
#2  XrdCl::SyncQueue<XrdCl::JobManager::JobHelper>::Get (this=0x7f49b2cd3a38) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/./XrdCl/XrdClSyncQueue.hh:67
#3  XrdCl::JobManager::RunJobs (this=0x7f49b2cd3a20) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdCl/XrdClJobManager.cc:146
#4  0x00007f49b016b679 in RunRunnerThread (arg=<optimized out>) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdCl/XrdClJobManager.cc:33
#5  0x00000032b6c07aa1 in start_thread () from /lib64/libpthread.so.0
#6  0x00000032b68e8aad in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f49ad738700 (LWP 20027)):
#0  0x00000032b68e5309 in syscall () from /lib64/libc.so.6
#1  0x00007f49b016b4e1 in XrdSys::LinuxSemaphore::Wait (this=0x7f49b4c7d388) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/./XrdSys/XrdSysLinuxSemaphore.hh:161
#2  XrdCl::SyncQueue<XrdCl::JobManager::JobHelper>::Get (this=0x7f49b2cd3a38) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/./XrdCl/XrdClSyncQueue.hh:67
#3  XrdCl::JobManager::RunJobs (this=0x7f49b2cd3a20) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdCl/XrdClJobManager.cc:146
#4  0x00007f49b016b679 in RunRunnerThread (arg=<optimized out>) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdCl/XrdClJobManager.cc:33
#5  0x00000032b6c07aa1 in start_thread () from /lib64/libpthread.so.0
#6  0x00000032b68e8aad in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f4987b3f700 (LWP 20213)):
#0  0x00000032b68acbdd in nanosleep () from /lib64/libc.so.6
#1  0x00000032b68aca50 in sleep () from /lib64/libc.so.6
#2  0x00007f49c7753b7e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-15-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00000032b6c0e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#5  0x00000032b6c095d8 in _L_lock_854 () from /lib64/libpthread.so.0
#6  0x00000032b6c094a7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#7  0x00007f49ccc9a378 in edm::one::OutputModuleBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#8  0x00007f49ccc7c2f2 in edm::WorkerT<edm::one::OutputModuleBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#9  0x00007f49ccbb3be4 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#10 0x00007f49ccbb3e78 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#11 0x00007f49ccbb41ed in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#12 0x00007f49ccbd29ce in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#13 0x00007f49ccbd2c82 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#14 0x00007f49ccbd31bc in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#15 0x00007f49ccbd339b in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007f49ccbd360f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007f49ccc55f92 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007f49ccc5680a in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007f49ccc5cf24 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007f49cb9789b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f49c9c4fe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#21 0x00007f49cb9721a4 in tbb::internal::arena::process (this=this@entry=0x7f49c9dff780, s=...) at ../../src/tbb/arena.cpp:156
#22 0x00007f49cb971264 in tbb::internal::market::process (this=0x7f49c9e27c00, j=...) at ../../src/tbb/market.cpp:502
#23 0x00007f49cb96d736 in tbb::internal::rml::private_worker::run (this=0x7f49c9c46100) at ../../src/tbb/private_server.cpp:275
#24 0x00007f49cb96d989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#25 0x00000032b6c07aa1 in start_thread () from /lib64/libpthread.so.0
#26 0x00000032b68e8aad in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f498713e700 (LWP 20214)):
#0  0x00000032b68acbdd in nanosleep () from /lib64/libc.so.6
#1  0x00000032b68aca50 in sleep () from /lib64/libc.so.6
#2  0x00007f49c7753b7e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-15-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00000032b6c0e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#5  0x00000032b6c095f3 in _L_lock_892 () from /lib64/libpthread.so.0
#6  0x00000032b6c094d7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#7  0x00007f49cc0b1064 in TListOfFunctionsIter::Next() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-15-1100/external/slc6_amd64_gcc530/lib/libCore.so
#8  0x00007f49cc8d3d59 in edm::IterWithDictBase::advance() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-15-1100/lib/slc6_amd64_gcc530/libFWCoreUtilities.so
#9  0x00007f49c3c54329 in reco::findMethod(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > >&, char const*, int&) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#10 0x00007f49c3c541e4 in reco::findMethod(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > >&, char const*, int&) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#11 0x00007f49c3c19270 in reco::parser::MethodSetter::push(std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, char const*, bool) const () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#12 0x00007f49c3c6dc40 in reco::parser::SingleInvoker::SingleInvoker(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#13 0x00007f49c3c6ed86 in reco::parser::LazyInvoker::invoker(edm::TypeWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#14 0x00007f49c3c711cb in reco::parser::LazyInvoker::invokeLast(edm::ObjectWithDict const&, std::vector<edm::ObjectWithDict, std::allocator<edm::ObjectWithDict> >&) const () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#15 0x00007f49c3c693ff in reco::parser::ExpressionLazyVar::value(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#16 0x00007f49c3c42416 in reco::parser::BinarySelector::operator()(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#17 0x00007f49a4a9dc89 in TrackingMonitor::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#18 0x00007f49ccca02fa in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007f49ccc7baa2 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007f49ccbb3be4 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007f49ccbb3e78 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007f49ccbb41ed in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007f49ccbd29ce in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#24 0x00007f49ccbd2c82 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#25 0x00007f49ccbd30a5 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#26 0x00007f49ccbd339b in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#27 0x00007f49ccbd360f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#28 0x00007f49ccc55f92 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#29 0x00007f49ccc566c7 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#30 0x00007f49ccc5cf24 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#31 0x00007f49cb9789b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f49c9c57e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#32 0x00007f49cb9721a4 in tbb::internal::arena::process (this=this@entry=0x7f49c9dff780, s=...) at ../../src/tbb/arena.cpp:156
#33 0x00007f49cb971264 in tbb::internal::market::process (this=0x7f49c9e27c00, j=...) at ../../src/tbb/market.cpp:502
#34 0x00007f49cb96d736 in tbb::internal::rml::private_worker::run (this=0x7f49c9c46180) at ../../src/tbb/private_server.cpp:275
#35 0x00007f49cb96d989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#36 0x00000032b6c07aa1 in start_thread () from /lib64/libpthread.so.0
#37 0x00000032b68e8aad in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f49866bf700 (LWP 20215)):
#0  0x00000032b68df283 in poll () from /lib64/libc.so.6
#1  0x00007f49c7754534 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-15-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007f49c775479a in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-15-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007f49c775508e in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-15-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007f498672a08e in ?? ()
#6  0x00007f49866b93b8 in ?? ()
#7  0x00007f49258e1800 in ?? ()
#8  0x0000000000000000 in ?? ()
Thread 1 (Thread 0x7f49cae43bc0 (LWP 19786)):
#0  0x00000032b68acbdd in nanosleep () from /lib64/libc.so.6
#1  0x00000032b68aca50 in sleep () from /lib64/libc.so.6
#2  0x00007f49c7753b7e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-15-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f49cbb9c813 in je_tcache_dalloc_small (slow_path=false, binind=<optimized out>, ptr=0x7f4940672000, tcache=<optimized out>, tsd=<optimized out>) at include/jemalloc/internal/tcache.h:415
#5  je_arena_dalloc (slow_path=false, tcache=<optimized out>, ptr=0x7f4940672000, tsdn=<optimized out>) at include/jemalloc/internal/arena.h:1428
#6  je_idalloctm (slow_path=false, is_metadata=false, tcache=<optimized out>, ptr=0x7f4940672000, tsdn=<optimized out>) at include/jemalloc/internal/jemalloc_internal.h:1070
#7  je_iqalloc (slow_path=false, tcache=<optimized out>, ptr=0x7f4940672000, tsd=<optimized out>) at include/jemalloc/internal/jemalloc_internal.h:1087
#8  ifree (tsd=<optimized out>, slow_path=false, tcache=<optimized out>, ptr=0x7f4940672000) at src/jemalloc.c:1811
#9  free (ptr=0x7f4940672000) at src/jemalloc.c:1931
#10 0x00007f49cc428486 in TStreamerInfo::GetDataMemberOffset(TDataMember*, TMemberStreamer*&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-15-1100/external/slc6_amd64_gcc530/lib/libRIO.so
#11 0x00007f49cc43d4e5 in TStreamerInfo::Build() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-15-1100/external/slc6_amd64_gcc530/lib/libRIO.so
#12 0x00007f49cc2c474d in TBufferFile::WriteClassBuffer(TClass const*, void*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-15-1100/external/slc6_amd64_gcc530/lib/libRIO.so
#13 0x00007f49cc2f73d1 in TGenCollectionStreamer::WriteMap(int, TBuffer&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-15-1100/external/slc6_amd64_gcc530/lib/libRIO.so
#14 0x00007f49cc2f9a16 in TGenCollectionStreamer::Streamer(TBuffer&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-15-1100/external/slc6_amd64_gcc530/lib/libRIO.so
#15 0x00007f49cc2cc33f in TCollectionStreamer::Streamer(TBuffer&, void*, int, TClass*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-15-1100/external/slc6_amd64_gcc530/lib/libRIO.so
#16 0x00007f49cc2c4075 in TBufferFile::WriteFastArray(void*, TClass const*, int, TMemberStreamer*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-15-1100/external/slc6_amd64_gcc530/lib/libRIO.so
#17 0x00007f49cc2e554b in int TStreamerInfo::WriteBufferAux<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-15-1100/external/slc6_amd64_gcc530/lib/libRIO.so
#18 0x00007f49cc38aabd in TStreamerInfoActions::GenericWriteAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-15-1100/external/slc6_amd64_gcc530/lib/libRIO.so
#19 0x00007f49cc2c2d15 in TBufferFile::ApplySequence(TStreamerInfoActions::TActionSequence const&, void*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-15-1100/external/slc6_amd64_gcc530/lib/libRIO.so
#20 0x00007f49cc2c453e in TBufferFile::WriteClassBuffer(TClass const*, void*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-15-1100/external/slc6_amd64_gcc530/lib/libRIO.so
#21 0x00007f49cc2c4075 in TBufferFile::WriteFastArray(void*, TClass const*, int, TMemberStreamer*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-15-1100/external/slc6_amd64_gcc530/lib/libRIO.so
#22 0x00007f49cc2e524c in int TStreamerInfo::WriteBufferAux<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-15-1100/external/slc6_amd64_gcc530/lib/libRIO.so
#23 0x00007f49cc38aabd in TStreamerInfoActions::GenericWriteAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-15-1100/external/slc6_amd64_gcc530/lib/libRIO.so
#24 0x00007f49cc2c2d15 in TBufferFile::ApplySequence(TStreamerInfoActions::TActionSequence const&, void*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-15-1100/external/slc6_amd64_gcc530/lib/libRIO.so
#25 0x00007f49cc7d26ca in TBranch::Fill() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-15-1100/external/slc6_amd64_gcc530/lib/libTree.so
#26 0x00007f49cc80a0d0 in TBranchElement::Fill() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-15-1100/external/slc6_amd64_gcc530/lib/libTree.so
#27 0x00007f49cc809e58 in TBranchElement::Fill() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-15-1100/external/slc6_amd64_gcc530/lib/libTree.so
#28 0x00007f49cc7bd031 in TTree::Fill() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-15-1100/external/slc6_amd64_gcc530/lib/libTree.so
#29 0x00007f4992639a0b in edm::RootOutputFile::fillBranches(edm::BranchType const&, edm::OccurrenceForOutput const&, std::vector<edm::StoredProductProvenance, std::allocator<edm::StoredProductProvenance> >*, edm::ProductProvenanceRetriever const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libIOPoolOutput.so
#30 0x00007f499263b4fb in edm::RootOutputFile::writeOne(edm::EventForOutput const&) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libIOPoolOutput.so
#31 0x00007f499264a26b in edm::PoolOutputModule::write(edm::EventForOutput const&) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libIOPoolOutput.so
#32 0x00007f49ccc9a40e in edm::one::OutputModuleBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#33 0x00007f49ccc7c2f2 in edm::WorkerT<edm::one::OutputModuleBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#34 0x00007f49ccbb3be4 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#35 0x00007f49ccbb3e78 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#36 0x00007f49ccbb41ed in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#37 0x00007f49ccbd29ce in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#38 0x00007f49ccbd2c82 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#39 0x00007f49ccbd31bc in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#40 0x00007f49ccbd339b in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#41 0x00007f49ccbd360f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#42 0x00007f49ccc55f92 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#43 0x00007f49ccc566c7 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#44 0x00007f49ccc5cf24 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#45 0x00007f49cb9789b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f49c9e16600, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#46 0x00007f49ccc5651e in edm::EventProcessor::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#47 0x00007f49ccbb6f0a in statemachine::HandleEvent::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#48 0x00007f49ccbb8498 in statemachine::HandleEvent::HandleEvent(boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::my_context) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#49 0x00007f49ccbc01de in boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::shallow_construct(boost::intrusive_ptr<statemachine::HandleLumis> const&, boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>&) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#50 0x00007f49ccbc03aa in boost::statechart::simple_state<statemachine::FirstLumi, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#51 0x00007f49ccc5c9d8 in boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#52 0x00007f49ccc4fafd in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#53 0x000000000040c844 in main::{lambda()#1}::operator()() const ()
#54 0x000000000040acc4 in main ()

Current Modules:

Module: TrackingMonitor (crashed)
Module: PoolOutputModule
Module: PoolOutputModule
Module: TrackingMonitor

A fatal system signal has occurred: segmentation violation

[dan131riley]

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc530/CMSSW_8_1_X_2016-08-16-2300/pyRelValMatrixLogs/run/1004.0_RunHI2011+RunHI2011+TIER0EXPHI+ALCAEXPHI+ALCAHARVD1HI+ALCAHARVD2HI+ALCAHARVD3HI+ALCAHARVD5HI/step2_RunHI2011+RunHI2011+TIER0EXPHI+ALCAEXPHI+ALCAHARVD1HI+ALCAHARVD2HI+ALCAHARVD3HI+ALCAHARVD5HI.log

and another, pointing strongly to an issue with calling in to TCling

Thread 10 (Thread 0x7fbf73dbf700 (LWP 24511)):
#0  0x0000003144c0f37d in waitpid () from /lib64/libpthread.so.0
#1  0x00007fbf86292d17 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-16-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007fbf862936a2 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-16-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007fbf8a7d5a50 in std::(anonymous namespace)::execute_native_thread_routine (__p=<optimized out>) at ../../../../../libstdc++-v3/src/c++11/thread.cc:84
#4  0x0000003144c07aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000031448e8aad in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fbf6d51d700 (LWP 24636)):
#0  0x00000031448e90a3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fbf6d5f13ba in XrdSys::IOEvents::PollE::Begin (this=0x7fbf70421b40, syncsem=<optimized out>, retcode=<optimized out>, eTxt=<optimized out>) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/./XrdSys/XrdSysIOEventsPollE.icc:213
#2  0x00007fbf6d5eda8d in XrdSys::IOEvents::BootStrap::Start (parg=0x7ffed45cdbc0) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdSys/XrdSysIOEvents.cc:131
#3  0x00007fbf6d5ec647 in XrdSysThread_Xeq (myargs=0x7fbf7256c160) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdSys/XrdSysPthread.cc:86
#4  0x0000003144c07aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000031448e8aad in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fbf6cb1c700 (LWP 24637)):
#0  0x0000003144c0f00d in nanosleep () from /lib64/libpthread.so.0
#1  0x00007fbf6d5ecdeb in XrdSysTimer::Wait (mills=<optimized out>) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdSys/XrdSysTimer.cc:239
#2  0x00007fbf6d700caa in XrdCl::TaskManager::RunTasks (this=0x7fbf6eef81c0) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdCl/XrdClTaskManager.cc:244
#3  0x00007fbf6d700e29 in RunRunnerThread (arg=<optimized out>) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdCl/XrdClTaskManager.cc:37
#4  0x0000003144c07aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000031448e8aad in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fbf6c11b700 (LWP 24638)):
#0  0x00000031448e5309 in syscall () from /lib64/libc.so.6
#1  0x00007fbf6d75c4e1 in XrdSys::LinuxSemaphore::Wait (this=0x7fbf6ed63ba0) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/./XrdSys/XrdSysLinuxSemaphore.hh:161
#2  XrdCl::SyncQueue<XrdCl::JobManager::JobHelper>::Get (this=0x7fbf70421a78) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/./XrdCl/XrdClSyncQueue.hh:67
#3  XrdCl::JobManager::RunJobs (this=0x7fbf70421a60) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdCl/XrdClJobManager.cc:146
#4  0x00007fbf6d75c679 in RunRunnerThread (arg=<optimized out>) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdCl/XrdClJobManager.cc:33
#5  0x0000003144c07aa1 in start_thread () from /lib64/libpthread.so.0
#6  0x00000031448e8aad in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fbf6b71a700 (LWP 24639)):
#0  0x00000031448e5309 in syscall () from /lib64/libc.so.6
#1  0x00007fbf6d75c4e1 in XrdSys::LinuxSemaphore::Wait (this=0x7fbf6ed63ba0) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/./XrdSys/XrdSysLinuxSemaphore.hh:161
#2  XrdCl::SyncQueue<XrdCl::JobManager::JobHelper>::Get (this=0x7fbf70421a78) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/./XrdCl/XrdClSyncQueue.hh:67
#3  XrdCl::JobManager::RunJobs (this=0x7fbf70421a60) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdCl/XrdClJobManager.cc:146
#4  0x00007fbf6d75c679 in RunRunnerThread (arg=<optimized out>) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdCl/XrdClJobManager.cc:33
#5  0x0000003144c07aa1 in start_thread () from /lib64/libpthread.so.0
#6  0x00000031448e8aad in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fbf6ad19700 (LWP 24640)):
#0  0x00000031448e5309 in syscall () from /lib64/libc.so.6
#1  0x00007fbf6d75c4e1 in XrdSys::LinuxSemaphore::Wait (this=0x7fbf6ed63ba0) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/./XrdSys/XrdSysLinuxSemaphore.hh:161
#2  XrdCl::SyncQueue<XrdCl::JobManager::JobHelper>::Get (this=0x7fbf70421a78) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/./XrdCl/XrdClSyncQueue.hh:67
#3  XrdCl::JobManager::RunJobs (this=0x7fbf70421a60) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdCl/XrdClJobManager.cc:146
#4  0x00007fbf6d75c679 in RunRunnerThread (arg=<optimized out>) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdCl/XrdClJobManager.cc:33
#5  0x0000003144c07aa1 in start_thread () from /lib64/libpthread.so.0
#6  0x00000031448e8aad in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fbf3dc3f700 (LWP 24896)):
#0  0x00000031448df283 in poll () from /lib64/libc.so.6
#1  0x00007fbf86293504 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-16-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007fbf8629376a in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-16-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007fbf8629405e in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-16-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007fbf8750108e in ?? ()
#6  0x00007fbf3dc393b8 in ?? ()
#7  0x00007fbe7d3acbf0 in ?? ()
#8  0x0000000000000000 in ?? ()
Thread 3 (Thread 0x7fbf3d23e700 (LWP 24897)):
#0  0x00000031448acbdd in nanosleep () from /lib64/libc.so.6
#1  0x00000031448aca50 in sleep () from /lib64/libc.so.6
#2  0x00007fbf86292b4e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-16-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000003144c0e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#5  0x0000003144c095f3 in _L_lock_892 () from /lib64/libpthread.so.0
#6  0x0000003144c094d7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#7  0x00007fbf7e1ae0cb in TCling::CallFunc_Factory() const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-16-2300/external/slc6_amd64_gcc530/lib/libCling.so
#8  0x00007fbf8b74d972 in TMethodCall::Init(TFunction const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-16-2300/external/slc6_amd64_gcc530/lib/libCore.so
#9  0x00007fbf8b74eb5b in TMethodCall::TMethodCall(TFunction const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-16-2300/external/slc6_amd64_gcc530/lib/libCore.so
#10 0x00007fbf8bf3c486 in edm::FunctionWithDict::FunctionWithDict(TMethod*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-16-2300/lib/slc6_amd64_gcc530/libFWCoreUtilities.so
#11 0x00007fbf814f5368 in reco::findMethod(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > >&, char const*, int&) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#12 0x00007fbf814f51e4 in reco::findMethod(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > >&, char const*, int&) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#13 0x00007fbf814ba270 in reco::parser::MethodSetter::push(std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, char const*, bool) const () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#14 0x00007fbf8150ec40 in reco::parser::SingleInvoker::SingleInvoker(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#15 0x00007fbf8150fd86 in reco::parser::LazyInvoker::invoker(edm::TypeWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#16 0x00007fbf815121cb in reco::parser::LazyInvoker::invokeLast(edm::ObjectWithDict const&, std::vector<edm::ObjectWithDict, std::allocator<edm::ObjectWithDict> >&) const () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#17 0x00007fbf8150a3ff in reco::parser::ExpressionLazyVar::value(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#18 0x00007fbf814e3416 in reco::parser::BinarySelector::operator()(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#19 0x00007fbf4942ac89 in TrackingMonitor::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#20 0x00007fbf8c3462fa in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007fbf8c321aa2 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007fbf8c259be4 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007fbf8c259e78 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#24 0x00007fbf8c25a1ed in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#25 0x00007fbf8c2789ce in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#26 0x00007fbf8c278c82 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#27 0x00007fbf8c2790a5 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#28 0x00007fbf8c27939b in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#29 0x00007fbf8c27960f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#30 0x00007fbf8c2fbf92 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#31 0x00007fbf8c2fc6c7 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#32 0x00007fbf8c302f24 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#33 0x00007fbf8afc39b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fbf3e5afe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#34 0x00007fbf8afbd1a4 in tbb::internal::arena::process (this=this@entry=0x7fbf8943b780, s=...) at ../../src/tbb/arena.cpp:156
#35 0x00007fbf8afbc264 in tbb::internal::market::process (this=0x7fbf89463e00, j=...) at ../../src/tbb/market.cpp:502
#36 0x00007fbf8afb8736 in tbb::internal::rml::private_worker::run (this=0x7fbf8945ba80) at ../../src/tbb/private_server.cpp:275
#37 0x00007fbf8afb8989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#38 0x0000003144c07aa1 in start_thread () from /lib64/libpthread.so.0
#39 0x00000031448e8aad in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fbf3c77f700 (LWP 24898)):
#0  0x00000031448acbdd in nanosleep () from /lib64/libc.so.6
#1  0x00000031448aca50 in sleep () from /lib64/libc.so.6
#2  0x00007fbf86292b4e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-16-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007fbf7f1fa5c0 in clang::NestedNameSpecifier::Create(clang::ASTContext const&, clang::NestedNameSpecifier*, clang::NamespaceAliasDecl*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-16-2300/external/slc6_amd64_gcc530/lib/libCling.so
#5  0x00007fbf7e2c2c02 in cling::utils::TypeName::CreateNestedNameSpecifier(clang::ASTContext const&, clang::TagDecl const*, bool) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-16-2300/external/slc6_amd64_gcc530/lib/libCling.so
#6  0x00007fbf7e2c1dff in cling::utils::TypeName::GetFullyQualifiedType(clang::QualType, clang::ASTContext const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-16-2300/external/slc6_amd64_gcc530/lib/libCling.so
#7  0x00007fbf7e2c21a4 in cling::utils::TypeName::GetFullyQualifiedName(clang::QualType, clang::ASTContext const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-16-2300/external/slc6_amd64_gcc530/lib/libCling.so
#8  0x00007fbf7e169c1f in ROOT::TMetaUtils::GetFullyQualifiedTypeName(std::string&, clang::QualType const&, clang::ASTContext const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-16-2300/external/slc6_amd64_gcc530/lib/libCling.so
#9  0x00007fbf7e1cf2f3 in TClingTypeInfo::Name() const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-16-2300/external/slc6_amd64_gcc530/lib/libCling.so
#10 0x00007fbf7e1db581 in TClingMethodInfo::CreateSignature(TString&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-16-2300/external/slc6_amd64_gcc530/lib/libCling.so
#11 0x00007fbf8b728852 in TFunction::CreateSignature() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-16-2300/external/slc6_amd64_gcc530/lib/libCore.so
#12 0x00007fbf8b719289 in TMethod::CreateSignature() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-16-2300/external/slc6_amd64_gcc530/lib/libCore.so
#13 0x00007fbf8b728a16 in TFunction::GetSignature() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-16-2300/external/slc6_amd64_gcc530/lib/libCore.so
#14 0x00007fbf8b74d8b6 in TMethodCall::Init(TFunction const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-16-2300/external/slc6_amd64_gcc530/lib/libCore.so
#15 0x00007fbf8b74eb5b in TMethodCall::TMethodCall(TFunction const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-16-2300/external/slc6_amd64_gcc530/lib/libCore.so
#16 0x00007fbf8bf3c486 in edm::FunctionWithDict::FunctionWithDict(TMethod*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-16-2300/lib/slc6_amd64_gcc530/libFWCoreUtilities.so
#17 0x00007fbf814f5368 in reco::findMethod(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > >&, char const*, int&) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#18 0x00007fbf814f51e4 in reco::findMethod(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > >&, char const*, int&) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#19 0x00007fbf814ba270 in reco::parser::MethodSetter::push(std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, char const*, bool) const () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#20 0x00007fbf8150ec40 in reco::parser::SingleInvoker::SingleInvoker(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#21 0x00007fbf8150fd86 in reco::parser::LazyInvoker::invoker(edm::TypeWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#22 0x00007fbf815121cb in reco::parser::LazyInvoker::invokeLast(edm::ObjectWithDict const&, std::vector<edm::ObjectWithDict, std::allocator<edm::ObjectWithDict> >&) const () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#23 0x00007fbf8150a3ff in reco::parser::ExpressionLazyVar::value(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#24 0x00007fbf814e3416 in reco::parser::BinarySelector::operator()(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#25 0x00007fbf4942ac89 in TrackingMonitor::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#26 0x00007fbf8c3462fa in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#27 0x00007fbf8c321aa2 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#28 0x00007fbf8c259be4 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#29 0x00007fbf8c259e78 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#30 0x00007fbf8c25a1ed in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#31 0x00007fbf8c2789ce in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#32 0x00007fbf8c278c82 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#33 0x00007fbf8c2790a5 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#34 0x00007fbf8c27939b in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#35 0x00007fbf8c27960f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#36 0x00007fbf8c2fbf92 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#37 0x00007fbf8c2fc80a in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#38 0x00007fbf8c302f24 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#39 0x00007fbf8afc39b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fbf3e597e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#40 0x00007fbf8afbd1a4 in tbb::internal::arena::process (this=this@entry=0x7fbf8943b780, s=...) at ../../src/tbb/arena.cpp:156
#41 0x00007fbf8afbc264 in tbb::internal::market::process (this=0x7fbf89463e00, j=...) at ../../src/tbb/market.cpp:502
#42 0x00007fbf8afb8736 in tbb::internal::rml::private_worker::run (this=0x7fbf8945b980) at ../../src/tbb/private_server.cpp:275
#43 0x00007fbf8afb8989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#44 0x0000003144c07aa1 in start_thread () from /lib64/libpthread.so.0
#45 0x00000031448e8aad in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fbf8c3c4bc0 (LWP 23990)):
#0  0x00000031448acbdd in nanosleep () from /lib64/libc.so.6
#1  0x00000031448aca50 in sleep () from /lib64/libc.so.6
#2  0x00007fbf86292b4e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-16-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000003144c0e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#5  0x0000003144c095f3 in _L_lock_892 () from /lib64/libpthread.so.0
#6  0x0000003144c094d7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#7  0x00007fbf8b729103 in TFunction::~TFunction() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-16-2300/external/slc6_amd64_gcc530/lib/libCore.so
#8  0x00007fbf8b719269 in TMethod::~TMethod() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-16-2300/external/slc6_amd64_gcc530/lib/libCore.so
#9  0x00007fbf8b74ecf8 in TMethodCall::~TMethodCall() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-16-2300/external/slc6_amd64_gcc530/lib/libCore.so
#10 0x00007fbf8bf3c4be in edm::FunctionWithDict::FunctionWithDict(TMethod*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-16-2300/lib/slc6_amd64_gcc530/libFWCoreUtilities.so
#11 0x00007fbf814f5368 in reco::findMethod(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > >&, char const*, int&) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#12 0x00007fbf814f51e4 in reco::findMethod(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > >&, char const*, int&) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#13 0x00007fbf814ba270 in reco::parser::MethodSetter::push(std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, char const*, bool) const () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#14 0x00007fbf8150ec40 in reco::parser::SingleInvoker::SingleInvoker(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#15 0x00007fbf8150fd86 in reco::parser::LazyInvoker::invoker(edm::TypeWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#16 0x00007fbf815121cb in reco::parser::LazyInvoker::invokeLast(edm::ObjectWithDict const&, std::vector<edm::ObjectWithDict, std::allocator<edm::ObjectWithDict> >&) const () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#17 0x00007fbf8150a3ff in reco::parser::ExpressionLazyVar::value(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#18 0x00007fbf814e3416 in reco::parser::BinarySelector::operator()(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#19 0x00007fbf4942ac89 in TrackingMonitor::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#20 0x00007fbf8c3462fa in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007fbf8c321aa2 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007fbf8c259be4 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007fbf8c259e78 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#24 0x00007fbf8c25a1ed in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#25 0x00007fbf8c2789ce in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#26 0x00007fbf8c278c82 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#27 0x00007fbf8c2790a5 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#28 0x00007fbf8c27939b in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#29 0x00007fbf8c27960f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#30 0x00007fbf8c2fbf92 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#31 0x00007fbf8c2fc6c7 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#32 0x00007fbf8c302f24 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#33 0x00007fbf8afc39b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fbf89462400, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#34 0x00007fbf8c2fc51e in edm::EventProcessor::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#35 0x00007fbf8c25cf0a in statemachine::HandleEvent::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#36 0x00007fbf8c25e498 in statemachine::HandleEvent::HandleEvent(boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::my_context) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#37 0x00007fbf8c2661de in boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::shallow_construct(boost::intrusive_ptr<statemachine::HandleLumis> const&, boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>&) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#38 0x00007fbf8c2663aa in boost::statechart::simple_state<statemachine::FirstLumi, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#39 0x00007fbf8c3029d8 in boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&) () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#40 0x00007fbf8c2f5afd in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/2016-34/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-14-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#41 0x000000000040c844 in main::{lambda()#1}::operator()() const ()
#42 0x000000000040acc4 in main ()

Current Modules:

Module: TrackingMonitor (crashed)
Module: TrackingMonitor
Module: TrackingMonitor
Module: TrackingMonitor

A fatal system signal has occurred: segmentation violation

[dan131riley]

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc530/CMSSW_8_1_X_2016-08-22-1100/pyRelValMatrixLogs/run/4.22_RunCosmics2011A+RunCosmics2011A+RECOCOSD+ALCACOSD+SKIMCOSD+HARVESTDC/step2_RunCosmics2011A+RunCosmics2011A+RECOCOSD+ALCACOSD+SKIMCOSD+HARVESTDC.log

Similar to previous-1

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thread 10 (Thread 0x7fc2eadbf700 (LWP 7853)):
#0  0x0000003cee40f37d in waitpid () from /lib64/libpthread.so.0
#1  0x00007fc2fc193d47 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007fc2fc1946d2 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007fc3001e3a50 in std::(anonymous namespace)::execute_native_thread_routine (__p=<optimized out>) at ../../../../../libstdc++-v3/src/c++11/thread.cc:84
#4  0x0000003cee407aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003cee0e8aad in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fc2e483c700 (LWP 7870)):
#0  0x0000003cee0e90a3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fc2e49093ba in XrdSys::IOEvents::PollE::Begin (this=0x7fc2ecfd6860, syncsem=<optimized out>, retcode=<optimized out>, eTxt=<optimized out>) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/./XrdSys/XrdSysIOEventsPollE.icc:213
#2  0x00007fc2e4905a8d in XrdSys::IOEvents::BootStrap::Start (parg=0x7ffc4cb6aed0) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdSys/XrdSysIOEvents.cc:131
#3  0x00007fc2e4904647 in XrdSysThread_Xeq (myargs=0x7fc2e94cdda0) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdSys/XrdSysPthread.cc:86
#4  0x0000003cee407aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003cee0e8aad in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fc2e3e3b700 (LWP 7871)):
#0  0x0000003cee40f00d in nanosleep () from /lib64/libpthread.so.0
#1  0x00007fc2e4904deb in XrdSysTimer::Wait (mills=<optimized out>) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdSys/XrdSysTimer.cc:239
#2  0x00007fc2e4a0fcaa in XrdCl::TaskManager::RunTasks (this=0x7fc2e5fdfb00) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdCl/XrdClTaskManager.cc:244
#3  0x00007fc2e4a0fe29 in RunRunnerThread (arg=<optimized out>) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdCl/XrdClTaskManager.cc:37
#4  0x0000003cee407aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003cee0e8aad in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fc2e343a700 (LWP 7872)):
#0  0x0000003cee0e5309 in syscall () from /lib64/libc.so.6
#1  0x00007fc2e4a6b4e1 in XrdSys::LinuxSemaphore::Wait (this=0x7fc2e94ff388) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/./XrdSys/XrdSysLinuxSemaphore.hh:161
#2  XrdCl::SyncQueue<XrdCl::JobManager::JobHelper>::Get (this=0x7fc2ecfd6798) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/./XrdCl/XrdClSyncQueue.hh:67
#3  XrdCl::JobManager::RunJobs (this=0x7fc2ecfd6780) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdCl/XrdClJobManager.cc:146
#4  0x00007fc2e4a6b679 in RunRunnerThread (arg=<optimized out>) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdCl/XrdClJobManager.cc:33
#5  0x0000003cee407aa1 in start_thread () from /lib64/libpthread.so.0
#6  0x0000003cee0e8aad in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fc2e2a39700 (LWP 7873)):
#0  0x0000003cee0e5309 in syscall () from /lib64/libc.so.6
#1  0x00007fc2e4a6b4e1 in XrdSys::LinuxSemaphore::Wait (this=0x7fc2e94ff388) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/./XrdSys/XrdSysLinuxSemaphore.hh:161
#2  XrdCl::SyncQueue<XrdCl::JobManager::JobHelper>::Get (this=0x7fc2ecfd6798) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/./XrdCl/XrdClSyncQueue.hh:67
#3  XrdCl::JobManager::RunJobs (this=0x7fc2ecfd6780) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdCl/XrdClJobManager.cc:146
#4  0x00007fc2e4a6b679 in RunRunnerThread (arg=<optimized out>) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdCl/XrdClJobManager.cc:33
#5  0x0000003cee407aa1 in start_thread () from /lib64/libpthread.so.0
#6  0x0000003cee0e8aad in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fc2e2038700 (LWP 7874)):
#0  0x0000003cee0e5309 in syscall () from /lib64/libc.so.6
#1  0x00007fc2e4a6b4e1 in XrdSys::LinuxSemaphore::Wait (this=0x7fc2e94ff388) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/./XrdSys/XrdSysLinuxSemaphore.hh:161
#2  XrdCl::SyncQueue<XrdCl::JobManager::JobHelper>::Get (this=0x7fc2ecfd6798) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/./XrdCl/XrdClSyncQueue.hh:67
#3  XrdCl::JobManager::RunJobs (this=0x7fc2ecfd6780) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdCl/XrdClJobManager.cc:146
#4  0x00007fc2e4a6b679 in RunRunnerThread (arg=<optimized out>) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdCl/XrdClJobManager.cc:33
#5  0x0000003cee407aa1 in start_thread () from /lib64/libpthread.so.0
#6  0x0000003cee0e8aad in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fc2bc2ff700 (LWP 1932)):
#0  0x0000003cee0df283 in poll () from /lib64/libc.so.6
#1  0x00007fc2fc194534 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007fc2fc19479a in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007fc2fc19508e in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007fc2bcd4c08e in ?? ()
#6  0x00007fc2bc2f9358 in ?? ()
#7  0x00007fc2a4d5d838 in ?? ()
#8  0x0000000000000000 in ?? ()
Thread 3 (Thread 0x7fc2bb8fe700 (LWP 1933)):
#0  0x0000003cee0acbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003cee0aca50 in sleep () from /lib64/libc.so.6
#2  0x00007fc2fc193b7e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000003cee40a8a9 in pthread_mutex_unlock () from /lib64/libpthread.so.0
#5  0x00007fc3009ddcd0 in je_malloc_mutex_unlock (tsdn=0x7fc2bb8fa310, mutex=<optimized out>) at include/jemalloc/internal/mutex.h:115
#6  arena_bin_nonfull_run_get (bin=<optimized out>, arena=0x7fc2ffe17880, tsdn=0x7fc2bb8fa310) at src/arena.c:2290
#7  arena_bin_malloc_hard (tsdn=tsdn@entry=0x7fc2bb8fa310, arena=arena@entry=0x7fc2ffe17880, bin=bin@entry=0x7fc2ffe19848) at src/arena.c:2334
#8  0x00007fc3009dfc5b in je_arena_tcache_fill_small (tsdn=0x7fc2bb8fa310, arena=0x7fc2ffe17880, tbin=0x7fc2bc32d468, binind=<optimized out>, prof_accumbytes=<optimized out>) at src/arena.c:2394
#9  0x00007fc300a04f3a in je_tcache_alloc_small_hard (tsdn=<optimized out>, arena=arena@entry=0x7fc2ffe17880, tcache=tcache@entry=0x7fc2bc32d000, tbin=tbin@entry=0x7fc2bc32d468, binind=binind@entry=34, tcache_success=tcache_success@entry=0x7fc2bb8f865f) at src/tcache.c:79
#10 0x00007fc3009d298f in je_tcache_alloc_small (slow_path=false, zero=false, binind=34, size=140474357830760, tcache=<optimized out>, arena=<optimized out>, tsd=<optimized out>) at include/jemalloc/internal/tcache.h:298
#11 je_arena_malloc (slow_path=false, tcache=<optimized out>, zero=false, ind=34, size=140474357830760, arena=0x0, tsdn=<optimized out>) at include/jemalloc/internal/arena.h:1333
#12 je_iallocztm (slow_path=false, arena=0x0, is_metadata=false, tcache=<optimized out>, zero=false, ind=34, size=140474357830760, tsdn=<optimized out>) at include/jemalloc/internal/jemalloc_internal.h:967
#13 je_ialloc (slow_path=false, zero=false, ind=34, size=140474357830760, tsd=<optimized out>) at include/jemalloc/internal/jemalloc_internal.h:979
#14 ialloc_body (slow_path=false, usize=<synthetic pointer>, tsdn=<synthetic pointer>, zero=false, size=140474357830760) at src/jemalloc.c:1520
#15 malloc (size=size@entry=12072) at src/jemalloc.c:1559
#16 0x00007fc3001b9f68 in operator new (sz=12072) at ../../../../libstdc++-v3/libsupc++/new_op.cc:50
#17 0x00007fc3001ba009 in operator new[] (sz=<optimized out>) at ../../../../libstdc++-v3/libsupc++/new_opv.cc:32
#18 0x00007fc300e9afc4 in TExMap::TExMap(int) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/external/slc6_amd64_gcc530/lib/libCore.so
#19 0x00007fc3010f59a8 in TBufferFile::InitMap() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/external/slc6_amd64_gcc530/lib/libRIO.so
#20 0x00007fc3010fd3b1 in TBufferFile::MapObject(void const*, TClass const*, unsigned int) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/external/slc6_amd64_gcc530/lib/libRIO.so
#21 0x00007fc301631e5d in TBranchElement::FillLeavesMember(TBuffer&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/external/slc6_amd64_gcc530/lib/libTree.so
#22 0x00007fc3016056ca in TBranch::Fill() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/external/slc6_amd64_gcc530/lib/libTree.so
#23 0x00007fc30163d0d0 in TBranchElement::Fill() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/external/slc6_amd64_gcc530/lib/libTree.so
#24 0x00007fc30163ce58 in TBranchElement::Fill() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/external/slc6_amd64_gcc530/lib/libTree.so
#25 0x00007fc3015f0031 in TTree::Fill() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/external/slc6_amd64_gcc530/lib/libTree.so
#26 0x00007fc2c6e88a0b in edm::RootOutputFile::fillBranches(edm::BranchType const&, edm::OccurrenceForOutput const&, std::vector<edm::StoredProductProvenance, std::allocator<edm::StoredProductProvenance> >*, edm::ProductProvenanceRetriever const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libIOPoolOutput.so
#27 0x00007fc2c6e8a4fb in edm::RootOutputFile::writeOne(edm::EventForOutput const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libIOPoolOutput.so
#28 0x00007fc2c6e9926b in edm::PoolOutputModule::write(edm::EventForOutput const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libIOPoolOutput.so
#29 0x00007fc301acd21e in edm::one::OutputModuleBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#30 0x00007fc301a214d2 in edm::WorkerT<edm::one::OutputModuleBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#31 0x00007fc3019d4584 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#32 0x00007fc3019d4818 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#33 0x00007fc3019d4b8d in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#34 0x00007fc3019fcd0e in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#35 0x00007fc3019fcfc2 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#36 0x00007fc3019fd4fc in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#37 0x00007fc3019fd6db in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#38 0x00007fc3019fd94f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#39 0x00007fc301a7d902 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#40 0x00007fc301a7e17a in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#41 0x00007fc301a849f4 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#42 0x00007fc3007ab9b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fc2fec57e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#43 0x00007fc3007a51a4 in tbb::internal::arena::process (this=this@entry=0x7fc2fedff780, s=...) at ../../src/tbb/arena.cpp:156
#44 0x00007fc3007a4264 in tbb::internal::market::process (this=0x7fc2fee27c00, j=...) at ../../src/tbb/market.cpp:502
#45 0x00007fc3007a0736 in tbb::internal::rml::private_worker::run (this=0x7fc2fec46180) at ../../src/tbb/private_server.cpp:275
#46 0x00007fc3007a0989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#47 0x0000003cee407aa1 in start_thread () from /lib64/libpthread.so.0
#48 0x0000003cee0e8aad in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fc2bae3f700 (LWP 1934)):
#0  0x0000003cee0acbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003cee0aca50 in sleep () from /lib64/libc.so.6
#2  0x00007fc2fc193b7e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000003cee4094d1 in pthread_mutex_lock () from /lib64/libpthread.so.0
#5  0x00007fc300edefce in TFunction::TFunction(TFunction const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/external/slc6_amd64_gcc530/lib/libCore.so
#6  0x00007fc300ecf136 in TMethod::TMethod(TMethod const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/external/slc6_amd64_gcc530/lib/libCore.so
#7  0x00007fc300ecf1d2 in TMethod::Clone(char const*) const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/external/slc6_amd64_gcc530/lib/libCore.so
#8  0x00007fc300f03867 in TMethodCall::Init(TFunction const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/external/slc6_amd64_gcc530/lib/libCore.so
#9  0x00007fc300f04b5b in TMethodCall::TMethodCall(TFunction const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/external/slc6_amd64_gcc530/lib/libCore.so
#10 0x00007fc301791c16 in edm::FunctionWithDict::FunctionWithDict(TMethod*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreUtilities.so
#11 0x00007fc2f851d9d8 in reco::findMethod(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > >&, char const*, int&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#12 0x00007fc2f851d854 in reco::findMethod(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > >&, char const*, int&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#13 0x00007fc2f84d84a0 in reco::parser::MethodSetter::push(std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, char const*, bool) const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#14 0x00007fc2f84fb4e0 in reco::parser::SingleInvoker::SingleInvoker(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#15 0x00007fc2f84fc626 in reco::parser::LazyInvoker::invoker(edm::TypeWithDict const&) const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#16 0x00007fc2f84fea6b in reco::parser::LazyInvoker::invokeLast(edm::ObjectWithDict const&, std::vector<edm::ObjectWithDict, std::allocator<edm::ObjectWithDict> >&) const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#17 0x00007fc2f8529a1f in reco::parser::ExpressionLazyVar::value(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#18 0x00007fc2f8502ac6 in reco::parser::BinarySelector::operator()(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#19 0x00007fc2d931bcc9 in TrackingMonitor::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#20 0x00007fc301ad310a in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007fc301a20c82 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007fc3019d4584 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007fc3019d4818 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#24 0x00007fc3019d4b8d in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#25 0x00007fc3019fcd0e in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#26 0x00007fc3019fcfc2 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#27 0x00007fc3019fd3e5 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#28 0x00007fc3019fd6db in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#29 0x00007fc3019fd94f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#30 0x00007fc301a7d902 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#31 0x00007fc301a7e037 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#32 0x00007fc301a849f4 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#33 0x00007fc3007ab9b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fc2ba42fe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#34 0x00007fc3007a51a4 in tbb::internal::arena::process (this=this@entry=0x7fc2fedff780, s=...) at ../../src/tbb/arena.cpp:156
#35 0x00007fc3007a4264 in tbb::internal::market::process (this=0x7fc2fee27c00, j=...) at ../../src/tbb/market.cpp:502
#36 0x00007fc3007a0736 in tbb::internal::rml::private_worker::run (this=0x7fc2fec46080) at ../../src/tbb/private_server.cpp:275
#37 0x00007fc3007a0989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#38 0x0000003cee407aa1 in start_thread () from /lib64/libpthread.so.0
#39 0x0000003cee0e8aad in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fc2ffe7cbc0 (LWP 7747)):
#0  0x0000003cee0acbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003cee0aca50 in sleep () from /lib64/libc.so.6
#2  0x00007fc2fc193b7e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000003cee40e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#5  0x0000003cee4095d8 in _L_lock_854 () from /lib64/libpthread.so.0
#6  0x0000003cee4094a7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#7  0x00007fc301acd188 in edm::one::OutputModuleBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#8  0x00007fc301a214d2 in edm::WorkerT<edm::one::OutputModuleBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#9  0x00007fc3019d4584 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#10 0x00007fc3019d4818 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#11 0x00007fc3019d4b8d in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#12 0x00007fc3019fcd0e in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#13 0x00007fc3019fcfc2 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#14 0x00007fc3019fd4fc in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#15 0x00007fc3019fd6db in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007fc3019fd94f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007fc301a7d902 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007fc301a7e037 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007fc301a849f4 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007fc3007ab9b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fc2fee16600, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#21 0x00007fc301a7de8e in edm::EventProcessor::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007fc3019e24ca in statemachine::HandleEvent::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007fc3019e3a58 in statemachine::HandleEvent::HandleEvent(boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::my_context) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#24 0x00007fc3019eb7de in boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::shallow_construct(boost::intrusive_ptr<statemachine::HandleLumis> const&, boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#25 0x00007fc3019eb9aa in boost::statechart::simple_state<statemachine::FirstLumi, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#26 0x00007fc301a844a8 in boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#27 0x00007fc301a777dd in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-08-22-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#28 0x000000000040c844 in main::{lambda()#1}::operator()() const ()
#29 0x000000000040acc4 in main ()

Current Modules:

Module: TrackingMonitor (crashed)
Module: PoolOutputModule
Module: TrackingMonitor
Module: PoolOutputModule

A fatal system signal has occurred: segmentation violation

[Dr15Jones]

Looks like TrackingMonitor again. So probably something to do with the lazy evaluation mode of the expression parser. @dan131riley Could you try to create a simple job which just has TrackingMonitor and just enough stuff to allow TrackingMonitor to run the expression parser? They try to hammer that job over and over again to see if it can fail semi reliably? If you can do that, we can try switching the TrackingMonitor to not use the lazy evaluation of the parser and instead set the types at initialization time. After such a switch we again hammer the system to see if it fails.

[dan131riley]

Another one, all four threads in TrackingMonitor:

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc530/CMSSW_8_1_X_2016-08-24-2300/pyRelValMatrixLogs/run/136.733_RunCosmics2016B+RunCosmics2016B+RECOCOSDRUN2+ALCACOSDRUN2+HARVESTDCRUN2/step2_RunCosmics2016B+RunCosmics2016B+RECOCOSDRUN2+ALCACOSDRUN2+HARVESTDCRUN2.log

I believe we can declare this our current most frequent threading-related crash. Attempts to reproduce with a stripped-down config are so far not showing any success.

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thread 10 (Thread 0x7f51dcf7f700 (LWP 31266)):
#0  0x0000003de100f37d in waitpid () from /lib64/libpthread.so.0
#1  0x00007f51ee8ccd47 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007f51ee8cd6d2 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007f51f1727a50 in std::(anonymous namespace)::execute_native_thread_routine (__p=<optimized out>) at ../../../../../libstdc++-v3/src/c++11/thread.cc:84
#4  0x0000003de1007aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003de0ce8aad in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f51d605f700 (LWP 31275)):
#0  0x0000003de0ce90a3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f51d61333ba in XrdSys::IOEvents::PollE::Begin (this=0x7f51d9216240, syncsem=<optimized out>, retcode=<optimized out>, eTxt=<optimized out>) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/./XrdSys/XrdSysIOEventsPollE.icc:213
#2  0x00007f51d612fa8d in XrdSys::IOEvents::BootStrap::Start (parg=0x7ffe449878e0) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdSys/XrdSysIOEvents.cc:131
#3  0x00007f51d612e647 in XrdSysThread_Xeq (myargs=0x7f51db50f4a0) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdSys/XrdSysPthread.cc:86
#4  0x0000003de1007aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003de0ce8aad in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f51d565e700 (LWP 31276)):
#0  0x0000003de100f00d in nanosleep () from /lib64/libpthread.so.0
#1  0x00007f51d612edeb in XrdSysTimer::Wait (mills=<optimized out>) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdSys/XrdSysTimer.cc:239
#2  0x00007f51d624acaa in XrdCl::TaskManager::RunTasks (this=0x7f51d7971940) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdCl/XrdClTaskManager.cc:244
#3  0x00007f51d624ae29 in RunRunnerThread (arg=<optimized out>) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdCl/XrdClTaskManager.cc:37
#4  0x0000003de1007aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003de0ce8aad in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f51d4c5d700 (LWP 31277)):
#0  0x0000003de0ce5309 in syscall () from /lib64/libc.so.6
#1  0x00007f51d62a64e1 in XrdSys::LinuxSemaphore::Wait (this=0x7f51d7a53380) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/./XrdSys/XrdSysLinuxSemaphore.hh:161
#2  XrdCl::SyncQueue<XrdCl::JobManager::JobHelper>::Get (this=0x7f51d9216178) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/./XrdCl/XrdClSyncQueue.hh:67
#3  XrdCl::JobManager::RunJobs (this=0x7f51d9216160) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdCl/XrdClJobManager.cc:146
#4  0x00007f51d62a6679 in RunRunnerThread (arg=<optimized out>) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdCl/XrdClJobManager.cc:33
#5  0x0000003de1007aa1 in start_thread () from /lib64/libpthread.so.0
#6  0x0000003de0ce8aad in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f51d425c700 (LWP 31278)):
#0  0x0000003de0ce5309 in syscall () from /lib64/libc.so.6
#1  0x00007f51d62a64e1 in XrdSys::LinuxSemaphore::Wait (this=0x7f51d7a53380) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/./XrdSys/XrdSysLinuxSemaphore.hh:161
#2  XrdCl::SyncQueue<XrdCl::JobManager::JobHelper>::Get (this=0x7f51d9216178) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/./XrdCl/XrdClSyncQueue.hh:67
#3  XrdCl::JobManager::RunJobs (this=0x7f51d9216160) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdCl/XrdClJobManager.cc:146
#4  0x00007f51d62a6679 in RunRunnerThread (arg=<optimized out>) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdCl/XrdClJobManager.cc:33
#5  0x0000003de1007aa1 in start_thread () from /lib64/libpthread.so.0
#6  0x0000003de0ce8aad in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f51d385b700 (LWP 31279)):
#0  0x0000003de0ce5309 in syscall () from /lib64/libc.so.6
#1  0x00007f51d62a64e1 in XrdSys::LinuxSemaphore::Wait (this=0x7f51d7a53380) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/./XrdSys/XrdSysLinuxSemaphore.hh:161
#2  XrdCl::SyncQueue<XrdCl::JobManager::JobHelper>::Get (this=0x7f51d9216178) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/./XrdCl/XrdClSyncQueue.hh:67
#3  XrdCl::JobManager::RunJobs (this=0x7f51d9216160) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdCl/XrdClJobManager.cc:146
#4  0x00007f51d62a6679 in RunRunnerThread (arg=<optimized out>) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre6-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre6-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0/xrootd-4.3.0/src/XrdCl/XrdClJobManager.cc:33
#5  0x0000003de1007aa1 in start_thread () from /lib64/libpthread.so.0
#6  0x0000003de0ce8aad in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f51ad7ff700 (LWP 31320)):
#0  0x0000003de0cacbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003de0caca50 in sleep () from /lib64/libc.so.6
#2  0x00007f51ee8ccb7e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000003de100e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#5  0x0000003de10095f3 in _L_lock_892 () from /lib64/libpthread.so.0
#6  0x0000003de10094d7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#7  0x00007f51e74378cb in TCling::CallFunc_Factory() const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/external/slc6_amd64_gcc530/lib/libCling.so
#8  0x00007f51f270c642 in TMethodCall::Init(TFunction const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/external/slc6_amd64_gcc530/lib/libCore.so
#9  0x00007f51f270d82b in TMethodCall::TMethodCall(TFunction const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/external/slc6_amd64_gcc530/lib/libCore.so
#10 0x00007f51f2f55c16 in edm::FunctionWithDict::FunctionWithDict(TMethod*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreUtilities.so
#11 0x00007f51ea85d9d8 in reco::findMethod(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > >&, char const*, int&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#12 0x00007f51ea85d854 in reco::findMethod(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > >&, char const*, int&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#13 0x00007f51ea8184a0 in reco::parser::MethodSetter::push(std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, char const*, bool) const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#14 0x00007f51ea83b4e0 in reco::parser::SingleInvoker::SingleInvoker(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#15 0x00007f51ea83c626 in reco::parser::LazyInvoker::invoker(edm::TypeWithDict const&) const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#16 0x00007f51ea83ea6b in reco::parser::LazyInvoker::invokeLast(edm::ObjectWithDict const&, std::vector<edm::ObjectWithDict, std::allocator<edm::ObjectWithDict> >&) const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#17 0x00007f51ea869a1f in reco::parser::ExpressionLazyVar::value(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#18 0x00007f51ea842ac6 in reco::parser::BinarySelector::operator()(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#19 0x00007f51ccb6ecc9 in TrackingMonitor::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#20 0x00007f51f33a010a in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007f51f32edc82 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007f51f32a1584 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007f51f32a1818 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#24 0x00007f51f32a1b8d in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#25 0x00007f51f32c9d0e in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#26 0x00007f51f32c9fc2 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#27 0x00007f51f32ca3e5 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#28 0x00007f51f32ca6db in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#29 0x00007f51f32ca94f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#30 0x00007f51f334a902 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#31 0x00007f51f334b037 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#32 0x00007f51f33519f4 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#33 0x00007f51f1f3d9b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f51f0bfbe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#34 0x00007f51f1f371a4 in tbb::internal::arena::process (this=this@entry=0x7f51f0d73780, s=...) at ../../src/tbb/arena.cpp:156
#35 0x00007f51f1f36264 in tbb::internal::market::process (this=0x7f51f0d9be00, j=...) at ../../src/tbb/market.cpp:502
#36 0x00007f51f1f32736 in tbb::internal::rml::private_worker::run (this=0x7f51f0d93a00) at ../../src/tbb/private_server.cpp:275
#37 0x00007f51f1f32989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#38 0x0000003de1007aa1 in start_thread () from /lib64/libpthread.so.0
#39 0x0000003de0ce8aad in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f51acdfe700 (LWP 31321)):
#0  0x0000003de0cdf283 in poll () from /lib64/libc.so.6
#1  0x00007f51ee8cd534 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007f51ee8cd79a in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007f51ee8ce08e in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007f51ef27708e in ?? ()
#6  0x00007f51acdf8358 in ?? ()
#7  0x00007f514ffe2240 in ?? ()
#8  0x0000000000000000 in ?? ()
Thread 2 (Thread 0x7f51ac37f700 (LWP 31322)):
#0  0x0000003de0cacbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003de0caca50 in sleep () from /lib64/libc.so.6
#2  0x00007f51ee8ccb7e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f51f2bc39b0 in ROOT::Math::chisquared_cdf_c(double, double, double) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/external/slc6_amd64_gcc530/lib/libMathCore.so
#5  0x00007f51ccbcfcc7 in TrackAnalyzer::analyze(edm::Event const&, edm::EventSetup const&, reco::Track const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#6  0x00007f51ccb6ee5b in TrackingMonitor::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#7  0x00007f51f33a010a in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#8  0x00007f51f32edc82 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#9  0x00007f51f32a1584 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#10 0x00007f51f32a1818 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#11 0x00007f51f32a1b8d in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#12 0x00007f51f32c9d0e in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#13 0x00007f51f32c9fc2 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#14 0x00007f51f32ca3e5 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#15 0x00007f51f32ca6db in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007f51f32ca94f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007f51f334a902 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007f51f334b17a in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007f51f33519f4 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007f51f1f3d9b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f51f0be3e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#21 0x00007f51f1f371a4 in tbb::internal::arena::process (this=this@entry=0x7f51f0d73780, s=...) at ../../src/tbb/arena.cpp:156
#22 0x00007f51f1f36264 in tbb::internal::market::process (this=0x7f51f0d9be00, j=...) at ../../src/tbb/market.cpp:502
#23 0x00007f51f1f32736 in tbb::internal::rml::private_worker::run (this=0x7f51f0d93980) at ../../src/tbb/private_server.cpp:275
#24 0x00007f51f1f32989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#25 0x0000003de1007aa1 in start_thread () from /lib64/libpthread.so.0
#26 0x0000003de0ce8aad in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f51f3602bc0 (LWP 31185)):
#0  0x0000003de0cacbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003de0caca50 in sleep () from /lib64/libc.so.6
#2  0x00007f51ee8ccb7e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f51f2bc39b0 in ROOT::Math::chisquared_cdf_c(double, double, double) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/external/slc6_amd64_gcc530/lib/libMathCore.so
#5  0x00007f51ccbcfcc7 in TrackAnalyzer::analyze(edm::Event const&, edm::EventSetup const&, reco::Track const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#6  0x00007f51ccb6ee5b in TrackingMonitor::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#7  0x00007f51f33a010a in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#8  0x00007f51f32edc82 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#9  0x00007f51f32a1584 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#10 0x00007f51f32a1818 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#11 0x00007f51f32a1b8d in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#12 0x00007f51f32c9d0e in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#13 0x00007f51f32c9fc2 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#14 0x00007f51f32ca3e5 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#15 0x00007f51f32ca6db in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007f51f32ca94f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007f51f334a902 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007f51f334b037 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007f51f33519f4 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007f51f1f3d9b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f51f0d9a400, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#21 0x00007f51f334ae8e in edm::EventProcessor::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007f51f32af4ca in statemachine::HandleEvent::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007f51f32b0a58 in statemachine::HandleEvent::HandleEvent(boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::my_context) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#24 0x00007f51f32b87de in boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::shallow_construct(boost::intrusive_ptr<statemachine::HandleLumis> const&, boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#25 0x00007f51f32b89aa in boost::statechart::simple_state<statemachine::FirstLumi, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#26 0x00007f51f33514a8 in boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#27 0x00007f51f33447dd in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-08-24-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#28 0x000000000040c844 in main::{lambda()#1}::operator()() const ()
#29 0x000000000040acc4 in main ()

Current Modules:

Module: TrackingMonitor (crashed)
Module: TrackingMonitor
Module: TrackingMonitor
Module: TrackingMonitor

A fatal system signal has occurred: segmentation violation

[Dr15Jones]

What about running the job under gdb and putting a break point on TrackingMonitor::analyze then stepping through the module until you can actually cause the '????' back trace? That way at least we will know what call chain the crash happens in (assuming the '????' also appear in normal running).

[Dr15Jones]

I took a look at TrackingMonitor and it appears that the strings being parsed are [https://github.com/cms-sw/cmssw/blob/b2d14d7371fb80ecd6fce9613d2e847b94b95ee7/DQM/TrackingMonitor/python/TrackingMonitor_cfi.py#L8]

numCut           = cms.string(" pt >= 1 & quality('highPurity') ")
denCut           = cms.string(" pt >= 1 ")

an interesting thing to note is 'highPurity' is a string translated into an enum where that is also done via calls to ROOT. So maybe the function generated by ROOT is fine but we are feeding it bad values?

[Dr15Jones]

The enum highPurity comes from reco::TrackBase::TrackQuality. This is not a global enum and therefore ROOT builds the set of enum values as an immutable list. Looking with Philippe we didn't find any obvious problem that could account for the crash.

[dan131riley]

Another one from one of the HI workflows that only does 2 events, so if there's interference we know it's from the other thread

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc530/CMSSW_8_1_X_2016-09-02-1100/pyRelValMatrixLogs/run/140.1_QCD_Pt_80_120_13_HI+QCD_Pt_80_120_13_HIINPUT+DIGIHI+RECOHI+HARVESTHI/step3_QCD_Pt_80_120_13_HI+QCD_Pt_80_120_13_HIINPUT+DIGIHI+RECOHI+HARVESTHI.log

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thread 5 (Thread 0x7f11561bf700 (LWP 9164)):
#0  0x0000003144c0f37d in waitpid () from /lib64/libpthread.so.0
#1  0x00007f11672d2d17 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/2016-36/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-01-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007f11672d36a2 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/2016-36/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-01-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007f116c295a50 in std::(anonymous namespace)::execute_native_thread_routine (__p=<optimized out>) at ../../../../../libstdc++-v3/src/c++11/thread.cc:84
#4  0x0000003144c07aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000031448e8aad in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f11212bf700 (LWP 9282)):
#0  0x00000031448df283 in poll () from /lib64/libc.so.6
#1  0x00007f11672d3504 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/2016-36/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-01-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007f11672d376a in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/2016-36/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-01-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007f11672d405e in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/2016-36/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-01-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007f11692ff08e in ?? ()
#6  0x00007f11212b9308 in ?? ()
#7  0x00007f105d5ffcd0 in ?? ()
#8  0x0000000000000000 in ?? ()
Thread 3 (Thread 0x7f11208be700 (LWP 9283)):
#0  0x00000031448acbdd in nanosleep () from /lib64/libc.so.6
#1  0x00000031448aca50 in sleep () from /lib64/libc.so.6
#2  0x00007f11672d2b4e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/2016-36/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-01-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00000031448e5309 in syscall () from /lib64/libc.so.6
#5  0x00007f116ca78962 in tbb::internal::futex_wait (comparand=2, futex=0x7f116af1baac) at ../../include/tbb/machine/linux_common.h:60
#6  tbb::internal::binary_semaphore::P (this=0x7f116af1baac) at ../../src/tbb/semaphore.h:206
#7  rml::internal::thread_monitor::commit_wait (c=<synthetic pointer>, this=0x7f116af1baa0) at ../../src/rml/include/../server/thread_monitor.h:259
#8  tbb::internal::rml::private_worker::run (this=0x7f116af1ba80) at ../../src/tbb/private_server.cpp:282
#9  0x00007f116ca78989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#10 0x0000003144c07aa1 in start_thread () from /lib64/libpthread.so.0
#11 0x00000031448e8aad in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f111fdff700 (LWP 9284)):
#0  0x00000031448acbdd in nanosleep () from /lib64/libc.so.6
#1  0x00000031448aca50 in sleep () from /lib64/libc.so.6
#2  0x00007f11672d2b4e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/2016-36/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-01-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00000031448e5309 in syscall () from /lib64/libc.so.6
#5  0x00007f116ca78962 in tbb::internal::futex_wait (comparand=2, futex=0x7f116af1b9ac) at ../../include/tbb/machine/linux_common.h:60
#6  tbb::internal::binary_semaphore::P (this=0x7f116af1b9ac) at ../../src/tbb/semaphore.h:206
#7  rml::internal::thread_monitor::commit_wait (c=<synthetic pointer>, this=0x7f116af1b9a0) at ../../src/rml/include/../server/thread_monitor.h:259
#8  tbb::internal::rml::private_worker::run (this=0x7f116af1b980) at ../../src/tbb/private_server.cpp:282
#9  0x00007f116ca78989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#10 0x0000003144c07aa1 in start_thread () from /lib64/libpthread.so.0
#11 0x00000031448e8aad in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f116daecbc0 (LWP 8899)):
#0  0x00000031448acbdd in nanosleep () from /lib64/libc.so.6
#1  0x00000031448aca50 in sleep () from /lib64/libc.so.6
#2  0x00007f11672d2b4e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/2016-36/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-01-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  operator new (sz=33) at ../../../../libstdc++-v3/libsupc++/new_op.cc:43
#5  0x00007f116c2aa639 in __gnu_cxx::new_allocator<char>::allocate (this=<optimized out>, __n=<optimized out>) at /mnt/build/davidlt/gcc530_final/b/BUILD/slc6_amd64_gcc530/external/gcc/5.3.0/gcc-tags_gcc_5_3_0_release-231255/obj/x86_64-pc-linux-gnu/libstdc++-v3/include/ext/new_allocator.h:104
#6  std::string::_Rep::_S_create (__capacity=8, __old_capacity=<optimized out>, __alloc=...) at /mnt/build/davidlt/gcc530_final/b/BUILD/slc6_amd64_gcc530/external/gcc/5.3.0/gcc-tags_gcc_5_3_0_release-231255/obj/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/basic_string.tcc:1051
#7  0x00007f116d935a95 in char* std::string::_S_construct<char const*>(char const*, char const*, std::allocator<char> const&, std::forward_iterator_tag) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-09-02-1100/external/slc6_amd64_gcc530/lib/libTree.so
#8  0x00007f116c2ac3a6 in std::string::_S_construct_aux<char const*> (__a=..., __end=<optimized out>, __beg=0x7f105deac939 "isLooper") at /mnt/build/davidlt/gcc530_final/b/BUILD/slc6_amd64_gcc530/external/gcc/5.3.0/gcc-tags_gcc_5_3_0_release-231255/obj/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/basic_string.h:4143
#9  std::string::_S_construct<char const*> (__a=..., __end=<optimized out>, __beg=0x7f105deac939 "isLooper") at /mnt/build/davidlt/gcc530_final/b/BUILD/slc6_amd64_gcc530/external/gcc/5.3.0/gcc-tags_gcc_5_3_0_release-231255/obj/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/basic_string.h:4164
#10 std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string (this=0x7ffd481f5020, __s=0x7f105deac939 "isLooper", __a=...) at /mnt/build/davidlt/gcc530_final/b/BUILD/slc6_amd64_gcc530/external/gcc/5.3.0/gcc-tags_gcc_5_3_0_release-231255/obj/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/basic_string.tcc:659
#11 0x00007f116d9fa0f8 in edm::FunctionWithDict::name() const () from /cvmfs/cms-ib.cern.ch/2016-36/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-01-1100/lib/slc6_amd64_gcc530/libFWCoreUtilities.so
#12 0x00007f116292d283 in reco::findMethod(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > >&, char const*, int&) () from /cvmfs/cms-ib.cern.ch/2016-36/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-01-1100/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#13 0x00007f116292d0f4 in reco::findMethod(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > >&, char const*, int&) () from /cvmfs/cms-ib.cern.ch/2016-36/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-01-1100/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#14 0x00007f116291ef60 in reco::parser::MethodSetter::push(std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, char const*, bool) const () from /cvmfs/cms-ib.cern.ch/2016-36/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-01-1100/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#15 0x00007f11629485c0 in reco::parser::SingleInvoker::SingleInvoker(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&) () from /cvmfs/cms-ib.cern.ch/2016-36/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-01-1100/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#16 0x00007f1162949706 in reco::parser::LazyInvoker::invoker(edm::TypeWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-36/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-01-1100/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#17 0x00007f116294bb4b in reco::parser::LazyInvoker::invokeLast(edm::ObjectWithDict const&, std::vector<edm::ObjectWithDict, std::allocator<edm::ObjectWithDict> >&) const () from /cvmfs/cms-ib.cern.ch/2016-36/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-01-1100/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#18 0x00007f116292f15f in reco::parser::ExpressionLazyVar::value(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-36/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-01-1100/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#19 0x00007f116290a4a6 in reco::parser::BinarySelector::operator()(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-36/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-01-1100/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#20 0x00007f113310cd19 in TrackingMonitor::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/2016-36/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-01-1100/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#21 0x00007f116de2139a in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-36/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-01-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007f116ddf7262 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-36/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-01-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007f116dd13a84 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-36/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-01-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#24 0x00007f116dd14195 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/2016-36/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-01-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#25 0x00007f116dd1450d in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-36/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-01-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#26 0x00007f116dd4013e in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-36/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-01-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#27 0x00007f116dd403f2 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-36/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-01-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#28 0x00007f116dd40815 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-36/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-01-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#29 0x00007f116dd40b0b in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-36/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-01-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#30 0x00007f116dd40d7f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-36/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-01-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#31 0x00007f116ddb8202 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-36/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-01-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#32 0x00007f116ddb8937 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-36/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-01-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#33 0x00007f116ddbf314 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-36/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-01-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#34 0x00007f116ca839b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f116af22400, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#35 0x00007f116ddb878e in edm::EventProcessor::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/2016-36/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-01-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#36 0x00007f116dd238ba in statemachine::HandleEvent::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/2016-36/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-01-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#37 0x00007f116dd24e48 in statemachine::HandleEvent::HandleEvent(boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::my_context) () from /cvmfs/cms-ib.cern.ch/2016-36/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-01-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#38 0x00007f116dd2cbce in boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::shallow_construct(boost::intrusive_ptr<statemachine::HandleLumis> const&, boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>&) () from /cvmfs/cms-ib.cern.ch/2016-36/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-01-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#39 0x00007f116dd2cd9a in boost::statechart::simple_state<statemachine::FirstLumi, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*) () from /cvmfs/cms-ib.cern.ch/2016-36/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-01-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#40 0x00007f116ddbec16 in boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&) () from /cvmfs/cms-ib.cern.ch/2016-36/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-01-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#41 0x00007f116ddb1f4d in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/2016-36/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-01-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#42 0x000000000040c844 in main::{lambda()#1}::operator()() const ()
#43 0x000000000040acc4 in main ()

Current Modules:

Module: TrackingMonitor (crashed)
Module: none
Module: TrackingMonitor
Module: none

A fatal system signal has occurred: segmentation violation

[dan131riley]

New variation

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc530/CMSSW_8_1_X_2016-09-04-1100/pyRelValMatrixLogs/run/140.4_ZEEMM_13_HI+ZEEMM_13_HIINPUT+DIGIHI+RECOHI+HARVESTHI/step3_ZEEMM_13_HI+ZEEMM_13_HIINPUT+DIGIHI+RECOHI+HARVESTHI.log

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thread 5 (Thread 0x7fa3dc63f700 (LWP 8911)):
#0  0x0000003fcce0f37d in waitpid () from /lib64/libpthread.so.0
#1  0x00007fa3ec613d17 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-04-0000/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007fa3ec6146a2 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-04-0000/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007fa3f215aa50 in std::(anonymous namespace)::execute_native_thread_routine (__p=<optimized out>) at ../../../../../libstdc++-v3/src/c++11/thread.cc:84
#4  0x0000003fcce07aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003fccae8aad in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fa3a83bf700 (LWP 9016)):
#0  0x0000003fccaacbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003fccaaca50 in sleep () from /lib64/libc.so.6
#2  0x00007fa3ec613b4e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-04-0000/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000003fccae5309 in syscall () from /lib64/libc.so.6
#5  0x00007fa3f2716962 in tbb::internal::futex_wait (comparand=2, futex=0x7fa3f0be612c) at ../../include/tbb/machine/linux_common.h:60
#6  tbb::internal::binary_semaphore::P (this=0x7fa3f0be612c) at ../../src/tbb/semaphore.h:206
#7  rml::internal::thread_monitor::commit_wait (c=<synthetic pointer>, this=0x7fa3f0be6120) at ../../src/rml/include/../server/thread_monitor.h:259
#8  tbb::internal::rml::private_worker::run (this=0x7fa3f0be6100) at ../../src/tbb/private_server.cpp:282
#9  0x00007fa3f2716989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#10 0x0000003fcce07aa1 in start_thread () from /lib64/libpthread.so.0
#11 0x0000003fccae8aad in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fa3a79be700 (LWP 9017)):
#0  0x0000003fccadf283 in poll () from /lib64/libc.so.6
#1  0x00007fa3ec614504 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-04-0000/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007fa3ec61476a in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-04-0000/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007fa3ec61505e in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-04-0000/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007fa3af74308e in ?? ()
#6  0x00007fa3a79b8308 in ?? ()
#7  0x00007fa2c4fd6310 in ?? ()
#8  0x0000000000000000 in ?? ()
Thread 2 (Thread 0x7fa3a6eff700 (LWP 9018)):
#0  0x0000003fccaacbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003fccaaca50 in sleep () from /lib64/libc.so.6
#2  0x00007fa3ec613b4e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-04-0000/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000003fccae5309 in syscall () from /lib64/libc.so.6
#5  0x00007fa3f2716962 in tbb::internal::futex_wait (comparand=2, futex=0x7fa3f0be60ac) at ../../include/tbb/machine/linux_common.h:60
#6  tbb::internal::binary_semaphore::P (this=0x7fa3f0be60ac) at ../../src/tbb/semaphore.h:206
#7  rml::internal::thread_monitor::commit_wait (c=<synthetic pointer>, this=0x7fa3f0be60a0) at ../../src/rml/include/../server/thread_monitor.h:259
#8  tbb::internal::rml::private_worker::run (this=0x7fa3f0be6080) at ../../src/tbb/private_server.cpp:282
#9  0x00007fa3f2716989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#10 0x0000003fcce07aa1 in start_thread () from /lib64/libpthread.so.0
#11 0x0000003fccae8aad in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fa3f1df3bc0 (LWP 8772)):
#0  0x0000003fccaacbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003fccaaca50 in sleep () from /lib64/libc.so.6
#2  0x00007fa3ec613b4e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-04-0000/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  do_lookup_x (new_hash=2276275598, old_hash=0x7ffea7073788, ref=<optimized out>, result=0x7ffea7073770, scope=<optimized out>, i=80, flags=5, skip=0x0, undef_map=0x7fa3f083a800) at dl-lookup.c:235
#5  0x00007fa3f3aecf0a in _dl_lookup_symbol_x (undef_name=0x7fa3ee1d7cd9 "_ZNK4reco10HitPattern19getTrackerLayerCaseENS0_11HitCategoryEtt", undef_map=0x7fa3f083a800, ref=0x7ffea70737e8, symbol_scope=0x7fa3d4f7c000, version=0x0, type_class=1, flags=5, skip_map=0x0) at dl-lookup.c:747
#6  0x00007fa3f3af10c0 in _dl_fixup (l=<optimized out>, reloc_arg=<optimized out>) at ../elf/dl-runtime.c:118
#7  0x00007fa3f3af7825 in _dl_runtime_resolve () at ../sysdeps/x86_64/dl-trampoline.S:41
#8  0x00007fa3ee20893f in reco::HitPattern::pixelBarrelLayersTotallyOffOrBad() const () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-04-0000/lib/slc6_amd64_gcc530/libDataFormatsTrackReco.so
#9  0x00007fa3b9c776dc in TrackAnalyzer::analyze(edm::Event const&, edm::EventSetup const&, reco::Track const&) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-04-0000/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#10 0x00007fa3b9c15eab in TrackingMonitor::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-04-0000/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#11 0x00007fa3f3a6239a in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-04-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#12 0x00007fa3f3a38262 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-04-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#13 0x00007fa3f3954a84 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-04-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#14 0x00007fa3f3955195 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-04-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#15 0x00007fa3f395550d in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-04-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007fa3f398113e in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-04-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007fa3f39813f2 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-04-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007fa3f3981815 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-04-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007fa3f3981b0b in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-04-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007fa3f3981d7f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-04-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007fa3f39f9202 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-04-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007fa3f39f9937 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-04-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007fa3f3a00314 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-04-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#24 0x00007fa3f27219b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fa3f0db6600, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#25 0x00007fa3f39f978e in edm::EventProcessor::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-04-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#26 0x00007fa3f39648ba in statemachine::HandleEvent::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-04-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#27 0x00007fa3f3965e48 in statemachine::HandleEvent::HandleEvent(boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::my_context) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-04-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#28 0x00007fa3f396dbce in boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::shallow_construct(boost::intrusive_ptr<statemachine::HandleLumis> const&, boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>&) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-04-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#29 0x00007fa3f396dd9a in boost::statechart::simple_state<statemachine::FirstLumi, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-04-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#30 0x00007fa3f39ffc16 in boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-04-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#31 0x00007fa3f39f2f4d in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-04-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#32 0x000000000040c844 in main::{lambda()#1}::operator()() const ()
#33 0x000000000040acc4 in main ()

Current Modules:

Module: TrackingMonitor (crashed)
Module: none
Module: none
Module: TrackingMonitor

A fatal system signal has occurred: segmentation violation

[dan131riley]

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc530/CMSSW_8_1_X_2016-09-07-1100/pyRelValMatrixLogs/run/136.733_RunCosmics2016B+RunCosmics2016B+RECOCOSDRUN2+ALCACOSDRUN2+HARVESTDCRUN2/step2_RunCosmics2016B+RunCosmics2016B+RECOCOSDRUN2+ALCACOSDRUN2+HARVESTDCRUN2.log

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thread 10 (Thread 0x7f3356e3f700 (LWP 27400)):
#0  0x0000003fcce0f37d in waitpid () from /lib64/libpthread.so.0
#1  0x00007f3368193d47 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007f33681946d2 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007f336bc1da50 in std::(anonymous namespace)::execute_native_thread_routine (__p=<optimized out>) at ../../../../../libstdc++-v3/src/c++11/thread.cc:84
#4  0x0000003fcce07aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003fccae8aad in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f33507bb700 (LWP 27430)):
#0  0x0000003fccae90a3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f33508883ba in XrdSys::IOEvents::PollE::Begin (this=0x7f33535d5780, syncsem=<optimized out>, retcode=<optimized out>, eTxt=<optimized out>) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/./XrdSys/XrdSysIOEventsPollE.icc:213
#2  0x00007f3350884a8d in XrdSys::IOEvents::BootStrap::Start (parg=0x7fff6da22860) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/XrdSys/XrdSysIOEvents.cc:131
#3  0x00007f3350883647 in XrdSysThread_Xeq (myargs=0x7f335554f0a0) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/XrdSys/XrdSysPthread.cc:86
#4  0x0000003fcce07aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003fccae8aad in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f334fdba700 (LWP 27431)):
#0  0x0000003fcce0f00d in nanosleep () from /lib64/libpthread.so.0
#1  0x00007f3350883deb in XrdSysTimer::Wait (mills=<optimized out>) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/XrdSys/XrdSysTimer.cc:239
#2  0x00007f335098ecaa in XrdCl::TaskManager::RunTasks (this=0x7f335554b600) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/XrdCl/XrdClTaskManager.cc:244
#3  0x00007f335098ee29 in RunRunnerThread (arg=<optimized out>) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/XrdCl/XrdClTaskManager.cc:37
#4  0x0000003fcce07aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003fccae8aad in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f334f3b9700 (LWP 27432)):
#0  0x0000003fccae5309 in syscall () from /lib64/libc.so.6
#1  0x00007f33509ea4e1 in XrdSys::LinuxSemaphore::Wait (this=0x7f33554db410) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/./XrdSys/XrdSysLinuxSemaphore.hh:161
#2  XrdCl::SyncQueue<XrdCl::JobManager::JobHelper>::Get (this=0x7f33535d56b8) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/./XrdCl/XrdClSyncQueue.hh:67
#3  XrdCl::JobManager::RunJobs (this=0x7f33535d56a0) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/XrdCl/XrdClJobManager.cc:146
#4  0x00007f33509ea679 in RunRunnerThread (arg=<optimized out>) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/XrdCl/XrdClJobManager.cc:33
#5  0x0000003fcce07aa1 in start_thread () from /lib64/libpthread.so.0
#6  0x0000003fccae8aad in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f334e9b8700 (LWP 27433)):
#0  0x0000003fccae5309 in syscall () from /lib64/libc.so.6
#1  0x00007f33509ea4e1 in XrdSys::LinuxSemaphore::Wait (this=0x7f33554db410) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/./XrdSys/XrdSysLinuxSemaphore.hh:161
#2  XrdCl::SyncQueue<XrdCl::JobManager::JobHelper>::Get (this=0x7f33535d56b8) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/./XrdCl/XrdClSyncQueue.hh:67
#3  XrdCl::JobManager::RunJobs (this=0x7f33535d56a0) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/XrdCl/XrdClJobManager.cc:146
#4  0x00007f33509ea679 in RunRunnerThread (arg=<optimized out>) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/XrdCl/XrdClJobManager.cc:33
#5  0x0000003fcce07aa1 in start_thread () from /lib64/libpthread.so.0
#6  0x0000003fccae8aad in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f334dfb7700 (LWP 27434)):
#0  0x0000003fccae5309 in syscall () from /lib64/libc.so.6
#1  0x00007f33509ea4e1 in XrdSys::LinuxSemaphore::Wait (this=0x7f33554db410) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/./XrdSys/XrdSysLinuxSemaphore.hh:161
#2  XrdCl::SyncQueue<XrdCl::JobManager::JobHelper>::Get (this=0x7f33535d56b8) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/./XrdCl/XrdClSyncQueue.hh:67
#3  XrdCl::JobManager::RunJobs (this=0x7f33535d56a0) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/XrdCl/XrdClJobManager.cc:146
#4  0x00007f33509ea679 in RunRunnerThread (arg=<optimized out>) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/XrdCl/XrdClJobManager.cc:33
#5  0x0000003fcce07aa1 in start_thread () from /lib64/libpthread.so.0
#6  0x0000003fccae8aad in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f3329f3f700 (LWP 27644)):
#0  0x0000003fccaacbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003fccaaca50 in sleep () from /lib64/libc.so.6
#2  0x00007f3368193b7e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  _dl_fixup (l=0x7f336d5d0000, reloc_arg=<optimized out>) at ../elf/dl-runtime.c:90
#5  0x00007f336d5b8825 in _dl_runtime_resolve () at ../sysdeps/x86_64/dl-trampoline.S:41
#6  0x00007f3346fbd247 in TrackAnalyzer::analyze(edm::Event const&, edm::EventSetup const&, reco::Track const&) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#7  0x00007f3346f5beab in TrackingMonitor::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#8  0x00007f336d52270a in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#9  0x00007f336d46eec2 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#10 0x00007f336d41bdf4 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#11 0x00007f336d41c505 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#12 0x00007f336d41c87d in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#13 0x00007f336d4470de in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#14 0x00007f336d447392 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#15 0x00007f336d4477b5 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007f336d447aab in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007f336d447d1f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007f336d4ccf12 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007f336d4cd647 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007f336d4d4024 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007f336c1e49b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f336a71fe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#22 0x00007f336c1de1a4 in tbb::internal::arena::process (this=this@entry=0x7f336a87f780, s=...) at ../../src/tbb/arena.cpp:156
#23 0x00007f336c1dd264 in tbb::internal::market::process (this=0x7f336a8a7c00, j=...) at ../../src/tbb/market.cpp:502
#24 0x00007f336c1d9736 in tbb::internal::rml::private_worker::run (this=0x7f336a6c6100) at ../../src/tbb/private_server.cpp:275
#25 0x00007f336c1d9989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#26 0x0000003fcce07aa1 in start_thread () from /lib64/libpthread.so.0
#27 0x0000003fccae8aad in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f332953e700 (LWP 27645)):
#0  0x0000003fccaacbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003fccaaca50 in sleep () from /lib64/libc.so.6
#2  0x00007f3368193b7e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f336cdc7644 in ROOT::Math::chisquared_cdf_c(double, double, double) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-09-07-1100/external/slc6_amd64_gcc530/lib/libMathCore.so
#5  0x00007f3346fbd247 in TrackAnalyzer::analyze(edm::Event const&, edm::EventSetup const&, reco::Track const&) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#6  0x00007f3346f5beab in TrackingMonitor::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#7  0x00007f336d52270a in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#8  0x00007f336d46eec2 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#9  0x00007f336d41bdf4 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#10 0x00007f336d41c505 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#11 0x00007f336d41c87d in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#12 0x00007f336d4470de in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#13 0x00007f336d447392 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#14 0x00007f336d4477b5 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#15 0x00007f336d447aab in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007f336d447d1f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007f336d4ccf12 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007f336d4cd7aa in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007f336d4d4024 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007f336c1e49b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f336a70fe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#21 0x00007f336c1de1a4 in tbb::internal::arena::process (this=this@entry=0x7f336a87f780, s=...) at ../../src/tbb/arena.cpp:156
#22 0x00007f336c1dd264 in tbb::internal::market::process (this=0x7f336a8a7c00, j=...) at ../../src/tbb/market.cpp:502
#23 0x00007f336c1d9736 in tbb::internal::rml::private_worker::run (this=0x7f336a6c6180) at ../../src/tbb/private_server.cpp:275
#24 0x00007f336c1d9989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#25 0x0000003fcce07aa1 in start_thread () from /lib64/libpthread.so.0
#26 0x0000003fccae8aad in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f3328a3f700 (LWP 27646)):
#0  0x0000003fccadf283 in poll () from /lib64/libc.so.6
#1  0x00007f3368194534 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007f336819479a in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007f336819508e in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007f333ba4208e in ?? ()
#6  0x00007f3328a39308 in ?? ()
#7  0x00007f32cc108e80 in ?? ()
#8  0x0000000000000000 in ?? ()
Thread 1 (Thread 0x7f336b8b6bc0 (LWP 27154)):
#0  0x0000003fccaacbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003fccaaca50 in sleep () from /lib64/libc.so.6
#2  0x00007f3368193b7e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f336cdc7644 in ROOT::Math::chisquared_cdf_c(double, double, double) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-09-07-1100/external/slc6_amd64_gcc530/lib/libMathCore.so
#5  0x00007f3346fbd247 in TrackAnalyzer::analyze(edm::Event const&, edm::EventSetup const&, reco::Track const&) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#6  0x00007f3346f5beab in TrackingMonitor::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#7  0x00007f336d52270a in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#8  0x00007f336d46eec2 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#9  0x00007f336d41bdf4 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#10 0x00007f336d41c505 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#11 0x00007f336d41c87d in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#12 0x00007f336d4470de in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#13 0x00007f336d447392 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#14 0x00007f336d4477b5 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#15 0x00007f336d447aab in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007f336d447d1f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007f336d4ccf12 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007f336d4cd647 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007f336d4d4024 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007f336c1e49b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f336a896600, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#21 0x00007f336d4cd49e in edm::EventProcessor::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007f336d42b53a in statemachine::HandleEvent::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007f336d42cac8 in statemachine::HandleEvent::HandleEvent(boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::my_context) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#24 0x00007f336d43484e in boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::shallow_construct(boost::intrusive_ptr<statemachine::HandleLumis> const&, boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>&) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#25 0x00007f336d434a1a in boost::statechart::simple_state<statemachine::FirstLumi, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#26 0x00007f336d4d3926 in boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#27 0x00007f336d4c6c5d in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-06-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#28 0x000000000040c844 in main::{lambda()#1}::operator()() const ()
#29 0x000000000040acc4 in main ()

Current Modules:

Module: TrackingMonitor (crashed)
Module: TrackingMonitor
Module: TrackingMonitor
Module: TrackingMonitor

A fatal system signal has occurred: segmentation violation

[dan131riley]

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc530/CMSSW_8_1_X_2016-09-08-1100/pyRelValMatrixLogs/run/140.2_PhotonJets_Pt_10_13_HI+PhotonJets_Pt_10_13_HIINPUT+DIGIHI+RECOHI+HARVESTHI/step3_PhotonJets_Pt_10_13_HI+PhotonJets_Pt_10_13_HIINPUT+DIGIHI+RECOHI+HARVESTHI.log

Another one with only two threads, will have to try using these to narrow down what the other thread was just doing...

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thread 5 (Thread 0x7f6eb9cbf700 (LWP 19138)):
#0  0x0000003cee40f37d in waitpid () from /lib64/libpthread.so.0
#1  0x00007f6ec9a53ed7 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-08-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007f6ec9a54862 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-08-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007f6ecf5d4a50 in std::(anonymous namespace)::execute_native_thread_routine (__p=<optimized out>) at ../../../../../libstdc++-v3/src/c++11/thread.cc:84
#4  0x0000003cee407aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003cee0e8aad in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f6e85c7f700 (LWP 19530)):
#0  0x0000003cee0acbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003cee0aca50 in sleep () from /lib64/libc.so.6
#2  0x00007f6ec9a53d0e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-08-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000003cee0e5309 in syscall () from /lib64/libc.so.6
#5  0x00007f6ecfb90962 in tbb::internal::futex_wait (comparand=2, futex=0x7f6ece06612c) at ../../include/tbb/machine/linux_common.h:60
#6  tbb::internal::binary_semaphore::P (this=0x7f6ece06612c) at ../../src/tbb/semaphore.h:206
#7  rml::internal::thread_monitor::commit_wait (c=<synthetic pointer>, this=0x7f6ece066120) at ../../src/rml/include/../server/thread_monitor.h:259
#8  tbb::internal::rml::private_worker::run (this=0x7f6ece066100) at ../../src/tbb/private_server.cpp:282
#9  0x00007f6ecfb90989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#10 0x0000003cee407aa1 in start_thread () from /lib64/libpthread.so.0
#11 0x0000003cee0e8aad in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f6e8527e700 (LWP 19531)):
#0  0x0000003cee0acbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003cee0aca50 in sleep () from /lib64/libc.so.6
#2  0x00007f6ec9a53d0e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-08-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000003cee089cc0 in _wordcopy_fwd_aligned () from /lib64/libc.so.6
#5  0x0000003cee083b22 in memmove () from /lib64/libc.so.6
#6  0x00007f6ec779a6f2 in TProfile::Copy(TObject&) const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-08-1100/external/slc6_amd64_gcc530/lib/libHist.so
#7  0x00007f6ec77a293a in TProfile* TProfileHelper::ExtendAxis<TProfile>(TProfile*, double, TAxis*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-08-1100/external/slc6_amd64_gcc530/lib/libHist.so
#8  0x00007f6ec779f319 in TProfile::ExtendAxis(double, TAxis*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-08-1100/external/slc6_amd64_gcc530/lib/libHist.so
#9  0x00007f6ec77f9290 in TAxis::FindBin(double) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-08-1100/external/slc6_amd64_gcc530/lib/libHist.so
#10 0x00007f6ec779e059 in TProfile::Fill(double, double, double) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-08-1100/external/slc6_amd64_gcc530/lib/libHist.so
#11 0x00007f6e973d603d in TrackingMonitor::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-08-1100/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#12 0x00007f6ed0ed91fa in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-08-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#13 0x00007f6ed0e3a7a2 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-08-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#14 0x00007f6ed0df5f84 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-08-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#15 0x00007f6ed0df6695 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-08-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007f6ed0df6a0d in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-08-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007f6ed0e1507e in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-08-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007f6ed0e15332 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-08-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007f6ed0e15755 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-08-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007f6ed0e15a4b in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-08-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007f6ed0e15cbf in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-08-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007f6ed0e96b02 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-08-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007f6ed0e9739a in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-08-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#24 0x00007f6ed0e9db04 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-08-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#25 0x00007f6ecfb9b9b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f6e873c3e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#26 0x00007f6ecfb951a4 in tbb::internal::arena::process (this=this@entry=0x7f6ece21f780, s=...) at ../../src/tbb/arena.cpp:156
#27 0x00007f6ecfb94264 in tbb::internal::market::process (this=0x7f6ece247c00, j=...) at ../../src/tbb/market.cpp:502
#28 0x00007f6ecfb90736 in tbb::internal::rml::private_worker::run (this=0x7f6ece066180) at ../../src/tbb/private_server.cpp:275
#29 0x00007f6ecfb90989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#30 0x0000003cee407aa1 in start_thread () from /lib64/libpthread.so.0
#31 0x0000003cee0e8aad in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f6e847ff700 (LWP 19532)):
#0  0x0000003cee0acbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003cee0aca50 in sleep () from /lib64/libc.so.6
#2  0x00007f6ec9a53d0e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-08-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000003cee0e5309 in syscall () from /lib64/libc.so.6
#5  0x00007f6ecfb90962 in tbb::internal::futex_wait (comparand=2, futex=0x7f6ece0660ac) at ../../include/tbb/machine/linux_common.h:60
#6  tbb::internal::binary_semaphore::P (this=0x7f6ece0660ac) at ../../src/tbb/semaphore.h:206
#7  rml::internal::thread_monitor::commit_wait (c=<synthetic pointer>, this=0x7f6ece0660a0) at ../../src/rml/include/../server/thread_monitor.h:259
#8  tbb::internal::rml::private_worker::run (this=0x7f6ece066080) at ../../src/tbb/private_server.cpp:282
#9  0x00007f6ecfb90989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#10 0x0000003cee407aa1 in start_thread () from /lib64/libpthread.so.0
#11 0x0000003cee0e8aad in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f6ecf26ebc0 (LWP 18918)):
#0  0x0000003cee0df283 in poll () from /lib64/libc.so.6
#1  0x00007f6ec9a546c4 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-08-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007f6ec9a5492a in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-08-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007f6ec9a5521e in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-08-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007f6e8d9c608e in ?? ()
#6  0x00007ffeb0124a08 in ?? ()
#7  0x00007f6db05d5578 in ?? ()
#8  0x0000000000000000 in ?? ()

Current Modules:

Module: TrackingMonitor (crashed)
Module: none
Module: TrackingMonitor
Module: none

A fatal system signal has occurred: segmentation violation

[dan131riley]

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc530/CMSSW_8_1_DEVEL_X_2016-09-05-1700/pyRelValMatrixLogs/run/134.813_RunCosmics2015C+RunCosmics2015C+RECOCOSDRUN2+ALCACOSDRUN2+HARVESTDCRUN2/step2_RunCosmics2015C+RunCosmics2015C+RECOCOSDRUN2+ALCACOSDRUN2+HARVESTDCRUN2.log

Threads 4 and 2 are both in TListOfFunctionsIter::Next, thread 4 has the mutex, thread 2 is waiting on it. I've been looking at the locking scheme, suspecting that we have another case of a race being amplified by a mutex, so any stack trace with a mutex held is of special interest.

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thread 10 (Thread 0x7fe5a3b3f700 (LWP 27095)):
#0  0x0000003cee40f37d in waitpid () from /lib64/libpthread.so.0
#1  0x00007fe5b4d53d17 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007fe5b4d546a2 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007fe5b8647a50 in std::(anonymous namespace)::execute_native_thread_routine (__p=<optimized out>) at ../../../../../libstdc++-v3/src/c++11/thread.cc:84
#4  0x0000003cee407aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003cee0e8aad in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fe59d4bb700 (LWP 27111)):
#0  0x0000003cee0e90a3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fe59d5883ba in XrdSys::IOEvents::PollE::Begin (this=0x7fe5a02ea240, syncsem=<optimized out>, retcode=<optimized out>, eTxt=<optimized out>) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/./XrdSys/XrdSysIOEventsPollE.icc:213
#2  0x00007fe59d584a8d in XrdSys::IOEvents::BootStrap::Start (parg=0x7ffef8bfe220) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/XrdSys/XrdSysIOEvents.cc:131
#3  0x00007fe59d583647 in XrdSysThread_Xeq (myargs=0x7fe5a229ed40) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/XrdSys/XrdSysPthread.cc:86
#4  0x0000003cee407aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003cee0e8aad in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fe59caba700 (LWP 27112)):
#0  0x0000003cee40f00d in nanosleep () from /lib64/libpthread.so.0
#1  0x00007fe59d583deb in XrdSysTimer::Wait (mills=<optimized out>) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/XrdSys/XrdSysTimer.cc:239
#2  0x00007fe59d68ecaa in XrdCl::TaskManager::RunTasks (this=0x7fe5a229ac00) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/XrdCl/XrdClTaskManager.cc:244
#3  0x00007fe59d68ee29 in RunRunnerThread (arg=<optimized out>) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/XrdCl/XrdClTaskManager.cc:37
#4  0x0000003cee407aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003cee0e8aad in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fe59c0b9700 (LWP 27113)):
#0  0x0000003cee0e5309 in syscall () from /lib64/libc.so.6
#1  0x00007fe59d6ea4e1 in XrdSys::LinuxSemaphore::Wait (this=0x7fe59ed30410) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/./XrdSys/XrdSysLinuxSemaphore.hh:161
#2  XrdCl::SyncQueue<XrdCl::JobManager::JobHelper>::Get (this=0x7fe5a02ea178) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/./XrdCl/XrdClSyncQueue.hh:67
#3  XrdCl::JobManager::RunJobs (this=0x7fe5a02ea160) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/XrdCl/XrdClJobManager.cc:146
#4  0x00007fe59d6ea679 in RunRunnerThread (arg=<optimized out>) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/XrdCl/XrdClJobManager.cc:33
#5  0x0000003cee407aa1 in start_thread () from /lib64/libpthread.so.0
#6  0x0000003cee0e8aad in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fe59b6b8700 (LWP 27114)):
#0  0x0000003cee0e5309 in syscall () from /lib64/libc.so.6
#1  0x00007fe59d6ea4e1 in XrdSys::LinuxSemaphore::Wait (this=0x7fe59ed30410) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/./XrdSys/XrdSysLinuxSemaphore.hh:161
#2  XrdCl::SyncQueue<XrdCl::JobManager::JobHelper>::Get (this=0x7fe5a02ea178) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/./XrdCl/XrdClSyncQueue.hh:67
#3  XrdCl::JobManager::RunJobs (this=0x7fe5a02ea160) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/XrdCl/XrdClJobManager.cc:146
#4  0x00007fe59d6ea679 in RunRunnerThread (arg=<optimized out>) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/XrdCl/XrdClJobManager.cc:33
#5  0x0000003cee407aa1 in start_thread () from /lib64/libpthread.so.0
#6  0x0000003cee0e8aad in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fe59acb7700 (LWP 27115)):
#0  0x0000003cee0e5309 in syscall () from /lib64/libc.so.6
#1  0x00007fe59d6ea4e1 in XrdSys::LinuxSemaphore::Wait (this=0x7fe59ed30410) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/./XrdSys/XrdSysLinuxSemaphore.hh:161
#2  XrdCl::SyncQueue<XrdCl::JobManager::JobHelper>::Get (this=0x7fe5a02ea178) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/./XrdCl/XrdClSyncQueue.hh:67
#3  XrdCl::JobManager::RunJobs (this=0x7fe5a02ea160) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/XrdCl/XrdClJobManager.cc:146
#4  0x00007fe59d6ea679 in RunRunnerThread (arg=<optimized out>) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/XrdCl/XrdClJobManager.cc:33
#5  0x0000003cee407aa1 in start_thread () from /lib64/libpthread.so.0
#6  0x0000003cee0e8aad in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fe576c3f700 (LWP 27330)):
#0  0x0000003cee0acbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003cee0aca50 in sleep () from /lib64/libc.so.6
#2  0x00007fe5b4d53b4e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007fe5ae1022e1 in cling::Transaction::Initialize(clang::Sema&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/external/slc6_amd64_gcc530/lib/libCling.so
#5  0x00007fe5ae139027 in cling::IncrementalParser::beginTransaction(cling::CompilationOptions const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/external/slc6_amd64_gcc530/lib/libCling.so
#6  0x00007fe5ae0ee7b4 in cling::Interpreter::PushTransactionRAII::PushTransactionRAII(cling::Interpreter const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/external/slc6_amd64_gcc530/lib/libCling.so
#7  0x00007fe5ae0683cb in TClingMethodInfo::TClingMethodInfo(cling::Interpreter*, TClingClassInfo*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/external/slc6_amd64_gcc530/lib/libCling.so
#8  0x00007fe5ae070bea in TCling::MethodInfo_Factory(ClassInfo_t*) const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/external/slc6_amd64_gcc530/lib/libCling.so
#9  0x00007fe5b933caeb in TListOfFunctions::Load() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/external/slc6_amd64_gcc530/lib/libCore.so
#10 0x00007fe5b9348f12 in TClass::GetListOfMethods(bool) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/external/slc6_amd64_gcc530/lib/libCore.so
#11 0x00007fe5b9b265c6 in edm::TypeFunctionMembers::begin() const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libFWCoreUtilities.so
#12 0x00007fe5b100c2d1 in reco::findMethod(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > >&, char const*, int&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#13 0x00007fe5b0fd1270 in reco::parser::MethodSetter::push(std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, char const*, bool) const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#14 0x00007fe5b1025cc0 in reco::parser::SingleInvoker::SingleInvoker(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#15 0x00007fe5b1026e06 in reco::parser::LazyInvoker::invoker(edm::TypeWithDict const&) const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#16 0x00007fe5b102924b in reco::parser::LazyInvoker::invokeLast(edm::ObjectWithDict const&, std::vector<edm::ObjectWithDict, std::allocator<edm::ObjectWithDict> >&) const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#17 0x00007fe5b102147f in reco::parser::ExpressionLazyVar::value(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#18 0x00007fe5b0ffa416 in reco::parser::BinarySelector::operator()(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#19 0x00007fe593c7bd19 in TrackingMonitor::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#20 0x00007fe5b9f4bfda in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007fe5b9f27722 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007fe5b9e58e34 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007fe5b9e59545 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#24 0x00007fe5b9e598bd in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#25 0x00007fe5b9e7a17e in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#26 0x00007fe5b9e7a432 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#27 0x00007fe5b9e7a855 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#28 0x00007fe5b9e7ab4b in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#29 0x00007fe5b9e7adbf in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#30 0x00007fe5b9f01b22 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#31 0x00007fe5b9f02257 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#32 0x00007fe5b9f08b74 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#33 0x00007fe5b8c0e9b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fe5b7137e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#34 0x00007fe5b8c081a4 in tbb::internal::arena::process (this=this@entry=0x7fe5b729f780, s=...) at ../../src/tbb/arena.cpp:156
#35 0x00007fe5b8c07264 in tbb::internal::market::process (this=0x7fe5b72c7c00, j=...) at ../../src/tbb/market.cpp:502
#36 0x00007fe5b8c03736 in tbb::internal::rml::private_worker::run (this=0x7fe5b70e6100) at ../../src/tbb/private_server.cpp:275
#37 0x00007fe5b8c03989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#38 0x0000003cee407aa1 in start_thread () from /lib64/libpthread.so.0
#39 0x0000003cee0e8aad in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fe57623e700 (LWP 27331)):
#0  0x0000003cee0acbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003cee0aca50 in sleep () from /lib64/libc.so.6
#2  0x00007fe5b4d53b4e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007fe5b97d7a44 in ROOT::Math::Cephes::lgam(double) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/external/slc6_amd64_gcc530/lib/libMathCore.so
#5  0x00007fe5b97d7e55 in ROOT::Math::Cephes::igamc(double, double) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/external/slc6_amd64_gcc530/lib/libMathCore.so
#6  0x00007fe593cdd247 in TrackAnalyzer::analyze(edm::Event const&, edm::EventSetup const&, reco::Track const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#7  0x00007fe593c7beab in TrackingMonitor::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#8  0x00007fe5b9f4bfda in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#9  0x00007fe5b9f27722 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#10 0x00007fe5b9e58e34 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#11 0x00007fe5b9e59545 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#12 0x00007fe5b9e598bd in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#13 0x00007fe5b9e7a17e in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#14 0x00007fe5b9e7a432 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#15 0x00007fe5b9e7a855 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007fe5b9e7ab4b in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007fe5b9e7adbf in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007fe5b9f01b22 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007fe5b9f023ba in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007fe5b9f08b74 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007fe5b8c0e9b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fe5b713fe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#22 0x00007fe5b8c081a4 in tbb::internal::arena::process (this=this@entry=0x7fe5b729f780, s=...) at ../../src/tbb/arena.cpp:156
#23 0x00007fe5b8c07264 in tbb::internal::market::process (this=0x7fe5b72c7c00, j=...) at ../../src/tbb/market.cpp:502
#24 0x00007fe5b8c03736 in tbb::internal::rml::private_worker::run (this=0x7fe5b70e6180) at ../../src/tbb/private_server.cpp:275
#25 0x00007fe5b8c03989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#26 0x0000003cee407aa1 in start_thread () from /lib64/libpthread.so.0
#27 0x0000003cee0e8aad in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fe5757bf700 (LWP 27332)):
#0  0x0000003cee0acbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003cee0aca50 in sleep () from /lib64/libc.so.6
#2  0x00007fe5b4d53b4e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000003cee40e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#5  0x0000003cee4095f3 in _L_lock_892 () from /lib64/libpthread.so.0
#6  0x0000003cee4094d7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#7  0x00007fe5b933b824 in TListOfFunctionsIter::Next() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/external/slc6_amd64_gcc530/lib/libCore.so
#8  0x00007fe5b9b647e9 in edm::IterWithDictBase::advance() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libFWCoreUtilities.so
#9  0x00007fe5b100c34d in reco::findMethod(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > >&, char const*, int&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#10 0x00007fe5b100c1e4 in reco::findMethod(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > >&, char const*, int&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#11 0x00007fe5b0fd1270 in reco::parser::MethodSetter::push(std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, char const*, bool) const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#12 0x00007fe5b1025cc0 in reco::parser::SingleInvoker::SingleInvoker(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#13 0x00007fe5b1026e06 in reco::parser::LazyInvoker::invoker(edm::TypeWithDict const&) const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#14 0x00007fe5b102924b in reco::parser::LazyInvoker::invokeLast(edm::ObjectWithDict const&, std::vector<edm::ObjectWithDict, std::allocator<edm::ObjectWithDict> >&) const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#15 0x00007fe5b102147f in reco::parser::ExpressionLazyVar::value(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#16 0x00007fe5b0ffa416 in reco::parser::BinarySelector::operator()(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#17 0x00007fe593c7bd19 in TrackingMonitor::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#18 0x00007fe5b9f4bfda in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007fe5b9f27722 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007fe5b9e58e34 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007fe5b9e59545 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007fe5b9e598bd in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007fe5b9e7a17e in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#24 0x00007fe5b9e7a432 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#25 0x00007fe5b9e7a855 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#26 0x00007fe5b9e7ab4b in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#27 0x00007fe5b9e7adbf in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#28 0x00007fe5b9f01b22 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#29 0x00007fe5b9f02257 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#30 0x00007fe5b9f08b74 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#31 0x00007fe5b8c0e9b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fe5b711fe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#32 0x00007fe5b8c081a4 in tbb::internal::arena::process (this=this@entry=0x7fe5b729f780, s=...) at ../../src/tbb/arena.cpp:156
#33 0x00007fe5b8c07264 in tbb::internal::market::process (this=0x7fe5b72c7c00, j=...) at ../../src/tbb/market.cpp:502
#34 0x00007fe5b8c03736 in tbb::internal::rml::private_worker::run (this=0x7fe5b70e6080) at ../../src/tbb/private_server.cpp:275
#35 0x00007fe5b8c03989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#36 0x0000003cee407aa1 in start_thread () from /lib64/libpthread.so.0
#37 0x0000003cee0e8aad in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fe5b82e0bc0 (LWP 26923)):
#0  0x0000003cee0df283 in poll () from /lib64/libc.so.6
#1  0x00007fe5b4d54504 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007fe5b4d5476a in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007fe5b4d5505e in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-05-1700/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007fe58878108e in ?? ()
#6  0x00007ffef8bfdd88 in ?? ()
#7  0x00007fe5368bbf08 in ?? ()
#8  0x0000000000000000 in ?? ()

Current Modules:

Module: TrackingMonitor (crashed)
Module: TrackingMonitor
Module: TrackingMonitor
Module: TrackingMonitor

A fatal system signal has occurred: segmentation violation

[dan131riley]

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc530/CMSSW_8_1_DEVEL_X_2016-09-08-1100/pyRelValMatrixLogs/run/140.53_RunHI2011+RunHI2011+RECOHID11+HARVESTDHI/step2_RunHI2011+RunHI2011+RECOHID11+HARVESTDHI.log

Another interesting one

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thread 10 (Thread 0x7f580dbbf700 (LWP 27479)):
#0  0x0000003cee40f37d in waitpid () from /lib64/libpthread.so.0
#1  0x00007f5821253ed7 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-08-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007f5821254862 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-08-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007f58235c4a50 in std::(anonymous namespace)::execute_native_thread_routine (__p=<optimized out>) at ../../../../../libstdc++-v3/src/c++11/thread.cc:84
#4  0x0000003cee407aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003cee0e8aad in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f580746d700 (LWP 27501)):
#0  0x0000003cee0e90a3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f580753a3ba in XrdSys::IOEvents::PollE::Begin (this=0x7f580b4cde40, syncsem=<optimized out>, retcode=<optimized out>, eTxt=<optimized out>) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/./XrdSys/XrdSysIOEventsPollE.icc:213
#2  0x00007f5807536a8d in XrdSys::IOEvents::BootStrap::Start (parg=0x7ffe61ef3c20) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/XrdSys/XrdSysIOEvents.cc:131
#3  0x00007f5807535647 in XrdSysThread_Xeq (myargs=0x7f580c309980) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/XrdSys/XrdSysPthread.cc:86
#4  0x0000003cee407aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003cee0e8aad in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f5806a6c700 (LWP 27502)):
#0  0x0000003cee40f00d in nanosleep () from /lib64/libpthread.so.0
#1  0x00007f5807535deb in XrdSysTimer::Wait (mills=<optimized out>) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/XrdSys/XrdSysTimer.cc:239
#2  0x00007f5807640caa in XrdCl::TaskManager::RunTasks (this=0x7f5808c18d00) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/XrdCl/XrdClTaskManager.cc:244
#3  0x00007f5807640e29 in RunRunnerThread (arg=<optimized out>) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/XrdCl/XrdClTaskManager.cc:37
#4  0x0000003cee407aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003cee0e8aad in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f580606b700 (LWP 27503)):
#0  0x0000003cee0e5309 in syscall () from /lib64/libc.so.6
#1  0x00007f580769c4e1 in XrdSys::LinuxSemaphore::Wait (this=0x7f58091eeb60) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/./XrdSys/XrdSysLinuxSemaphore.hh:161
#2  XrdCl::SyncQueue<XrdCl::JobManager::JobHelper>::Get (this=0x7f580b4cdd78) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/./XrdCl/XrdClSyncQueue.hh:67
#3  XrdCl::JobManager::RunJobs (this=0x7f580b4cdd60) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/XrdCl/XrdClJobManager.cc:146
#4  0x00007f580769c679 in RunRunnerThread (arg=<optimized out>) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/XrdCl/XrdClJobManager.cc:33
#5  0x0000003cee407aa1 in start_thread () from /lib64/libpthread.so.0
#6  0x0000003cee0e8aad in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f580566a700 (LWP 27504)):
#0  0x0000003cee0e5309 in syscall () from /lib64/libc.so.6
#1  0x00007f580769c4e1 in XrdSys::LinuxSemaphore::Wait (this=0x7f58091eeb60) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/./XrdSys/XrdSysLinuxSemaphore.hh:161
#2  XrdCl::SyncQueue<XrdCl::JobManager::JobHelper>::Get (this=0x7f580b4cdd78) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/./XrdCl/XrdClSyncQueue.hh:67
#3  XrdCl::JobManager::RunJobs (this=0x7f580b4cdd60) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/XrdCl/XrdClJobManager.cc:146
#4  0x00007f580769c679 in RunRunnerThread (arg=<optimized out>) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/XrdCl/XrdClJobManager.cc:33
#5  0x0000003cee407aa1 in start_thread () from /lib64/libpthread.so.0
#6  0x0000003cee0e8aad in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f5804c69700 (LWP 27505)):
#0  0x0000003cee0e5309 in syscall () from /lib64/libc.so.6
#1  0x00007f580769c4e1 in XrdSys::LinuxSemaphore::Wait (this=0x7f58091eeb60) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/./XrdSys/XrdSysLinuxSemaphore.hh:161
#2  XrdCl::SyncQueue<XrdCl::JobManager::JobHelper>::Get (this=0x7f580b4cdd78) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/./XrdCl/XrdClSyncQueue.hh:67
#3  XrdCl::JobManager::RunJobs (this=0x7f580b4cdd60) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/XrdCl/XrdClJobManager.cc:146
#4  0x00007f580769c679 in RunRunnerThread (arg=<optimized out>) at /build/cmsbuild/auto-builds/CMSSW_8_1_0_pre11-slc6_amd64_gcc530/build/CMSSW_8_1_0_pre11-build/BUILD/slc6_amd64_gcc530/external/xrootd/4.3.0-giojec/xrootd-4.3.0/src/XrdCl/XrdClJobManager.cc:33
#5  0x0000003cee407aa1 in start_thread () from /lib64/libpthread.so.0
#6  0x0000003cee0e8aad in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f57d72ff700 (LWP 27677)):
#0  0x0000003cee0acbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003cee0aca50 in sleep () from /lib64/libc.so.6
#2  0x00007f5821253d0e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-08-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000003cee40e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#5  0x0000003cee4095f3 in _L_lock_892 () from /lib64/libpthread.so.0
#6  0x0000003cee4094d7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#7  0x00007f58242b90ab in TClass::GetClass(char const*, bool, bool) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-08-1100/external/slc6_amd64_gcc530/lib/libCore.so
#8  0x00007f582509ff1b in edm::setRefCoreStreamer(bool) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libDataFormatsCommon.so
#9  0x00007f5807716f37 in edm::RootDelayedReader::getProduct_(edm::BranchKey const&, edm::EDProductGetter const*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/pluginIOPoolInput.so
#10 0x00007f5824df1ec9 in edm::DelayedReader::getProduct(edm::BranchKey const&, edm::EDProductGetter const*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#11 0x00007f5824ddc234 in void edm::SerialTaskQueueChain::actionToRun<edm::InputProductResolver::prefetchAsync_(edm::WaitingTask*, edm::Principal const&, bool, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const::{lambda()#1}>(edm::InputProductResolver::prefetchAsync_(edm::WaitingTask*, edm::Principal const&, bool, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const::{lambda()#1} const&) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#12 0x00007f5824ddc571 in edm::SerialTaskQueue::QueuedTask<void edm::SerialTaskQueueChain::push<edm::InputProductResolver::prefetchAsync_(edm::WaitingTask*, edm::Principal const&, bool, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const::{lambda()#1}>(edm::InputProductResolver::prefetchAsync_(edm::WaitingTask*, edm::Principal const&, bool, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const::{lambda()#1} const&)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#13 0x00007f5823b8c9b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f57d86dfe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#14 0x00007f5824de0303 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#15 0x00007f5824de081d in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007f5824e0e40e in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007f5824e0e6c2 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007f5824e0eae5 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007f5824e0eddb in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007f5824e0f04f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007f5824e87012 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007f5824e87747 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007f5824e8df44 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#24 0x00007f5823b8c9b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f57d86dfe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#25 0x00007f5823b861a4 in tbb::internal::arena::process (this=this@entry=0x7f582221f780, s=...) at ../../src/tbb/arena.cpp:156
#26 0x00007f5823b85264 in tbb::internal::market::process (this=0x7f5822247c00, j=...) at ../../src/tbb/market.cpp:502
#27 0x00007f5823b81736 in tbb::internal::rml::private_worker::run (this=0x7f5822066100) at ../../src/tbb/private_server.cpp:275
#28 0x00007f5823b81989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#29 0x0000003cee407aa1 in start_thread () from /lib64/libpthread.so.0
#30 0x0000003cee0e8aad in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f57d68fe700 (LWP 27678)):
#0  0x0000003cee0acbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003cee0aca50 in sleep () from /lib64/libc.so.6
#2  0x00007f5821253d0e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-08-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  _dl_fixup (l=0x7f5824f825b8, reloc_arg=296) at ../elf/dl-runtime.c:79
#5  0x00007f5824f60825 in _dl_runtime_resolve () at ../sysdeps/x86_64/dl-trampoline.S:41
#6  0x00007f5824adb205 in edm::TypeWithDict::byName(std::string const&, long) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreUtilities.so
#7  0x00007f5824adb92b in edm::TypeWithDict::TypeWithDict(TMethodArg*, long) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreUtilities.so
#8  0x00007f581af6af6b in reco::checkMethod(edm::FunctionWithDict const&, edm::TypeWithDict const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > >&) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#9  0x00007f581af6c3dc in reco::findMethod(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > >&, char const*, int&) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#10 0x00007f581af6c144 in reco::findMethod(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > >&, char const*, int&) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#11 0x00007f581af22300 in reco::parser::MethodSetter::push(std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, char const*, bool) const () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#12 0x00007f581af4b480 in reco::parser::SingleInvoker::SingleInvoker(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#13 0x00007f581af4c5c6 in reco::parser::LazyInvoker::invoker(edm::TypeWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#14 0x00007f581af4ea0b in reco::parser::LazyInvoker::invokeLast(edm::ObjectWithDict const&, std::vector<edm::ObjectWithDict, std::allocator<edm::ObjectWithDict> >&) const () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#15 0x00007f581af786df in reco::parser::ExpressionLazyVar::value(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#16 0x00007f581af51b06 in reco::parser::BinarySelector::operator()(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#17 0x00007f57e8e2ad19 in TrackingMonitor::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#18 0x00007f5824ecb07a in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007f5824eb3602 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007f5824ddfd94 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007f5824de04a5 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007f5824de081d in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007f5824e0e40e in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#24 0x00007f5824e0e6c2 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#25 0x00007f5824e0eae5 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#26 0x00007f5824e0eddb in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#27 0x00007f5824e0f04f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#28 0x00007f5824e87012 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#29 0x00007f5824e87747 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#30 0x00007f5824e8df44 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#31 0x00007f5823b8c9b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f57d86cfe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#32 0x00007f5823b861a4 in tbb::internal::arena::process (this=this@entry=0x7f582221f780, s=...) at ../../src/tbb/arena.cpp:156
#33 0x00007f5823b85264 in tbb::internal::market::process (this=0x7f5822247c00, j=...) at ../../src/tbb/market.cpp:502
#34 0x00007f5823b81736 in tbb::internal::rml::private_worker::run (this=0x7f5822066180) at ../../src/tbb/private_server.cpp:275
#35 0x00007f5823b81989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#36 0x0000003cee407aa1 in start_thread () from /lib64/libpthread.so.0
#37 0x0000003cee0e8aad in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f57d5d80700 (LWP 27679)):
#0  0x0000003cee0df283 in poll () from /lib64/libc.so.6
#1  0x00007f58212546c4 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-08-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007f582125492a in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-08-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007f582125521e in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-08-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007f57df24008e in ?? ()
#6  0x00007f57d5d7a308 in ?? ()
#7  0x00007f57180308d0 in ?? ()
#8  0x0000000000000000 in ?? ()
Thread 1 (Thread 0x7f582325ebc0 (LWP 27352)):
#0  0x0000003cee0acbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003cee0aca50 in sleep () from /lib64/libc.so.6
#2  0x00007f5821253d0e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-08-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  fastjet::LazyTiling9::_set_NN (jets_for_minheap=..., jetI=<optimized out>, this=0x7ffe61ef3570) at LazyTiling9.cc:470
#5  fastjet::LazyTiling9::run (this=this@entry=0x7ffe61ef3570) at LazyTiling9.cc:698
#6  0x00007f57fe607eb7 in fastjet::ClusterSequence::_initialise_and_run_no_decant (this=this@entry=0x7ffe61ef3930) at ClusterSequence.cc:384
#7  0x00007f57fe6084c2 in fastjet::ClusterSequence::_initialise_and_run (this=this@entry=0x7ffe61ef3930, jet_def_in=..., writeout_combinations=@0x7ffe61ef3870: false) at ClusterSequence.cc:228
#8  0x00007f57fe6bd2f0 in fastjet::ClusterSequenceActiveAreaExplicitGhosts::_initialise<fastjet::PseudoJet> (this=0x7ffe61ef3930, pseudojets=..., jet_def_in=..., ghost_spec=0x7f571cadf2a8, ghosts=0x0, ghost_area=0, writeout_combinations=@0x7ffe61ef3870: false) at ./../include/fastjet/ClusterSequenceActiveAreaExplicitGhosts.hh:227
#9  0x00007f57fe63390b in fastjet::ClusterSequenceActiveAreaExplicitGhosts::ClusterSequenceActiveAreaExplicitGhosts<fastjet::PseudoJet> (writeout_combinations=@0x7ffe61ef3870: false, ghost_spec=..., jet_def_in=..., pseudojets=..., this=0x7ffe61ef3930) at ./../include/fastjet/ClusterSequenceActiveAreaExplicitGhosts.hh:69
#10 fastjet::ClusterSequenceActiveArea::_run_AA (this=this@entry=0x7f571caecfc0, ghost_spec=...) at ClusterSequenceActiveArea.cc:139
#11 0x00007f57fe633c58 in fastjet::ClusterSequenceActiveArea::_initialise_and_run_AA (this=0x7f571caecfc0, jet_def_in=..., ghost_spec=..., writeout_combinations=<optimized out>) at ClusterSequenceActiveArea.cc:67
#12 0x00007f57fe940de7 in void fastjet::ClusterSequenceArea::initialize_and_run_cswa<fastjet::PseudoJet>(std::vector<fastjet::PseudoJet, std::allocator<fastjet::PseudoJet> > const&, fastjet::JetDefinition const&) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/pluginRecoJetsJetProducers_plugins.so
#13 0x00007f57fe944aa4 in FastjetJetProducer::runAlgorithm(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/pluginRecoJetsJetProducers_plugins.so
#14 0x00007f57fe902da4 in VirtualJetProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/pluginRecoJetsJetProducers_plugins.so
#15 0x00007f57fe946f12 in FastjetJetProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/pluginRecoJetsJetProducers_plugins.so
#16 0x00007f5824ec7aa2 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007f5824eb3862 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007f5824ddfd94 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007f5824de04a5 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007f5824de081d in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007f5824e0e40e in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007f5824e0e6c2 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007f5824e0eae5 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#24 0x00007f5824e0eddb in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#25 0x00007f5824e0f04f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#26 0x00007f5824e87012 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#27 0x00007f5824e87747 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#28 0x00007f5824e8df44 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#29 0x00007f5823b8c9b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f5822236600, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#30 0x00007f5824e8759e in edm::EventProcessor::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#31 0x00007f5824de3d4a in statemachine::HandleEvent::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#32 0x00007f5824de52d8 in statemachine::HandleEvent::HandleEvent(boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::my_context) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#33 0x00007f5824ded01e in boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::shallow_construct(boost::intrusive_ptr<statemachine::HandleLumis> const&, boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>&) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#34 0x00007f5824ded1ea in boost::statechart::simple_state<statemachine::FirstLumi, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#35 0x00007f5824e8d846 in boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&) () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#36 0x00007f5824e80d5d in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/2016-37/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-07-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#37 0x000000000040c844 in main::{lambda()#1}::operator()() const ()
#38 0x000000000040acc4 in main ()

Current Modules:

Module: TrackingMonitor (crashed)
Module: TrackingMonitor
Module: none
Module: FastjetJetProducer

A fatal system signal has occurred: segmentation violation

[dan131riley]

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc530/CMSSW_8_1_DEVEL_X_2016-09-16-1100/pyRelValMatrixLogs/run/1325.0_TTbar_13+TTbar_13INPUT+DIGIUP15+RECOUP15+HARVESTUP15+ALCATTUP15/step5_TTbar_13+TTbar_13INPUT+DIGIUP15+RECOUP15+HARVESTUP15+ALCATTUP15.log

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thread 5 (Thread 0x7f22c623f700 (LWP 8856)):
#0  0x0000003cee40f37d in waitpid () from /lib64/libpthread.so.0
#1  0x00007f22d28d3ed7 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007f22d28d4862 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007f22d5e00a50 in std::(anonymous namespace)::execute_native_thread_routine (__p=<optimized out>) at ../../../../../libstdc++-v3/src/c++11/thread.cc:84
#4  0x0000003cee407aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003cee0e8aad in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f22a58bf700 (LWP 8955)):
#0  0x0000003cee0acbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003cee0aca50 in sleep () from /lib64/libc.so.6
#2  0x00007f22d28d3d0e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f22d3a75110 in checkRootObject(std::string const&, TObject*, char const*, int) [clone .constprop.117] () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libDQMServicesCore.so
#5  0x00007f22d3a76fc7 in MonitorElement::Fill(double) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libDQMServicesCore.so
#6  0x00007f22a721bc97 in TrackAnalyzer::fillHistosForEfficiencyFromHitPatter(reco::Track const&, std::string, float) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#7  0x00007f22a721dc0a in TrackAnalyzer::analyze(edm::Event const&, edm::EventSetup const&, reco::Track const&) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#8  0x00007f22a71baeab in TrackingMonitor::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#9  0x00007f22d770739a in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#10 0x00007f22d764b672 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#11 0x00007f22d75ff1c4 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#12 0x00007f22d75ff8d5 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#13 0x00007f22d75ffc4d in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#14 0x00007f22d763530e in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#15 0x00007f22d76355c2 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007f22d76359e5 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007f22d7635cdb in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007f22d7635f4f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007f22d76a50d2 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007f22d76a596a in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007f22d76ac1e4 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007f22d63c89b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f22b7b27e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#23 0x00007f22d63c21a4 in tbb::internal::arena::process (this=this@entry=0x7f22d4a5f780, s=...) at ../../src/tbb/arena.cpp:156
#24 0x00007f22d63c1264 in tbb::internal::market::process (this=0x7f22d4a87c00, j=...) at ../../src/tbb/market.cpp:502
#25 0x00007f22d63bd736 in tbb::internal::rml::private_worker::run (this=0x7f22d48a6100) at ../../src/tbb/private_server.cpp:275
#26 0x00007f22d63bd989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#27 0x0000003cee407aa1 in start_thread () from /lib64/libpthread.so.0
#28 0x0000003cee0e8aad in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f22a4ebe700 (LWP 8956)):
#0  0x0000003cee0acbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003cee0aca50 in sleep () from /lib64/libc.so.6
#2  0x00007f22d28d3d0e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f22d1b4ffd6 in TH1::GetNbinsX() const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-16-1100/external/slc6_amd64_gcc530/lib/libHist.so
#5  0x00007f22a721dd03 in TrackAnalyzer::analyze(edm::Event const&, edm::EventSetup const&, reco::Track const&) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#6  0x00007f22a71baeab in TrackingMonitor::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#7  0x00007f22d770739a in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#8  0x00007f22d764b672 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#9  0x00007f22d75ff1c4 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#10 0x00007f22d75ff8d5 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#11 0x00007f22d75ffc4d in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#12 0x00007f22d763530e in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#13 0x00007f22d76355c2 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#14 0x00007f22d76359e5 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#15 0x00007f22d7635cdb in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007f22d7635f4f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007f22d76a50d2 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007f22d76a5807 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007f22d76ac1e4 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007f22d63c89b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f22b7b2fe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#21 0x00007f22d63c21a4 in tbb::internal::arena::process (this=this@entry=0x7f22d4a5f780, s=...) at ../../src/tbb/arena.cpp:156
#22 0x00007f22d63c1264 in tbb::internal::market::process (this=0x7f22d4a87c00, j=...) at ../../src/tbb/market.cpp:502
#23 0x00007f22d63bd736 in tbb::internal::rml::private_worker::run (this=0x7f22d48a6180) at ../../src/tbb/private_server.cpp:275
#24 0x00007f22d63bd989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#25 0x0000003cee407aa1 in start_thread () from /lib64/libpthread.so.0
#26 0x0000003cee0e8aad in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f22a443f700 (LWP 8957)):
#0  0x0000003cee0df283 in poll () from /lib64/libc.so.6
#1  0x00007f22d28d46c4 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007f22d28d492a in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007f22d28d521e in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007f22bb44008e in ?? ()
#6  0x00007f22a4439308 in ?? ()
#7  0x00007f2298a6a800 in ?? ()
#8  0x0000000000000000 in ?? ()
Thread 1 (Thread 0x7f22d5a9abc0 (LWP 8829)):
#0  0x0000003cee0acbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003cee0aca50 in sleep () from /lib64/libc.so.6
#2  0x00007f22d28d3d0e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000003cee084458 in __memset_sse2 () from /lib64/libc.so.6
#5  0x00007f22d6ab608c in TArrayD::Set(int) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-16-1100/external/slc6_amd64_gcc530/lib/libCore.so
#6  0x00007f22d1bebf47 in TH1::Copy(TObject&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-16-1100/external/slc6_amd64_gcc530/lib/libHist.so
#7  0x00007f22d1c85a59 in TProfile::Copy(TObject&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-16-1100/external/slc6_amd64_gcc530/lib/libHist.so
#8  0x00007f22d1c8cd4a in TProfile* TProfileHelper::ExtendAxis<TProfile>(TProfile*, double, TAxis*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-16-1100/external/slc6_amd64_gcc530/lib/libHist.so
#9  0x00007f22d1c8a699 in TProfile::ExtendAxis(double, TAxis*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-16-1100/external/slc6_amd64_gcc530/lib/libHist.so
#10 0x00007f22d1b597a0 in TAxis::FindBin(double) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-16-1100/external/slc6_amd64_gcc530/lib/libHist.so
#11 0x00007f22d1c893d9 in TProfile::Fill(double, double, double) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-16-1100/external/slc6_amd64_gcc530/lib/libHist.so
#12 0x00007f22a71bb03d in TrackingMonitor::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#13 0x00007f22d770739a in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#14 0x00007f22d764b672 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#15 0x00007f22d75ff1c4 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007f22d75ff8d5 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007f22d75ffc4d in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007f22d763530e in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007f22d76355c2 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007f22d76359e5 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007f22d7635cdb in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007f22d7635f4f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007f22d76a50d2 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#24 0x00007f22d76a5807 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#25 0x00007f22d76ac1e4 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#26 0x00007f22d63c89b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f22d4a76600, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#27 0x00007f22d76a565e in edm::EventProcessor::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#28 0x00007f22d760ee4a in statemachine::HandleEvent::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#29 0x00007f22d76103d8 in statemachine::HandleEvent::HandleEvent(boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::my_context) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#30 0x00007f22d761815e in boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::shallow_construct(boost::intrusive_ptr<statemachine::HandleLumis> const&, boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>&) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#31 0x00007f22d761832a in boost::statechart::simple_state<statemachine::FirstLumi, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#32 0x00007f22d76abae6 in boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#33 0x00007f22d769ee1d in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#34 0x000000000040c844 in main::{lambda()#1}::operator()() const ()
#35 0x000000000040acc4 in main ()

Current Modules:

Module: TrackingMonitor (crashed)
Module: TrackingMonitor
Module: TrackingMonitor
Module: TrackingMonitor

A fatal system signal has occurred: segmentation violation

[dan131riley]

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc530/CMSSW_8_1_DEVEL_X_2016-09-16-1100/pyRelValMatrixLogs/run/25.0_TTbar+TTbarINPUT+DIGI+RECOAlCaCalo+HARVEST+ALCATT/step5_TTbar+TTbarINPUT+DIGI+RECOAlCaCalo+HARVEST+ALCATT.log

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thread 5 (Thread 0x7f0c0667f700 (LWP 12914)):
#0  0x0000003cee40f37d in waitpid () from /lib64/libpthread.so.0
#1  0x00007f0c12d13ed7 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007f0c12d14862 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007f0c162b5a50 in std::(anonymous namespace)::execute_native_thread_routine (__p=<optimized out>) at ../../../../../libstdc++-v3/src/c++11/thread.cc:84
#4  0x0000003cee407aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003cee0e8aad in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f0be623f700 (LWP 12940)):
#0  0x0000003cee0df283 in poll () from /lib64/libc.so.6
#1  0x00007f0c12d146c4 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007f0c12d1492a in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007f0c12d1521e in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007f0bf42c908e in ?? ()
#6  0x00007f0be6239308 in ?? ()
#7  0x00007f0bcc77fe38 in ?? ()
#8  0x0000000000000000 in ?? ()
Thread 3 (Thread 0x7f0be583e700 (LWP 12941)):
#0  0x0000003cee0acbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003cee0aca50 in sleep () from /lib64/libc.so.6
#2  0x00007f0c12d13d0e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  std::string::_M_mutate (this=0x7f0bcc7b3748, __pos=0, __len1=0, __len2=0) at /mnt/build/davidlt/gcc530_final/b/BUILD/slc6_amd64_gcc530/external/gcc/5.3.0/gcc-tags_gcc_5_3_0_release-231255/obj/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/basic_string.tcc:912
#5  0x00007f0c16f8e87b in TClassEdit::TSplitType::ShortType(std::string&, int) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-16-1100/external/slc6_amd64_gcc530/lib/libCore.so
#6  0x00007f0c0f653c21 in ROOT::TMetaUtils::GetFullyQualifiedTypeName(std::string&, clang::QualType const&, clang::ASTContext const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-16-1100/external/slc6_amd64_gcc530/lib/libCling.so
#7  0x00007f0c0f6c7293 in TClingTypeInfo::Name() const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-16-1100/external/slc6_amd64_gcc530/lib/libCling.so
#8  0x00007f0c16faa076 in TMethodArg::GetTypeName() const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-16-1100/external/slc6_amd64_gcc530/lib/libCore.so
#9  0x00007f0c1779ddfd in edm::TypeWithDict::TypeWithDict(TMethodArg*, long) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreUtilities.so
#10 0x00007f0c0393b9cb in reco::checkMethod(edm::FunctionWithDict const&, edm::TypeWithDict const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > >&) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#11 0x00007f0c0393ce3c in reco::findMethod(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > >&, char const*, int&) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#12 0x00007f0c0393cba4 in reco::findMethod(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > >&, char const*, int&) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#13 0x00007f0c039283a0 in reco::parser::MethodSetter::push(std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, char const*, bool) const () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#14 0x00007f0c039549b0 in reco::parser::SingleInvoker::SingleInvoker(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#15 0x00007f0c03955af6 in reco::parser::LazyInvoker::invoker(edm::TypeWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#16 0x00007f0c03957f3b in reco::parser::LazyInvoker::invokeLast(edm::ObjectWithDict const&, std::vector<edm::ObjectWithDict, std::allocator<edm::ObjectWithDict> >&) const () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#17 0x00007f0c0397ea9f in reco::parser::ExpressionLazyVar::value(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#18 0x00007f0c0395b036 in reco::parser::BinarySelector::operator()(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#19 0x00007f0be9d69d19 in TrackingMonitor::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#20 0x00007f0c17bbc39a in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007f0c17b00672 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007f0c17ab41c4 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007f0c17ab48d5 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#24 0x00007f0c17ab4c4d in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#25 0x00007f0c17aea30e in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#26 0x00007f0c17aea5c2 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#27 0x00007f0c17aea9e5 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#28 0x00007f0c17aeacdb in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#29 0x00007f0c17aeaf4f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#30 0x00007f0c17b5a0d2 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#31 0x00007f0c17b5a807 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#32 0x00007f0c17b611e4 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#33 0x00007f0c1687d9b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f0be4377e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#34 0x00007f0c168771a4 in tbb::internal::arena::process (this=this@entry=0x7f0c14f1f780, s=...) at ../../src/tbb/arena.cpp:156
#35 0x00007f0c16876264 in tbb::internal::market::process (this=0x7f0c14f47c00, j=...) at ../../src/tbb/market.cpp:502
#36 0x00007f0c16872736 in tbb::internal::rml::private_worker::run (this=0x7f0c14d66180) at ../../src/tbb/private_server.cpp:275
#37 0x00007f0c16872989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#38 0x0000003cee407aa1 in start_thread () from /lib64/libpthread.so.0
#39 0x0000003cee0e8aad in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f0be4d7f700 (LWP 12942)):
#0  0x0000003cee0acbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003cee0aca50 in sleep () from /lib64/libc.so.6
#2  0x00007f0c12d13d0e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f0c176bbe4c in TBranch::GetSubBranch(TBranch const*) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-16-1100/external/slc6_amd64_gcc530/lib/libTree.so
#5  0x00007f0c176bbf03 in TBranch::GetMother() const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-16-1100/external/slc6_amd64_gcc530/lib/libTree.so
#6  0x00007f0c176cb16f in TBranchElement::InitializeOffsets() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-16-1100/external/slc6_amd64_gcc530/lib/libTree.so
#7  0x00007f0c176cd634 in TBranchElement::SetAddress(void*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-16-1100/external/slc6_amd64_gcc530/lib/libTree.so
#8  0x00007f0bfd8f0a92 in edm::RootDelayedReader::getProduct_(edm::BranchKey const&, edm::EDProductGetter const*) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/pluginIOPoolInput.so
#9  0x00007f0c17acd7f9 in edm::DelayedReader::getProduct(edm::BranchKey const&, edm::EDProductGetter const*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#10 0x00007f0c17ab0b05 in edm::InputProductResolver::resolveProduct_(edm::Principal const&, bool, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#11 0x00007f0c17b1a7cb in edm::EventPrincipal::getByProductID(edm::ProductID const&) const () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#12 0x00007f0c17b1aa58 in edm::EventPrincipal::getIt(edm::ProductID const&) const () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#13 0x00007f0be71a343d in AlCaECALRecHitReducer::AddMiniRecHitCollection(reco::SuperCluster const&, std::set<DetId, std::less<DetId>, std::allocator<DetId> >&, CaloTopology const*) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/pluginCalibrationEcalAlCaRecoProducersAuto.so
#14 0x00007f0be71a464c in AlCaECALRecHitReducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/pluginCalibrationEcalAlCaRecoProducersAuto.so
#15 0x00007f0c17b91f11 in edm::EDProducer::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007f0c17b01712 in edm::WorkerT<edm::EDProducer>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007f0c17ab41c4 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007f0c17ab65e6 in void edm::SerialTaskQueueChain::actionToRun<void edm::SerialTaskQueueChain::pushAndWait<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}::operator()() const::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}::operator()() const::{lambda()#2} const&)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}::operator()() const::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007f0c17ab6791 in edm::SerialTaskQueue::QueuedTask<void edm::SerialTaskQueueChain::push<void edm::SerialTaskQueueChain::pushAndWait<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}::operator()() const::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}::operator()() const::{lambda()#2} const&)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}::operator()() const::{lambda()#2})::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007f0c1687d9b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f0be4367e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#21 0x00007f0c17ab4590 in void edm::SerialTaskQueueChain::pushAndWait<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}::operator()() const::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}::operator()() const::{lambda()#2} const&) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007f0c17ab4824 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007f0c17ab4c4d in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#24 0x00007f0c17aea30e in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#25 0x00007f0c17aea5c2 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#26 0x00007f0c17aea9e5 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#27 0x00007f0c17aeacdb in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#28 0x00007f0c17aeaf4f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#29 0x00007f0c17b5a0d2 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#30 0x00007f0c17b5a96a in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#31 0x00007f0c17b611e4 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#32 0x00007f0c1687d9b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f0be4367e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#33 0x00007f0c168771a4 in tbb::internal::arena::process (this=this@entry=0x7f0c14f1f780, s=...) at ../../src/tbb/arena.cpp:156
#34 0x00007f0c16876264 in tbb::internal::market::process (this=0x7f0c14f47c00, j=...) at ../../src/tbb/market.cpp:502
#35 0x00007f0c16872736 in tbb::internal::rml::private_worker::run (this=0x7f0c14d66080) at ../../src/tbb/private_server.cpp:275
#36 0x00007f0c16872989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#37 0x0000003cee407aa1 in start_thread () from /lib64/libpthread.so.0
#38 0x0000003cee0e8aad in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f0c15f4fbc0 (LWP 12889)):
#0  0x0000003cee0acbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003cee0aca50 in sleep () from /lib64/libc.so.6
#2  0x00007f0c12d13d0e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f0c120caa20 in TProfile::RetrieveBinContent(int) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-16-1100/external/slc6_amd64_gcc530/lib/libHist.so
#5  0x00007f0c1202bf72 in TH1::Copy(TObject&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-16-1100/external/slc6_amd64_gcc530/lib/libHist.so
#6  0x00007f0c120c5a59 in TProfile::Copy(TObject&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-16-1100/external/slc6_amd64_gcc530/lib/libHist.so
#7  0x00007f0c120ccd4a in TProfile* TProfileHelper::ExtendAxis<TProfile>(TProfile*, double, TAxis*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-16-1100/external/slc6_amd64_gcc530/lib/libHist.so
#8  0x00007f0c120ca699 in TProfile::ExtendAxis(double, TAxis*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-16-1100/external/slc6_amd64_gcc530/lib/libHist.so
#9  0x00007f0c11f997a0 in TAxis::FindBin(double) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-16-1100/external/slc6_amd64_gcc530/lib/libHist.so
#10 0x00007f0c120c93d9 in TProfile::Fill(double, double, double) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-16-1100/external/slc6_amd64_gcc530/lib/libHist.so
#11 0x00007f0be9d6a03d in TrackingMonitor::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#12 0x00007f0c17bbc39a in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#13 0x00007f0c17b00672 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#14 0x00007f0c17ab41c4 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#15 0x00007f0c17ab48d5 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007f0c17ab4c4d in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007f0c17aea30e in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007f0c17aea5c2 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007f0c17aea9e5 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007f0c17aeacdb in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007f0c17aeaf4f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007f0c17b5a0d2 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007f0c17b5a807 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#24 0x00007f0c17b611e4 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#25 0x00007f0c1687d9b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f0c14f36600, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#26 0x00007f0c17b5a65e in edm::EventProcessor::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#27 0x00007f0c17ac3e4a in statemachine::HandleEvent::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#28 0x00007f0c17ac53d8 in statemachine::HandleEvent::HandleEvent(boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::my_context) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#29 0x00007f0c17acd15e in boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::shallow_construct(boost::intrusive_ptr<statemachine::HandleLumis> const&, boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>&) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#30 0x00007f0c17acd32a in boost::statechart::simple_state<statemachine::FirstLumi, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#31 0x00007f0c17b60ae6 in boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&) () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#32 0x00007f0c17b53e1d in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/2016-38/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-15-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#33 0x000000000040c844 in main::{lambda()#1}::operator()() const ()
#34 0x000000000040acc4 in main ()

Current Modules:

Module: TrackingMonitor (crashed)
Module: AlCaECALRecHitReducer
Module: TrackingMonitor
Module: TrackingMonitor

A fatal system signal has occurred: segmentation violation

[dan131riley]

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc530/CMSSW_8_1_X_2016-09-25-1100/pyRelValMatrixLogs/run/1325.0_TTbar_13+TTbar_13INPUT+DIGIUP15+RECOUP15+HARVESTUP15+ALCATTUP15/step5_TTbar_13+TTbar_13INPUT+DIGIUP15+RECOUP15+HARVESTUP15+ALCATTUP15.log

Similar to one last week. I've been trying putting loops in reco::findMethod() and a few other places to stress the locking scheme, but still no success replicating.

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thread 5 (Thread 0x7fab1f13f700 (LWP 7270)):
#0  0x0000003cee40f37d in waitpid () from /lib64/libpthread.so.0
#1  0x00007fab2b613ed7 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007fab2b614862 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007fab2eb61a50 in std::(anonymous namespace)::execute_native_thread_routine (__p=<optimized out>) at ../../../../../libstdc++-v3/src/c++11/thread.cc:84
#4  0x0000003cee407aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003cee0e8aad in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7faafe1ff700 (LWP 7364)):
#0  0x0000003cee0acbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003cee0aca50 in sleep () from /lib64/libc.so.6
#2  0x00007fab2b613d0e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007fab290aeccb in clang::Lexer::LexTokenInternal(clang::Token&, bool) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-09-25-1100/external/slc6_amd64_gcc530/lib/libCling.so
#5  0x00007fab29102a6f in clang::Preprocessor::Lex(clang::Token&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-09-25-1100/external/slc6_amd64_gcc530/lib/libCling.so
#6  0x00007fab28053f10 in cling::prepareForParsing(clang::Parser&, cling::Interpreter const*, llvm::StringRef, llvm::StringRef, cling::LookupHelper::DiagSetting) [clone .constprop.308] () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-09-25-1100/external/slc6_amd64_gcc530/lib/libCling.so
#7  0x00007fab28058f83 in clang::FunctionDecl const* cling::execFindFunction<cling::ParseProto, clang::FunctionDecl const*>(clang::Parser&, cling::Interpreter*, clang::Decl const*, llvm::StringRef, cling::ParseProto::ArgsInput const&, bool, clang::FunctionDecl const* (*)(clang::DeclContext*, bool, llvm::SmallVectorImpl<clang::Expr*> const&, clang::LookupResult&, clang::DeclarationNameInfo&, clang::TemplateArgumentListInfo const*, clang::ASTContext&, clang::Parser&, clang::Sema&), cling::LookupHelper::DiagSetting) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-09-25-1100/external/slc6_amd64_gcc530/lib/libCling.so
#8  0x00007fab2805958b in cling::LookupHelper::findFunctionProto(clang::Decl const*, llvm::StringRef, llvm::StringRef, cling::LookupHelper::DiagSetting, bool) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-09-25-1100/external/slc6_amd64_gcc530/lib/libCling.so
#9  0x00007fab27faddab in TClingClassInfo::GetMethod(char const*, char const*, bool, long*, ROOT::EFunctionMatchMode, TClingClassInfo::EInheritanceMode) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-09-25-1100/external/slc6_amd64_gcc530/lib/libCling.so
#10 0x00007fab27fcfff0 in TCling::GetFunctionWithPrototype(ClassInfo_t*, char const*, char const*, bool, ROOT::EFunctionMatchMode) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-09-25-1100/external/slc6_amd64_gcc530/lib/libCling.so
#11 0x00007fab2f864584 in TClass::GetMethodWithPrototype(char const*, char const*, bool, ROOT::EFunctionMatchMode) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-09-25-1100/external/slc6_amd64_gcc530/lib/libCore.so
#12 0x00007fab3005c629 in edm::TypeWithDict::functionMemberByName(std::string const&, std::string const&, bool) const () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreUtilities.so
#13 0x00007fab1c1315da in reco::findMethod(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > >&, char const*, int&) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#14 0x00007fab1c131784 in reco::findMethod(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > >&, char const*, int&) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#15 0x00007fab1c138ef0 in reco::parser::MethodSetter::push(std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, char const*, bool) const () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#16 0x00007fab1c15a6e0 in reco::parser::SingleInvoker::SingleInvoker(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#17 0x00007fab1c15b826 in reco::parser::LazyInvoker::invoker(edm::TypeWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#18 0x00007fab1c15dc6b in reco::parser::LazyInvoker::invokeLast(edm::ObjectWithDict const&, std::vector<edm::ObjectWithDict, std::allocator<edm::ObjectWithDict> >&) const () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#19 0x00007fab1c1349ef in reco::parser::ExpressionLazyVar::value(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#20 0x00007fab1c161b96 in reco::parser::BinarySelector::operator()(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#21 0x00007faaffb43d19 in TrackingMonitor::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#22 0x00007fab3046d48a in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007fab303ab9b2 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#24 0x00007fab3035d7f4 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#25 0x00007fab3035df05 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#26 0x00007fab3035e27d in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#27 0x00007fab30394c9e in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#28 0x00007fab30394f52 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#29 0x00007fab30395375 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#30 0x00007fab3039566b in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#31 0x00007fab303958df in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#32 0x00007fab304068f2 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#33 0x00007fab3040718a in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#34 0x00007fab3040da04 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#35 0x00007fab2f1289b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fab10d4fe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#36 0x00007fab2f1221a4 in tbb::internal::arena::process (this=this@entry=0x7fab2d7bf780, s=...) at ../../src/tbb/arena.cpp:156
#37 0x00007fab2f121264 in tbb::internal::market::process (this=0x7fab2d7e7c00, j=...) at ../../src/tbb/market.cpp:502
#38 0x00007fab2f11d736 in tbb::internal::rml::private_worker::run (this=0x7fab2d606100) at ../../src/tbb/private_server.cpp:275
#39 0x00007fab2f11d989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#40 0x0000003cee407aa1 in start_thread () from /lib64/libpthread.so.0
#41 0x0000003cee0e8aad in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7faafd7fe700 (LWP 7365)):
#0  0x0000003cee0df283 in poll () from /lib64/libc.so.6
#1  0x00007fab2b6146c4 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007fab2b61492a in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007fab2b61521e in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007fab138c108e in ?? ()
#6  0x00007faafd7f8308 in ?? ()
#7  0x00007faaef43f8a8 in ?? ()
#8  0x0000000000000000 in ?? ()
Thread 2 (Thread 0x7faafcd3f700 (LWP 7366)):
#0  0x0000003cee0acbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003cee0aca50 in sleep () from /lib64/libc.so.6
#2  0x00007fab2b613d0e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007fab1c134810 in reco::parser::ExpressionVar::isValidReturnType(reco::method::TypeCode) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#5  0x00007fab1c15a5a3 in reco::parser::SingleInvoker::retToDouble(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#6  0x00007fab1c15dd4c in reco::parser::LazyInvoker::invokeLast(edm::ObjectWithDict const&, std::vector<edm::ObjectWithDict, std::allocator<edm::ObjectWithDict> >&) const () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#7  0x00007fab1c1349ef in reco::parser::ExpressionLazyVar::value(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#8  0x00007fab1c161b96 in reco::parser::BinarySelector::operator()(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#9  0x00007faaffb43d19 in TrackingMonitor::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#10 0x00007fab3046d48a in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#11 0x00007fab303ab9b2 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#12 0x00007fab3035d7f4 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#13 0x00007fab3035df05 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#14 0x00007fab3035e27d in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#15 0x00007fab30394c9e in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007fab30394f52 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007fab30395375 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007fab3039566b in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007fab303958df in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007fab304068f2 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007fab30407027 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007fab3040da04 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007fab2f1289b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fab10d2fe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#24 0x00007fab2f1221a4 in tbb::internal::arena::process (this=this@entry=0x7fab2d7bf780, s=...) at ../../src/tbb/arena.cpp:156
#25 0x00007fab2f121264 in tbb::internal::market::process (this=0x7fab2d7e7c00, j=...) at ../../src/tbb/market.cpp:502
#26 0x00007fab2f11d736 in tbb::internal::rml::private_worker::run (this=0x7fab2d606080) at ../../src/tbb/private_server.cpp:275
#27 0x00007fab2f11d989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#28 0x0000003cee407aa1 in start_thread () from /lib64/libpthread.so.0
#29 0x0000003cee0e8aad in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fab2e7fabc0 (LWP 7247)):
#0  0x0000003cee0acbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003cee0aca50 in sleep () from /lib64/libc.so.6
#2  0x00007fab2b613d0e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000003cee40e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#5  0x0000003cee4095f3 in _L_lock_892 () from /lib64/libpthread.so.0
#6  0x0000003cee4094d7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#7  0x00007fab27fd5e0b in TCling::CallFunc_Factory() const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-09-25-1100/external/slc6_amd64_gcc530/lib/libCling.so
#8  0x00007fab2f857312 in TMethodCall::Init(TFunction const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-09-25-1100/external/slc6_amd64_gcc530/lib/libCore.so
#9  0x00007fab2f8584fb in TMethodCall::TMethodCall(TFunction const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-09-25-1100/external/slc6_amd64_gcc530/lib/libCore.so
#10 0x00007fab30068c96 in edm::FunctionWithDict::FunctionWithDict(TMethod*) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreUtilities.so
#11 0x00007fab1c131a03 in reco::findMethod(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > >&, char const*, int&) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#12 0x00007fab1c131784 in reco::findMethod(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > >&, char const*, int&) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#13 0x00007fab1c138ef0 in reco::parser::MethodSetter::push(std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, char const*, bool) const () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#14 0x00007fab1c15a6e0 in reco::parser::SingleInvoker::SingleInvoker(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#15 0x00007fab1c15b826 in reco::parser::LazyInvoker::invoker(edm::TypeWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#16 0x00007fab1c15dc6b in reco::parser::LazyInvoker::invokeLast(edm::ObjectWithDict const&, std::vector<edm::ObjectWithDict, std::allocator<edm::ObjectWithDict> >&) const () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#17 0x00007fab1c1349ef in reco::parser::ExpressionLazyVar::value(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#18 0x00007fab1c161b96 in reco::parser::BinarySelector::operator()(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#19 0x00007faaffb43d19 in TrackingMonitor::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#20 0x00007fab3046d48a in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007fab303ab9b2 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007fab3035d7f4 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007fab3035df05 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#24 0x00007fab3035e27d in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#25 0x00007fab30394c9e in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#26 0x00007fab30394f52 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#27 0x00007fab30395375 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#28 0x00007fab3039566b in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#29 0x00007fab303958df in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#30 0x00007fab304068f2 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#31 0x00007fab30407027 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#32 0x00007fab3040da04 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#33 0x00007fab2f1289b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fab2d7d6600, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#34 0x00007fab30406e7e in edm::EventProcessor::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#35 0x00007fab3036bf8a in statemachine::HandleEvent::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#36 0x00007fab3036d518 in statemachine::HandleEvent::HandleEvent(boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::my_context) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#37 0x00007fab3037529e in boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::shallow_construct(boost::intrusive_ptr<statemachine::HandleLumis> const&, boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>&) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#38 0x00007fab3037546a in boost::statechart::simple_state<statemachine::FirstLumi, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#39 0x00007fab3040d306 in boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#40 0x00007fab304005dd in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#41 0x000000000040c844 in main::{lambda()#1}::operator()() const ()
#42 0x000000000040acc4 in main ()

Current Modules:

Module: TrackingMonitor (crashed)
Module: TrackingMonitor
Module: TrackingMonitor
Module: TrackingMonitor

A fatal system signal has occurred: segmentation violation

[dan131riley]

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc530/CMSSW_8_1_DEVEL_X_2016-09-29-0000/pyRelValMatrixLogs/run/1311.0_MinBias_13+MinBias_13INPUT+DIGIUP15+RECOMINUP15+HARVESTMINUP15+ALCAMINUP15/step5_MinBias_13+MinBias_13INPUT+DIGIUP15+RECOMINUP15+HARVESTMINUP15+ALCAMINUP15.log

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thread 5 (Thread 0x7fd6cd17f700 (LWP 17333)):
#0  0x000000302a80f37d in waitpid () from /lib64/libpthread.so.0
#1  0x00007fd6d960dea7 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-25-0000/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007fd6d960e832 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-25-0000/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007fd6dc518a50 in std::(anonymous namespace)::execute_native_thread_routine (__p=<optimized out>) at ../../../../../libstdc++-v3/src/c++11/thread.cc:84
#4  0x000000302a807aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x000000302a4e8aad in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fd6ad0e8700 (LWP 17355)):
#0  0x000000302a4df283 in poll () from /lib64/libc.so.6
#1  0x00007fd6d960e694 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-25-0000/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007fd6d960e8fa in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-25-0000/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007fd6d960f1ee in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-25-0000/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007fd6dde5708e in ?? ()
#6  0x00007fd6ad0e2488 in ?? ()
#7  0x00007fd6a4cc7870 in ?? ()
#8  0x0000000000000000 in ?? ()
Thread 3 (Thread 0x7fd6ac6e7700 (LWP 17356)):
#0  0x000000302a4acbdd in nanosleep () from /lib64/libc.so.6
#1  0x000000302a4aca50 in sleep () from /lib64/libc.so.6
#2  0x00007fd6d960dcde in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-25-0000/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  do_lookup_x (new_hash=2688881856, old_hash=0x7fd6ac6e19d8, ref=<optimized out>, result=0x7fd6ac6e19c0, scope=<optimized out>, i=248, flags=5, skip=0x0, undef_map=0x7fd6c02a9c00) at dl-lookup.c:235
#5  0x00007fd6de23af0a in _dl_lookup_symbol_x (undef_name=0x7fd6ad50d762 "_ZNK3edm5Event10getByTokenISt6vectorIN4reco5TrackESaIS4_EEEEbNS_11EDGetTokenTIT_EERNS_6HandleIS8_EE", undef_map=0x7fd6c02a9c00, ref=0x7fd6ac6e1a38, symbol_scope=0x7fd6c02a9f58, version=0x0, type_class=1, flags=5, skip_map=0x0) at dl-lookup.c:747
#6  0x00007fd6de23f0c0 in _dl_fixup (l=<optimized out>, reloc_arg=<optimized out>) at ../elf/dl-runtime.c:118
#7  0x00007fd6de245825 in _dl_runtime_resolve () at ../sysdeps/x86_64/dl-trampoline.S:41
#8  0x00007fd6ad52ee7b in TkAlCaRecoMonitor::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-25-0000/lib/slc6_amd64_gcc530/pluginDQMOfflineAlignment.so
#9  0x00007fd6de1a338a in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-29-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#10 0x00007fd6de176902 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-29-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#11 0x00007fd6de089f7b in decltype ({parm#1}()) edm::convertException::wrap<void edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-29-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#12 0x00007fd6de08a16c in void edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-29-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#13 0x00007fd6de08a4d9 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-29-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#14 0x00007fd6dcd1e9b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fd6bdb9fe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#15 0x00007fd6de0d8fb3 in edm::StreamSchedule::processOneEvent(edm::EventPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-29-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007fd6de133b6c in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-29-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007fd6de134287 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-29-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007fd6de13ad54 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-29-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007fd6dcd1e9b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fd6bdb9fe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#20 0x00007fd6dcd181a4 in tbb::internal::arena::process (this=this@entry=0x7fd6dbb77780, s=...) at ../../src/tbb/arena.cpp:156
#21 0x00007fd6dcd17264 in tbb::internal::market::process (this=0x7fd6dbb9fe00, j=...) at ../../src/tbb/market.cpp:502
#22 0x00007fd6dcd13736 in tbb::internal::rml::private_worker::run (this=0x7fd6dbb97a80) at ../../src/tbb/private_server.cpp:275
#23 0x00007fd6dcd13989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#24 0x000000302a807aa1 in start_thread () from /lib64/libpthread.so.0
#25 0x000000302a4e8aad in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fd6abc7f700 (LWP 17357)):
#0  0x000000302a4acbdd in nanosleep () from /lib64/libc.so.6
#1  0x000000302a4aca50 in sleep () from /lib64/libc.so.6
#2  0x00007fd6d960dcde in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-25-0000/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000000302a80e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#5  0x000000302a8095f3 in _L_lock_892 () from /lib64/libpthread.so.0
#6  0x000000302a8094d7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#7  0x00007fd6dd4e078b in TClass::GetClass(char const*, bool, bool) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-29-0000/external/slc6_amd64_gcc530/lib/libCore.so
#8  0x00007fd6de350099 in edm::setRefCoreStreamer(edm::EDProductGetter const*) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libDataFormatsCommon.so
#9  0x00007fd6c373d1ca in edm::RootDelayedReader::getProduct_(edm::BranchKey const&, edm::EDProductGetter const*) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-25-0000/lib/slc6_amd64_gcc530/pluginIOPoolInput.so
#10 0x00007fd6de0a78f9 in edm::DelayedReader::getProduct(edm::BranchKey const&, edm::EDProductGetter const*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-29-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#11 0x00007fd6de097165 in edm::InputProductResolver::resolveProduct_(edm::Principal const&, bool, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-29-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#12 0x00007fd6de18acdb in edm::EventPrincipal::getByProductID(edm::ProductID const&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-29-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#13 0x00007fd6de18af68 in edm::EventPrincipal::getIt(edm::ProductID const&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-29-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#14 0x00007fd6de3547ab in edm::RefCore::tryToGetProductPtr(std::type_info const&, edm::EDProductGetter const*) const () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libDataFormatsCommon.so
#15 0x00007fd6ad9bfb87 in void helper::ClusterStorer::processClusters<SiStripRecHit2D, SiStripCluster>(std::vector<helper::ClusterStorer::ClusterHitRecord<SiStripRecHit2D::ClusterRef>, std::allocator<helper::ClusterStorer::ClusterHitRecord<SiStripRecHit2D::ClusterRef> > >&, edmNew::DetSetVector<SiStripCluster>&, edm::RefProd<edmNew::DetSetVector<SiStripCluster> >&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-29-0000/lib/slc6_amd64_gcc530/libCommonToolsRecoAlgos.so
#16 0x00007fd6ad9f2690 in void helper::TrackCollectionStoreManager::cloneAndStore<__gnu_cxx::__normal_iterator<reco::Track const* const*, std::vector<reco::Track const*, std::allocator<reco::Track const*> > > >(__gnu_cxx::__normal_iterator<reco::Track const* const*, std::vector<reco::Track const*, std::allocator<reco::Track const*> > > const&, __gnu_cxx::__normal_iterator<reco::Track const* const*, std::vector<reco::Track const*, std::allocator<reco::Track const*> > > const&, edm::Event&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-29-0000/lib/slc6_amd64_gcc530/pluginCommonAlignmentProducerSelectors.so
#17 0x00007fd6ad9fa256 in ObjectSelector<TrackConfigSelector, std::vector<reco::Track, std::allocator<reco::Track> >, NonNullNumberSelector, helper::NullPostProcessor<std::vector<reco::Track, std::allocator<reco::Track> >, edm::stream::EDFilter<> >, helper::TrackCollectionStoreManager, helper::TrackSelectorBase, reco::modules::NoEventSetupInit<TrackConfigSelector> >::filter(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-29-0000/lib/slc6_amd64_gcc530/pluginCommonAlignmentProducerSelectors.so
#18 0x00007fd6de1a1bc6 in edm::stream::EDFilterAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-29-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007fd6de176a32 in edm::WorkerT<edm::stream::EDFilterAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-29-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007fd6de089f7b in decltype ({parm#1}()) edm::convertException::wrap<void edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-29-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007fd6de08a16c in void edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-29-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007fd6de08a4d9 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-29-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007fd6dcd1e9b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fd6bdb87e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#24 0x00007fd6de0d8fb3 in edm::StreamSchedule::processOneEvent(edm::EventPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-29-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#25 0x00007fd6de133b6c in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-29-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#26 0x00007fd6de134287 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-29-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#27 0x00007fd6de13ad54 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-29-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#28 0x00007fd6dcd1e9b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fd6bdb87e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#29 0x00007fd6dcd181a4 in tbb::internal::arena::process (this=this@entry=0x7fd6dbb77780, s=...) at ../../src/tbb/arena.cpp:156
#30 0x00007fd6dcd17264 in tbb::internal::market::process (this=0x7fd6dbb9fe00, j=...) at ../../src/tbb/market.cpp:502
#31 0x00007fd6dcd13736 in tbb::internal::rml::private_worker::run (this=0x7fd6dbb97980) at ../../src/tbb/private_server.cpp:275
#32 0x00007fd6dcd13989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#33 0x000000302a807aa1 in start_thread () from /lib64/libpthread.so.0
#34 0x000000302a4e8aad in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fd6de43bbc0 (LWP 17299)):
#0  0x000000302a4acbdd in nanosleep () from /lib64/libc.so.6
#1  0x000000302a4aca50 in sleep () from /lib64/libc.so.6
#2  0x00007fd6d960dcde in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-25-0000/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  je_tcache_alloc_easy (tcache_success=<synthetic pointer>, tbin=0x7fd6dc1020e8) at include/jemalloc/internal/tcache.h:270
#5  je_tcache_alloc_small (slow_path=false, zero=false, binind=6, size=96, tcache=<optimized out>, arena=0x0, tsd=<optimized out>) at include/jemalloc/internal/tcache.h:290
#6  je_arena_malloc (slow_path=false, tcache=<optimized out>, zero=false, ind=6, size=96, arena=0x0, tsdn=<optimized out>) at include/jemalloc/internal/arena.h:1333
#7  je_iallocztm (slow_path=false, arena=0x0, is_metadata=false, tcache=<optimized out>, zero=false, ind=6, size=96, tsdn=<optimized out>) at include/jemalloc/internal/jemalloc_internal.h:967
#8  je_ialloc (slow_path=false, zero=false, ind=6, size=96, tsd=<optimized out>) at include/jemalloc/internal/jemalloc_internal.h:979
#9  ialloc_body (slow_path=false, usize=<synthetic pointer>, tsdn=<synthetic pointer>, zero=false, size=96) at src/jemalloc.c:1520
#10 malloc (size=size@entry=96) at src/jemalloc.c:1559
#11 0x00007fd6dc4eef68 in operator new (sz=96) at ../../../../libstdc++-v3/libsupc++/new_op.cc:50
#12 0x00007fd6c9dfdec2 in void std::vector<edm::TypeWithDict, std::allocator<edm::TypeWithDict> >::_M_emplace_back_aux<edm::TypeWithDict>(edm::TypeWithDict&&) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#13 0x00007fd6c9dfb595 in reco::parser::MethodSetter::push(std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, char const*, bool) const () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#14 0x00007fd6c9de9d60 in reco::parser::SingleInvoker::SingleInvoker(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#15 0x00007fd6c9deaea6 in reco::parser::LazyInvoker::invoker(edm::TypeWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#16 0x00007fd6c9ded2eb in reco::parser::LazyInvoker::invokeLast(edm::ObjectWithDict const&, std::vector<edm::ObjectWithDict, std::allocator<edm::ObjectWithDict> >&) const () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#17 0x00007fd6c9df4e6f in reco::parser::ExpressionLazyVar::value(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#18 0x00007fd6c9e176b6 in reco::parser::BinarySelector::operator()(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-25-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#19 0x00007fd6ad6d7d19 in TrackingMonitor::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/2016-40/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-09-25-0000/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#20 0x00007fd6de1a338a in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-29-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007fd6de176902 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-29-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007fd6de089f7b in decltype ({parm#1}()) edm::convertException::wrap<void edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-29-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007fd6de08a16c in void edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-29-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#24 0x00007fd6de08a4d9 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-29-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#25 0x00007fd6dcd1e9b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fd6dbb9e400, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#26 0x00007fd6de0d8fb3 in edm::StreamSchedule::processOneEvent(edm::EventPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-29-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#27 0x00007fd6de133b6c in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-29-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#28 0x00007fd6de134287 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-29-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#29 0x00007fd6de13ad54 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-29-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#30 0x00007fd6dcd1e9b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fd6dbb9e400, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#31 0x00007fd6de1340de in edm::EventProcessor::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-29-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#32 0x00007fd6de09d7ea in statemachine::HandleEvent::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-29-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#33 0x00007fd6de09ed78 in statemachine::HandleEvent::HandleEvent(boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::my_context) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-29-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#34 0x00007fd6de0a6afe in boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::shallow_construct(boost::intrusive_ptr<statemachine::HandleLumis> const&, boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-29-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#35 0x00007fd6de0a6cca in boost::statechart::simple_state<statemachine::FirstLumi, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-29-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#36 0x00007fd6de13a656 in boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-29-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#37 0x00007fd6de12d8cd in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-09-29-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#38 0x000000000040c844 in main::{lambda()#1}::operator()() const ()
#39 0x000000000040acc4 in main ()

Current Modules:

Module: TrackingMonitor (crashed)
Module: TrackingMonitor
Module: AlignmentTrackSelectorModule
Module: TkAlCaRecoMonitor

A fatal system signal has occurred: segmentation violation

[dan131riley]

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc530/CMSSW_8_1_X_2016-10-02-2300/pyRelValMatrixLogs/run/140.2_PhotonJets_Pt_10_13_HI+PhotonJets_Pt_10_13_HIINPUT+DIGIHI+RECOHI+HARVESTHI/step3_PhotonJets_Pt_10_13_HI+PhotonJets_Pt_10_13_HIINPUT+DIGIHI+RECOHI+HARVESTHI.log

still there...

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thread 5 (Thread 0x7fcf2d97f700 (LWP 17890)):
#0  0x0000003cee40f37d in waitpid () from /lib64/libpthread.so.0
#1  0x00007fcf3e7d3ed7 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-02-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007fcf3e7d4862 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-02-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007fcf438f2a50 in std::(anonymous namespace)::execute_native_thread_routine (__p=<optimized out>) at ../../../../../libstdc++-v3/src/c++11/thread.cc:84
#4  0x0000003cee407aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003cee0e8aad in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fcef96ff700 (LWP 17898)):
#0  0x0000003cee0acbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003cee0aca50 in sleep () from /lib64/libc.so.6
#2  0x00007fcf3e7d3d0e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-02-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000003cee0e5309 in syscall () from /lib64/libc.so.6
#5  0x00007fcf43eae962 in tbb::internal::futex_wait (comparand=2, futex=0x7fcf423a612c) at ../../include/tbb/machine/linux_common.h:60
#6  tbb::internal::binary_semaphore::P (this=0x7fcf423a612c) at ../../src/tbb/semaphore.h:206
#7  rml::internal::thread_monitor::commit_wait (c=<synthetic pointer>, this=0x7fcf423a6120) at ../../src/rml/include/../server/thread_monitor.h:259
#8  tbb::internal::rml::private_worker::run (this=0x7fcf423a6100) at ../../src/tbb/private_server.cpp:282
#9  0x00007fcf43eae989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#10 0x0000003cee407aa1 in start_thread () from /lib64/libpthread.so.0
#11 0x0000003cee0e8aad in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fcef8cfe700 (LWP 17899)):
#0  0x0000003cee0acbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003cee0aca50 in sleep () from /lib64/libc.so.6
#2  0x00007fcf3e7d3d0e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-02-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000003cee0e5309 in syscall () from /lib64/libc.so.6
#5  0x00007fcf43eae962 in tbb::internal::futex_wait (comparand=2, futex=0x7fcf423a61ac) at ../../include/tbb/machine/linux_common.h:60
#6  tbb::internal::binary_semaphore::P (this=0x7fcf423a61ac) at ../../src/tbb/semaphore.h:206
#7  rml::internal::thread_monitor::commit_wait (c=<synthetic pointer>, this=0x7fcf423a61a0) at ../../src/rml/include/../server/thread_monitor.h:259
#8  tbb::internal::rml::private_worker::run (this=0x7fcf423a6180) at ../../src/tbb/private_server.cpp:282
#9  0x00007fcf43eae989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#10 0x0000003cee407aa1 in start_thread () from /lib64/libpthread.so.0
#11 0x0000003cee0e8aad in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fcef827f700 (LWP 17900)):
#0  0x0000003cee0acbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003cee0aca50 in sleep () from /lib64/libc.so.6
#2  0x00007fcf3e7d3d0e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-02-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000003cee089cc4 in _wordcopy_fwd_aligned () from /lib64/libc.so.6
#5  0x0000003cee083b22 in memmove () from /lib64/libc.so.6
#6  0x00007fcf3b685a06 in TH1::Copy(TObject&) const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-10-02-2300/external/slc6_amd64_gcc530/lib/libHist.so
#7  0x00007fcf3b5d95b9 in TProfile::Copy(TObject&) const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-10-02-2300/external/slc6_amd64_gcc530/lib/libHist.so
#8  0x00007fcf3b5e092a in TProfile* TProfileHelper::ExtendAxis<TProfile>(TProfile*, double, TAxis*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-10-02-2300/external/slc6_amd64_gcc530/lib/libHist.so
#9  0x00007fcf3b5de1f9 in TProfile::ExtendAxis(double, TAxis*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-10-02-2300/external/slc6_amd64_gcc530/lib/libHist.so
#10 0x00007fcf3b6aad00 in TAxis::FindBin(double) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-10-02-2300/external/slc6_amd64_gcc530/lib/libHist.so
#11 0x00007fcf3b5dcf39 in TProfile::Fill(double, double, double) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-10-02-2300/external/slc6_amd64_gcc530/lib/libHist.so
#12 0x00007fcf06253e0d in TrackingMonitor::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-02-1100/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#13 0x00007fcf451f9a7a in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-02-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#14 0x00007fcf45161a32 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-02-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#15 0x00007fcf45105a34 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-02-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007fcf45106145 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-02-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007fcf451064bd in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-02-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007fcf451492ee in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-02-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007fcf451495a2 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-02-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007fcf45149adc in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-02-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007fcf45149cbb in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-02-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007fcf45149f2f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-02-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007fcf451ae012 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-02-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#24 0x00007fcf451ae8aa in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-02-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#25 0x00007fcf451b5014 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-02-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#26 0x00007fcf43eb99b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fcefb9d3e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#27 0x00007fcf43eb31a4 in tbb::internal::arena::process (this=this@entry=0x7fcf4255f780, s=...) at ../../src/tbb/arena.cpp:156
#28 0x00007fcf43eb2264 in tbb::internal::market::process (this=0x7fcf42587c00, j=...) at ../../src/tbb/market.cpp:502
#29 0x00007fcf43eae736 in tbb::internal::rml::private_worker::run (this=0x7fcf423a6080) at ../../src/tbb/private_server.cpp:275
#30 0x00007fcf43eae989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#31 0x0000003cee407aa1 in start_thread () from /lib64/libpthread.so.0
#32 0x0000003cee0e8aad in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fcf4358bbc0 (LWP 17862)):
#0  0x0000003cee0df283 in poll () from /lib64/libc.so.6
#1  0x00007fcf3e7d46c4 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-02-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007fcf3e7d492a in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-02-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007fcf3e7d521e in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-02-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007fcf082c008e in ?? ()
#6  0x00007ffdc5d1c0c8 in ?? ()
#7  0x00007fce35e7fc70 in ?? ()
#8  0x0000000000000000 in ?? ()

Current Modules:

Module: TrackingMonitor (crashed)
Module: none
Module: TrackingMonitor
Module: none

A fatal system signal has occurred: segmentation violation

[dan131riley]

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc530/CMSSW_8_1_X_2016-10-04-1100/pyRelValMatrixLogs/run/1307.0_CosmicsSPLoose_UP15+CosmicsSPLoose_UP15INPUT+DIGICOS_UP15+RECOCOS_UP15+ALCACOS_UP15+HARVESTCOS_UP15/step4_CosmicsSPLoose_UP15+CosmicsSPLoose_UP15INPUT+DIGICOS_UP15+RECOCOS_UP15+ALCACOS_UP15+HARVESTCOS_UP15.log

This one is kind of interesting because two of the threads are waiting gInterpreterMutex in TListOfFunctionsIter::Next(), with thread 1 in TClass::GetClass holding the mutex.

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thread 5 (Thread 0x7f66492bf700 (LWP 214364)):
#0  0x0000003e8cc0f37d in waitpid () from /lib64/libpthread.so.0
#1  0x00007f6655793ed7 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007f6655794862 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007f6658c9fa50 in std::(anonymous namespace)::execute_native_thread_routine (__p=<optimized out>) at ../../../../../libstdc++-v3/src/c++11/thread.cc:84
#4  0x0000003e8cc07aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003e8c8e8aad in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f663162b700 (LWP 215085)):
#0  0x0000003e8c8acbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003e8c8aca50 in sleep () from /lib64/libc.so.6
#2  0x00007f6655793d0e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000003e8cc0e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#5  0x0000003e8cc095f3 in _L_lock_892 () from /lib64/libpthread.so.0
#6  0x0000003e8cc094d7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#7  0x00007f6659b8dbb4 in TListOfFunctionsIter::Next() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-10-04-1100/external/slc6_amd64_gcc530/lib/libCore.so
#8  0x00007f665a3c4429 in edm::IterWithDictBase::advance() () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreUtilities.so
#9  0x00007f66462bd82d in reco::findMethod(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > >&, char const*, int&) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#10 0x00007f66462bd6c4 in reco::findMethod(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > >&, char const*, int&) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#11 0x00007f66462b4b30 in reco::parser::MethodSetter::push(std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, char const*, bool) const () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#12 0x00007f664629bb50 in reco::parser::SingleInvoker::SingleInvoker(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#13 0x00007f664629cc96 in reco::parser::LazyInvoker::invoker(edm::TypeWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#14 0x00007f664629f0db in reco::parser::LazyInvoker::invokeLast(edm::ObjectWithDict const&, std::vector<edm::ObjectWithDict, std::allocator<edm::ObjectWithDict> >&) const () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#15 0x00007f66462d29ff in reco::parser::ExpressionLazyVar::value(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#16 0x00007f66462a44f6 in reco::parser::BinarySelector::operator()(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#17 0x00007f6631a5eae9 in TrackingMonitor::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#18 0x00007f665a7aea7a in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007f665a716a32 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007f665a6baa34 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007f665a6bb145 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007f665a6bb4bd in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007f665a6fe2ee in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#24 0x00007f665a6fe5a2 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#25 0x00007f665a6fe9c5 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#26 0x00007f665a6fecbb in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#27 0x00007f665a6fef2f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#28 0x00007f665a763012 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#29 0x00007f665a763747 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#30 0x00007f665a76a014 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#31 0x00007f665946e9b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f663af0fe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#32 0x00007f66594681a4 in tbb::internal::arena::process (this=this@entry=0x7f665790b780, s=...) at ../../src/tbb/arena.cpp:156
#33 0x00007f6659467264 in tbb::internal::market::process (this=0x7f6657927900, j=...) at ../../src/tbb/market.cpp:502
#34 0x00007f6659463736 in tbb::internal::rml::private_worker::run (this=0x7f6657747100) at ../../src/tbb/private_server.cpp:275
#35 0x00007f6659463989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#36 0x0000003e8cc07aa1 in start_thread () from /lib64/libpthread.so.0
#37 0x0000003e8c8e8aad in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f6630c2a700 (LWP 215086)):
#0  0x0000003e8c8acbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003e8c8aca50 in sleep () from /lib64/libc.so.6
#2  0x00007f6655793d0e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000003e8cc0e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#5  0x0000003e8cc095f3 in _L_lock_892 () from /lib64/libpthread.so.0
#6  0x0000003e8cc094d7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#7  0x00007f6659b8dbb4 in TListOfFunctionsIter::Next() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-10-04-1100/external/slc6_amd64_gcc530/lib/libCore.so
#8  0x00007f665a3c4429 in edm::IterWithDictBase::advance() () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreUtilities.so
#9  0x00007f66462bd82d in reco::findMethod(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > >&, char const*, int&) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#10 0x00007f66462bd6c4 in reco::findMethod(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > >&, char const*, int&) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#11 0x00007f66462b4b30 in reco::parser::MethodSetter::push(std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, char const*, bool) const () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#12 0x00007f664629bb50 in reco::parser::SingleInvoker::SingleInvoker(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#13 0x00007f664629cc96 in reco::parser::LazyInvoker::invoker(edm::TypeWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#14 0x00007f664629f0db in reco::parser::LazyInvoker::invokeLast(edm::ObjectWithDict const&, std::vector<edm::ObjectWithDict, std::allocator<edm::ObjectWithDict> >&) const () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#15 0x00007f66462d29ff in reco::parser::ExpressionLazyVar::value(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#16 0x00007f66462a44f6 in reco::parser::BinarySelector::operator()(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#17 0x00007f6631a5eae9 in TrackingMonitor::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#18 0x00007f665a7aea7a in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007f665a716a32 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007f665a6baa34 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007f665a6bb145 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007f665a6bb4bd in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007f665a6fe2ee in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#24 0x00007f665a6fe5a2 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#25 0x00007f665a6fe9c5 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#26 0x00007f665a6fecbb in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#27 0x00007f665a6fef2f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#28 0x00007f665a763012 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#29 0x00007f665a7638aa in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#30 0x00007f665a76a014 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#31 0x00007f665946e9b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f663af1fe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#32 0x00007f66594681a4 in tbb::internal::arena::process (this=this@entry=0x7f665790b780, s=...) at ../../src/tbb/arena.cpp:156
#33 0x00007f6659467264 in tbb::internal::market::process (this=0x7f6657927900, j=...) at ../../src/tbb/market.cpp:502
#34 0x00007f6659463736 in tbb::internal::rml::private_worker::run (this=0x7f6657747180) at ../../src/tbb/private_server.cpp:275
#35 0x00007f6659463989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#36 0x0000003e8cc07aa1 in start_thread () from /lib64/libpthread.so.0
#37 0x0000003e8c8e8aad in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f663017f700 (LWP 215087)):
#0  0x0000003e8c8df283 in poll () from /lib64/libc.so.6
#1  0x00007f66557946c4 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007f665579492a in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007f665579521e in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007f6644ec108e in ?? ()
#6  0x00007f6630179308 in ?? ()
#7  0x00007f662f723678 in ?? ()
#8  0x0000000000000000 in ?? ()
Thread 1 (Thread 0x7f6658938bc0 (LWP 213529)):
#0  0x0000003e8c8acbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003e8c8aca50 in sleep () from /lib64/libc.so.6
#2  0x00007f6655793d0e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000003e8c928bd2 in __strcmp_sse42 () from /lib64/libc.so.6
#5  0x00007f6659b58129 in TList::FindObject(char const*) const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-10-04-1100/external/slc6_amd64_gcc530/lib/libCore.so
#6  0x00007f66520df769 in TCling::GetClassSharedLibs(char const*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-10-04-1100/external/slc6_amd64_gcc530/lib/libCling.so
#7  0x00007f66520e084e in TCling::AutoLoad(char const*, bool) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-10-04-1100/external/slc6_amd64_gcc530/lib/libCling.so
#8  0x00007f6659ba0d4b in TClass::GetClass(char const*, bool, bool) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-10-04-1100/external/slc6_amd64_gcc530/lib/libCore.so
#9  0x00007f665a3b7b59 in edm::TypeWithDict::byName(std::string const&, long) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreUtilities.so
#10 0x00007f665a3b85d2 in edm::TypeWithDict::byName(std::string const&) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreUtilities.so
#11 0x00007f665a3ab5d7 in edm::FunctionWithDict::finalReturnType() const () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreUtilities.so
#12 0x00007f66462bf489 in reco::returnType(edm::FunctionWithDict const&) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#13 0x00007f66462b4b8c in reco::parser::MethodSetter::push(std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, char const*, bool) const () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#14 0x00007f664629bb50 in reco::parser::SingleInvoker::SingleInvoker(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#15 0x00007f664629cc96 in reco::parser::LazyInvoker::invoker(edm::TypeWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#16 0x00007f664629f0db in reco::parser::LazyInvoker::invokeLast(edm::ObjectWithDict const&, std::vector<edm::ObjectWithDict, std::allocator<edm::ObjectWithDict> >&) const () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#17 0x00007f66462d29ff in reco::parser::ExpressionLazyVar::value(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#18 0x00007f66462a44f6 in reco::parser::BinarySelector::operator()(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#19 0x00007f6631a5eae9 in TrackingMonitor::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#20 0x00007f665a7aea7a in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007f665a716a32 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007f665a6baa34 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007f665a6bb145 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#24 0x00007f665a6bb4bd in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#25 0x00007f665a6fe2ee in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#26 0x00007f665a6fe5a2 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#27 0x00007f665a6fe9c5 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#28 0x00007f665a6fecbb in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#29 0x00007f665a6fef2f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#30 0x00007f665a763012 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#31 0x00007f665a763747 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#32 0x00007f665a76a014 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#33 0x00007f665946e9b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f6657922600, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#34 0x00007f665a76359e in edm::EventProcessor::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#35 0x00007f665a6c599a in statemachine::HandleEvent::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#36 0x00007f665a6c6f28 in statemachine::HandleEvent::HandleEvent(boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::my_context) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#37 0x00007f665a6cec6e in boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::shallow_construct(boost::intrusive_ptr<statemachine::HandleLumis> const&, boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>&) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#38 0x00007f665a6cee3a in boost::statechart::simple_state<statemachine::FirstLumi, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#39 0x00007f665a769916 in boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&) () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#40 0x00007f665a75ccfd in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/2016-41/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-03-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#41 0x000000000040c844 in main::{lambda()#1}::operator()() const ()
#42 0x000000000040acc4 in main ()

Current Modules:

Module: TrackingMonitor (crashed)
Module: TrackingMonitor
Module: TrackingMonitor
Module: TrackingMonitor

A fatal system signal has occurred: segmentation violation

[dan131riley]

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc530/CMSSW_8_1_X_2016-10-10-1100/pyRelValMatrixLogs/run/1325.0_TTbar_13+TTbar_13INPUT+DIGIUP15+RECOUP15+HARVESTUP15+ALCATTUP15/step5_TTbar_13+TTbar_13INPUT+DIGIUP15+RECOUP15+HARVESTUP15+ALCATTUP15.log

Two threads waiting on mutex, and one unlocking:

#3  <signal handler called>
#4  0x0000003a8820e4c5 in __lll_unlock_wake () from /lib64/libpthread.so.0
#5  0x0000003a8820a947 in _L_unlock_657 () from /lib64/libpthread.so.0
#6  0x0000003a8820a8af in pthread_mutex_unlock () from /lib64/libpthread.so.0
#7  0x00007fd99f313c39 in TMethod::CreateSignature() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-10-10-1100/external/slc6_amd64_gcc530/lib/libCore.so
#8  0x00007fd99f314726 in TFunction::GetSignature() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-10-10-1100/external/slc6_amd64_gcc530/lib/libCore.so
#9  0x00007fd99f321216 in TMethodCall::Init(TFunction const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-10-10-1100/external/slc6_amd64_gcc530/lib/libCore.so
#10 0x00007fd99f3224bb in TMethodCall::TMethodCall(TFunction const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-10-10-1100/external/slc6_amd64_gcc530/lib/libCore.so
#11 0x00007fd99fb59746 in edm::FunctionWithDict::FunctionWithDict(TMethod*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-10-10-1100/lib/slc6_amd64_gcc530/libFWCoreUtilities.so

leads eventually to

https://github.com/root-mirror/root/blob/4fa59f4fb80fd914cb84144e5218e55397810c01/core/meta/src/TFunction.cxx#L132

TList *TFunction::GetListOfMethodArgs()
{
   if (!fMethodArgs && fInfo) {
      if (!gInterpreter)
         Fatal("GetListOfMethodArgs", "gInterpreter not initialized");

      gInterpreter->CreateListOfMethodArgs(this);
   }
   return fMethodArgs;
}

https://github.com/root-mirror/root/blob/418cbd5954626b1789c7b53fa3150803f1e42385/core/meta/src/TCling.cxx#L3675

void TCling::CreateListOfMethodArgs(TFunction* m) const
{
   R__LOCKGUARD2(gInterpreterMutex);
   if (m->fMethodArgs) {
      return;
   }
   m->fMethodArgs = new TList;
   TClingMethodArgInfo t(fInterpreter, (TClingMethodInfo*)m->fInfo);
   while (t.Next()) {
      if (t.IsValid()) {
         TClingMethodArgInfo* a = new TClingMethodArgInfo(t);
         m->fMethodArgs->Add(new TMethodArg((MethodArgInfo_t*)a, m));
      }
   }
}

and it looks like we have a set-before-initalized race on fMethodArgs.

Full stack trace:

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thread 5 (Thread 0x7fd98f07f700 (LWP 17855)):
#0  0x0000003a8820f37d in waitpid () from /lib64/libpthread.so.0
#1  0x00007fd99b708737 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-10-10-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007fd99b7090c2 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-10-10-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007fd99e3aaa50 in std::(anonymous namespace)::execute_native_thread_routine (__p=<optimized out>) at ../../../../../libstdc++-v3/src/c++11/thread.cc:84
#4  0x0000003a88207aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003a87ee8aad in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fd96d7ff700 (LWP 17880)):
#0  0x0000003a87eacbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003a87eaca50 in sleep () from /lib64/libc.so.6
#2  0x00007fd99b70856e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-10-10-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000003a8820e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#5  0x0000003a882095f3 in _L_lock_892 () from /lib64/libpthread.so.0
#6  0x0000003a882094d7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#7  0x00007fd9980dbf93 in TCling::GetFunctionWithPrototype(ClassInfo_t*, char const*, char const*, bool, ROOT::EFunctionMatchMode) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-10-10-1100/external/slc6_amd64_gcc530/lib/libCling.so
#8  0x00007fd99f32e544 in TClass::GetMethodWithPrototype(char const*, char const*, bool, ROOT::EFunctionMatchMode) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-10-10-1100/external/slc6_amd64_gcc530/lib/libCore.so
#9  0x00007fd99fb70b59 in edm::TypeWithDict::functionMemberByName(std::string const&, std::string const&, bool) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-10-10-1100/lib/slc6_amd64_gcc530/libFWCoreUtilities.so
#10 0x00007fd98be4a03a in reco::findMethod(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > >&, char const*, int&) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#11 0x00007fd98be0f270 in reco::parser::MethodSetter::push(std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, char const*, bool) const () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#12 0x00007fd98be63cc0 in reco::parser::SingleInvoker::SingleInvoker(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#13 0x00007fd98be64e06 in reco::parser::LazyInvoker::invoker(edm::TypeWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#14 0x00007fd98be6724b in reco::parser::LazyInvoker::invokeLast(edm::ObjectWithDict const&, std::vector<edm::ObjectWithDict, std::allocator<edm::ObjectWithDict> >&) const () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#15 0x00007fd98be5f47f in reco::parser::ExpressionLazyVar::value(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#16 0x00007fd98be38416 in reco::parser::BinarySelector::operator()(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#17 0x00007fd96f0f2ae9 in TrackingMonitor::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#18 0x00007fd99ffa13ca in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007fd99ff664f2 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007fd99fe93b44 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007fd99fe94255 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007fd99fe945cd in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007fd99feb59ce in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#24 0x00007fd99feb5c82 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#25 0x00007fd99feb60a5 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#26 0x00007fd99feb639b in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#27 0x00007fd99feb660f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#28 0x00007fd99ff40b82 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#29 0x00007fd99ff412b7 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#30 0x00007fd99ff47ce4 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#31 0x00007fd99eba09b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fd97fb87e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#32 0x00007fd99eb9a1a4 in tbb::internal::arena::process (this=this@entry=0x7fd99d9f7780, s=...) at ../../src/tbb/arena.cpp:156
#33 0x00007fd99eb99264 in tbb::internal::market::process (this=0x7fd99da1fe00, j=...) at ../../src/tbb/market.cpp:502
#34 0x00007fd99eb95736 in tbb::internal::rml::private_worker::run (this=0x7fd99da17a00) at ../../src/tbb/private_server.cpp:275
#35 0x00007fd99eb95989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#36 0x0000003a88207aa1 in start_thread () from /lib64/libpthread.so.0
#37 0x0000003a87ee8aad in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fd96cdfe700 (LWP 17881)):
#0  0x0000003a87edf283 in poll () from /lib64/libc.so.6
#1  0x00007fd99b708f24 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-10-10-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007fd99b70918a in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-10-10-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007fd99b709a7e in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-10-10-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007fd99eee208e in ?? ()
#6  0x00007fd96cdf8308 in ?? ()
#7  0x00007fd95defd870 in ?? ()
#8  0x0000000000000000 in ?? ()
Thread 2 (Thread 0x7fd96c37f700 (LWP 17882)):
#0  0x0000003a87eacbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003a87eaca50 in sleep () from /lib64/libc.so.6
#2  0x00007fd99b70856e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-10-10-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000003a8820e4c5 in __lll_unlock_wake () from /lib64/libpthread.so.0
#5  0x0000003a8820a947 in _L_unlock_657 () from /lib64/libpthread.so.0
#6  0x0000003a8820a8af in pthread_mutex_unlock () from /lib64/libpthread.so.0
#7  0x00007fd99f313c39 in TMethod::CreateSignature() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-10-10-1100/external/slc6_amd64_gcc530/lib/libCore.so
#8  0x00007fd99f314726 in TFunction::GetSignature() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-10-10-1100/external/slc6_amd64_gcc530/lib/libCore.so
#9  0x00007fd99f321216 in TMethodCall::Init(TFunction const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-10-10-1100/external/slc6_amd64_gcc530/lib/libCore.so
#10 0x00007fd99f3224bb in TMethodCall::TMethodCall(TFunction const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-10-10-1100/external/slc6_amd64_gcc530/lib/libCore.so
#11 0x00007fd99fb59746 in edm::FunctionWithDict::FunctionWithDict(TMethod*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-10-10-1100/lib/slc6_amd64_gcc530/libFWCoreUtilities.so
#12 0x00007fd98be4a463 in reco::findMethod(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > >&, char const*, int&) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#13 0x00007fd98be4a1e4 in reco::findMethod(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > >&, char const*, int&) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#14 0x00007fd98be0f270 in reco::parser::MethodSetter::push(std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, char const*, bool) const () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#15 0x00007fd98be63cc0 in reco::parser::SingleInvoker::SingleInvoker(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#16 0x00007fd98be64e06 in reco::parser::LazyInvoker::invoker(edm::TypeWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#17 0x00007fd98be6724b in reco::parser::LazyInvoker::invokeLast(edm::ObjectWithDict const&, std::vector<edm::ObjectWithDict, std::allocator<edm::ObjectWithDict> >&) const () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#18 0x00007fd98be5f47f in reco::parser::ExpressionLazyVar::value(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#19 0x00007fd98be38416 in reco::parser::BinarySelector::operator()(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#20 0x00007fd96f0f2ae9 in TrackingMonitor::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#21 0x00007fd99ffa13ca in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007fd99ff664f2 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007fd99fe93b44 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#24 0x00007fd99fe94255 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#25 0x00007fd99fe945cd in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#26 0x00007fd99feb59ce in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#27 0x00007fd99feb5c82 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#28 0x00007fd99feb60a5 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#29 0x00007fd99feb639b in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#30 0x00007fd99feb660f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#31 0x00007fd99ff40b82 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#32 0x00007fd99ff4141a in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#33 0x00007fd99ff47ce4 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#34 0x00007fd99eba09b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fd97fb6fe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#35 0x00007fd99eb9a1a4 in tbb::internal::arena::process (this=this@entry=0x7fd99d9f7780, s=...) at ../../src/tbb/arena.cpp:156
#36 0x00007fd99eb99264 in tbb::internal::market::process (this=0x7fd99da1fe00, j=...) at ../../src/tbb/market.cpp:502
#37 0x00007fd99eb95736 in tbb::internal::rml::private_worker::run (this=0x7fd99da17980) at ../../src/tbb/private_server.cpp:275
#38 0x00007fd99eb95989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#39 0x0000003a88207aa1 in start_thread () from /lib64/libpthread.so.0
#40 0x0000003a87ee8aad in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fd99fc90bc0 (LWP 17804)):
#0  0x0000003a87eacbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003a87eaca50 in sleep () from /lib64/libc.so.6
#2  0x00007fd99b70856e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-10-10-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000003a8820e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#5  0x0000003a882095f3 in _L_lock_892 () from /lib64/libpthread.so.0
#6  0x0000003a882094d7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#7  0x00007fd99f31f884 in TListOfFunctionsIter::Next() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-10-10-1100/external/slc6_amd64_gcc530/lib/libCore.so
#8  0x00007fd99fb7fa69 in edm::IterWithDictBase::advance() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-10-10-1100/lib/slc6_amd64_gcc530/libFWCoreUtilities.so
#9  0x00007fd98be4a34d in reco::findMethod(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > >&, char const*, int&) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#10 0x00007fd98be4a1e4 in reco::findMethod(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > >&, char const*, int&) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#11 0x00007fd98be0f270 in reco::parser::MethodSetter::push(std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&, char const*, bool) const () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#12 0x00007fd98be63cc0 in reco::parser::SingleInvoker::SingleInvoker(edm::TypeWithDict const&, std::string const&, std::vector<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string>, std::allocator<boost::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::string> > > const&) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#13 0x00007fd98be64e06 in reco::parser::LazyInvoker::invoker(edm::TypeWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#14 0x00007fd98be6724b in reco::parser::LazyInvoker::invokeLast(edm::ObjectWithDict const&, std::vector<edm::ObjectWithDict, std::allocator<edm::ObjectWithDict> >&) const () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#15 0x00007fd98be5f47f in reco::parser::ExpressionLazyVar::value(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#16 0x00007fd98be38416 in reco::parser::BinarySelector::operator()(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#17 0x00007fd96f0f2ae9 in TrackingMonitor::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#18 0x00007fd99ffa13ca in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007fd99ff664f2 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007fd99fe93b44 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007fd99fe94255 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007fd99fe945cd in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007fd99feb59ce in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#24 0x00007fd99feb5c82 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#25 0x00007fd99feb60a5 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#26 0x00007fd99feb639b in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#27 0x00007fd99feb660f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#28 0x00007fd99ff40b82 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#29 0x00007fd99ff412b7 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#30 0x00007fd99ff47ce4 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#31 0x00007fd99eba09b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fd99da1e400, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#32 0x00007fd99ff4110e in edm::EventProcessor::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#33 0x00007fd99fe98a7a in statemachine::HandleEvent::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#34 0x00007fd99fe9a008 in statemachine::HandleEvent::HandleEvent(boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::my_context) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#35 0x00007fd99fea1d8e in boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::shallow_construct(boost::intrusive_ptr<statemachine::HandleLumis> const&, boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>&) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#36 0x00007fd99fea1f5a in boost::statechart::simple_state<statemachine::FirstLumi, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#37 0x00007fd99ff475e6 in boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&) () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#38 0x00007fd99ff3a86d in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/2016-42/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-10-09-0000/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#39 0x000000000040c844 in main::{lambda()#1}::operator()() const ()
#40 0x000000000040acc4 in main ()

Current Modules:

Module: TrackingMonitor (crashed)
Module: TrackingMonitor
Module: TrackingMonitor
Module: TrackingMonitor

A fatal system signal has occurred: segmentation violation

[dan131riley]

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc530/CMSSW_8_1_DEVEL_X_2016-10-23-1100/pyRelValMatrixLogs/run/10024.0_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2017_GenSimFullINPUT+DigiFull_2017+RecoFull_2017+ALCAFull_2017+HARVESTFull_2017/step4_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2017_GenSimFullINPUT+DigiFull_2017+RecoFull_2017+ALCAFull_2017+HARVESTFull_2017.log

AFAIK this doesn't yet have the fMethodArgs bug fix (it looks like cms-sw/root and root.spec were just updated to add it, after this IB), but this crash is still interesting because it isn't all "???"--there's a stack trace with "__cf_16" crashing, which I'm pretty sure is cling-generated:

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thread 5 (Thread 0x7fafb48bf700 (LWP 18182)):
#0  0x0000003fccadf283 in poll () from /lib64/libc.so.6
#1  0x00007fafe1ed0f64 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007fafe1ed11ca in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007fafe1ed2782 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007fafc870208e in __cf_16 ()
#6  0x00007fafd2ab6f59 in reco::parser::MethodInvoker::invoke(edm::ObjectWithDict const&, edm::ObjectWithDict&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#7  0x00007fafd2ab7c92 in reco::parser::SingleInvoker::invoke(edm::ObjectWithDict const&, std::vector<edm::ObjectWithDict, std::allocator<edm::ObjectWithDict> >&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#8  0x00007fafd2abb047 in reco::parser::LazyInvoker::invokeLast(edm::ObjectWithDict const&, std::vector<edm::ObjectWithDict, std::allocator<edm::ObjectWithDict> >&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#9  0x00007fafd2ae1b7f in reco::parser::ExpressionLazyVar::value(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#10 0x00007fafd2abe116 in reco::parser::BinarySelector::operator()(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#11 0x00007fafb69deb99 in TrackingMonitor::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#12 0x00007fafe6d62eca in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#13 0x00007fafe6ca9932 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#14 0x00007fafe6c50965 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) [clone .isra.79] [clone .constprop.155] () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#15 0x00007fafe6c58347 in decltype ({parm#1}()) edm::convertException::wrap<void edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007fafe6c5847c in void edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007fafe6c587b9 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007fafe5a208f3 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fafb8d63e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:495
#19 0x00007fafe6cb5913 in edm::StreamSchedule::processOneEvent(edm::EventPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007fafe6d03f7c in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007fafe6d047fa in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007fafe6d0b1e4 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007fafe5a208f3 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fafb8d63e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:495
#24 0x00007fafe5a19a86 in tbb::internal::arena::process (this=0x7fafe407b780, s=...) at ../../src/tbb/arena.cpp:160
#25 0x00007fafe5a18524 in tbb::internal::market::process (this=0x7fafe4097c00, j=...) at ../../src/tbb/market.cpp:677
#26 0x00007fafe5a148a6 in tbb::internal::rml::private_worker::run (this=0x7fafe3eb6080) at ../../src/tbb/private_server.cpp:275
#27 0x00007fafe5a14af9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#28 0x0000003fcce07aa1 in start_thread () from /lib64/libpthread.so.0
#29 0x0000003fccae8aad in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fafb533e700 (LWP 18181)):
#0  0x0000003fccaacbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003fccaaca50 in sleep () from /lib64/libc.so.6
#2  0x00007fafe1ed05ae in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000003fcca89849 in memcpy () from /lib64/libc.so.6
#5  0x00007fafe6125b75 in TArrayD::Set(int) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/external/slc6_amd64_gcc530/lib/libCore.so
#6  0x00007fafe108c937 in TH1::Copy(TObject&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/external/slc6_amd64_gcc530/lib/libHist.so
#7  0x00007fafe11acb79 in TProfile::Copy(TObject&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/external/slc6_amd64_gcc530/lib/libHist.so
#8  0x00007fafe11b3e6a in TProfile* TProfileHelper::ExtendAxis<TProfile>(TProfile*, double, TAxis*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/external/slc6_amd64_gcc530/lib/libHist.so
#9  0x00007fafe11b17b9 in TProfile::ExtendAxis(double, TAxis*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/external/slc6_amd64_gcc530/lib/libHist.so
#10 0x00007fafe11367a0 in TAxis::FindBin(double) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/external/slc6_amd64_gcc530/lib/libHist.so
#11 0x00007fafe11b04f9 in TProfile::Fill(double, double, double) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/external/slc6_amd64_gcc530/lib/libHist.so
#12 0x00007fafb69deebd in TrackingMonitor::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/pluginDQMTrackingMonitor.so
#13 0x00007fafe6d62eca in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#14 0x00007fafe6ca9932 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#15 0x00007fafe6c50965 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) [clone .isra.79] [clone .constprop.155] () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007fafe6c58347 in decltype ({parm#1}()) edm::convertException::wrap<void edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007fafe6c5847c in void edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007fafe6c587b9 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007fafe5a208f3 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fafb902be00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:495
#20 0x00007fafe6cb5913 in edm::StreamSchedule::processOneEvent(edm::EventPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007fafe6d03f7c in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007fafe6d04697 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007fafe6d0b1e4 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#24 0x00007fafe5a208f3 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fafb902be00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:495
#25 0x00007fafe5a19a86 in tbb::internal::arena::process (this=0x7fafe407b780, s=...) at ../../src/tbb/arena.cpp:160
#26 0x00007fafe5a18524 in tbb::internal::market::process (this=0x7fafe4097c00, j=...) at ../../src/tbb/market.cpp:677
#27 0x00007fafe5a148a6 in tbb::internal::rml::private_worker::run (this=0x7fafe3eb6180) at ../../src/tbb/private_server.cpp:275
#28 0x00007fafe5a14af9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#29 0x0000003fcce07aa1 in start_thread () from /lib64/libpthread.so.0
#30 0x0000003fccae8aad in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fafb5d3f700 (LWP 18180)):
#0  0x0000003fccaacbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003fccaaca50 in sleep () from /lib64/libc.so.6
#2  0x00007fafe1ed05ae in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000003fcce0e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#5  0x0000003fcce095f3 in _L_lock_892 () from /lib64/libpthread.so.0
#6  0x0000003fcce094d7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#7  0x00007fafe6c53194 in edm::InputProductResolver::resolveProduct_(edm::Principal const&, bool, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#8  0x00007fafe6cc209b in edm::EventPrincipal::getByProductID(edm::ProductID const&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#9  0x00007fafe6cc2328 in edm::EventPrincipal::getIt(edm::ProductID const&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#10 0x00007fafe6f3e58b in edm::RefCore::tryToGetProductPtr(std::type_info const&, edm::EDProductGetter const*) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libDataFormatsCommon.so
#11 0x00007fafc34e4d99 in void helper::ClusterStorer::processClusters<SiPixelRecHit, SiPixelCluster>(std::vector<helper::ClusterStorer::ClusterHitRecord<SiPixelRecHit::ClusterRef>, std::allocator<helper::ClusterStorer::ClusterHitRecord<SiPixelRecHit::ClusterRef> > >&, edmNew::DetSetVector<SiPixelCluster>&, edm::RefProd<edmNew::DetSetVector<SiPixelCluster> >&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libCommonToolsRecoAlgos.so
#12 0x00007fafc34e0654 in helper::ClusterStorer::processAllClusters(edmNew::DetSetVector<SiPixelCluster>&, edm::RefProd<edmNew::DetSetVector<SiPixelCluster> >, edmNew::DetSetVector<SiStripCluster>&, edm::RefProd<edmNew::DetSetVector<SiStripCluster> >) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libCommonToolsRecoAlgos.so
#13 0x00007fafb9689c40 in void helper::TrackCollectionStoreManager::cloneAndStore<__gnu_cxx::__normal_iterator<reco::Track const* const*, std::vector<reco::Track const*, std::allocator<reco::Track const*> > > >(__gnu_cxx::__normal_iterator<reco::Track const* const*, std::vector<reco::Track const*, std::allocator<reco::Track const*> > > const&, __gnu_cxx::__normal_iterator<reco::Track const* const*, std::vector<reco::Track const*, std::allocator<reco::Track const*> > > const&, edm::Event&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/pluginCommonToolsRecoAlgos_plugins.so
#14 0x00007fafb8c4d436 in ObjectSelector<TrackConfigSelector, std::vector<reco::Track, std::allocator<reco::Track> >, NonNullNumberSelector, helper::NullPostProcessor<std::vector<reco::Track, std::allocator<reco::Track> >, edm::stream::EDFilter<> >, helper::TrackCollectionStoreManager, helper::TrackSelectorBase, reco::modules::NoEventSetupInit<TrackConfigSelector> >::filter(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/pluginCommonAlignmentProducerSelectors.so
#15 0x00007fafe6d61706 in edm::stream::EDFilterAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007fafe6ca9a62 in edm::WorkerT<edm::stream::EDFilterAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007fafe6c50965 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) [clone .isra.79] [clone .constprop.155] () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007fafe6c58347 in decltype ({parm#1}()) edm::convertException::wrap<void edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007fafe6c5847c in void edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007fafe6c587b9 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007fafe5a208f3 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fafb9023e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:495
#22 0x00007fafe6cb5913 in edm::StreamSchedule::processOneEvent(edm::EventPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007fafe6d03f7c in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#24 0x00007fafe6d04697 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#25 0x00007fafe6d0b1e4 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#26 0x00007fafe5a208f3 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fafb9023e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:495
#27 0x00007fafe5a19a86 in tbb::internal::arena::process (this=0x7fafe407b780, s=...) at ../../src/tbb/arena.cpp:160
#28 0x00007fafe5a18524 in tbb::internal::market::process (this=0x7fafe4097c00, j=...) at ../../src/tbb/market.cpp:677
#29 0x00007fafe5a148a6 in tbb::internal::rml::private_worker::run (this=0x7fafe3eb6100) at ../../src/tbb/private_server.cpp:275
#30 0x00007fafe5a14af9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#31 0x0000003fcce07aa1 in start_thread () from /lib64/libpthread.so.0
#32 0x0000003fccae8aad in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fafd5a7f700 (LWP 18166)):
#0  0x0000003fcce0f37d in waitpid () from /lib64/libpthread.so.0
#1  0x00007fafe1ed0777 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007fafe1ed1102 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007fafe5458a50 in std::(anonymous namespace)::execute_native_thread_routine (__p=<optimized out>) at ../../../../../libstdc++-v3/src/c++11/thread.cc:84
#4  0x0000003fcce07aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003fccae8aad in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fafe50f1bc0 (LWP 18149)):
#0  0x0000003fccaacbdd in nanosleep () from /lib64/libc.so.6
#1  0x0000003fccaaca50 in sleep () from /lib64/libc.so.6
#2  0x00007fafe1ed05ae in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007fafe5558375 in ?? () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/external/slc6_amd64_gcc530/lib/libz.so.1
#5  0x00007fafe555b9cb in inflate () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/external/slc6_amd64_gcc530/lib/libz.so.1
#6  0x00007fafe612a567 in R__unzip () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/external/slc6_amd64_gcc530/lib/libCore.so
#7  0x00007fafe68a4b19 in TBasket::ReadBasketBuffers(long long, int, TFile*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/external/slc6_amd64_gcc530/lib/libTree.so
#8  0x00007fafe68955df in TBranch::GetBasket(int) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/external/slc6_amd64_gcc530/lib/libTree.so
#9  0x00007fafe6895b63 in TBranch::GetEntry(long long, int) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/external/slc6_amd64_gcc530/lib/libTree.so
#10 0x00007fafe687ca30 in TBranchElement::GetEntry(long long, int) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/external/slc6_amd64_gcc530/lib/libTree.so
#11 0x00007fafe687c9d5 in TBranchElement::GetEntry(long long, int) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/external/slc6_amd64_gcc530/lib/libTree.so
#12 0x00007fafcc9ff71f in edm::RootTree::getEntry(TBranch*, long long) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/pluginIOPoolInput.so
#13 0x00007fafcca08b69 in edm::RootDelayedReader::getProduct_(edm::BranchKey const&, edm::EDProductGetter const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/pluginIOPoolInput.so
#14 0x00007fafe6c6fec9 in edm::DelayedReader::getProduct(edm::BranchKey const&, edm::EDProductGetter const*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#15 0x00007fafe6c53325 in edm::InputProductResolver::resolveProduct_(edm::Principal const&, bool, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007fafe6cc209b in edm::EventPrincipal::getByProductID(edm::ProductID const&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007fafe6cc2328 in edm::EventPrincipal::getIt(edm::ProductID const&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007fafe6f3e58b in edm::RefCore::tryToGetProductPtr(std::type_info const&, edm::EDProductGetter const*) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libDataFormatsCommon.so
#19 0x00007fafc34e6c27 in void helper::ClusterStorer::processClusters<SiStripRecHit2D, SiStripCluster>(std::vector<helper::ClusterStorer::ClusterHitRecord<SiStripRecHit2D::ClusterRef>, std::allocator<helper::ClusterStorer::ClusterHitRecord<SiStripRecHit2D::ClusterRef> > >&, edmNew::DetSetVector<SiStripCluster>&, edm::RefProd<edmNew::DetSetVector<SiStripCluster> >&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libCommonToolsRecoAlgos.so
#20 0x00007fafb9689c40 in void helper::TrackCollectionStoreManager::cloneAndStore<__gnu_cxx::__normal_iterator<reco::Track const* const*, std::vector<reco::Track const*, std::allocator<reco::Track const*> > > >(__gnu_cxx::__normal_iterator<reco::Track const* const*, std::vector<reco::Track const*, std::allocator<reco::Track const*> > > const&, __gnu_cxx::__normal_iterator<reco::Track const* const*, std::vector<reco::Track const*, std::allocator<reco::Track const*> > > const&, edm::Event&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/pluginCommonToolsRecoAlgos_plugins.so
#21 0x00007fafb8c4d436 in ObjectSelector<TrackConfigSelector, std::vector<reco::Track, std::allocator<reco::Track> >, NonNullNumberSelector, helper::NullPostProcessor<std::vector<reco::Track, std::allocator<reco::Track> >, edm::stream::EDFilter<> >, helper::TrackCollectionStoreManager, helper::TrackSelectorBase, reco::modules::NoEventSetupInit<TrackConfigSelector> >::filter(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/pluginCommonAlignmentProducerSelectors.so
#22 0x00007fafe6d61706 in edm::stream::EDFilterAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007fafe6ca9a62 in edm::WorkerT<edm::stream::EDFilterAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#24 0x00007fafe6c50965 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) [clone .isra.79] [clone .constprop.155] () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#25 0x00007fafe6c58347 in decltype ({parm#1}()) edm::convertException::wrap<void edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#26 0x00007fafe6c5847c in void edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#27 0x00007fafe6c587b9 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#28 0x00007fafe5a208f3 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fafe4086600, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:495
#29 0x00007fafe6cb5913 in edm::StreamSchedule::processOneEvent(edm::EventPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#30 0x00007fafe6d03f7c in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#31 0x00007fafe6d04697 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#32 0x00007fafe6d0b1e4 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#33 0x00007fafe5a208f3 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fafe4086600, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:495
#34 0x00007fafe6d044ee in edm::EventProcessor::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#35 0x00007fafe6c6651a in statemachine::HandleEvent::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#36 0x00007fafe6c67aa8 in statemachine::HandleEvent::HandleEvent(boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::my_context) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#37 0x00007fafe6c6f82e in boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::shallow_construct(boost::intrusive_ptr<statemachine::HandleLumis> const&, boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#38 0x00007fafe6c6f9fa in boost::statechart::simple_state<statemachine::FirstLumi, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#39 0x00007fafe6d0aae6 in boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#40 0x00007fafe6cfdcdd in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-10-23-1100/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#41 0x000000000040c844 in main::{lambda()#1}::operator()() const ()
#42 0x000000000040acc4 in main ()

Current Modules:

Module: TrackingMonitor (crashed)
Module: AlignmentTrackSelectorModule
Module: TrackingMonitor
Module: AlignmentTrackSelectorModule

A fatal system signal has occurred: segmentation violation

[Dr15Jones]

I talked with Shahzad today about this (I'm at CERN) and he's updating our version of ROOT to include the needed fix.

[dan131riley]

This bug hasn't recurred in a month, so I think it is safe to conclude that the fMethodArgs bug was the cause, and it is now fixed.

[dan131riley]

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc700/CMSSW_9_3_X_2017-07-19-2300/pyRelValMatrixLogs/run/45.0_ZpTT_1500_8TeV+ZpTT_1500_8TeVINPUT+DIGI+RECO+HARVEST/step3_ZpTT_1500_8TeV+ZpTT_1500_8TeVINPUT+DIGI+RECO+HARVEST.log

Superficially looks like the previous crashes, so maybe still a race condition somewhere that's harder to hit?

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thread 5 (Thread 0x7f1862203700 (LWP 19247)):
#0  0x000000334fcdf383 in poll () from /lib64/libc.so.6
#1  0x00007f18bc3f8b37 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/pluginFWCoreServicesPlugins.so
#2  0x00007f18bc3f910c in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  0x00007f18bc3f9fcc in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007f187c694000 in ?? ()
#6  0x00007f18b2bdb637 in reco::parser::MethodInvoker::invoke(edm::ObjectWithDict const&, edm::ObjectWithDict&) const () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libCommonToolsUtils.so
#7  0x00007f18b2bb5c39 in reco::parser::ExpressionVar::value(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libCommonToolsUtils.so
#8  0x00007f18b2be2f06 in reco::parser::BinarySelector::operator()(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libCommonToolsUtils.so
#9  0x00007f18b2be398e in reco::parser::LogicalUnaryOperator<std::logical_not<bool> >::operator()(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libCommonToolsUtils.so
#10 0x00007f18b2c06a66 in reco::parser::LogicalBinaryOperator<std::logical_and<bool> >::operator()(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libCommonToolsUtils.so
#11 0x00007f187c7311d7 in ObjectSelector<SingleElementCollectionSelector<edm::View<reco::GsfElectron>, StringCutObjectSelector<reco::GsfElectron, false>, std::vector<reco::GsfElectron, std::allocator<reco::GsfElectron> >, std::vector<reco::GsfElectron const*, std::allocator<reco::GsfElectron const*> >, helper::SelectionPointerAdder<std::vector<reco::GsfElectron const*, std::allocator<reco::GsfElectron const*> > > >, std::vector<reco::GsfElectron, std::allocator<reco::GsfElectron> >, NonNullNumberSelector, helper::NullPostProcessor<std::vector<reco::GsfElectron, std::allocator<reco::GsfElectron> >, edm::stream::EDFilter<> >, helper::CollectionStoreManager<std::vector<reco::GsfElectron, std::allocator<reco::GsfElectron> >, helper::IteratorToObjectConverter<std::vector<reco::GsfElectron, std::allocator<reco::GsfElectron> > > >, helper::ObjectSelectorBase<std::vector<reco::GsfElectron, std::allocator<reco::GsfElectron> >, edm::stream::EDFilter<> >, reco::modules::SingleElementCollectionSelectorEventSetupInit<SingleElementCollectionSelector<edm::View<reco::GsfElectron>, StringCutObjectSelector<reco::GsfElectron, false>, std::vector<reco::GsfElectron, std::allocator<reco::GsfElectron> >, std::vector<reco::GsfElectron const*, std::allocator<reco::GsfElectron const*> >, helper::SelectionPointerAdder<std::vector<reco::GsfElectron const*, std::allocator<reco::GsfElectron const*> > > > > >::filter(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/pluginValidationRecoTauPlugin.so
#12 0x00007f18c2dda3c6 in edm::stream::EDFilterAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#13 0x00007f18c2d14d22 in edm::WorkerT<edm::stream::EDFilterAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#14 0x00007f18c2cac1c7 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#15 0x00007f18c2cac37d in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#16 0x00007f18c2cac70b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#17 0x00007f18c2cacbb5 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#18 0x00007f18c16988ae in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f1870d7fe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:501
#19 0x00007f18c1691c18 in tbb::internal::arena::process (this=0x7f18c0107780, s=...) at ../../src/tbb/arena.cpp:159
#20 0x00007f18c16906d3 in tbb::internal::market::process (this=0x7f18c0123e00, j=...) at ../../src/tbb/market.cpp:697
#21 0x00007f18c168c925 in tbb::internal::rml::private_worker::run (this=0x7f18c011b980) at ../../src/tbb/private_server.cpp:271
#22 0x00007f18c168cb69 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:224
#23 0x0000003350007aa1 in start_thread () from /lib64/libpthread.so.0
#24 0x000000334fce8bcd in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f1863404700 (LWP 19246)):
#0  0x000000334fcacc4d in nanosleep () from /lib64/libc.so.6
#1  0x000000334fcacac0 in sleep () from /lib64/libc.so.6
#2  0x00007f18bc3f83f1 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f18c18d74b1 in isalloc (tsdn=0x7f18633fdb68, ptr=0x7f17a6350680) at include/jemalloc/internal/jemalloc_internal_inlines_c.h:17
#5  0x00007f18c18dfa4f in sdallocx (ptr=0x7f17a6350680, size=616, flags=0) at src/jemalloc.c:2842
#6  0x00007f18c194c9c9 in operator delete (ptr=0x7f17a6350680, size=616) at src/jemalloc_cpp.cpp:122
#7  0x00007f18a1a171a4 in KalmanVertexTrackCompatibilityEstimator<5u>::estimate(CachingVertex<5u> const&, ReferenceCountingPointer<LinearizedTrackState<5u> >, unsigned int) const () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libRecoVertexKalmanVertexFit.so
#8  0x00007f18a1a308c9 in AdaptiveVertexFitter::reWeightTracks(std::vector<ReferenceCountingPointer<LinearizedTrackState<5u> >, std::allocator<ReferenceCountingPointer<LinearizedTrackState<5u> > > > const&, CachingVertex<5u> const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libRecoVertexAdaptiveVertexFit.so
#9  0x00007f18a1a310e3 in AdaptiveVertexFitter::reWeightTracks(std::vector<ReferenceCountingPointer<VertexTrack<5u> >, std::allocator<ReferenceCountingPointer<VertexTrack<5u> > > > const&, CachingVertex<5u> const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libRecoVertexAdaptiveVertexFit.so
#10 0x00007f18a1a333eb in AdaptiveVertexFitter::fit(std::vector<ReferenceCountingPointer<VertexTrack<5u> >, std::allocator<ReferenceCountingPointer<VertexTrack<5u> > > > const&, VertexState const&, bool) const () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libRecoVertexAdaptiveVertexFit.so
#11 0x00007f18a1a3498b in AdaptiveVertexFitter::vertex(std::vector<reco::TransientTrack, std::allocator<reco::TransientTrack> > const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libRecoVertexAdaptiveVertexFit.so
#12 0x00007f188b2be7ef in PrimaryVertexProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/pluginRecoVertexPrimaryVertexProducerPlugins.so
#13 0x00007f18c2dd84a3 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#14 0x00007f18c2d14c02 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#15 0x00007f18c2cac1c7 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#16 0x00007f18c2cac37d in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#17 0x00007f18c2cac70b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#18 0x00007f18c2cacbb5 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#19 0x00007f18c16988ae in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f1870a97e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:501
#20 0x00007f18c1691c18 in tbb::internal::arena::process (this=0x7f18c0107780, s=...) at ../../src/tbb/arena.cpp:159
#21 0x00007f18c16906d3 in tbb::internal::market::process (this=0x7f18c0123e00, j=...) at ../../src/tbb/market.cpp:697
#22 0x00007f18c168c925 in tbb::internal::rml::private_worker::run (this=0x7f18c011ba80) at ../../src/tbb/private_server.cpp:271
#23 0x00007f18c168cb69 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:224
#24 0x0000003350007aa1 in start_thread () from /lib64/libpthread.so.0
#25 0x000000334fce8bcd in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f1863e05700 (LWP 19245)):
#0  0x000000334fcacc4d in nanosleep () from /lib64/libc.so.6
#1  0x000000334fcacac0 in sleep () from /lib64/libc.so.6
#2  0x00007f18bc3f83f1 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f18c18d1fe5 in tsd_reentrancy_levelp_get (tsd=0x7f1863dfeb68) at include/jemalloc/internal/tsd.h:198
#5  0x00007f18c18d2953 in tsd_reentrancy_level_get (tsd=0x7f1863dfeb68) at include/jemalloc/internal/tsd.h:223
#6  0x00007f18c18d9c70 in check_entry_exit_locking (tsdn=0x7f1863dfeb68) at src/jemalloc.c:681
#7  0x00007f18c18dd7fc in ifree (tsd=0x7f1863dfeb68, ptr=0x7f17bddb8980, tcache=0x7f1863dfed20, slow_path=true) at src/jemalloc.c:2116
#8  0x00007f18c18de2a4 in free (ptr=0x7f17bddb8980) at src/jemalloc.c:2290
#9  0x00007f18c194c935 in operator delete (ptr=0x7f17bddb8980) at src/jemalloc_cpp.cpp:98
#10 0x00007f1884673e47 in std::_Rb_tree<CSCDetId, std::pair<CSCDetId const, GlobalErrorBase<double, ErrorMatrixTag> >, std::_Select1st<std::pair<CSCDetId const, GlobalErrorBase<double, ErrorMatrixTag> > >, std::less<CSCDetId>, std::allocator<std::pair<CSCDetId const, GlobalErrorBase<double, ErrorMatrixTag> > > >::_M_erase(std::_Rb_tree_node<std::pair<CSCDetId const, GlobalErrorBase<double, ErrorMatrixTag> > >*) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libRecoMuonGlobalTrackingTools.so
#11 0x00007f1884673e3b in std::_Rb_tree<CSCDetId, std::pair<CSCDetId const, GlobalErrorBase<double, ErrorMatrixTag> >, std::_Select1st<std::pair<CSCDetId const, GlobalErrorBase<double, ErrorMatrixTag> > >, std::less<CSCDetId>, std::allocator<std::pair<CSCDetId const, GlobalErrorBase<double, ErrorMatrixTag> > > >::_M_erase(std::_Rb_tree_node<std::pair<CSCDetId const, GlobalErrorBase<double, ErrorMatrixTag> > >*) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libRecoMuonGlobalTrackingTools.so
#12 0x00007f1884673e3b in std::_Rb_tree<CSCDetId, std::pair<CSCDetId const, GlobalErrorBase<double, ErrorMatrixTag> >, std::_Select1st<std::pair<CSCDetId const, GlobalErrorBase<double, ErrorMatrixTag> > >, std::less<CSCDetId>, std::allocator<std::pair<CSCDetId const, GlobalErrorBase<double, ErrorMatrixTag> > > >::_M_erase(std::_Rb_tree_node<std::pair<CSCDetId const, GlobalErrorBase<double, ErrorMatrixTag> > >*) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libRecoMuonGlobalTrackingTools.so
#13 0x00007f1884673e3b in std::_Rb_tree<CSCDetId, std::pair<CSCDetId const, GlobalErrorBase<double, ErrorMatrixTag> >, std::_Select1st<std::pair<CSCDetId const, GlobalErrorBase<double, ErrorMatrixTag> > >, std::less<CSCDetId>, std::allocator<std::pair<CSCDetId const, GlobalErrorBase<double, ErrorMatrixTag> > > >::_M_erase(std::_Rb_tree_node<std::pair<CSCDetId const, GlobalErrorBase<double, ErrorMatrixTag> > >*) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libRecoMuonGlobalTrackingTools.so
#14 0x00007f1884673e3b in std::_Rb_tree<CSCDetId, std::pair<CSCDetId const, GlobalErrorBase<double, ErrorMatrixTag> >, std::_Select1st<std::pair<CSCDetId const, GlobalErrorBase<double, ErrorMatrixTag> > >, std::less<CSCDetId>, std::allocator<std::pair<CSCDetId const, GlobalErrorBase<double, ErrorMatrixTag> > > >::_M_erase(std::_Rb_tree_node<std::pair<CSCDetId const, GlobalErrorBase<double, ErrorMatrixTag> > >*) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libRecoMuonGlobalTrackingTools.so
#15 0x00007f1884673e3b in std::_Rb_tree<CSCDetId, std::pair<CSCDetId const, GlobalErrorBase<double, ErrorMatrixTag> >, std::_Select1st<std::pair<CSCDetId const, GlobalErrorBase<double, ErrorMatrixTag> > >, std::less<CSCDetId>, std::allocator<std::pair<CSCDetId const, GlobalErrorBase<double, ErrorMatrixTag> > > >::_M_erase(std::_Rb_tree_node<std::pair<CSCDetId const, GlobalErrorBase<double, ErrorMatrixTag> > >*) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libRecoMuonGlobalTrackingTools.so
#16 0x00007f1884673e3b in std::_Rb_tree<CSCDetId, std::pair<CSCDetId const, GlobalErrorBase<double, ErrorMatrixTag> >, std::_Select1st<std::pair<CSCDetId const, GlobalErrorBase<double, ErrorMatrixTag> > >, std::less<CSCDetId>, std::allocator<std::pair<CSCDetId const, GlobalErrorBase<double, ErrorMatrixTag> > > >::_M_erase(std::_Rb_tree_node<std::pair<CSCDetId const, GlobalErrorBase<double, ErrorMatrixTag> > >*) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libRecoMuonGlobalTrackingTools.so
#17 0x00007f1884673e3b in std::_Rb_tree<CSCDetId, std::pair<CSCDetId const, GlobalErrorBase<double, ErrorMatrixTag> >, std::_Select1st<std::pair<CSCDetId const, GlobalErrorBase<double, ErrorMatrixTag> > >, std::less<CSCDetId>, std::allocator<std::pair<CSCDetId const, GlobalErrorBase<double, ErrorMatrixTag> > > >::_M_erase(std::_Rb_tree_node<std::pair<CSCDetId const, GlobalErrorBase<double, ErrorMatrixTag> > >*) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libRecoMuonGlobalTrackingTools.so
#18 0x00007f1884673e3b in std::_Rb_tree<CSCDetId, std::pair<CSCDetId const, GlobalErrorBase<double, ErrorMatrixTag> >, std::_Select1st<std::pair<CSCDetId const, GlobalErrorBase<double, ErrorMatrixTag> > >, std::less<CSCDetId>, std::allocator<std::pair<CSCDetId const, GlobalErrorBase<double, ErrorMatrixTag> > > >::_M_erase(std::_Rb_tree_node<std::pair<CSCDetId const, GlobalErrorBase<double, ErrorMatrixTag> > >*) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libRecoMuonGlobalTrackingTools.so
#19 0x00007f1884672871 in ThrParameters::~ThrParameters() () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libRecoMuonGlobalTrackingTools.so
#20 0x00007f1884676014 in DynamicTruncation::~DynamicTruncation() () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libRecoMuonGlobalTrackingTools.so
#21 0x00007f188466b2ae in GlobalMuonRefitter::refit(reco::Track const&, reco::TransientTrack, std::vector<std::shared_ptr<TrackingRecHit const>, std::allocator<std::shared_ptr<TrackingRecHit const> > > const&, int, TrackerTopology const*) const () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libRecoMuonGlobalTrackingTools.so
#22 0x00007f188466bc25 in GlobalMuonRefitter::refit(reco::Track const&, int, TrackerTopology const*) const () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libRecoMuonGlobalTrackingTools.so
#23 0x00007f18846c6bbb in TevMuonProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/pluginRecoMuonGlobalMuonProducer.so
#24 0x00007f18c2dd84a3 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#25 0x00007f18c2d14c02 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#26 0x00007f18c2cac1c7 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#27 0x00007f18c2cac37d in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#28 0x00007f18c2cac70b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#29 0x00007f18c2cacbb5 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#30 0x00007f18c16988ae in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f1870c57e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:501
#31 0x00007f18c1691c18 in tbb::internal::arena::process (this=0x7f18c0107780, s=...) at ../../src/tbb/arena.cpp:159
#32 0x00007f18c16906d3 in tbb::internal::market::process (this=0x7f18c0123e00, j=...) at ../../src/tbb/market.cpp:697
#33 0x00007f18c168c925 in tbb::internal::rml::private_worker::run (this=0x7f18c011ba00) at ../../src/tbb/private_server.cpp:271
#34 0x00007f18c168cb69 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:224
#35 0x0000003350007aa1 in start_thread () from /lib64/libpthread.so.0
#36 0x000000334fce8bcd in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f18a34ee700 (LWP 19224)):
#0  0x000000335000f37d in waitpid () from /lib64/libpthread.so.0
#1  0x00007f18bc3f8677 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/pluginFWCoreServicesPlugins.so
#2  0x00007f18bc3f8caa in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  0x00007f18c0f85c2f in std::execute_native_thread_routine (__p=0x7f18bcc7bae0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#4  0x0000003350007aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x000000334fce8bcd in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f18c0c3b360 (LWP 19203)):
#0  0x000000334fcacc4d in nanosleep () from /lib64/libc.so.6
#1  0x000000334fcacac0 in sleep () from /lib64/libc.so.6
#2  0x00007f18bc3f83f1 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f18a0c49cd6 in CaloDetIdAssociator::crossedElement(Point3DBase<float, GlobalTag> const&, Point3DBase<float, GlobalTag> const&, DetId const&, double, SteppingHelixStateInfo const*) const () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/pluginTrackingToolsTrackAssociatorPlugins.so
#5  0x00007f18a0c1bfa2 in DetIdAssociator::getCrossedDetIds(std::set<DetId, std::less<DetId>, std::allocator<DetId> > const&, std::vector<Point3DBase<float, GlobalTag>, std::allocator<Point3DBase<float, GlobalTag> > > const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libTrackingToolsTrackAssociator.so
#6  0x00007f18a0c0cc47 in TrackDetectorAssociator::fillCaloTowers(edm::Event const&, TrackDetMatchInfo&, TrackAssociatorParameters const&) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libTrackingToolsTrackAssociator.so
#7  0x00007f18a0c107f2 in TrackDetectorAssociator::associate(edm::Event const&, edm::EventSetup const&, TrackAssociatorParameters const&, FreeTrajectoryState const*, FreeTrajectoryState const*) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libTrackingToolsTrackAssociator.so
#8  0x00007f18a0c10888 in TrackDetectorAssociator::associate(edm::Event const&, edm::EventSetup const&, FreeTrajectoryState const&, TrackAssociatorParameters const&) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libTrackingToolsTrackAssociator.so
#9  0x00007f18845e69c4 in muonisolation::CaloExtractorByAssociator::deposits(edm::Event const&, edm::EventSetup const&, reco::Track const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/pluginRecoMuonMuonIsolationPlugins.so
#10 0x00007f188a2f56b9 in MuonIdProducer::fillMuonIsolation(edm::Event&, edm::EventSetup const&, reco::Muon&, reco::IsoDeposit&, reco::IsoDeposit&, reco::IsoDeposit&, reco::IsoDeposit&, reco::IsoDeposit&) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/pluginRecoMuonMuonIdentificationPlugins.so
#11 0x00007f188a2fbb25 in MuonIdProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/pluginRecoMuonMuonIdentificationPlugins.so
#12 0x00007f18c2dd84a3 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#13 0x00007f18c2d14c02 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#14 0x00007f18c2cac1c7 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#15 0x00007f18c2cac37d in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#16 0x00007f18c2cac70b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#17 0x00007f18c2cacbb5 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#18 0x00007f18c16988ae in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f18c0122400, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:501
#19 0x00007f18c2d591ea in edm::EventProcessor::readAndProcessEvents() () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#20 0x00007f18c2d63969 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-18-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#21 0x000000000040da1a in main::{lambda()#1}::operator()() const ()
#22 0x000000000040c2ca in main ()

Current Modules:

Module: TauValElectronSelector:idElectrons (crashed)
Module: MuonIdProducer:muons1stStep
Module: TevMuonProducer:tevMuons
Module: PrimaryVertexProducer:unsortedOfflinePrimaryVertices

A fatal system signal has occurred: segmentation violation

[dan131riley]

Another one along the same lines

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc7_amd64_gcc630/CMSSW_9_3_X_2017-08-25-2300/pyRelValMatrixLogs/run/5.5_ZTT+ZTTFS+HARVESTFS/step1_ZTT+ZTTFS+HARVESTFS.log

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thread 5 (Thread 0x7f06d963f700 (LWP 17877)):
#0  0x00007f073bdedefd in nanosleep () from /lib64/libc.so.6
#1  0x00007f073bdedd94 in sleep () from /lib64/libc.so.6
#2  0x00007f0732f2a9ce in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f073e980595 in do_lookup_x () from /lib64/ld-linux-x86-64.so.2
#5  0x00007f073e980e6f in _dl_lookup_symbol_x () from /lib64/ld-linux-x86-64.so.2
#6  0x00007f073e985776 in _dl_fixup () from /lib64/ld-linux-x86-64.so.2
#7  0x00007f073e98c260 in _dl_runtime_resolve () from /lib64/ld-linux-x86-64.so.2
#8  0x00007f06df9a5571 in SingleTopTChannelLepton::MonitorEnsemble::fill(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/pluginDQMPhysics.so
#9  0x00007f06df9ab741 in SingleTopTChannelLeptonDQM::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/pluginDQMPhysics.so
#10 0x00007f073e8ea1da in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#11 0x00007f073e81c612 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#12 0x00007f073e7c6ce7 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#13 0x00007f073e7c6ebb in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#14 0x00007f073e7c83db in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#15 0x00007f073e7c8985 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#16 0x00007f073d445983 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f06e166fe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:501
#17 0x00007f073d43e9d2 in tbb::internal::arena::process (this=0x7f0739f77780, s=...) at ../../src/tbb/arena.cpp:159
#18 0x00007f073d43d4cb in tbb::internal::market::process (this=0x7f0739f97e00, j=...) at ../../src/tbb/market.cpp:677
#19 0x00007f073d4397c6 in tbb::internal::rml::private_worker::run (this=0x7f0739f8f980) at ../../src/tbb/private_server.cpp:271
#20 0x00007f073d4399f9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:224
#21 0x00007f073c0f9dc5 in start_thread () from /lib64/libpthread.so.0
#22 0x00007f073be26ced in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f06da0be700 (LWP 17876)):
#0  0x00007f073be1c69d in poll () from /lib64/libc.so.6
#1  0x00007f0732f2b044 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#2  0x00007f0732f2b81a in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  0x00007f0732f2c875 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007f06e6102000 in ?? ()
#6  0x00007f0723b7ae59 in reco::parser::MethodInvoker::invoke(edm::ObjectWithDict const&, edm::ObjectWithDict&) const () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/libCommonToolsUtils.so
#7  0x00007f0723b4ce81 in reco::parser::ExpressionVar::value(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/libCommonToolsUtils.so
#8  0x00007f0723b788fe in reco::parser::ExpressionUnaryOperator<reco::parser::abs_f>::value(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/libCommonToolsUtils.so
#9  0x00007f0723b67a09 in reco::parser::BinarySelector::operator()(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/libCommonToolsUtils.so
#10 0x00007f0723b99b36 in reco::parser::LogicalBinaryOperator<std::logical_and<bool> >::operator()(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/libCommonToolsUtils.so
#11 0x00007f0723b99b36 in reco::parser::LogicalBinaryOperator<std::logical_and<bool> >::operator()(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/libCommonToolsUtils.so
#12 0x00007f0723b99b36 in reco::parser::LogicalBinaryOperator<std::logical_and<bool> >::operator()(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/libCommonToolsUtils.so
#13 0x00007f06df91ada8 in StringCutObjectSelector<reco::PFCandidate, false>::operator()(reco::PFCandidate const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/pluginDQMPhysics.so
#14 0x00007f06df9283b7 in SelectionStep<reco::PFCandidate>::select(edm::Event const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/pluginDQMPhysics.so
#15 0x00007f06df913cef in TopSingleLeptonDQM::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/pluginDQMPhysics.so
#16 0x00007f073e8ea1da in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#17 0x00007f073e81c612 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#18 0x00007f073e7c6ce7 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#19 0x00007f073e7c6ebb in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#20 0x00007f073e7c83db in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#21 0x00007f073e7c8985 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#22 0x00007f073d445983 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f06e1677e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:501
#23 0x00007f073d43e9d2 in tbb::internal::arena::process (this=0x7f0739f77780, s=...) at ../../src/tbb/arena.cpp:159
#24 0x00007f073d43d4cb in tbb::internal::market::process (this=0x7f0739f97e00, j=...) at ../../src/tbb/market.cpp:677
#25 0x00007f073d4397c6 in tbb::internal::rml::private_worker::run (this=0x7f0739f8fa80) at ../../src/tbb/private_server.cpp:271
#26 0x00007f073d4399f9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:224
#27 0x00007f073c0f9dc5 in start_thread () from /lib64/libpthread.so.0
#28 0x00007f073be26ced in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f06daabf700 (LWP 17875)):
#0  0x00007f073bdedefd in nanosleep () from /lib64/libc.so.6
#1  0x00007f073bdedd94 in sleep () from /lib64/libc.so.6
#2  0x00007f0732f2a9ce in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f073e980595 in do_lookup_x () from /lib64/ld-linux-x86-64.so.2
#5  0x00007f073e980e6f in _dl_lookup_symbol_x () from /lib64/ld-linux-x86-64.so.2
#6  0x00007f073e985776 in _dl_fixup () from /lib64/ld-linux-x86-64.so.2
#7  0x00007f073e98c260 in _dl_runtime_resolve () from /lib64/ld-linux-x86-64.so.2
#8  0x00007f06e709741e in bool edm::Event::getByToken<reco::Vertex>(edm::EDGetTokenT<edm::View<reco::Vertex> >, edm::Handle<edm::View<reco::Vertex> >&) const () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/pluginRecoBTagSecondaryVertexProducer.so
#9  0x00007f06df9a5571 in SingleTopTChannelLepton::MonitorEnsemble::fill(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/pluginDQMPhysics.so
#10 0x00007f06df9ab741 in SingleTopTChannelLeptonDQM::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/pluginDQMPhysics.so
#11 0x00007f073e8ea1da in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#12 0x00007f073e81c612 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#13 0x00007f073e7c6ce7 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#14 0x00007f073e7c6ebb in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#15 0x00007f073e7c83db in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#16 0x00007f073e7c8985 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#17 0x00007f073d445983 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f06e18cfe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:501
#18 0x00007f073d43e9d2 in tbb::internal::arena::process (this=0x7f0739f77780, s=...) at ../../src/tbb/arena.cpp:159
#19 0x00007f073d43d4cb in tbb::internal::market::process (this=0x7f0739f97e00, j=...) at ../../src/tbb/market.cpp:677
#20 0x00007f073d4397c6 in tbb::internal::rml::private_worker::run (this=0x7f0739f8fa00) at ../../src/tbb/private_server.cpp:271
#21 0x00007f073d4399f9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:224
#22 0x00007f073c0f9dc5 in start_thread () from /lib64/libpthread.so.0
#23 0x00007f073be26ced in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f0726dbf700 (LWP 17840)):
#0  0x00007f073c100ca9 in waitpid () from /lib64/libpthread.so.0
#1  0x00007f0732f2ac57 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#2  0x00007f0732f2b1f5 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  0x00007f073c6e2c3f in std::execute_native_thread_routine (__p=0x7f0734ffffc0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#4  0x00007f073c0f9dc5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f073be26ced in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f073a4fbc80 (LWP 17781)):
#0  0x00007f073bdedefd in nanosleep () from /lib64/libc.so.6
#1  0x00007f073bdedd94 in sleep () from /lib64/libc.so.6
#2  0x00007f0732f2a9ce in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f073e8147e0 in edm::PrincipalGetAdapter::getByToken_(edm::TypeID const&, edm::KindOfType, edm::EDGetToken, edm::ModuleCallingContext const*) const () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#5  0x00007f06e709739e in bool edm::Event::getByToken<reco::Vertex>(edm::EDGetTokenT<edm::View<reco::Vertex> >, edm::Handle<edm::View<reco::Vertex> >&) const () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/pluginRecoBTagSecondaryVertexProducer.so
#6  0x00007f06df9a5571 in SingleTopTChannelLepton::MonitorEnsemble::fill(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/pluginDQMPhysics.so
#7  0x00007f06df9ab741 in SingleTopTChannelLeptonDQM::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/pluginDQMPhysics.so
#8  0x00007f073e8ea1da in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#9  0x00007f073e81c612 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#10 0x00007f073e7c6ce7 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#11 0x00007f073e7c6ebb in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#12 0x00007f073e7c83db in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#13 0x00007f073e7c8985 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#14 0x00007f073d445983 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f0739f96400, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:501
#15 0x00007f073e86c5ac in edm::EventProcessor::readAndProcessEvents() () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#16 0x00007f073e876f01 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/nweek-02486/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_X_2017-08-25-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#17 0x000000000040e6f2 in main::{lambda()#1}::operator()() const ()
#18 0x000000000040d0d5 in main ()

Current Modules:

Module: TopSingleLeptonDQM:topSingleElectronMediumDQM (crashed)
Module: SingleTopTChannelLeptonDQM:singleTopMuonMediumDQM
Module: SingleTopTChannelLeptonDQM:singleTopMuonMediumDQM
Module: SingleTopTChannelLeptonDQM:singleTopMuonMediumDQM

A fatal system signal has occurred: segmentation violation

[dan131riley]

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc7_amd64_gcc700/CMSSW_10_1_X_2018-03-06-2300/pyRelValMatrixLogs/run/500207.0_NuGun_UP15+NuGun_UP15INPUT+DIGIPRMXUP15_PU50+RECOPRMXUP15_PU50+HARVESTUP15_PU50/step3_NuGun_UP15+NuGun_UP15INPUT+DIGIPRMXUP15_PU50+RECOPRMXUP15_PU50+HARVESTUP15_PU50.log

Apparently still not quite fixed, but really rare:

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thread 6 (Thread 0x7f9c8ad44700 (LWP 25970)):
#0  0x00007f9cf00bb945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f9cf069bcec in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /build/cmsbld/auto-builds/CMSSW_10_1_0_pre2-slc7_amd64_gcc700/build/CMSSW_10_1_0_pre2-build/BUILD/slc7_amd64_gcc700/external/gcc/7.0.0-omkpbe2/gcc-branches_gcc-7-branch-257125/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/x86_64-unknown-linux-gnu/bits/gthr-default.h:864
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00007f9c8b228f82 in Eigen::NonBlockingThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_1_X_2018-03-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#4  0x00007f9c8b226767 in std::_Function_handler<void (), tensorflow::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_1_X_2018-03-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#5  0x00007f9cf06a18ff in std::execute_native_thread_routine (__p=0x7f9c8f8c6880) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#6  0x00007f9cf00b7e25 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f9cefde534d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f9cc31ff700 (LWP 25486)):
#0  0x00007f9cefdac1ad in nanosleep () from /lib64/libc.so.6
#1  0x00007f9cefdac044 in sleep () from /lib64/libc.so.6
#2  0x00007f9ce616a401 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f9cdd5ee901 in PerigeeConversions::jacobianCurvilinear2Perigee(FreeTrajectoryState const&) () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libTrackingToolsTrajectoryState.so
#5  0x00007f9cdd5eed01 in PerigeeConversions::ftsToPerigeeError(FreeTrajectoryState const&) () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libTrackingToolsTrajectoryState.so
#6  0x00007f9cdd5fce53 in TrajectoryStateClosestToPoint::TrajectoryStateClosestToPoint(FreeTrajectoryState const&, Point3DBase<float, GlobalTag> const&) () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libTrackingToolsTrajectoryState.so
#7  0x00007f9cdd5ee42b in PerigeeConversions::trajectoryStateClosestToPoint(ROOT::Math::SVector<double, 3u> const&, Point3DBase<float, GlobalTag> const&, int const&, ROOT::Math::SMatrix<double, 6u, 6u, ROOT::Math::MatRepSym<double, 6u> > const&, MagneticField const*) () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libTrackingToolsTrajectoryState.so
#8  0x00007f9ccecd4c95 in PerigeeLinearizedTrackState::createRefittedTrackState(Point3DBase<float, GlobalTag> const&, ROOT::Math::SVector<double, 3u> const&, ROOT::Math::SMatrix<double, 6u, 6u, ROOT::Math::MatRepSym<double, 6u> > const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libRecoVertexVertexTools.so
#9  0x00007f9cce7f220e in KalmanVertexTrackUpdator<5u>::trackRefit(VertexState const&, ReferenceCountingPointer<LinearizedTrackState<5u> >, float) const () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libRecoVertexKalmanVertexFit.so
#10 0x00007f9cce7f22f3 in KalmanVertexTrackUpdator<5u>::update(CachingVertex<5u> const&, ReferenceCountingPointer<VertexTrack<5u> >) const () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libRecoVertexKalmanVertexFit.so
#11 0x00007f9cce813b90 in KalmanVertexTrackCompatibilityEstimator<5u>::estimateFittedTrack(CachingVertex<5u> const&, ReferenceCountingPointer<VertexTrack<5u> >) const () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libRecoVertexKalmanVertexFit.so
#12 0x00007f9cce814f8a in KalmanVertexTrackCompatibilityEstimator<5u>::estimate(CachingVertex<5u> const&, ReferenceCountingPointer<VertexTrack<5u> >, unsigned int) const () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libRecoVertexKalmanVertexFit.so
#13 0x00007f9cce813de7 in KalmanVertexTrackCompatibilityEstimator<5u>::estimate(CachingVertex<5u> const&, ReferenceCountingPointer<LinearizedTrackState<5u> >, unsigned int) const () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libRecoVertexKalmanVertexFit.so
#14 0x00007f9cce82dd6d in AdaptiveVertexFitter::reWeightTracks(std::vector<ReferenceCountingPointer<LinearizedTrackState<5u> >, std::allocator<ReferenceCountingPointer<LinearizedTrackState<5u> > > > const&, CachingVertex<5u> const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libRecoVertexAdaptiveVertexFit.so
#15 0x00007f9cce82e523 in AdaptiveVertexFitter::reWeightTracks(std::vector<ReferenceCountingPointer<VertexTrack<5u> >, std::allocator<ReferenceCountingPointer<VertexTrack<5u> > > > const&, CachingVertex<5u> const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libRecoVertexAdaptiveVertexFit.so
#16 0x00007f9cce831b83 in AdaptiveVertexFitter::fit(std::vector<ReferenceCountingPointer<VertexTrack<5u> >, std::allocator<ReferenceCountingPointer<VertexTrack<5u> > > > const&, VertexState const&, bool) const () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libRecoVertexAdaptiveVertexFit.so
#17 0x00007f9cce832de8 in AdaptiveVertexFitter::vertex(std::vector<reco::TransientTrack, std::allocator<reco::TransientTrack> > const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libRecoVertexAdaptiveVertexFit.so
#18 0x00007f9c956ba869 in PrimaryVertexProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/pluginRecoVertexPrimaryVertexProducerPlugins.so
#19 0x00007f9cf28ffc43 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#20 0x00007f9cf283bbe2 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#21 0x00007f9cf27be00a in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#22 0x00007f9cf27be1cd in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#23 0x00007f9cf27be55b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#24 0x00007f9cf27bfc5e in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#25 0x00007f9cf13f9466 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f9cedcd3e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:509
#26 0x00007f9cf13f2dd8 in tbb::internal::arena::process (this=0x7f9ceddd2d00, s=...) at ../../src/tbb/arena.cpp:160
#27 0x00007f9cf13f18cb in tbb::internal::market::process (this=0x7f9ceddd3580, j=...) at ../../src/tbb/market.cpp:693
#28 0x00007f9cf13edb45 in tbb::internal::rml::private_worker::run (this=0x7f9cedbf5080) at ../../src/tbb/private_server.cpp:270
#29 0x00007f9cf13edd89 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#30 0x00007f9cf00b7e25 in start_thread () from /lib64/libpthread.so.0
#31 0x00007f9cefde534d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f9cc41fe700 (LWP 25485)):
#0  0x00007f9cefddaa3d in poll () from /lib64/libc.so.6
#1  0x00007f9ce616ab37 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#2  0x00007f9ce616b20c in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  0x00007f9ce616c279 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007f9c86616000 in ?? ()
#6  0x00007f9cdd50a647 in reco::parser::MethodInvoker::invoke(edm::ObjectWithDict const&, edm::ObjectWithDict&) const () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libCommonToolsUtils.so
#7  0x00007f9cdd53f4e9 in reco::parser::ExpressionVar::value(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libCommonToolsUtils.so
#8  0x00007f9cdd5123b9 in reco::parser::BinarySelector::operator()(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libCommonToolsUtils.so
#9  0x00007f9cdd53e0d6 in reco::parser::LogicalBinaryOperator<std::logical_and<bool> >::operator()(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libCommonToolsUtils.so
#10 0x00007f9cdd53e0d6 in reco::parser::LogicalBinaryOperator<std::logical_and<bool> >::operator()(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libCommonToolsUtils.so
#11 0x00007f9cdd53e0d6 in reco::parser::LogicalBinaryOperator<std::logical_and<bool> >::operator()(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libCommonToolsUtils.so
#12 0x00007f9cc221bdbf in ObjectSelector<SingleElementCollectionSelector<std::vector<reco::Vertex, std::allocator<reco::Vertex> >, StringCutObjectSelector<reco::Vertex, false>, std::vector<reco::Vertex, std::allocator<reco::Vertex> >, std::vector<reco::Vertex const*, std::allocator<reco::Vertex const*> >, helper::SelectionPointerAdder<std::vector<reco::Vertex const*, std::allocator<reco::Vertex const*> > > >, std::vector<reco::Vertex, std::allocator<reco::Vertex> >, NonNullNumberSelector, helper::NullPostProcessor<std::vector<reco::Vertex, std::allocator<reco::Vertex> >, edm::stream::EDFilter<> >, helper::CollectionStoreManager<std::vector<reco::Vertex, std::allocator<reco::Vertex> >, helper::IteratorToObjectConverter<std::vector<reco::Vertex, std::allocator<reco::Vertex> > > >, helper::ObjectSelectorBase<std::vector<reco::Vertex, std::allocator<reco::Vertex> >, edm::stream::EDFilter<> >, reco::modules::SingleElementCollectionSelectorEventSetupInit<SingleElementCollectionSelector<std::vector<reco::Vertex, std::allocator<reco::Vertex> >, StringCutObjectSelector<reco::Vertex, false>, std::vector<reco::Vertex, std::allocator<reco::Vertex> >, std::vector<reco::Vertex const*, std::allocator<reco::Vertex const*> >, helper::SelectionPointerAdder<std::vector<reco::Vertex const*, std::allocator<reco::Vertex const*> > > > > >::filter(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/pluginCommonToolsRecoAlgos_plugins.so
#13 0x00007f9cf2901e46 in edm::stream::EDFilterAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#14 0x00007f9cf283bd02 in edm::WorkerT<edm::stream::EDFilterAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#15 0x00007f9cf27be00a in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#16 0x00007f9cf27be1cd in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#17 0x00007f9cf27be55b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#18 0x00007f9cf27bfc5e in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#19 0x00007f9cf13f9466 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f9cedcebe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:509
#20 0x00007f9cf13f2dd8 in tbb::internal::arena::process (this=0x7f9ceddd2d00, s=...) at ../../src/tbb/arena.cpp:160
#21 0x00007f9cf13f18cb in tbb::internal::market::process (this=0x7f9ceddd3580, j=...) at ../../src/tbb/market.cpp:693
#22 0x00007f9cf13edb45 in tbb::internal::rml::private_worker::run (this=0x7f9cedbf5180) at ../../src/tbb/private_server.cpp:270
#23 0x00007f9cf13edd89 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#24 0x00007f9cf00b7e25 in start_thread () from /lib64/libpthread.so.0
#25 0x00007f9cefde534d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f9cc4bff700 (LWP 25484)):
#0  0x00007f9cefdac1ad in nanosleep () from /lib64/libc.so.6
#1  0x00007f9cefdac044 in sleep () from /lib64/libc.so.6
#2  0x00007f9ce616a401 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007ffe223c6a11 in clock_gettime ()
#5  0x00007f9cefdf90cd in clock_gettime () from /lib64/libc.so.6
#6  0x00007f9ce9b45ac1 in boost::chrono::thread_clock::now() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_1_X_2018-03-06-2300/external/slc7_amd64_gcc700/lib/libboost_chrono.so.1.63.0
#7  0x00007f9cdf043d53 in FastTimerService::Measurement::measure_and_accumulate(FastTimerService::AtomicResources&) () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libHLTriggerTimer.so
#8  0x00007f9cf28ffc27 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#9  0x00007f9cf283bbe2 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#10 0x00007f9cf27be00a in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#11 0x00007f9cf27be1cd in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#12 0x00007f9cf27be55b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#13 0x00007f9cf27bfc5e in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#14 0x00007f9cf13f9466 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f9cedcdbe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:509
#15 0x00007f9cf13f2dd8 in tbb::internal::arena::process (this=0x7f9ceddd2d00, s=...) at ../../src/tbb/arena.cpp:160
#16 0x00007f9cf13f18cb in tbb::internal::market::process (this=0x7f9ceddd3580, j=...) at ../../src/tbb/market.cpp:693
#17 0x00007f9cf13edb45 in tbb::internal::rml::private_worker::run (this=0x7f9cedbf5100) at ../../src/tbb/private_server.cpp:270
#18 0x00007f9cf13edd89 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#19 0x00007f9cf00b7e25 in start_thread () from /lib64/libpthread.so.0
#20 0x00007f9cefde534d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f9cd09ff700 (LWP 24292)):
#0  0x00007f9cf00bf189 in waitpid () from /lib64/libpthread.so.0
#1  0x00007f9ce616a607 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#2  0x00007f9ce616b12a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  0x00007f9cf06a18ff in std::execute_native_thread_routine (__p=0x7f9ce15ffa40) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#4  0x00007f9cf00b7e25 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f9cefde534d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f9cee4b6c80 (LWP 22603)):
#0  0x00007f9cefdac1ad in nanosleep () from /lib64/libc.so.6
#1  0x00007f9cefdac044 in sleep () from /lib64/libc.so.6
#2  0x00007f9ce616a401 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f9c9c88abc6 in MuonAssociatorByHitsHelper::associateRecoToSimIndices(std::vector<std::pair<__gnu_cxx::__normal_iterator<TrackingRecHit* const*, std::vector<TrackingRecHit*, std::allocator<TrackingRecHit*> > >, __gnu_cxx::__normal_iterator<TrackingRecHit* const*, std::vector<TrackingRecHit*, std::allocator<TrackingRecHit*> > > >, std::allocator<std::pair<__gnu_cxx::__normal_iterator<TrackingRecHit* const*, std::vector<TrackingRecHit*, std::allocator<TrackingRecHit*> > >, __gnu_cxx::__normal_iterator<TrackingRecHit* const*, std::vector<TrackingRecHit*, std::allocator<TrackingRecHit*> > > > > > const&, edm::RefVector<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle, edm::refhelper::FindUsingAdvance<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle> > const&, MuonAssociatorByHitsHelper::Resources const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libSimMuonMCTruth.so
#5  0x00007f9c9c85eb85 in MuonAssociatorByHits::associateRecoToSim(edm::RefToBaseVector<reco::Track> const&, edm::RefVector<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle, edm::refhelper::FindUsingAdvance<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle> > const&, edm::Event const*, edm::EventSetup const*) const () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libSimMuonMCTruth.so
#6  0x00007f9c94b24ba9 in MuonAssociatorByHits::associateRecoToSim(edm::Handle<edm::View<reco::Track> >&, edm::Handle<std::vector<TrackingParticle, std::allocator<TrackingParticle> > >&, edm::Event const*, edm::EventSetup const*) const () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/pluginSimMuonMCTruthPlugins.so
#7  0x00007f9c94b1f2dd in MuonAssociatorEDProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/pluginSimMuonMCTruthPlugins.so
#8  0x00007f9cf28ffc43 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#9  0x00007f9cf283bbe2 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#10 0x00007f9cf27be00a in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#11 0x00007f9cf27be1cd in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#12 0x00007f9cf27be55b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#13 0x00007f9cf27bfc5e in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#14 0x00007f9cf13f9466 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f9ceddc2600, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:509
#15 0x00007f9cf288b548 in edm::EventProcessor::processLumis(std::shared_ptr<void> const&) () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#16 0x00007f9cf289469a in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/nweek-02514/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-03-05-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#17 0x000000000040ea69 in main::{lambda()#1}::operator()() const ()
#18 0x000000000040d331 in main ()

Current Modules:

Module: VertexSelector:selectedVerticesForPFMEtCorrType0 (crashed)
Module: MuonAssociatorEDProducer:tpToStaSeedAssociation
Module: PrimaryVertexProducer:unsortedOfflinePrimaryVertices
Module: VertexCompositeCandidateCollectionSelector:KshortWlxy16

A fatal system signal has occurred: segmentation violation

[dan131riley]

Still seeing these at a low rate, this is partly a reminder to myself to continue looking at these, especially checking the other threads for clues.

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc700/CMSSW_10_4_X_2018-11-30-2300/pyRelValMatrixLogs/run/134.704_RunJetHT2015B+RunJetHT2015B+HLTDR2_50ns+RECODR2_50nsreHLT_HIPM+HARVESTDR2/step3_RunJetHT2015B+RunJetHT2015B+HLTDR2_50ns+RECODR2_50nsreHLT_HIPM+HARVESTDR2.log

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Sat Dec  1 10:37:40 CET 2018
Thread 6 (Thread 0x7faa86013700 (LWP 29157)):
#0  0x00000038c320b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007faaf6e66cbc in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /build/cmsbld/auto-builds/CMSSW_10_1_0_pre2-slc6_amd64_gcc700/build/CMSSW_10_1_0_pre2-build/BUILD/slc6_amd64_gcc700/external/gcc/7.0.0-omkpbe2/gcc-branches_gcc-7-branch-257125/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/x86_64-unknown-linux-gnu/bits/gthr-default.h:864
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00007faa9e7d7b3a in Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/external/slc6_amd64_gcc700/lib/libtensorflow_framework.so
#4  0x00007faa9e7d4677 in std::_Function_handler<void (), tensorflow::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/external/slc6_amd64_gcc700/lib/libtensorflow_framework.so
#5  0x00007faaf6e6c8cf in std::execute_native_thread_routine (__p=0x7faa97d619d0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#6  0x00000038c3207aa1 in start_thread () from /lib64/libpthread.so.0
#7  0x00000038c2ee8c4d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7faac57ff700 (LWP 29150)):
#0  0x00000038c2eaccbd in nanosleep () from /lib64/libc.so.6
#1  0x00000038c2eacb30 in sleep () from /lib64/libc.so.6
#2  0x00007faae6534de1 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007faad5a04968 in std::pair<std::vector<Cluster1D<void>, std::allocator<Cluster1D<void> > >::const_iterator, std::vector<Cluster1D<void>, std::allocator<Cluster1D<void> > >::const_iterator> FsmwClusterizer1DNameSpace::fsmw<void>(std::vector<Cluster1D<void>, std::allocator<Cluster1D<void> > > const&, double) () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libRecoVertexVertexTools.so
#5  0x00007faad5a05e46 in FsmwClusterizer1D<void>::operator()(std::vector<Cluster1D<void>, std::allocator<Cluster1D<void> > > const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libRecoVertexVertexTools.so
#6  0x00007faad5a039f1 in FsmwModeFinder3d::operator()(std::vector<std::pair<Point3DBase<float, GlobalTag>, float>, std::allocator<std::pair<Point3DBase<float, GlobalTag>, float> > > const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libRecoVertexVertexTools.so
#7  0x00007faad3b120bd in CrossingPtBasedLinearizationPointFinder::find(std::vector<std::pair<Point3DBase<float, GlobalTag>, float>, std::allocator<std::pair<Point3DBase<float, GlobalTag>, float> > > const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libRecoVertexLinearizationPointFinders.so
#8  0x00007faad3b13fee in CrossingPtBasedLinearizationPointFinder::getLinearizationPoint(std::vector<reco::TransientTrack, std::allocator<reco::TransientTrack> > const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libRecoVertexLinearizationPointFinders.so
#9  0x00007faad3b72dd1 in AdaptiveVertexFitter::vertex(std::vector<reco::TransientTrack, std::allocator<reco::TransientTrack> > const&, reco::BeamSpot const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libRecoVertexAdaptiveVertexFit.so
#10 0x00007faab6ee2165 in PFTauPrimaryVertexProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/pluginRecoTauTagRecoTauPlugins.so
#11 0x00007faaf8d6c393 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#12 0x00007faaf8ca8cc2 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#13 0x00007faaf8c449ea in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#14 0x00007faaf8c44bad in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#15 0x00007faaf8c462ab in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#16 0x00007faaf8c47354 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#17 0x00007faaf7579096 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7faaf5ea3e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:509
#18 0x00007faaf75728e0 in tbb::internal::arena::process (this=0x7faaf5fc2d00, s=...) at ../../src/tbb/arena.cpp:160
#19 0x00007faaf757125b in tbb::internal::market::process (this=0x7faaf5fc3580, j=...) at ../../src/tbb/market.cpp:693
#20 0x00007faaf756d4f3 in tbb::internal::rml::private_worker::run (this=0x7faaf5de8080) at ../../src/tbb/private_server.cpp:270
#21 0x00007faaf756d759 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#22 0x00000038c3207aa1 in start_thread () from /lib64/libpthread.so.0
#23 0x00000038c2ee8c4d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7faac69f6700 (LWP 29149)):
#0  0x00000038c2edf403 in poll () from /lib64/libc.so.6
#1  0x00007faae6535517 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/pluginFWCoreServicesPlugins.so
#2  0x00007faae6535bac in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  0x00007faae6536c17 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007faa9443a000 in ?? ()
#6  0x00007faae30657f9 in reco::parser::MethodInvoker::invoke(edm::ObjectWithDict const&, edm::ObjectWithDict&) const () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libCommonToolsUtils.so
#7  0x00007faae304b4d9 in reco::parser::ExpressionVar::value(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libCommonToolsUtils.so
#8  0x00007faae306cae9 in reco::parser::BinarySelector::operator()(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libCommonToolsUtils.so
#9  0x00007faab6f13e8d in PFRecoTauDiscriminationByIsolation::discriminate(edm::Ref<std::vector<reco::PFTau, std::allocator<reco::PFTau> >, reco::PFTau, edm::refhelper::FindUsingAdvance<std::vector<reco::PFTau, std::allocator<reco::PFTau> >, reco::PFTau> > const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/pluginRecoTauTagRecoTauPlugins.so
#10 0x00007faab6d1e9fa in TauDiscriminationProducerBase<reco::PFTau, reco::PFTauDiscriminator>::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libRecoTauTagRecoTau.so
#11 0x00007faaf8d6c393 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#12 0x00007faaf8ca8cc2 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#13 0x00007faaf8c449ea in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#14 0x00007faaf8c44bad in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#15 0x00007faaf8c462ab in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#16 0x00007faaf8c47354 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#17 0x00007faaf7579096 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7faaf5ec3e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:509
#18 0x00007faaf75728e0 in tbb::internal::arena::process (this=0x7faaf5fc2d00, s=...) at ../../src/tbb/arena.cpp:160
#19 0x00007faaf757125b in tbb::internal::market::process (this=0x7faaf5fc3580, j=...) at ../../src/tbb/market.cpp:693
#20 0x00007faaf756d4f3 in tbb::internal::rml::private_worker::run (this=0x7faaf5de8180) at ../../src/tbb/private_server.cpp:270
#21 0x00007faaf756d759 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#22 0x00000038c3207aa1 in start_thread () from /lib64/libpthread.so.0
#23 0x00000038c2ee8c4d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7faac73f7700 (LWP 29148)):
#0  0x00000038c2eaccbd in nanosleep () from /lib64/libc.so.6
#1  0x00000038c2eacb30 in sleep () from /lib64/libc.so.6
#2  0x00007faae6534de1 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007faad5a04986 in std::pair<std::vector<Cluster1D<void>, std::allocator<Cluster1D<void> > >::const_iterator, std::vector<Cluster1D<void>, std::allocator<Cluster1D<void> > >::const_iterator> FsmwClusterizer1DNameSpace::fsmw<void>(std::vector<Cluster1D<void>, std::allocator<Cluster1D<void> > > const&, double) () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libRecoVertexVertexTools.so
#5  0x00007faad5a05e46 in FsmwClusterizer1D<void>::operator()(std::vector<Cluster1D<void>, std::allocator<Cluster1D<void> > > const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libRecoVertexVertexTools.so
#6  0x00007faad5a03ad9 in FsmwModeFinder3d::operator()(std::vector<std::pair<Point3DBase<float, GlobalTag>, float>, std::allocator<std::pair<Point3DBase<float, GlobalTag>, float> > > const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libRecoVertexVertexTools.so
#7  0x00007faad3b120bd in CrossingPtBasedLinearizationPointFinder::find(std::vector<std::pair<Point3DBase<float, GlobalTag>, float>, std::allocator<std::pair<Point3DBase<float, GlobalTag>, float> > > const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libRecoVertexLinearizationPointFinders.so
#8  0x00007faad3b13fee in CrossingPtBasedLinearizationPointFinder::getLinearizationPoint(std::vector<reco::TransientTrack, std::allocator<reco::TransientTrack> > const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libRecoVertexLinearizationPointFinders.so
#9  0x00007faad3b72dd1 in AdaptiveVertexFitter::vertex(std::vector<reco::TransientTrack, std::allocator<reco::TransientTrack> > const&, reco::BeamSpot const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libRecoVertexAdaptiveVertexFit.so
#10 0x00007faab6ee2165 in PFTauPrimaryVertexProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/pluginRecoTauTagRecoTauPlugins.so
#11 0x00007faaf8d6c393 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#12 0x00007faaf8ca8cc2 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#13 0x00007faaf8c449ea in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#14 0x00007faaf8c44bad in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#15 0x00007faaf8c462ab in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#16 0x00007faaf8c47354 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#17 0x00007faaf7579096 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7faaf5ebbe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:509
#18 0x00007faaf75728e0 in tbb::internal::arena::process (this=0x7faaf5fc2d00, s=...) at ../../src/tbb/arena.cpp:160
#19 0x00007faaf757125b in tbb::internal::market::process (this=0x7faaf5fc3580, j=...) at ../../src/tbb/market.cpp:693
#20 0x00007faaf756d4f3 in tbb::internal::rml::private_worker::run (this=0x7faaf5de8100) at ../../src/tbb/private_server.cpp:270
#21 0x00007faaf756d759 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#22 0x00000038c3207aa1 in start_thread () from /lib64/libpthread.so.0
#23 0x00000038c2ee8c4d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7faad72a3700 (LWP 29136)):
#0  0x00000038c320f37d in waitpid () from /lib64/libpthread.so.0
#1  0x00007faae6534fe7 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/pluginFWCoreServicesPlugins.so
#2  0x00007faae6535aca in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  0x00007faaf6e6c8cf in std::execute_native_thread_routine (__p=0x7faae7dabe50) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#4  0x00000038c3207aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000038c2ee8c4d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7faaf6b223e0 (LWP 29022)):
#0  0x00000038c2eaccbd in nanosleep () from /lib64/libc.so.6
#1  0x00000038c2eacb30 in sleep () from /lib64/libc.so.6
#2  0x00007faae6534de1 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  arena_dissociate_bin_slab (bin=0x7faaf6806218, slab=0x7faa84f535c0, arena=0x7faaf6800980) at src/arena.c:1468
#5  arena_dalloc_bin_locked_impl (junked=true, ptr=<optimized out>, slab=0x7faa84f535c0, arena=0x7faaf6800980, arena@entry=0x1c008607000, tsdn=0x7faaf6b1b7d8, tsdn@entry=0x0) at src/arena.c:1543
#6  je_arena_dalloc_bin_junked_locked (tsdn=tsdn@entry=0x7faaf6b1b7d8, arena=arena@entry=0x7faaf6800980, extent=0x7faa84f535c0, ptr=<optimized out>) at src/arena.c:1559
#7  0x00007faaf7826599 in je_tcache_bin_flush_small (tsd=0x7faaf6b1b7d8, tcache=<optimized out>, tbin=0x7faaf6b1bbe8, binind=24, rem=4) at src/tcache.c:149
#8  0x00007faaf78274ff in je_tcache_event_hard (tsd=<optimized out>, tcache=0x7faaf6b1b998) at src/tcache.c:54
#9  0x00007faae4a9ccbc in CaloGeometry::getPosition(DetId const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libGeometryCaloGeometry.so
#10 0x00007faab6e978fa in PFRecoTauDiscriminationAgainstElectronDeadECAL::updateBadTowers(edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/pluginRecoTauTagRecoTauPlugins.so
#11 0x00007faab6d1ecd6 in TauDiscriminationProducerBase<reco::PFTau, reco::PFTauDiscriminator>::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libRecoTauTagRecoTau.so
#12 0x00007faaf8d6c393 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#13 0x00007faaf8ca8cc2 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#14 0x00007faaf8c449ea in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#15 0x00007faaf8c44bad in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#16 0x00007faaf8c462ab in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#17 0x00007faaf8c47354 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#18 0x00007faaf7579096 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7faaf5fb2600, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:509
#19 0x00007faaf8cf3aa0 in edm::EventProcessor::processLumis(std::shared_ptr<void> const&) () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#20 0x00007faaf8cfcbd2 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/nweek-02552/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-11-30-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#21 0x000000000040feda in main::{lambda()#1}::operator()() const ()
#22 0x000000000040e192 in main ()

Current Modules:

Module: PFRecoTauDiscriminationByIsolation:hpsPFTauFootprintCorrectiondR03 (crashed)
Module: PFTauPrimaryVertexProducer:hpsPFTauPrimaryVertexProducer
Module: PFTauPrimaryVertexProducer:hpsPFTauPrimaryVertexProducer
Module: PFRecoTauDiscriminationAgainstElectronDeadECAL:hpsPFTauDiscriminationByDeadECALElectronRejection

A fatal system signal has occurred: segmentation violation

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc7_amd64_gcc700/CMSSW_10_4_X_2018-12-03-1100/pyRelValMatrixLogs/run/25210.0_TTbar012Jets_NLO_Mad_py8_Evt_13+TTbar012Jets_NLO_Mad_py8_Evt_13+DIGIUP15_PU25+RECOUP15_PU25+HARVESTUP15_PU25/step3_TTbar012Jets_NLO_Mad_py8_Evt_13+TTbar012Jets_NLO_Mad_py8_Evt_13+DIGIUP15_PU25+RECOUP15_PU25+HARVESTUP15_PU25.log

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Mon Dec  3 14:19:43 CET 2018
Thread 11 (Thread 0x7fd9f331e700 (LWP 2771)):
#0  0x00007fda7ca39995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fda7d019cec in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /build/cmsbld/auto-builds/CMSSW_10_1_0_pre2-slc7_amd64_gcc700/build/CMSSW_10_1_0_pre2-build/BUILD/slc7_amd64_gcc700/external/gcc/7.0.0-omkpbe2/gcc-branches_gcc-7-branch-257125/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/x86_64-unknown-linux-gnu/bits/gthr-default.h:864
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00007fda1cd9dbda in Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_4_X_2018-12-03-1100/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#4  0x00007fda1cd9a717 in std::_Function_handler<void (), tensorflow::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_4_X_2018-12-03-1100/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#5  0x00007fda7d01f8ff in std::execute_native_thread_routine (__p=0x7fd9f78c1850) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#6  0x00007fda7ca35e25 in start_thread () from /lib64/libpthread.so.0
#7  0x00007fda7c75fbad in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fda0f192700 (LWP 2724)):
#0  0x00007fda7c759ec9 in syscall () from /lib64/libc.so.6
#1  0x00007fda746be86e in XrdSys::LinuxSemaphore::Wait (this=0x7fda15c4b920) at /build/cmsbld/auto-builds/CMSSW_10_3_0_pre1-slc7_amd64_gcc700/build/CMSSW_10_3_0_pre1-build/BUILD/slc7_amd64_gcc700/external/xrootd/4.8.4/xrootd-4.8.4/src/./XrdSys/XrdSysLinuxSemaphore.hh:161
#2  XrdCl::SyncQueue<XrdCl::JobManager::JobHelper>::Get (this=0x7fda18241158) at /build/cmsbld/auto-builds/CMSSW_10_3_0_pre1-slc7_amd64_gcc700/build/CMSSW_10_3_0_pre1-build/BUILD/slc7_amd64_gcc700/external/xrootd/4.8.4/xrootd-4.8.4/src/./XrdCl/XrdClSyncQueue.hh:67
#3  XrdCl::JobManager::RunJobs (this=0x7fda18241140) at /build/cmsbld/auto-builds/CMSSW_10_3_0_pre1-slc7_amd64_gcc700/build/CMSSW_10_3_0_pre1-build/BUILD/slc7_amd64_gcc700/external/xrootd/4.8.4/xrootd-4.8.4/src/XrdCl/XrdClJobManager.cc:146
#4  0x00007fda746bea59 in RunRunnerThread (arg=<optimized out>) at /build/cmsbld/auto-builds/CMSSW_10_3_0_pre1-slc7_amd64_gcc700/build/CMSSW_10_3_0_pre1-build/BUILD/slc7_amd64_gcc700/external/xrootd/4.8.4/xrootd-4.8.4/src/XrdCl/XrdClJobManager.cc:33
#5  0x00007fda7ca35e25 in start_thread () from /lib64/libpthread.so.0
#6  0x00007fda7c75fbad in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fda0fb93700 (LWP 2723)):
#0  0x00007fda7c759ec9 in syscall () from /lib64/libc.so.6
#1  0x00007fda746be86e in XrdSys::LinuxSemaphore::Wait (this=0x7fda15c4b920) at /build/cmsbld/auto-builds/CMSSW_10_3_0_pre1-slc7_amd64_gcc700/build/CMSSW_10_3_0_pre1-build/BUILD/slc7_amd64_gcc700/external/xrootd/4.8.4/xrootd-4.8.4/src/./XrdSys/XrdSysLinuxSemaphore.hh:161
#2  XrdCl::SyncQueue<XrdCl::JobManager::JobHelper>::Get (this=0x7fda18241158) at /build/cmsbld/auto-builds/CMSSW_10_3_0_pre1-slc7_amd64_gcc700/build/CMSSW_10_3_0_pre1-build/BUILD/slc7_amd64_gcc700/external/xrootd/4.8.4/xrootd-4.8.4/src/./XrdCl/XrdClSyncQueue.hh:67
#3  XrdCl::JobManager::RunJobs (this=0x7fda18241140) at /build/cmsbld/auto-builds/CMSSW_10_3_0_pre1-slc7_amd64_gcc700/build/CMSSW_10_3_0_pre1-build/BUILD/slc7_amd64_gcc700/external/xrootd/4.8.4/xrootd-4.8.4/src/XrdCl/XrdClJobManager.cc:146
#4  0x00007fda746bea59 in RunRunnerThread (arg=<optimized out>) at /build/cmsbld/auto-builds/CMSSW_10_3_0_pre1-slc7_amd64_gcc700/build/CMSSW_10_3_0_pre1-build/BUILD/slc7_amd64_gcc700/external/xrootd/4.8.4/xrootd-4.8.4/src/XrdCl/XrdClJobManager.cc:33
#5  0x00007fda7ca35e25 in start_thread () from /lib64/libpthread.so.0
#6  0x00007fda7c75fbad in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fda10594700 (LWP 2722)):
#0  0x00007fda7c759ec9 in syscall () from /lib64/libc.so.6
#1  0x00007fda746be86e in XrdSys::LinuxSemaphore::Wait (this=0x7fda15c4b920) at /build/cmsbld/auto-builds/CMSSW_10_3_0_pre1-slc7_amd64_gcc700/build/CMSSW_10_3_0_pre1-build/BUILD/slc7_amd64_gcc700/external/xrootd/4.8.4/xrootd-4.8.4/src/./XrdSys/XrdSysLinuxSemaphore.hh:161
#2  XrdCl::SyncQueue<XrdCl::JobManager::JobHelper>::Get (this=0x7fda18241158) at /build/cmsbld/auto-builds/CMSSW_10_3_0_pre1-slc7_amd64_gcc700/build/CMSSW_10_3_0_pre1-build/BUILD/slc7_amd64_gcc700/external/xrootd/4.8.4/xrootd-4.8.4/src/./XrdCl/XrdClSyncQueue.hh:67
#3  XrdCl::JobManager::RunJobs (this=0x7fda18241140) at /build/cmsbld/auto-builds/CMSSW_10_3_0_pre1-slc7_amd64_gcc700/build/CMSSW_10_3_0_pre1-build/BUILD/slc7_amd64_gcc700/external/xrootd/4.8.4/xrootd-4.8.4/src/XrdCl/XrdClJobManager.cc:146
#4  0x00007fda746bea59 in RunRunnerThread (arg=<optimized out>) at /build/cmsbld/auto-builds/CMSSW_10_3_0_pre1-slc7_amd64_gcc700/build/CMSSW_10_3_0_pre1-build/BUILD/slc7_amd64_gcc700/external/xrootd/4.8.4/xrootd-4.8.4/src/XrdCl/XrdClJobManager.cc:33
#5  0x00007fda7ca35e25 in start_thread () from /lib64/libpthread.so.0
#6  0x00007fda7c75fbad in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fda10f95700 (LWP 2721)):
#0  0x00007fda7ca3cf3d in nanosleep () from /lib64/libpthread.so.0
#1  0x00007fda720d1ded in XrdSysTimer::Wait (mills=<optimized out>) at /build/cmsbld/auto-builds/CMSSW_10_3_0_pre1-slc7_amd64_gcc700/build/CMSSW_10_3_0_pre1-build/BUILD/slc7_amd64_gcc700/external/xrootd/4.8.4/xrootd-4.8.4/src/XrdSys/XrdSysTimer.cc:239
#2  0x00007fda746621c8 in XrdCl::TaskManager::RunTasks (this=0x7fda181ef440) at /build/cmsbld/auto-builds/CMSSW_10_3_0_pre1-slc7_amd64_gcc700/build/CMSSW_10_3_0_pre1-build/BUILD/slc7_amd64_gcc700/external/xrootd/4.8.4/xrootd-4.8.4/src/XrdCl/XrdClTaskManager.cc:244
#3  0x00007fda74662319 in RunRunnerThread (arg=<optimized out>) at /build/cmsbld/auto-builds/CMSSW_10_3_0_pre1-slc7_amd64_gcc700/build/CMSSW_10_3_0_pre1-build/BUILD/slc7_amd64_gcc700/external/xrootd/4.8.4/xrootd-4.8.4/src/XrdCl/XrdClTaskManager.cc:37
#4  0x00007fda7ca35e25 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fda7c75fbad in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fda11996700 (LWP 2720)):
#0  0x00007fda7c760183 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fda720d6c17 in XrdSys::IOEvents::PollE::Begin (this=0x7fda18241220, syncsem=<optimized out>, retcode=<optimized out>, eTxt=<optimized out>) at /build/cmsbld/auto-builds/CMSSW_10_3_0_pre1-slc7_amd64_gcc700/build/CMSSW_10_3_0_pre1-build/BUILD/slc7_amd64_gcc700/external/xrootd/4.8.4/xrootd-4.8.4/src/./XrdSys/XrdSysIOEventsPollE.icc:213
#2  0x00007fda720d34f5 in XrdSys::IOEvents::BootStrap::Start (parg=0x7fffdb7f7670) at /build/cmsbld/auto-builds/CMSSW_10_3_0_pre1-slc7_amd64_gcc700/build/CMSSW_10_3_0_pre1-build/BUILD/slc7_amd64_gcc700/external/xrootd/4.8.4/xrootd-4.8.4/src/XrdSys/XrdSysIOEvents.cc:131
#3  0x00007fda720d15e8 in XrdSysThread_Xeq (myargs=0x7fda15ca5a80) at /build/cmsbld/auto-builds/CMSSW_10_3_0_pre1-slc7_amd64_gcc700/build/CMSSW_10_3_0_pre1-build/BUILD/slc7_amd64_gcc700/external/xrootd/4.8.4/xrootd-4.8.4/src/XrdSys/XrdSysPthread.cc:86
#4  0x00007fda7ca35e25 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fda7c75fbad in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fda493ff700 (LWP 2711)):
#0  0x00007fda7c72656d in nanosleep () from /lib64/libc.so.6
#1  0x00007fda7c726404 in sleep () from /lib64/libc.so.6
#2  0x00007fda68a9d111 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007fda120835a7 in ClusterTPAssociationProducer::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/pluginSimTrackerTrackerHitAssociationPlugins.so
#5  0x00007fda7f55d4f8 in edm::global::EDProducerBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#6  0x00007fda7f530452 in edm::WorkerT<edm::global::EDProducerBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#7  0x00007fda7f4420da in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#8  0x00007fda7f44229d in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#9  0x00007fda7f44399b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#10 0x00007fda7f444a44 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#11 0x00007fda7dd760c6 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fda7a22be00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:509
#12 0x00007fda7dd6f910 in tbb::internal::arena::process (this=0x7fda7a342d00, s=...) at ../../src/tbb/arena.cpp:160
#13 0x00007fda7dd6e28b in tbb::internal::market::process (this=0x7fda7a343580, j=...) at ../../src/tbb/market.cpp:693
#14 0x00007fda7dd6a523 in tbb::internal::rml::private_worker::run (this=0x7fda7a166080) at ../../src/tbb/private_server.cpp:270
#15 0x00007fda7dd6a789 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#16 0x00007fda7ca35e25 in start_thread () from /lib64/libpthread.so.0
#17 0x00007fda7c75fbad in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fda4a50c700 (LWP 2710)):
#0  0x00007fda7c754f0d in poll () from /lib64/libc.so.6
#1  0x00007fda68a9d847 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#2  0x00007fda68a9dedc in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  0x00007fda68a9ef47 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007fda12e90020 in ?? ()
#6  0x00007fda6559db79 in reco::parser::MethodInvoker::invoke(edm::ObjectWithDict const&, edm::ObjectWithDict&) const () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/libCommonToolsUtils.so
#7  0x00007fda6557fab9 in reco::parser::ExpressionVar::value(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/libCommonToolsUtils.so
#8  0x00007fda3a78f76d in ElectronMVAEstimatorRun2::mvaValue(edm::Ptr<reco::Candidate> const&, edm::EventBase const&, int&) const () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/libRecoEgammaElectronIdentification.so
#9  0x00007fda2fcee7e7 in MVAValueMapProducer<reco::GsfElectron>::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/pluginRecoEgammaElectronIdentificationPlugins.so
#10 0x00007fda7f55d4f8 in edm::global::EDProducerBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#11 0x00007fda7f530452 in edm::WorkerT<edm::global::EDProducerBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#12 0x00007fda7f4420da in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#13 0x00007fda7f44229d in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#14 0x00007fda7f44399b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#15 0x00007fda7f444a44 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#16 0x00007fda7dd760c6 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fda7a24be00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:509
#17 0x00007fda7dd6f910 in tbb::internal::arena::process (this=0x7fda7a342d00, s=...) at ../../src/tbb/arena.cpp:160
#18 0x00007fda7dd6e28b in tbb::internal::market::process (this=0x7fda7a343580, j=...) at ../../src/tbb/market.cpp:693
#19 0x00007fda7dd6a523 in tbb::internal::rml::private_worker::run (this=0x7fda7a166180) at ../../src/tbb/private_server.cpp:270
#20 0x00007fda7dd6a789 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#21 0x00007fda7ca35e25 in start_thread () from /lib64/libpthread.so.0
#22 0x00007fda7c75fbad in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fda4af0d700 (LWP 2709)):
#0  0x00007fda7c72656d in nanosleep () from /lib64/libc.so.6
#1  0x00007fda7c726404 in sleep () from /lib64/libc.so.6
#2  0x00007fda68a9d111 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  fastjet::LazyTiling25::_bj_dist<fastjet::TiledJet> (this=0x7fda4af04e90, jetA=0x7fd922e860c8, jetA=0x7fd922e860c8, jetB=0x7fd922ef32c8, jetB=0x7fd922ef32c8) at ./../include/fastjet/internal/LazyTiling25.hh:169
#5  fastjet::LazyTiling25::_set_NN (jets_for_minheap=..., jetI=<optimized out>, this=0x7fda4af04e90) at LazyTiling25.cc:484
#6  fastjet::LazyTiling25::run (this=this@entry=0x7fda4af04e90) at LazyTiling25.cc:728
#7  0x00007fda4877cffa in fastjet::ClusterSequence::_initialise_and_run_no_decant (this=this@entry=0x7fda4af05260) at ClusterSequence.cc:376
#8  0x00007fda4877d582 in fastjet::ClusterSequence::_initialise_and_run (this=this@entry=0x7fda4af05260, jet_def_in=..., writeout_combinations=@0x7fda4af051a0: false) at ClusterSequence.cc:228
#9  0x00007fda49a38cf4 in fastjet::ClusterSequenceActiveAreaExplicitGhosts::_initialise<fastjet::PseudoJet> (this=0x7fda4af05260, pseudojets=..., jet_def_in=..., ghost_spec=0x7fd92a516b28, ghosts=<optimized out>, ghost_area=0, writeout_combinations=@0x7fda4af051a0: false) at ./../include/fastjet/ClusterSequenceActiveAreaExplicitGhosts.hh:227
#10 0x00007fda487aaf54 in fastjet::ClusterSequenceActiveAreaExplicitGhosts::ClusterSequenceActiveAreaExplicitGhosts<fastjet::PseudoJet> (writeout_combinations=@0x7fda4af051a0: false, ghost_spec=..., jet_def_in=..., pseudojets=..., this=0x7fda4af05260) at ./../include/fastjet/ClusterSequenceActiveAreaExplicitGhosts.hh:69
#11 fastjet::ClusterSequenceActiveArea::_run_AA (this=this@entry=0x7fd92a516f40, ghost_spec=...) at ClusterSequenceActiveArea.cc:139
#12 0x00007fda487ab2d8 in fastjet::ClusterSequenceActiveArea::_initialise_and_run_AA (this=this@entry=0x7fd92a516f40, jet_def_in=..., ghost_spec=..., writeout_combinations=@0x7fda4af0547f: false) at ClusterSequenceActiveArea.cc:67
#13 0x00007fda49a3a4aa in fastjet::ClusterSequenceActiveArea::ClusterSequenceActiveArea<fastjet::PseudoJet> (writeout_combinations=@0x7fda4af0547f: false, ghost_spec=..., jet_def_in=..., pseudojets=..., this=0x7fd92a516f40) at ./../include/fastjet/ClusterSequenceActiveArea.hh:217
#14 fastjet::ClusterSequenceArea::initialize_and_run_cswa<fastjet::PseudoJet> (this=0x7fd92a516a00, pseudojets=..., jet_def_in=...) at ./../include/fastjet/ClusterSequenceArea.hh:239
#15 0x00007fda2e70afb9 in FastjetJetProducer::runAlgorithm(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/pluginRecoJetsJetProducers_plugins.so
#16 0x00007fda2e6a5c32 in VirtualJetProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/pluginRecoJetsJetProducers_plugins.so
#17 0x00007fda2e70cd67 in FastjetJetProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/pluginRecoJetsJetProducers_plugins.so
#18 0x00007fda7f573423 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#19 0x00007fda7f5308d2 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#20 0x00007fda7f4420da in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#21 0x00007fda7f44229d in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#22 0x00007fda7f44399b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#23 0x00007fda7f444a44 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#24 0x00007fda7dd760c6 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fda7a23be00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:509
#25 0x00007fda7dd6f910 in tbb::internal::arena::process (this=0x7fda7a342d00, s=...) at ../../src/tbb/arena.cpp:160
#26 0x00007fda7dd6e28b in tbb::internal::market::process (this=0x7fda7a343580, j=...) at ../../src/tbb/market.cpp:693
#27 0x00007fda7dd6a523 in tbb::internal::rml::private_worker::run (this=0x7fda7a166100) at ../../src/tbb/private_server.cpp:270
#28 0x00007fda7dd6a789 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#29 0x00007fda7ca35e25 in start_thread () from /lib64/libpthread.so.0
#30 0x00007fda7c75fbad in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fda5afae700 (LWP 2671)):
#0  0x00007fda7ca3d279 in waitpid () from /lib64/libpthread.so.0
#1  0x00007fda68a9d317 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#2  0x00007fda68a9ddfa in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  0x00007fda7d01f8ff in std::execute_native_thread_routine (__p=0x7fda6f0e6e90) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#4  0x00007fda7ca35e25 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fda7c75fbad in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fda7ae29480 (LWP 2459)):
#0  0x00007fda7c72656d in nanosleep () from /lib64/libc.so.6
#1  0x00007fda7c726404 in sleep () from /lib64/libc.so.6
#2  0x00007fda68a9d111 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007fda120885b6 in std::_Rb_tree<std::pair<unsigned long, EncodedEventId>, std::pair<std::pair<unsigned long, EncodedEventId> const, edm::Ref<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle, edm::refhelper::FindUsingAdvance<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle> > >, std::_Select1st<std::pair<std::pair<unsigned long, EncodedEventId> const, edm::Ref<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle, edm::refhelper::FindUsingAdvance<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle> > > >, std::less<std::pair<unsigned long, EncodedEventId> >, std::allocator<std::pair<std::pair<unsigned long, EncodedEventId> const, edm::Ref<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle, edm::refhelper::FindUsingAdvance<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle> > > > >::find(std::pair<unsigned long, EncodedEventId> const&) () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/pluginSimTrackerTrackerHitAssociationPlugins.so
#5  0x00007fda120844ce in ClusterTPAssociationProducer::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/pluginSimTrackerTrackerHitAssociationPlugins.so
#6  0x00007fda7f55d4f8 in edm::global::EDProducerBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#7  0x00007fda7f530452 in edm::WorkerT<edm::global::EDProducerBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#8  0x00007fda7f4420da in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#9  0x00007fda7f44229d in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#10 0x00007fda7f44399b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#11 0x00007fda7f444a44 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#12 0x00007fda7dd760c6 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fda7a332600, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:509
#13 0x00007fda7f4e3a00 in edm::EventProcessor::processLumis(std::shared_ptr<void> const&) () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#14 0x00007fda7f4ecb32 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/nweek-02553/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_X_2018-12-02-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#15 0x000000000040fe8a in main::{lambda()#1}::operator()() const ()
#16 0x000000000040e162 in main ()

Current Modules:

Module: ElectronMVAValueMapProducer:electronMVAValueMapProducer (crashed)
Module: ClusterTPAssociationProducer:tpClusterProducerPreSplitting
Module: FastjetJetProducer:ca8PFJetsCHSprunedForBoostedTaus
Module: ClusterTPAssociationProducer:tpClusterProducerPreSplitting

A fatal system signal has occurred: segmentation violation

[dan131riley]

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc700/CMSSW_10_6_CLANG_X_2019-05-06-2300/pyRelValMatrixLogs/run/1325.8_TTbar_13_94Xv1NanoAODINPUT+TTbar_13_94Xv1NanoAODINPUT+NANOEDMMC2017_94XMiniAODv1+HARVESTNANOAODMC2017_94XMiniAODv1/step2_TTbar_13_94Xv1NanoAODINPUT+TTbar_13_94Xv1NanoAODINPUT+NANOEDMMC2017_94XMiniAODv1+HARVESTNANOAODMC2017_94XMiniAODv1.log#/229-229

Haven't seen one of these in a while...no obvious clues, but adding it to the collection:

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Tue May  7 04:43:55 CEST 2019
Thread 11 (Thread 0x7f1ddf839700 (LWP 23746)):
#0  0x00007f1e76c06965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f1e771e70ac in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /build/cmsbld/auto-builds/CMSSW_10_6_0_pre4-slc7_amd64_gcc700/build/CMSSW_10_6_0_pre4-build/BUILD/slc7_amd64_gcc700/external/gcc/7.0.0-pafccj/gcc-branches_gcc-7-branch-268351/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/x86_64-unknown-linux-gnu/bits/gthr-default.h:864
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00007f1df90b5bba in Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#4  0x00007f1df90b2687 in std::_Function_handler<void (), tensorflow::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#5  0x00007f1e771eccbf in std::execute_native_thread_routine (__p=0x7f1ddfb08210) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#6  0x00007f1e76c02dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f1e7692bead in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7f1e10fff700 (LWP 23713)):
#0  0x00007f1e768f2e2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f1e768f2cc4 in sleep () from /lib64/libc.so.6
#2  0x00007f1e681e13b0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f1dfbb2f8a5 in ?? () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_cc.so
#5  0x00007f1dfbba2099 in ?? () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_cc.so
#6  0x00007f1dfbba2410 in ?? () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_cc.so
#7  0x00007f1dfbba5704 in tensorflow::MatMulOp<Eigen::ThreadPoolDevice, float, false>::Compute(tensorflow::OpKernelContext*) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_cc.so
#8  0x00007f1df94a0213 in tensorflow::ThreadPoolDevice::Compute(tensorflow::OpKernel*, tensorflow::OpKernelContext*) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#9  0x00007f1df94694b7 in tensorflow::(anonymous namespace)::ExecutorState::Process(tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#10 0x00007f1df945b795 in std::_Function_handler<void (), std::_Bind<void (tensorflow::(anonymous namespace)::ExecutorState::*(tensorflow::(anonymous namespace)::ExecutorState*, tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long))(tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long)> >::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#11 0x00007f1dfe20fa96 in std::_Function_handler<void (std::function<void ()>), tensorflow::NTSession::Run(tensorflow::RunOptions const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor> > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*, tensorflow::RunMetadata*)::$_2>::_M_invoke(std::_Any_data const&, std::function<void ()>&&) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libPhysicsToolsTensorFlow.so
#12 0x00007f1df945c8f7 in tensorflow::(anonymous namespace)::ExecutorState::ScheduleReady(tensorflow::gtl::InlinedVector<tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, 8> const&, tensorflow::(anonymous namespace)::ExecutorState::TaggedNodeReadyQueue*) [clone .part.247] () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#13 0x00007f1df9460d41 in tensorflow::(anonymous namespace)::ExecutorState::NodeDone(tensorflow::Status const&, tensorflow::Node const*, tensorflow::gtl::InlinedVector<tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, 8> const&, tensorflow::NodeExecStatsWrapper*, tensorflow::(anonymous namespace)::ExecutorState::TaggedNodeReadyQueue*) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#14 0x00007f1df9467fb3 in tensorflow::(anonymous namespace)::ExecutorState::Process(tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#15 0x00007f1df945b795 in std::_Function_handler<void (), std::_Bind<void (tensorflow::(anonymous namespace)::ExecutorState::*(tensorflow::(anonymous namespace)::ExecutorState*, tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long))(tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long)> >::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#16 0x00007f1dfe20fa96 in std::_Function_handler<void (std::function<void ()>), tensorflow::NTSession::Run(tensorflow::RunOptions const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor> > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*, tensorflow::RunMetadata*)::$_2>::_M_invoke(std::_Any_data const&, std::function<void ()>&&) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libPhysicsToolsTensorFlow.so
#17 0x00007f1df945c8f7 in tensorflow::(anonymous namespace)::ExecutorState::ScheduleReady(tensorflow::gtl::InlinedVector<tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, 8> const&, tensorflow::(anonymous namespace)::ExecutorState::TaggedNodeReadyQueue*) [clone .part.247] () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#18 0x00007f1df9460d41 in tensorflow::(anonymous namespace)::ExecutorState::NodeDone(tensorflow::Status const&, tensorflow::Node const*, tensorflow::gtl::InlinedVector<tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, 8> const&, tensorflow::NodeExecStatsWrapper*, tensorflow::(anonymous namespace)::ExecutorState::TaggedNodeReadyQueue*) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#19 0x00007f1df9467fb3 in tensorflow::(anonymous namespace)::ExecutorState::Process(tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#20 0x00007f1df945b795 in std::_Function_handler<void (), std::_Bind<void (tensorflow::(anonymous namespace)::ExecutorState::*(tensorflow::(anonymous namespace)::ExecutorState*, tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long))(tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long)> >::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#21 0x00007f1dfe20fa96 in std::_Function_handler<void (std::function<void ()>), tensorflow::NTSession::Run(tensorflow::RunOptions const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor> > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*, tensorflow::RunMetadata*)::$_2>::_M_invoke(std::_Any_data const&, std::function<void ()>&&) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libPhysicsToolsTensorFlow.so
#22 0x00007f1df945c8f7 in tensorflow::(anonymous namespace)::ExecutorState::ScheduleReady(tensorflow::gtl::InlinedVector<tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, 8> const&, tensorflow::(anonymous namespace)::ExecutorState::TaggedNodeReadyQueue*) [clone .part.247] () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#23 0x00007f1df9460d41 in tensorflow::(anonymous namespace)::ExecutorState::NodeDone(tensorflow::Status const&, tensorflow::Node const*, tensorflow::gtl::InlinedVector<tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, 8> const&, tensorflow::NodeExecStatsWrapper*, tensorflow::(anonymous namespace)::ExecutorState::TaggedNodeReadyQueue*) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#24 0x00007f1df9467fb3 in tensorflow::(anonymous namespace)::ExecutorState::Process(tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#25 0x00007f1df945b795 in std::_Function_handler<void (), std::_Bind<void (tensorflow::(anonymous namespace)::ExecutorState::*(tensorflow::(anonymous namespace)::ExecutorState*, tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long))(tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long)> >::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#26 0x00007f1dfe20fa96 in std::_Function_handler<void (std::function<void ()>), tensorflow::NTSession::Run(tensorflow::RunOptions const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor> > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*, tensorflow::RunMetadata*)::$_2>::_M_invoke(std::_Any_data const&, std::function<void ()>&&) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libPhysicsToolsTensorFlow.so
#27 0x00007f1df945c8f7 in tensorflow::(anonymous namespace)::ExecutorState::ScheduleReady(tensorflow::gtl::InlinedVector<tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, 8> const&, tensorflow::(anonymous namespace)::ExecutorState::TaggedNodeReadyQueue*) [clone .part.247] () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#28 0x00007f1df9460d41 in tensorflow::(anonymous namespace)::ExecutorState::NodeDone(tensorflow::Status const&, tensorflow::Node const*, tensorflow::gtl::InlinedVector<tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, 8> const&, tensorflow::NodeExecStatsWrapper*, tensorflow::(anonymous namespace)::ExecutorState::TaggedNodeReadyQueue*) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#29 0x00007f1df9467fb3 in tensorflow::(anonymous namespace)::ExecutorState::Process(tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#30 0x00007f1df945b795 in std::_Function_handler<void (), std::_Bind<void (tensorflow::(anonymous namespace)::ExecutorState::*(tensorflow::(anonymous namespace)::ExecutorState*, tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long))(tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long)> >::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#31 0x00007f1dfe20fa96 in std::_Function_handler<void (std::function<void ()>), tensorflow::NTSession::Run(tensorflow::RunOptions const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor> > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*, tensorflow::RunMetadata*)::$_2>::_M_invoke(std::_Any_data const&, std::function<void ()>&&) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libPhysicsToolsTensorFlow.so
#32 0x00007f1df945c750 in tensorflow::(anonymous namespace)::ExecutorState::ScheduleReady(tensorflow::gtl::InlinedVector<tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, 8> const&, tensorflow::(anonymous namespace)::ExecutorState::TaggedNodeReadyQueue*) [clone .part.247] () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#33 0x00007f1df9460d41 in tensorflow::(anonymous namespace)::ExecutorState::NodeDone(tensorflow::Status const&, tensorflow::Node const*, tensorflow::gtl::InlinedVector<tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, 8> const&, tensorflow::NodeExecStatsWrapper*, tensorflow::(anonymous namespace)::ExecutorState::TaggedNodeReadyQueue*) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#34 0x00007f1df9467fb3 in tensorflow::(anonymous namespace)::ExecutorState::Process(tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#35 0x00007f1df946978f in std::_Function_handler<void (), tensorflow::(anonymous namespace)::ExecutorState::ScheduleReady(tensorflow::gtl::InlinedVector<tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, 8> const&, tensorflow::(anonymous namespace)::ExecutorState::TaggedNodeReadyQueue*)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#36 0x00007f1dfe20fa96 in std::_Function_handler<void (std::function<void ()>), tensorflow::NTSession::Run(tensorflow::RunOptions const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor> > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*, tensorflow::RunMetadata*)::$_2>::_M_invoke(std::_Any_data const&, std::function<void ()>&&) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libPhysicsToolsTensorFlow.so
#37 0x00007f1df945cc2e in tensorflow::(anonymous namespace)::ExecutorState::ScheduleReady(tensorflow::gtl::InlinedVector<tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, 8> const&, tensorflow::(anonymous namespace)::ExecutorState::TaggedNodeReadyQueue*) [clone .part.247] () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#38 0x00007f1df9462bbd in tensorflow::(anonymous namespace)::ExecutorImpl::RunAsync(tensorflow::Executor::Args const&, std::function<void (tensorflow::Status const&)>) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#39 0x00007f1dfe205e18 in tensorflow::NTSession::Run(tensorflow::RunOptions const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor> > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*, tensorflow::RunMetadata*) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libPhysicsToolsTensorFlow.so
#40 0x00007f1dfe204e3a in tensorflow::NTSession::Run(std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor> > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libPhysicsToolsTensorFlow.so
#41 0x00007f1dfe202572 in tensorflow::run(tensorflow::Session*, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor> > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libPhysicsToolsTensorFlow.so
#42 0x00007f1dfe202b57 in tensorflow::run(tensorflow::Session*, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor> > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libPhysicsToolsTensorFlow.so
#43 0x00007f1dc2cc6ba4 in DeepFlavourTFJetTagsProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/pluginRecoBTagTensorFlowPlugins.so
#44 0x00007f1e7b0caad2 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetupImpl const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#45 0x00007f1e7aff42be in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetupImpl const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#46 0x00007f1e7af84360 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#47 0x00007f1e7af833ce in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#48 0x00007f1e7af84b6d in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#49 0x00007f1e7af84812 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#50 0x00007f1e77f44931 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f1e740dbe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:521
#51 0x00007f1e77f3e3d8 in tbb::internal::arena::process (this=0x7f1e741cad00, s=...) at ../../src/tbb/arena.cpp:160
#52 0x00007f1e77f3cd53 in tbb::internal::market::process (this=0x7f1e741cb580, j=...) at ../../src/tbb/market.cpp:710
#53 0x00007f1e77f39011 in tbb::internal::rml::private_worker::run (this=0x7f1e73fee080) at ../../src/tbb/private_server.cpp:270
#54 0x00007f1e77f39259 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#55 0x00007f1e76c02dd5 in start_thread () from /lib64/libpthread.so.0
#56 0x00007f1e7692bead in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f1e122bc700 (LWP 23712)):
#0  0x00007f1e7692120d in poll () from /lib64/libc.so.6
#1  0x00007f1e681e228c in (anonymous namespace)::full_read(int, char*, unsigned long, int) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#2  0x00007f1e681e1d4a in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  0x00007f1e681e172f in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007f1d79244020 in ?? ()
#6  0x00007f1e544413be in reco::parser::MethodInvoker::invoke(edm::ObjectWithDict const&, edm::ObjectWithDict&) const () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libCommonToolsUtils.so
#7  0x00007f1e544420ed in reco::parser::SingleInvoker::invoke(edm::ObjectWithDict const&, std::vector<edm::ObjectWithDict, std::allocator<edm::ObjectWithDict> >&) const () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libCommonToolsUtils.so
#8  0x00007f1e54442289 in reco::parser::LazyInvoker::invokeLast(edm::ObjectWithDict const&, std::vector<edm::ObjectWithDict, std::allocator<edm::ObjectWithDict> >&) const () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libCommonToolsUtils.so
#9  0x00007f1e5447790d in reco::parser::ExpressionLazyVar::value(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libCommonToolsUtils.so
#10 0x00007f1dfe77febd in StringObjectFunction<reco::Candidate, false>::operator()(reco::Candidate const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/pluginPhysicsToolsNanoAODPlugins.so
#11 0x00007f1dfe77fbaf in SimpleFlatTableProducerBase<reco::Candidate, edm::View<reco::Candidate> >::FuncVariable<StringObjectFunction<reco::Candidate, false>, int>::fill(std::vector<reco::Candidate const*, std::allocator<reco::Candidate const*> >, nanoaod::FlatTable&) const () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/pluginPhysicsToolsNanoAODPlugins.so
#12 0x00007f1dfe77f225 in SimpleFlatTableProducer<reco::Candidate>::fillTable(edm::Event const&, edm::Handle<edm::View<reco::Candidate> > const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/pluginPhysicsToolsNanoAODPlugins.so
#13 0x00007f1dfe77eb61 in SimpleFlatTableProducerBase<reco::Candidate, edm::View<reco::Candidate> >::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/pluginPhysicsToolsNanoAODPlugins.so
#14 0x00007f1e7b0caad2 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetupImpl const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#15 0x00007f1e7aff42be in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetupImpl const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#16 0x00007f1e7af84360 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#17 0x00007f1e7af833ce in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#18 0x00007f1e7af84b6d in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#19 0x00007f1e7af84812 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#20 0x00007f1e77f44931 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f1e740fbe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:521
#21 0x00007f1e77f3e3d8 in tbb::internal::arena::process (this=0x7f1e741cad00, s=...) at ../../src/tbb/arena.cpp:160
#22 0x00007f1e77f3cd53 in tbb::internal::market::process (this=0x7f1e741cb580, j=...) at ../../src/tbb/market.cpp:710
#23 0x00007f1e77f39011 in tbb::internal::rml::private_worker::run (this=0x7f1e73fee180) at ../../src/tbb/private_server.cpp:270
#24 0x00007f1e77f39259 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#25 0x00007f1e76c02dd5 in start_thread () from /lib64/libpthread.so.0
#26 0x00007f1e7692bead in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f1e12cbd700 (LWP 23711)):
#0  0x00007f1e768f2e2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f1e768f2cc4 in sleep () from /lib64/libc.so.6
#2  0x00007f1e681e13b0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  sz_index2size_lookup (index=2) at include/jemalloc/internal/sz.h:201
#5  sz_index2size (index=2) at include/jemalloc/internal/sz.h:209
#6  imalloc_body (tsd=0x7f1e12cb6af8, dopts=<synthetic pointer>, sopts=<synthetic pointer>) at src/jemalloc.c:1837
#7  imalloc (dopts=<synthetic pointer>, sopts=<synthetic pointer>) at src/jemalloc.c:2005
#8  malloc (size=size@entry=32) at src/jemalloc.c:2038
#9  0x00007f1e781eff19 in newImpl<false> (size=32) at src/jemalloc_cpp.cpp:78
#10 operator new (size=32) at src/jemalloc_cpp.cpp:87
#11 0x00007f1df8f81b7c in tensorflow::OpKernelContext::set_output(int, tensorflow::Tensor const&) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#12 0x00007f1dfb240d31 in tensorflow::SwitchOp::Compute(tensorflow::OpKernelContext*) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_cc.so
#13 0x00007f1df94a0213 in tensorflow::ThreadPoolDevice::Compute(tensorflow::OpKernel*, tensorflow::OpKernelContext*) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#14 0x00007f1df94694b7 in tensorflow::(anonymous namespace)::ExecutorState::Process(tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#15 0x00007f1df945b795 in std::_Function_handler<void (), std::_Bind<void (tensorflow::(anonymous namespace)::ExecutorState::*(tensorflow::(anonymous namespace)::ExecutorState*, tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long))(tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long)> >::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#16 0x00007f1dfe20fa96 in std::_Function_handler<void (std::function<void ()>), tensorflow::NTSession::Run(tensorflow::RunOptions const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor> > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*, tensorflow::RunMetadata*)::$_2>::_M_invoke(std::_Any_data const&, std::function<void ()>&&) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libPhysicsToolsTensorFlow.so
#17 0x00007f1df945c8f7 in tensorflow::(anonymous namespace)::ExecutorState::ScheduleReady(tensorflow::gtl::InlinedVector<tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, 8> const&, tensorflow::(anonymous namespace)::ExecutorState::TaggedNodeReadyQueue*) [clone .part.247] () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#18 0x00007f1df9460d41 in tensorflow::(anonymous namespace)::ExecutorState::NodeDone(tensorflow::Status const&, tensorflow::Node const*, tensorflow::gtl::InlinedVector<tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, 8> const&, tensorflow::NodeExecStatsWrapper*, tensorflow::(anonymous namespace)::ExecutorState::TaggedNodeReadyQueue*) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#19 0x00007f1df9467fb3 in tensorflow::(anonymous namespace)::ExecutorState::Process(tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#20 0x00007f1df945b795 in std::_Function_handler<void (), std::_Bind<void (tensorflow::(anonymous namespace)::ExecutorState::*(tensorflow::(anonymous namespace)::ExecutorState*, tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long))(tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long)> >::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#21 0x00007f1dfe20fa96 in std::_Function_handler<void (std::function<void ()>), tensorflow::NTSession::Run(tensorflow::RunOptions const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor> > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*, tensorflow::RunMetadata*)::$_2>::_M_invoke(std::_Any_data const&, std::function<void ()>&&) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libPhysicsToolsTensorFlow.so
#22 0x00007f1df945c8f7 in tensorflow::(anonymous namespace)::ExecutorState::ScheduleReady(tensorflow::gtl::InlinedVector<tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, 8> const&, tensorflow::(anonymous namespace)::ExecutorState::TaggedNodeReadyQueue*) [clone .part.247] () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#23 0x00007f1df9460d41 in tensorflow::(anonymous namespace)::ExecutorState::NodeDone(tensorflow::Status const&, tensorflow::Node const*, tensorflow::gtl::InlinedVector<tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, 8> const&, tensorflow::NodeExecStatsWrapper*, tensorflow::(anonymous namespace)::ExecutorState::TaggedNodeReadyQueue*) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#24 0x00007f1df9467fb3 in tensorflow::(anonymous namespace)::ExecutorState::Process(tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#25 0x00007f1df945b795 in std::_Function_handler<void (), std::_Bind<void (tensorflow::(anonymous namespace)::ExecutorState::*(tensorflow::(anonymous namespace)::ExecutorState*, tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long))(tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long)> >::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#26 0x00007f1dfe20fa96 in std::_Function_handler<void (std::function<void ()>), tensorflow::NTSession::Run(tensorflow::RunOptions const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor> > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*, tensorflow::RunMetadata*)::$_2>::_M_invoke(std::_Any_data const&, std::function<void ()>&&) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libPhysicsToolsTensorFlow.so
#27 0x00007f1df945c8f7 in tensorflow::(anonymous namespace)::ExecutorState::ScheduleReady(tensorflow::gtl::InlinedVector<tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, 8> const&, tensorflow::(anonymous namespace)::ExecutorState::TaggedNodeReadyQueue*) [clone .part.247] () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#28 0x00007f1df9460d41 in tensorflow::(anonymous namespace)::ExecutorState::NodeDone(tensorflow::Status const&, tensorflow::Node const*, tensorflow::gtl::InlinedVector<tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, 8> const&, tensorflow::NodeExecStatsWrapper*, tensorflow::(anonymous namespace)::ExecutorState::TaggedNodeReadyQueue*) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#29 0x00007f1df9467fb3 in tensorflow::(anonymous namespace)::ExecutorState::Process(tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#30 0x00007f1df945b795 in std::_Function_handler<void (), std::_Bind<void (tensorflow::(anonymous namespace)::ExecutorState::*(tensorflow::(anonymous namespace)::ExecutorState*, tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long))(tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long)> >::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#31 0x00007f1dfe20fa96 in std::_Function_handler<void (std::function<void ()>), tensorflow::NTSession::Run(tensorflow::RunOptions const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor> > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*, tensorflow::RunMetadata*)::$_2>::_M_invoke(std::_Any_data const&, std::function<void ()>&&) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libPhysicsToolsTensorFlow.so
#32 0x00007f1df945c750 in tensorflow::(anonymous namespace)::ExecutorState::ScheduleReady(tensorflow::gtl::InlinedVector<tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, 8> const&, tensorflow::(anonymous namespace)::ExecutorState::TaggedNodeReadyQueue*) [clone .part.247] () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#33 0x00007f1df9460d41 in tensorflow::(anonymous namespace)::ExecutorState::NodeDone(tensorflow::Status const&, tensorflow::Node const*, tensorflow::gtl::InlinedVector<tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, 8> const&, tensorflow::NodeExecStatsWrapper*, tensorflow::(anonymous namespace)::ExecutorState::TaggedNodeReadyQueue*) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#34 0x00007f1df9467fb3 in tensorflow::(anonymous namespace)::ExecutorState::Process(tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#35 0x00007f1df946978f in std::_Function_handler<void (), tensorflow::(anonymous namespace)::ExecutorState::ScheduleReady(tensorflow::gtl::InlinedVector<tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, 8> const&, tensorflow::(anonymous namespace)::ExecutorState::TaggedNodeReadyQueue*)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#36 0x00007f1dfe20fa96 in std::_Function_handler<void (std::function<void ()>), tensorflow::NTSession::Run(tensorflow::RunOptions const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor> > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*, tensorflow::RunMetadata*)::$_2>::_M_invoke(std::_Any_data const&, std::function<void ()>&&) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libPhysicsToolsTensorFlow.so
#37 0x00007f1df945cc2e in tensorflow::(anonymous namespace)::ExecutorState::ScheduleReady(tensorflow::gtl::InlinedVector<tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, 8> const&, tensorflow::(anonymous namespace)::ExecutorState::TaggedNodeReadyQueue*) [clone .part.247] () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#38 0x00007f1df9462bbd in tensorflow::(anonymous namespace)::ExecutorImpl::RunAsync(tensorflow::Executor::Args const&, std::function<void (tensorflow::Status const&)>) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libtensorflow_framework.so
#39 0x00007f1dfe205e18 in tensorflow::NTSession::Run(tensorflow::RunOptions const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor> > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*, tensorflow::RunMetadata*) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libPhysicsToolsTensorFlow.so
#40 0x00007f1dfe204e3a in tensorflow::NTSession::Run(std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor> > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libPhysicsToolsTensorFlow.so
#41 0x00007f1dfe202572 in tensorflow::run(tensorflow::Session*, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor> > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libPhysicsToolsTensorFlow.so
#42 0x00007f1dfe202b57 in tensorflow::run(tensorflow::Session*, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor> > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libPhysicsToolsTensorFlow.so
#43 0x00007f1dc2cc6ba4 in DeepFlavourTFJetTagsProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/pluginRecoBTagTensorFlowPlugins.so
#44 0x00007f1e7b0caad2 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetupImpl const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#45 0x00007f1e7aff42be in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetupImpl const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#46 0x00007f1e7af84360 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#47 0x00007f1e7af833ce in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#48 0x00007f1e7af84b6d in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#49 0x00007f1e7af84812 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#50 0x00007f1e77f44931 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f1e740ebe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:521
#51 0x00007f1e77f3e3d8 in tbb::internal::arena::process (this=0x7f1e741cad00, s=...) at ../../src/tbb/arena.cpp:160
#52 0x00007f1e77f3cd53 in tbb::internal::market::process (this=0x7f1e741cb580, j=...) at ../../src/tbb/market.cpp:710
#53 0x00007f1e77f39011 in tbb::internal::rml::private_worker::run (this=0x7f1e73fee100) at ../../src/tbb/private_server.cpp:270
#54 0x00007f1e77f39259 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#55 0x00007f1e76c02dd5 in start_thread () from /lib64/libpthread.so.0
#56 0x00007f1e7692bead in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f1e22552700 (LWP 23703)):
#0  0x00007f1e769261c9 in syscall () from /lib64/libc.so.6
#1  0x00007f1e6f6712ee in XrdSys::LinuxSemaphore::Wait (this=0x7f1e46800ef8) at /build/cmsbld/auto-builds/CMSSW_10_6_0_pre4-slc7_amd64_gcc700/build/CMSSW_10_6_0_pre4-build/BUILD/slc7_amd64_gcc700/external/xrootd/4.8.5-pafccj/xrootd-4.8.5/src/./XrdSys/XrdSysLinuxSemaphore.hh:161
#2  XrdCl::SyncQueue<XrdCl::JobManager::JobHelper>::Get (this=0x7f1e469500b8) at /build/cmsbld/auto-builds/CMSSW_10_6_0_pre4-slc7_amd64_gcc700/build/CMSSW_10_6_0_pre4-build/BUILD/slc7_amd64_gcc700/external/xrootd/4.8.5-pafccj/xrootd-4.8.5/src/./XrdCl/XrdClSyncQueue.hh:67
#3  XrdCl::JobManager::RunJobs (this=0x7f1e469500a0) at /build/cmsbld/auto-builds/CMSSW_10_6_0_pre4-slc7_amd64_gcc700/build/CMSSW_10_6_0_pre4-build/BUILD/slc7_amd64_gcc700/external/xrootd/4.8.5-pafccj/xrootd-4.8.5/src/XrdCl/XrdClJobManager.cc:146
#4  0x00007f1e6f6714d9 in RunRunnerThread (arg=<optimized out>) at /build/cmsbld/auto-builds/CMSSW_10_6_0_pre4-slc7_amd64_gcc700/build/CMSSW_10_6_0_pre4-build/BUILD/slc7_amd64_gcc700/external/xrootd/4.8.5-pafccj/xrootd-4.8.5/src/XrdCl/XrdClJobManager.cc:33
#5  0x00007f1e76c02dd5 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f1e7692bead in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f1e22f53700 (LWP 23702)):
#0  0x00007f1e769261c9 in syscall () from /lib64/libc.so.6
#1  0x00007f1e6f6712ee in XrdSys::LinuxSemaphore::Wait (this=0x7f1e46800ef8) at /build/cmsbld/auto-builds/CMSSW_10_6_0_pre4-slc7_amd64_gcc700/build/CMSSW_10_6_0_pre4-build/BUILD/slc7_amd64_gcc700/external/xrootd/4.8.5-pafccj/xrootd-4.8.5/src/./XrdSys/XrdSysLinuxSemaphore.hh:161
#2  XrdCl::SyncQueue<XrdCl::JobManager::JobHelper>::Get (this=0x7f1e469500b8) at /build/cmsbld/auto-builds/CMSSW_10_6_0_pre4-slc7_amd64_gcc700/build/CMSSW_10_6_0_pre4-build/BUILD/slc7_amd64_gcc700/external/xrootd/4.8.5-pafccj/xrootd-4.8.5/src/./XrdCl/XrdClSyncQueue.hh:67
#3  XrdCl::JobManager::RunJobs (this=0x7f1e469500a0) at /build/cmsbld/auto-builds/CMSSW_10_6_0_pre4-slc7_amd64_gcc700/build/CMSSW_10_6_0_pre4-build/BUILD/slc7_amd64_gcc700/external/xrootd/4.8.5-pafccj/xrootd-4.8.5/src/XrdCl/XrdClJobManager.cc:146
#4  0x00007f1e6f6714d9 in RunRunnerThread (arg=<optimized out>) at /build/cmsbld/auto-builds/CMSSW_10_6_0_pre4-slc7_amd64_gcc700/build/CMSSW_10_6_0_pre4-build/BUILD/slc7_amd64_gcc700/external/xrootd/4.8.5-pafccj/xrootd-4.8.5/src/XrdCl/XrdClJobManager.cc:33
#5  0x00007f1e76c02dd5 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f1e7692bead in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f1e23954700 (LWP 23701)):
#0  0x00007f1e769261c9 in syscall () from /lib64/libc.so.6
#1  0x00007f1e6f6712ee in XrdSys::LinuxSemaphore::Wait (this=0x7f1e46800ef8) at /build/cmsbld/auto-builds/CMSSW_10_6_0_pre4-slc7_amd64_gcc700/build/CMSSW_10_6_0_pre4-build/BUILD/slc7_amd64_gcc700/external/xrootd/4.8.5-pafccj/xrootd-4.8.5/src/./XrdSys/XrdSysLinuxSemaphore.hh:161
#2  XrdCl::SyncQueue<XrdCl::JobManager::JobHelper>::Get (this=0x7f1e469500b8) at /build/cmsbld/auto-builds/CMSSW_10_6_0_pre4-slc7_amd64_gcc700/build/CMSSW_10_6_0_pre4-build/BUILD/slc7_amd64_gcc700/external/xrootd/4.8.5-pafccj/xrootd-4.8.5/src/./XrdCl/XrdClSyncQueue.hh:67
#3  XrdCl::JobManager::RunJobs (this=0x7f1e469500a0) at /build/cmsbld/auto-builds/CMSSW_10_6_0_pre4-slc7_amd64_gcc700/build/CMSSW_10_6_0_pre4-build/BUILD/slc7_amd64_gcc700/external/xrootd/4.8.5-pafccj/xrootd-4.8.5/src/XrdCl/XrdClJobManager.cc:146
#4  0x00007f1e6f6714d9 in RunRunnerThread (arg=<optimized out>) at /build/cmsbld/auto-builds/CMSSW_10_6_0_pre4-slc7_amd64_gcc700/build/CMSSW_10_6_0_pre4-build/BUILD/slc7_amd64_gcc700/external/xrootd/4.8.5-pafccj/xrootd-4.8.5/src/XrdCl/XrdClJobManager.cc:33
#5  0x00007f1e76c02dd5 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f1e7692bead in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f1e24355700 (LWP 23700)):
#0  0x00007f1e76c09e3d in nanosleep () from /lib64/libpthread.so.0
#1  0x00007f1e71613ded in XrdSysTimer::Wait (mills=<optimized out>) at /build/cmsbld/auto-builds/CMSSW_10_6_0_pre4-slc7_amd64_gcc700/build/CMSSW_10_6_0_pre4-build/BUILD/slc7_amd64_gcc700/external/xrootd/4.8.5-pafccj/xrootd-4.8.5/src/XrdSys/XrdSysTimer.cc:239
#2  0x00007f1e6f6129a8 in XrdCl::TaskManager::RunTasks (this=0x7f1e46aa5b80) at /build/cmsbld/auto-builds/CMSSW_10_6_0_pre4-slc7_amd64_gcc700/build/CMSSW_10_6_0_pre4-build/BUILD/slc7_amd64_gcc700/external/xrootd/4.8.5-pafccj/xrootd-4.8.5/src/XrdCl/XrdClTaskManager.cc:244
#3  0x00007f1e6f612af9 in RunRunnerThread (arg=<optimized out>) at /build/cmsbld/auto-builds/CMSSW_10_6_0_pre4-slc7_amd64_gcc700/build/CMSSW_10_6_0_pre4-build/BUILD/slc7_amd64_gcc700/external/xrootd/4.8.5-pafccj/xrootd-4.8.5/src/XrdCl/XrdClTaskManager.cc:37
#4  0x00007f1e76c02dd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f1e7692bead in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f1e24d56700 (LWP 23699)):
#0  0x00007f1e7692c483 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f1e71618c27 in XrdSys::IOEvents::PollE::Begin (this=0x7f1e46950420, syncsem=<optimized out>, retcode=<optimized out>, eTxt=<optimized out>) at /build/cmsbld/auto-builds/CMSSW_10_6_0_pre4-slc7_amd64_gcc700/build/CMSSW_10_6_0_pre4-build/BUILD/slc7_amd64_gcc700/external/xrootd/4.8.5-pafccj/xrootd-4.8.5/src/./XrdSys/XrdSysIOEventsPollE.icc:213
#2  0x00007f1e71615505 in XrdSys::IOEvents::BootStrap::Start (parg=0x7ffe818cfec0) at /build/cmsbld/auto-builds/CMSSW_10_6_0_pre4-slc7_amd64_gcc700/build/CMSSW_10_6_0_pre4-build/BUILD/slc7_amd64_gcc700/external/xrootd/4.8.5-pafccj/xrootd-4.8.5/src/XrdSys/XrdSysIOEvents.cc:131
#3  0x00007f1e716135e8 in XrdSysThread_Xeq (myargs=0x7f1e46a0ee80) at /build/cmsbld/auto-builds/CMSSW_10_6_0_pre4-slc7_amd64_gcc700/build/CMSSW_10_6_0_pre4-build/BUILD/slc7_amd64_gcc700/external/xrootd/4.8.5-pafccj/xrootd-4.8.5/src/XrdSys/XrdSysPthread.cc:86
#4  0x00007f1e76c02dd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f1e7692bead in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f1e47fb3700 (LWP 23683)):
#0  0x00007f1e76c0a179 in waitpid () from /lib64/libpthread.so.0
#1  0x00007f1e681e1f9d in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#2  0x00007f1e771eccbf in std::execute_native_thread_routine (__p=0x7f1e70aaa800) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#3  0x00007f1e76c02dd5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f1e7692bead in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f1e74ff3480 (LWP 23551)):
#0  0x00007f1e768f2e2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f1e768f2cc4 in sleep () from /lib64/libc.so.6
#2  0x00007f1e681e13b0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f1e7077ef11 in cblas_sgemm () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libgslcblas.so.0
#5  0x00007f1dc4f0c3b0 in void linalg_gemm<mshadow::cpu, float>(mshadow::Tensor<mshadow::cpu, 2, float> const&, mshadow::Tensor<mshadow::cpu, 2, float> const&, mshadow::Tensor<mshadow::cpu, 2, float> const&, bool, bool, mshadow::Stream<mshadow::cpu>*, mxnet::OpReqType) [clone .constprop.894] () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libmxnetpredict.so
#6  0x00007f1dc4f1e39f in mxnet::op::ConvolutionOp<mshadow::cpu, float>::Forward(mxnet::OpContext const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libmxnetpredict.so
#7  0x00007f1dc4f24b9a in void mxnet::op::ConvolutionCompute<mshadow::cpu>(nnvm::NodeAttrs const&, mxnet::OpContext const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libmxnetpredict.so
#8  0x00007f1dc5583759 in mxnet::exec::FComputeExecutor::Run(mxnet::RunContext, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libmxnetpredict.so
#9  0x00007f1dc5544e13 in mxnet::exec::GraphExecutor::CreateCachedSegOpr(unsigned long, unsigned long)::{lambda(mxnet::RunContext, mxnet::engine::CallbackOnComplete)#1}::operator()(mxnet::RunContext, mxnet::engine::CallbackOnComplete) const () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libmxnetpredict.so
#10 0x00007f1dc5544ee1 in std::_Function_handler<void (mxnet::RunContext, mxnet::engine::CallbackOnComplete), mxnet::exec::GraphExecutor::CreateCachedSegOpr(unsigned long, unsigned long)::{lambda(mxnet::RunContext, mxnet::engine::CallbackOnComplete)#1}>::_M_invoke(std::_Any_data const&, mxnet::RunContext&&, mxnet::engine::CallbackOnComplete&&) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libmxnetpredict.so
#11 0x00007f1dc54bcb2e in mxnet::engine::NaiveEngine::Push(mxnet::engine::Opr*, mxnet::Context, int, bool)::{lambda(mxnet::RunContext, mxnet::engine::CallbackOnComplete)#1}::operator()(mxnet::RunContext, mxnet::engine::CallbackOnComplete) const () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libmxnetpredict.so
#12 0x00007f1dc54bd7f1 in std::_Function_handler<void (mxnet::RunContext, mxnet::engine::CallbackOnComplete), mxnet::engine::NaiveEngine::Push(mxnet::engine::Opr*, mxnet::Context, int, bool)::{lambda(mxnet::RunContext, mxnet::engine::CallbackOnComplete)#1}>::_M_invoke(std::_Any_data const&, mxnet::RunContext&&, mxnet::engine::CallbackOnComplete&&) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libmxnetpredict.so
#13 0x00007f1dc54b658e in mxnet::engine::NaiveEngine::PushAsync(std::function<void (mxnet::RunContext, mxnet::engine::CallbackOnComplete)>, mxnet::Context, std::vector<mxnet::engine::Var*, std::allocator<mxnet::engine::Var*> > const&, std::vector<mxnet::engine::Var*, std::allocator<mxnet::engine::Var*> > const&, mxnet::FnProperty, int, char const*, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libmxnetpredict.so
#14 0x00007f1dc54b9dc9 in mxnet::engine::NaiveEngine::Push(mxnet::engine::Opr*, mxnet::Context, int, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libmxnetpredict.so
#15 0x00007f1dc5548757 in mxnet::exec::GraphExecutor::RunOps(bool, unsigned long, unsigned long) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libmxnetpredict.so
#16 0x00007f1dc550c1a8 in MXExecutorForward () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/external/slc7_amd64_gcc700/lib/libmxnetpredict.so
#17 0x00007f1dc596bded in mxnet::cpp::Executor::Forward(bool) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libPhysicsToolsMXNet.so
#18 0x00007f1dc596972e in mxnet::cpp::Predictor::predict(std::vector<std::vector<float, std::allocator<float> >, std::allocator<std::vector<float, std::allocator<float> > > > const&) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libPhysicsToolsMXNet.so
#19 0x00007f1dc5b8c68d in DeepBoostedJetTagsProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/pluginRecoBTagDeepBoostedJetPlugins.so
#20 0x00007f1e7b0caad2 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetupImpl const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#21 0x00007f1e7aff42be in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetupImpl const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#22 0x00007f1e7af84360 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#23 0x00007f1e7af833ce in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#24 0x00007f1e7af84b6d in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#25 0x00007f1e7af84812 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#26 0x00007f1e77f44931 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f1e741ba600, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:521
#27 0x00007f1e7b06c231 in edm::EventProcessor::processLumis(std::shared_ptr<void> const&) () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#28 0x00007f1e7b069694 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/nweek-02575/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-05-06-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#29 0x000000000040cb89 in main::$_0::operator()() const ()
#30 0x000000000040a458 in main ()

Current Modules:

Module: SimpleCandidateFlatTableProducer:isoTrackTable (crashed)
Module: DeepBoostedJetTagsProducer:pfMassDecorrelatedDeepBoostedJetTagsAK8WithDeepInfo
Module: DeepFlavourTFJetTagsProducer:pfDeepFlavourJetTagsWithDeepInfo
Module: DeepFlavourTFJetTagsProducer:pfDeepFlavourJetTagsWithDeepInfo

A fatal system signal has occurred: segmentation violation

[smuzaffar]

closing this issue. Please open a new one if you still such such threading issues.