Likely memory corruption in recent IBs

[dan131riley]

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/raw/slc7_ppc64le_gcc9/CMSSW_12_1_X_2021-09-03-2300/pyRelValMatrixLogs/run/250399.17_FS_PREMIXUP17_PU50+FS_PREMIXUP17_PU50/step1_FS_PREMIXUP17_PU50+FS_PREMIXUP17_PU50.log

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/raw/slc7_aarch64_gcc9/CMSSW_12_1_X_2021-09-05-2300/pyRelValMatrixLogs/run/23461.97_PREMIXUP26D49_PU25+2026D49PU_PMXS1+PREMIX_PremixHLBeamSpotPU/step1_PREMIXUP26D49_PU25+2026D49PU_PMXS1+PREMIX_PremixHLBeamSpotPU.log

have similar crashes in jemalloc that probably indicate some kind of memory corruption. Both crashes are in the MixingModule deep in ROOT IO class instantiation. One is on ppc64, the other on aarch64. Not seen yet on amd64.

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Mon Sep  6 09:57:50 CEST 2021
Thread 15 (Thread 0x100958483c0 (LWP 86853) "cmsRun"):
#0  0x0000010023e44e24 in poll () from /lib64/libc.so.6
#1  0x0000010025ce90c0 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/pluginFWCoreServicesPlugins.so
#2  0x0000010025ce9a44 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/pluginFWCoreServicesPlugins.so
#3  0x0000010025cecc88 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x0000010023357594 in ffs_lu (bitmap=<optimized out>) at include/jemalloc/internal/bit_util.h:22
#6  bitmap_sfu (binfo=<optimized out>, bitmap=<optimized out>) at include/jemalloc/internal/bitmap.h:315
#7  arena_slab_reg_alloc_batch (ptrs=<optimized out>, cnt=1, bin_info=<optimized out>, slab=0x103a000b840) at src/arena.c:296
#8  je_arena_tcache_fill_small (tsdn=0x1009584df60, arena=0x103a0000c80, tcache=<optimized out>, tbin=0x1009584e258, binind=9, prof_accumbytes=<optimized out>) at src/arena.c:1402
#9  0x00000100233a0db4 in je_tcache_alloc_small_hard (tsdn=tsdn@entry=0x1009584df60, arena=arena@entry=0x103a0000c80, tcache=tcache@entry=0x1009584e170, tbin=tbin@entry=0x1009584e258, binind=<optimized out>, tcache_success=tcache_success@entry=0x100958444a8) at src/tcache.c:94
#10 0x000001002334d214 in tcache_alloc_small (slow_path=false, zero=false, binind=<optimized out>, size=<optimized out>, tcache=0x1009584e170, arena=0x103a0000c80, tsd=0x1009584df60) at include/jemalloc/internal/tcache_inlines.h:60
#11 arena_malloc (slow_path=false, tcache=0x1009584e170, zero=false, ind=<optimized out>, size=<optimized out>, arena=0x0, tsdn=<optimized out>) at include/jemalloc/internal/arena_inlines_b.h:165
#12 iallocztm (slow_path=false, arena=0x0, is_internal=false, tcache=0x1009584e170, zero=false, ind=<optimized out>, size=<optimized out>, tsdn=<optimized out>) at include/jemalloc/internal/jemalloc_internal_inlines_c.h:53
#13 imalloc_no_sample (ind=<optimized out>, usize=160, size=<optimized out>, tsd=<optimized out>, dopts=<synthetic pointer>, sopts=<synthetic pointer>) at src/jemalloc.c:1949
#14 imalloc_body (tsd=<optimized out>, dopts=<synthetic pointer>, sopts=<synthetic pointer>) at src/jemalloc.c:2149
#15 imalloc (dopts=<synthetic pointer>, sopts=<synthetic pointer>) at src/jemalloc.c:2260
#16 je_malloc_default (size=<optimized out>) at src/jemalloc.c:2291
#17 0x000001002334dab4 in malloc (size=size@entry=136) at src/jemalloc.c:2390
#18 0x00000100233a54dc in newImpl<false> (size=136) at src/jemalloc_cpp.cpp:77
#19 operator new (size=136) at src/jemalloc_cpp.cpp:87
#20 0x0000010022fc94ac in TStorage::ObjectAlloc(unsigned long) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libCore.so
#21 0x00000100225410d8 in ROOT::new_TLeafElement(void*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libTree.so
#22 0x00000100230663fc in TClass::NewObject(TClass::ENewType, bool) const () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libCore.so
#23 0x0000010023066814 in TClass::New(TClass::ENewType, bool) const () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libCore.so
#24 0x0000010022a9adc0 in TBufferFile::ReadObjectAny(TClass const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#25 0x0000010023034410 in TObjArray::Streamer(TBuffer&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libCore.so
#26 0x0000010022cd740c in int TStreamerInfo::ReadBuffer<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#27 0x0000010022b5b24c in TStreamerInfoActions::GenericReadAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#28 0x0000010022a99ab4 in TBufferFile::ReadClassBuffer(TClass const*, void*, int, unsigned int, unsigned int, TClass const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#29 0x0000010022563378 in TBranch::Streamer(TBuffer&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libTree.so
#30 0x00000100230a5fdc in TStreamerBase::ReadBuffer(TBuffer&, char*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libCore.so
#31 0x0000010022cd8604 in int TStreamerInfo::ReadBuffer<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#32 0x0000010022b5b24c in TStreamerInfoActions::GenericReadAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#33 0x0000010022a9a1ec in TBufferFile::ReadClassBuffer(TClass const*, void*, TClass const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#34 0x000001002257a090 in TBranchElement::Streamer(TBuffer&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libTree.so
#35 0x0000010022a9ae0c in TBufferFile::ReadObjectAny(TClass const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#36 0x0000010023034410 in TObjArray::Streamer(TBuffer&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libCore.so
#37 0x0000010022cd740c in int TStreamerInfo::ReadBuffer<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#38 0x0000010022b5b24c in TStreamerInfoActions::GenericReadAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#39 0x0000010022a99ab4 in TBufferFile::ReadClassBuffer(TClass const*, void*, int, unsigned int, unsigned int, TClass const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#40 0x0000010022563378 in TBranch::Streamer(TBuffer&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libTree.so
#41 0x00000100230a5fdc in TStreamerBase::ReadBuffer(TBuffer&, char*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libCore.so
#42 0x0000010022cd8604 in int TStreamerInfo::ReadBuffer<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#43 0x0000010022b5b24c in TStreamerInfoActions::GenericReadAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#44 0x0000010022a9a1ec in TBufferFile::ReadClassBuffer(TClass const*, void*, TClass const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#45 0x000001002257a090 in TBranchElement::Streamer(TBuffer&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libTree.so
#46 0x0000010022a9ae0c in TBufferFile::ReadObjectAny(TClass const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#47 0x0000010023034410 in TObjArray::Streamer(TBuffer&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libCore.so
#48 0x0000010022cd740c in int TStreamerInfo::ReadBuffer<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#49 0x0000010022b5b24c in TStreamerInfoActions::GenericReadAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#50 0x0000010022a99ab4 in TBufferFile::ReadClassBuffer(TClass const*, void*, int, unsigned int, unsigned int, TClass const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#51 0x00000100225d7d30 in TTree::Streamer(TBuffer&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libTree.so
#52 0x0000010022b33ad8 in TKey::ReadObj() () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#53 0x0000010022ae93a4 in TDirectoryFile::Get(char const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#54 0x0000010092ebff58 in edm::RootTree::init(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int, unsigned int) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/pluginIOPoolInput.so
#55 0x0000010092ec038c in edm::RootTree::RootTree(std::shared_ptr<edm::InputFile>, edm::BranchType const&, unsigned int, unsigned int, unsigned int, unsigned int, bool, edm::InputType) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/pluginIOPoolInput.so
#56 0x0000010092eaa1f0 in edm::RootFile::RootFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, edm::ProcessConfiguration const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<edm::InputFile>, std::shared_ptr<edm::EventSkipperByID>, bool, int, int, unsigned int, unsigned int, int, edm::InputSource::ProcessingMode, edm::RunHelperBase*, bool, edm::ProductSelectorRules const&, edm::InputType, std::shared_ptr<edm::BranchIDListHelper>, edm::ProcessBlockHelper*, std::shared_ptr<edm::ThinnedAssociationsHelper>, std::vector<edm::BranchID, std::allocator<edm::BranchID> > const*, std::shared_ptr<edm::DuplicateChecker>, bool, edm::ProcessHistoryRegistry&, std::vector<std::shared_ptr<edm::IndexIntoFile>, std::allocator<std::shared_ptr<edm::IndexIntoFile> > > const&, unsigned long, std::vector<edm::Hash<2>, std::allocator<edm::Hash<2> > >&, bool, bool, bool, bool, bool) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/pluginIOPoolInput.so
#57 0x0000010092e92ae8 in edm::RootEmbeddedFileSequence::makeRootFile(std::shared_ptr<edm::InputFile>) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/pluginIOPoolInput.so
#58 0x0000010092eb7120 in edm::RootInputFileSequence::initTheFile(bool, bool, edm::InputSource*, char const*, edm::InputType) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/pluginIOPoolInput.so
#59 0x0000010092e92120 in edm::RootEmbeddedFileSequence::readOneRandom(edm::EventPrincipal&, unsigned long&, CLHEP::HepRandomEngine*, edm::EventID const*, bool) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/pluginIOPoolInput.so
#60 0x0000010092d32230 in void edm::PileUp::readPileUp<std::_Bind<bool (edm::MixingModule::*(std::reference_wrapper<edm::MixingModule>, std::_Placeholder<1>, edm::ModuleCallingContext const*, int, std::_Placeholder<2>, int, std::reference_wrapper<edm::EventSetup const>, edm::StreamID))(edm::EventPrincipal const&, edm::ModuleCallingContext const*, int, int, int&, edm::EventSetup const&, edm::StreamID const&)> >(edm::EventID const&, std::vector<edm::SecondaryEventIDAndFileInfo, std::allocator<edm::SecondaryEventIDAndFileInfo> >&, std::_Bind<bool (edm::MixingModule::*(std::reference_wrapper<edm::MixingModule>, std::_Placeholder<1>, edm::ModuleCallingContext const*, int, std::_Placeholder<2>, int, std::reference_wrapper<edm::EventSetup const>, edm::StreamID))(edm::EventPrincipal const&, edm::ModuleCallingContext const*, int, int, int&, edm::EventSetup const&, edm::StreamID const&)>, int, edm::StreamID const&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/pluginSimGeneralMixingModulePlugins.so
#61 0x0000010092d29204 in edm::MixingModule::doPileUp(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/pluginSimGeneralMixingModulePlugins.so
#62 0x0000010092dd5b90 in edm::BMixingModule::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/libMixingBase.so
#63 0x0000010021f0fa20 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/libFWCoreFramework.so
#64 0x0000010021ef19fc in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/libFWCoreFramework.so
#65 0x0000010021e508a0 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/libFWCoreFramework.so
#66 0x0000010021e50a94 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/libFWCoreFramework.so
#67 0x0000010021e50e08 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/libFWCoreFramework.so
#68 0x0000010021e5342c in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/libFWCoreFramework.so
#69 0x0000010022408810 in tbb::detail::d1::function_task<edm::WaitingTaskWithArenaHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}::operator()() const::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/libFWCoreConcurrency.so
#70 0x000001002384f524 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (this=0x100248cfe80, t=0x100247fee00, t@entry=0x0, waiter=...) at /home/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_aarch64_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/task_dispatcher.h:321
#71 0x000001002384bcc8 in tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=..., t=0x0, this=<optimized out>) at /home/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_aarch64_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/governor.h:149
#72 tbb::detail::r1::arena::process (this=0x100248cf780, tls=...) at /home/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_aarch64_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/arena.cpp:138
#73 0x00000100238558b8 in tbb::detail::r1::market::process (this=0x100248cb080, j=...) at /home/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_aarch64_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/market.cpp:596
#74 0x0000010023859708 in tbb::detail::r1::rml::private_worker::run (this=0x10024fcc000) at /home/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_aarch64_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/private_server.cpp:267
#75 0x000001002385997c in tbb::detail::r1::rml::private_worker::thread_routine (arg=<optimized out>) at /home/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_aarch64_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/private_server.cpp:221
#76 0x0000010023d37d38 in start_thread () from /lib64/libpthread.so.0
#77 0x0000010023e4f690 in thread_start () from /lib64/libc.so.6
Thread 14 (Thread 0x10094e383c0 (LWP 86852) "cmsRun"):
#0  0x0000010023e1a9c4 in nanosleep () from /lib64/libc.so.6
#1  0x0000010023e1a678 in sleep () from /lib64/libc.so.6
#2  0x0000010025ce8a50 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000001002255db68 in TBranch::TBranch() () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libTree.so
#5  0x0000010022568b1c in TBranchElement::TBranchElement() () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libTree.so
#6  0x000001002253ebf0 in ROOT::new_TBranchElement(void*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libTree.so
#7  0x00000100230663fc in TClass::NewObject(TClass::ENewType, bool) const () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libCore.so
#8  0x0000010023066814 in TClass::New(TClass::ENewType, bool) const () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libCore.so
#9  0x0000010022a9adc0 in TBufferFile::ReadObjectAny(TClass const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#10 0x0000010023034410 in TObjArray::Streamer(TBuffer&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libCore.so
#11 0x0000010022cd740c in int TStreamerInfo::ReadBuffer<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#12 0x0000010022b5b24c in TStreamerInfoActions::GenericReadAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#13 0x0000010022a99ab4 in TBufferFile::ReadClassBuffer(TClass const*, void*, int, unsigned int, unsigned int, TClass const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#14 0x00000100225d7d30 in TTree::Streamer(TBuffer&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libTree.so
#15 0x0000010022b33ad8 in TKey::ReadObj() () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#16 0x0000010022ae93a4 in TDirectoryFile::Get(char const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#17 0x0000010092ebff58 in edm::RootTree::init(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int, unsigned int) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/pluginIOPoolInput.so
#18 0x0000010092ec038c in edm::RootTree::RootTree(std::shared_ptr<edm::InputFile>, edm::BranchType const&, unsigned int, unsigned int, unsigned int, unsigned int, bool, edm::InputType) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/pluginIOPoolInput.so
#19 0x0000010092eaa1f0 in edm::RootFile::RootFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, edm::ProcessConfiguration const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<edm::InputFile>, std::shared_ptr<edm::EventSkipperByID>, bool, int, int, unsigned int, unsigned int, int, edm::InputSource::ProcessingMode, edm::RunHelperBase*, bool, edm::ProductSelectorRules const&, edm::InputType, std::shared_ptr<edm::BranchIDListHelper>, edm::ProcessBlockHelper*, std::shared_ptr<edm::ThinnedAssociationsHelper>, std::vector<edm::BranchID, std::allocator<edm::BranchID> > const*, std::shared_ptr<edm::DuplicateChecker>, bool, edm::ProcessHistoryRegistry&, std::vector<std::shared_ptr<edm::IndexIntoFile>, std::allocator<std::shared_ptr<edm::IndexIntoFile> > > const&, unsigned long, std::vector<edm::Hash<2>, std::allocator<edm::Hash<2> > >&, bool, bool, bool, bool, bool) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/pluginIOPoolInput.so
#20 0x0000010092e92ae8 in edm::RootEmbeddedFileSequence::makeRootFile(std::shared_ptr<edm::InputFile>) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/pluginIOPoolInput.so
#21 0x0000010092eb7120 in edm::RootInputFileSequence::initTheFile(bool, bool, edm::InputSource*, char const*, edm::InputType) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/pluginIOPoolInput.so
#22 0x0000010092e92120 in edm::RootEmbeddedFileSequence::readOneRandom(edm::EventPrincipal&, unsigned long&, CLHEP::HepRandomEngine*, edm::EventID const*, bool) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/pluginIOPoolInput.so
#23 0x0000010092d32230 in void edm::PileUp::readPileUp<std::_Bind<bool (edm::MixingModule::*(std::reference_wrapper<edm::MixingModule>, std::_Placeholder<1>, edm::ModuleCallingContext const*, int, std::_Placeholder<2>, int, std::reference_wrapper<edm::EventSetup const>, edm::StreamID))(edm::EventPrincipal const&, edm::ModuleCallingContext const*, int, int, int&, edm::EventSetup const&, edm::StreamID const&)> >(edm::EventID const&, std::vector<edm::SecondaryEventIDAndFileInfo, std::allocator<edm::SecondaryEventIDAndFileInfo> >&, std::_Bind<bool (edm::MixingModule::*(std::reference_wrapper<edm::MixingModule>, std::_Placeholder<1>, edm::ModuleCallingContext const*, int, std::_Placeholder<2>, int, std::reference_wrapper<edm::EventSetup const>, edm::StreamID))(edm::EventPrincipal const&, edm::ModuleCallingContext const*, int, int, int&, edm::EventSetup const&, edm::StreamID const&)>, int, edm::StreamID const&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/pluginSimGeneralMixingModulePlugins.so
#24 0x0000010092d29204 in edm::MixingModule::doPileUp(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/pluginSimGeneralMixingModulePlugins.so
#25 0x0000010092dd5b90 in edm::BMixingModule::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/libMixingBase.so
#26 0x0000010021f0fa20 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/libFWCoreFramework.so
#27 0x0000010021ef19fc in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/libFWCoreFramework.so
#28 0x0000010021e508a0 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/libFWCoreFramework.so
#29 0x0000010021e50a94 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/libFWCoreFramework.so
#30 0x0000010021e50e08 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/libFWCoreFramework.so
#31 0x0000010021e5342c in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/libFWCoreFramework.so
#32 0x0000010022408810 in tbb::detail::d1::function_task<edm::WaitingTaskWithArenaHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}::operator()() const::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/libFWCoreConcurrency.so
#33 0x000001002384f524 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (this=0x100248cff00, t=0x100247f7b00, t@entry=0x0, waiter=...) at /home/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_aarch64_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/task_dispatcher.h:321
#34 0x000001002384bcc8 in tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=..., t=0x0, this=<optimized out>) at /home/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_aarch64_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/governor.h:149
#35 tbb::detail::r1::arena::process (this=0x100248cf780, tls=...) at /home/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_aarch64_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/arena.cpp:138
#36 0x00000100238558b8 in tbb::detail::r1::market::process (this=0x100248cb080, j=...) at /home/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_aarch64_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/market.cpp:596
#37 0x0000010023859708 in tbb::detail::r1::rml::private_worker::run (this=0x10024fcc100) at /home/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_aarch64_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/private_server.cpp:267
#38 0x000001002385997c in tbb::detail::r1::rml::private_worker::thread_routine (arg=<optimized out>) at /home/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_aarch64_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/private_server.cpp:221
#39 0x0000010023d37d38 in start_thread () from /lib64/libpthread.so.0
#40 0x0000010023e4f690 in thread_start () from /lib64/libc.so.6
Thread 13 (Thread 0x100944283c0 (LWP 86851) "cmsRun"):
#0  0x0000010023e1a9c4 in nanosleep () from /lib64/libc.so.6
#1  0x0000010023e1a678 in sleep () from /lib64/libc.so.6
#2  0x0000010025ce8a50 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000010022bcb838 in int TStreamerInfoActions::ReadBasicType<int>(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#5  0x0000010022a99ab4 in TBufferFile::ReadClassBuffer(TClass const*, void*, int, unsigned int, unsigned int, TClass const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#6  0x0000010022563378 in TBranch::Streamer(TBuffer&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libTree.so
#7  0x00000100230a5fdc in TStreamerBase::ReadBuffer(TBuffer&, char*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libCore.so
#8  0x0000010022cd8604 in int TStreamerInfo::ReadBuffer<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#9  0x0000010022b5b24c in TStreamerInfoActions::GenericReadAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#10 0x0000010022a9a1ec in TBufferFile::ReadClassBuffer(TClass const*, void*, TClass const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#11 0x000001002257a090 in TBranchElement::Streamer(TBuffer&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libTree.so
#12 0x0000010022a9ae0c in TBufferFile::ReadObjectAny(TClass const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#13 0x0000010023034410 in TObjArray::Streamer(TBuffer&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libCore.so
#14 0x0000010022cd740c in int TStreamerInfo::ReadBuffer<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#15 0x0000010022b5b24c in TStreamerInfoActions::GenericReadAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#16 0x0000010022a99ab4 in TBufferFile::ReadClassBuffer(TClass const*, void*, int, unsigned int, unsigned int, TClass const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#17 0x0000010022563378 in TBranch::Streamer(TBuffer&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libTree.so
#18 0x00000100230a5fdc in TStreamerBase::ReadBuffer(TBuffer&, char*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libCore.so
#19 0x0000010022cd8604 in int TStreamerInfo::ReadBuffer<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#20 0x0000010022b5b24c in TStreamerInfoActions::GenericReadAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#21 0x0000010022a9a1ec in TBufferFile::ReadClassBuffer(TClass const*, void*, TClass const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#22 0x000001002257a090 in TBranchElement::Streamer(TBuffer&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libTree.so
#23 0x0000010022a9ae0c in TBufferFile::ReadObjectAny(TClass const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#24 0x0000010023034410 in TObjArray::Streamer(TBuffer&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libCore.so
#25 0x0000010022cd740c in int TStreamerInfo::ReadBuffer<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#26 0x0000010022b5b24c in TStreamerInfoActions::GenericReadAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#27 0x0000010022a99ab4 in TBufferFile::ReadClassBuffer(TClass const*, void*, int, unsigned int, unsigned int, TClass const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#28 0x00000100225d7d30 in TTree::Streamer(TBuffer&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libTree.so
#29 0x0000010022b33ad8 in TKey::ReadObj() () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#30 0x0000010022ae93a4 in TDirectoryFile::Get(char const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#31 0x0000010092ebff58 in edm::RootTree::init(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int, unsigned int) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/pluginIOPoolInput.so
#32 0x0000010092ec038c in edm::RootTree::RootTree(std::shared_ptr<edm::InputFile>, edm::BranchType const&, unsigned int, unsigned int, unsigned int, unsigned int, bool, edm::InputType) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/pluginIOPoolInput.so
#33 0x0000010092eaa1f0 in edm::RootFile::RootFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, edm::ProcessConfiguration const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<edm::InputFile>, std::shared_ptr<edm::EventSkipperByID>, bool, int, int, unsigned int, unsigned int, int, edm::InputSource::ProcessingMode, edm::RunHelperBase*, bool, edm::ProductSelectorRules const&, edm::InputType, std::shared_ptr<edm::BranchIDListHelper>, edm::ProcessBlockHelper*, std::shared_ptr<edm::ThinnedAssociationsHelper>, std::vector<edm::BranchID, std::allocator<edm::BranchID> > const*, std::shared_ptr<edm::DuplicateChecker>, bool, edm::ProcessHistoryRegistry&, std::vector<std::shared_ptr<edm::IndexIntoFile>, std::allocator<std::shared_ptr<edm::IndexIntoFile> > > const&, unsigned long, std::vector<edm::Hash<2>, std::allocator<edm::Hash<2> > >&, bool, bool, bool, bool, bool) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/pluginIOPoolInput.so
#34 0x0000010092e92ae8 in edm::RootEmbeddedFileSequence::makeRootFile(std::shared_ptr<edm::InputFile>) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/pluginIOPoolInput.so
#35 0x0000010092eb7120 in edm::RootInputFileSequence::initTheFile(bool, bool, edm::InputSource*, char const*, edm::InputType) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/pluginIOPoolInput.so
#36 0x0000010092e92120 in edm::RootEmbeddedFileSequence::readOneRandom(edm::EventPrincipal&, unsigned long&, CLHEP::HepRandomEngine*, edm::EventID const*, bool) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/pluginIOPoolInput.so
#37 0x0000010092d32230 in void edm::PileUp::readPileUp<std::_Bind<bool (edm::MixingModule::*(std::reference_wrapper<edm::MixingModule>, std::_Placeholder<1>, edm::ModuleCallingContext const*, int, std::_Placeholder<2>, int, std::reference_wrapper<edm::EventSetup const>, edm::StreamID))(edm::EventPrincipal const&, edm::ModuleCallingContext const*, int, int, int&, edm::EventSetup const&, edm::StreamID const&)> >(edm::EventID const&, std::vector<edm::SecondaryEventIDAndFileInfo, std::allocator<edm::SecondaryEventIDAndFileInfo> >&, std::_Bind<bool (edm::MixingModule::*(std::reference_wrapper<edm::MixingModule>, std::_Placeholder<1>, edm::ModuleCallingContext const*, int, std::_Placeholder<2>, int, std::reference_wrapper<edm::EventSetup const>, edm::StreamID))(edm::EventPrincipal const&, edm::ModuleCallingContext const*, int, int, int&, edm::EventSetup const&, edm::StreamID const&)>, int, edm::StreamID const&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/pluginSimGeneralMixingModulePlugins.so
#38 0x0000010092d29204 in edm::MixingModule::doPileUp(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/pluginSimGeneralMixingModulePlugins.so
#39 0x0000010092dd5b90 in edm::BMixingModule::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/libMixingBase.so
#40 0x0000010021f0fa20 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/libFWCoreFramework.so
#41 0x0000010021ef19fc in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/libFWCoreFramework.so
#42 0x0000010021e508a0 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/libFWCoreFramework.so
#43 0x0000010021e50a94 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/libFWCoreFramework.so
#44 0x0000010021e50e08 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/libFWCoreFramework.so
#45 0x0000010021e5342c in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/libFWCoreFramework.so
#46 0x0000010022408810 in tbb::detail::d1::function_task<edm::WaitingTaskWithArenaHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}::operator()() const::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/libFWCoreConcurrency.so
#47 0x000001002384f524 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (this=0x100248cfe00, t=0x100247dfd00, t@entry=0x0, waiter=...) at /home/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_aarch64_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/task_dispatcher.h:321
#48 0x000001002384bcc8 in tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=..., t=0x0, this=<optimized out>) at /home/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_aarch64_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/governor.h:149
#49 tbb::detail::r1::arena::process (this=0x100248cf780, tls=...) at /home/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_aarch64_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/arena.cpp:138
#50 0x00000100238558b8 in tbb::detail::r1::market::process (this=0x100248cb080, j=...) at /home/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_aarch64_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/market.cpp:596
#51 0x0000010023859708 in tbb::detail::r1::rml::private_worker::run (this=0x10024fcc080) at /home/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_aarch64_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/private_server.cpp:267
#52 0x000001002385997c in tbb::detail::r1::rml::private_worker::thread_routine (arg=<optimized out>) at /home/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_aarch64_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/private_server.cpp:221
#53 0x0000010023d37d38 in start_thread () from /lib64/libpthread.so.0
#54 0x0000010023e4f690 in thread_start () from /lib64/libc.so.6
Thread 12 (Thread 0x100939b83c0 (LWP 86850) "cmsRun"):
#0  0x0000010023d3e334 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x0000010023d3e3d8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x0000010023d3e4b4 in sem_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
#3  0x0000010025939910 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libXrdCl.so.2
#4  0x00000100259399e4 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libXrdCl.so.2
#5  0x0000010023d37d38 in start_thread () from /lib64/libpthread.so.0
#6  0x0000010023e4f690 in thread_start () from /lib64/libc.so.6
Thread 11 (Thread 0x100937a83c0 (LWP 86849) "cmsRun"):
#0  0x0000010023d3e334 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x0000010023d3e3d8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x0000010023d3e4b4 in sem_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
#3  0x0000010025939910 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libXrdCl.so.2
#4  0x00000100259399e4 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libXrdCl.so.2
#5  0x0000010023d37d38 in start_thread () from /lib64/libpthread.so.0
#6  0x0000010023e4f690 in thread_start () from /lib64/libc.so.6
Thread 10 (Thread 0x100935983c0 (LWP 86848) "cmsRun"):
#0  0x0000010023d3e334 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x0000010023d3e3d8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x0000010023d3e4b4 in sem_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
#3  0x0000010025939910 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libXrdCl.so.2
#4  0x00000100259399e4 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libXrdCl.so.2
#5  0x0000010023d37d38 in start_thread () from /lib64/libpthread.so.0
#6  0x0000010023e4f690 in thread_start () from /lib64/libc.so.6
Thread 9 (Thread 0x100933883c0 (LWP 86847) "cmsRun"):
#0  0x0000010023d3fcb4 in nanosleep () from /lib64/libpthread.so.0
#1  0x00000100259d3568 in XrdSysTimer::Wait(int) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libXrdUtils.so.2
#2  0x00000100258e6130 in XrdCl::TaskManager::RunTasks() () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libXrdCl.so.2
#3  0x00000100258e62bc in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libXrdCl.so.2
#4  0x0000010023d37d38 in start_thread () from /lib64/libpthread.so.0
#5  0x0000010023e4f690 in thread_start () from /lib64/libc.so.6
Thread 8 (Thread 0x100931783c0 (LWP 86846) "cmsRun"):
#0  0x0000010023e4f864 in epoll_pwait () from /lib64/libc.so.6
#1  0x00000100259cdd94 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libXrdUtils.so.2
#2  0x00000100259ca6cc in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libXrdUtils.so.2
#3  0x00000100259d2cc0 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libXrdUtils.so.2
#4  0x0000010023d37d38 in start_thread () from /lib64/libpthread.so.0
#5  0x0000010023e4f690 in thread_start () from /lib64/libc.so.6
Thread 7 (Thread 0x100922083c0 (LWP 86836) "cmsRun"):
#0  0x0000010023d3c0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
#1  0x0000010023b23470 in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /home/cmsbld/muz/w/BUILD/slc7_aarch64_gcc9/external/gcc/9.3.0/gcc-9.3.0/obj/aarch64-unknown-linux-gnu/libstdc++-v3/include/aarch64-unknown-linux-gnu/bits/gthr-default.h:865
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x000001008183102c in omt::ThreadHandoff::threadLoop(void*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/biglib/slc7_aarch64_gcc9/pluginSimulation.so
#4  0x0000010023d37d38 in start_thread () from /lib64/libpthread.so.0
#5  0x0000010023e4f690 in thread_start () from /lib64/libc.so.6
Thread 6 (Thread 0x100917f83c0 (LWP 86835) "cmsRun"):
#0  0x0000010023d3c0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
#1  0x0000010023b23470 in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /home/cmsbld/muz/w/BUILD/slc7_aarch64_gcc9/external/gcc/9.3.0/gcc-9.3.0/obj/aarch64-unknown-linux-gnu/libstdc++-v3/include/aarch64-unknown-linux-gnu/bits/gthr-default.h:865
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x000001008183102c in omt::ThreadHandoff::threadLoop(void*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/biglib/slc7_aarch64_gcc9/pluginSimulation.so
#4  0x0000010023d37d38 in start_thread () from /lib64/libpthread.so.0
#5  0x0000010023e4f690 in thread_start () from /lib64/libc.so.6
Thread 5 (Thread 0x10090de83c0 (LWP 86834) "cmsRun"):
#0  0x0000010023d3c0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
#1  0x0000010023b23470 in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /home/cmsbld/muz/w/BUILD/slc7_aarch64_gcc9/external/gcc/9.3.0/gcc-9.3.0/obj/aarch64-unknown-linux-gnu/libstdc++-v3/include/aarch64-unknown-linux-gnu/bits/gthr-default.h:865
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x000001008183102c in omt::ThreadHandoff::threadLoop(void*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/biglib/slc7_aarch64_gcc9/pluginSimulation.so
#4  0x0000010023d37d38 in start_thread () from /lib64/libpthread.so.0
#5  0x0000010023e4f690 in thread_start () from /lib64/libc.so.6
Thread 4 (Thread 0x100903d83c0 (LWP 86833) "cmsRun"):
#0  0x0000010023d3c0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
#1  0x0000010023b23470 in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /home/cmsbld/muz/w/BUILD/slc7_aarch64_gcc9/external/gcc/9.3.0/gcc-9.3.0/obj/aarch64-unknown-linux-gnu/libstdc++-v3/include/aarch64-unknown-linux-gnu/bits/gthr-default.h:865
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x000001008183102c in omt::ThreadHandoff::threadLoop(void*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/biglib/slc7_aarch64_gcc9/pluginSimulation.so
#4  0x0000010023d37d38 in start_thread () from /lib64/libpthread.so.0
#5  0x0000010023e4f690 in thread_start () from /lib64/libc.so.6
Thread 3 (Thread 0x1008f9c83c0 (LWP 86832) "cmsRun"):
#0  0x0000010023d3c0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
#1  0x0000010023b23470 in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /home/cmsbld/muz/w/BUILD/slc7_aarch64_gcc9/external/gcc/9.3.0/gcc-9.3.0/obj/aarch64-unknown-linux-gnu/libstdc++-v3/include/aarch64-unknown-linux-gnu/bits/gthr-default.h:865
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00000100818189ec in OscarMTMasterThread::OscarMTMasterThread(edm::ParameterSet const&)::{lambda()#1}::operator()() const [clone .isra.0] () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/biglib/slc7_aarch64_gcc9/pluginSimulation.so
#4  0x0000010023b28bec in std::execute_native_thread_routine (__p=0x1006710e340) at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
#5  0x0000010023d37d38 in start_thread () from /lib64/libpthread.so.0
#6  0x0000010023e4f690 in thread_start () from /lib64/libc.so.6
Thread 2 (Thread 0x100331b83c0 (LWP 86605) "cmsRun"):
#0  0x0000010023d404d8 in waitpid () from /lib64/libpthread.so.0
#1  0x0000010025ce8c50 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/pluginFWCoreServicesPlugins.so
#2  0x0000010025ce99b0 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/pluginFWCoreServicesPlugins.so
#3  0x0000010023b28bec in std::execute_native_thread_routine (__p=0x100603ac140) at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
#4  0x0000010023d37d38 in start_thread () from /lib64/libpthread.so.0
#5  0x0000010023e4f690 in thread_start () from /lib64/libc.so.6
Thread 1 (Thread 0x10024600000 (LWP 86514) "cmsRun"):
#0  0x0000010023e1a9c4 in nanosleep () from /lib64/libc.so.6
#1  0x0000010023e1a678 in sleep () from /lib64/libc.so.6
#2  0x0000010025ce8a50 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000010022a92100 in TBufferFile::ReadFastArray(long long*, int) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#5  0x0000010022cd3d60 in int TStreamerInfo::ReadBuffer<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#6  0x0000010022b5b24c in TStreamerInfoActions::GenericReadAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#7  0x0000010022a99ab4 in TBufferFile::ReadClassBuffer(TClass const*, void*, int, unsigned int, unsigned int, TClass const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#8  0x0000010022563378 in TBranch::Streamer(TBuffer&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libTree.so
#9  0x00000100230a5fdc in TStreamerBase::ReadBuffer(TBuffer&, char*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libCore.so
#10 0x0000010022cd8604 in int TStreamerInfo::ReadBuffer<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#11 0x0000010022b5b24c in TStreamerInfoActions::GenericReadAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#12 0x0000010022a9a1ec in TBufferFile::ReadClassBuffer(TClass const*, void*, TClass const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#13 0x000001002257a090 in TBranchElement::Streamer(TBuffer&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libTree.so
#14 0x0000010022a9ae0c in TBufferFile::ReadObjectAny(TClass const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#15 0x0000010023034410 in TObjArray::Streamer(TBuffer&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libCore.so
#16 0x0000010022cd740c in int TStreamerInfo::ReadBuffer<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#17 0x0000010022b5b24c in TStreamerInfoActions::GenericReadAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#18 0x0000010022a99ab4 in TBufferFile::ReadClassBuffer(TClass const*, void*, int, unsigned int, unsigned int, TClass const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#19 0x0000010022563378 in TBranch::Streamer(TBuffer&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libTree.so
#20 0x00000100230a5fdc in TStreamerBase::ReadBuffer(TBuffer&, char*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libCore.so
#21 0x0000010022cd8604 in int TStreamerInfo::ReadBuffer<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#22 0x0000010022b5b24c in TStreamerInfoActions::GenericReadAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#23 0x0000010022a9a1ec in TBufferFile::ReadClassBuffer(TClass const*, void*, TClass const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#24 0x000001002257a090 in TBranchElement::Streamer(TBuffer&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libTree.so
#25 0x0000010022a9ae0c in TBufferFile::ReadObjectAny(TClass const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#26 0x0000010023034410 in TObjArray::Streamer(TBuffer&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libCore.so
#27 0x0000010022cd740c in int TStreamerInfo::ReadBuffer<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#28 0x0000010022b5b24c in TStreamerInfoActions::GenericReadAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#29 0x0000010022a99ab4 in TBufferFile::ReadClassBuffer(TClass const*, void*, int, unsigned int, unsigned int, TClass const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#30 0x00000100225d7d30 in TTree::Streamer(TBuffer&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libTree.so
#31 0x0000010022b33ad8 in TKey::ReadObj() () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#32 0x0000010022ae93a4 in TDirectoryFile::Get(char const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/external/slc7_aarch64_gcc9/lib/libRIO.so
#33 0x0000010092ebff58 in edm::RootTree::init(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int, unsigned int) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/pluginIOPoolInput.so
#34 0x0000010092ec038c in edm::RootTree::RootTree(std::shared_ptr<edm::InputFile>, edm::BranchType const&, unsigned int, unsigned int, unsigned int, unsigned int, bool, edm::InputType) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/pluginIOPoolInput.so
#35 0x0000010092eaa1f0 in edm::RootFile::RootFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, edm::ProcessConfiguration const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<edm::InputFile>, std::shared_ptr<edm::EventSkipperByID>, bool, int, int, unsigned int, unsigned int, int, edm::InputSource::ProcessingMode, edm::RunHelperBase*, bool, edm::ProductSelectorRules const&, edm::InputType, std::shared_ptr<edm::BranchIDListHelper>, edm::ProcessBlockHelper*, std::shared_ptr<edm::ThinnedAssociationsHelper>, std::vector<edm::BranchID, std::allocator<edm::BranchID> > const*, std::shared_ptr<edm::DuplicateChecker>, bool, edm::ProcessHistoryRegistry&, std::vector<std::shared_ptr<edm::IndexIntoFile>, std::allocator<std::shared_ptr<edm::IndexIntoFile> > > const&, unsigned long, std::vector<edm::Hash<2>, std::allocator<edm::Hash<2> > >&, bool, bool, bool, bool, bool) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/pluginIOPoolInput.so
#36 0x0000010092e92ae8 in edm::RootEmbeddedFileSequence::makeRootFile(std::shared_ptr<edm::InputFile>) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/pluginIOPoolInput.so
#37 0x0000010092eb7120 in edm::RootInputFileSequence::initTheFile(bool, bool, edm::InputSource*, char const*, edm::InputType) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/pluginIOPoolInput.so
#38 0x0000010092e92120 in edm::RootEmbeddedFileSequence::readOneRandom(edm::EventPrincipal&, unsigned long&, CLHEP::HepRandomEngine*, edm::EventID const*, bool) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/pluginIOPoolInput.so
#39 0x0000010092d32230 in void edm::PileUp::readPileUp<std::_Bind<bool (edm::MixingModule::*(std::reference_wrapper<edm::MixingModule>, std::_Placeholder<1>, edm::ModuleCallingContext const*, int, std::_Placeholder<2>, int, std::reference_wrapper<edm::EventSetup const>, edm::StreamID))(edm::EventPrincipal const&, edm::ModuleCallingContext const*, int, int, int&, edm::EventSetup const&, edm::StreamID const&)> >(edm::EventID const&, std::vector<edm::SecondaryEventIDAndFileInfo, std::allocator<edm::SecondaryEventIDAndFileInfo> >&, std::_Bind<bool (edm::MixingModule::*(std::reference_wrapper<edm::MixingModule>, std::_Placeholder<1>, edm::ModuleCallingContext const*, int, std::_Placeholder<2>, int, std::reference_wrapper<edm::EventSetup const>, edm::StreamID))(edm::EventPrincipal const&, edm::ModuleCallingContext const*, int, int, int&, edm::EventSetup const&, edm::StreamID const&)>, int, edm::StreamID const&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/pluginSimGeneralMixingModulePlugins.so
#40 0x0000010092d29204 in edm::MixingModule::doPileUp(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/pluginSimGeneralMixingModulePlugins.so
#41 0x0000010092dd5b90 in edm::BMixingModule::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/libMixingBase.so
#42 0x0000010021f0fa20 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/libFWCoreFramework.so
#43 0x0000010021ef19fc in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/libFWCoreFramework.so
#44 0x0000010021e508a0 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/libFWCoreFramework.so
#45 0x0000010021e50a94 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/libFWCoreFramework.so
#46 0x0000010021e50e08 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/libFWCoreFramework.so
#47 0x0000010021e5342c in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/libFWCoreFramework.so
#48 0x0000010022408810 in tbb::detail::d1::function_task<edm::WaitingTaskWithArenaHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}::operator()() const::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/libFWCoreConcurrency.so
#49 0x000001002385ed74 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (this=this@entry=0x100248cfd80, t=0x10024742000, t@entry=0x0, waiter=...) at /home/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_aarch64_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/task_dispatcher.h:321
#50 0x000001002385cb54 in tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x100248cfd80) at /home/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_aarch64_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/task_dispatcher.h:462
#51 tbb::detail::r1::task_dispatcher::execute_and_wait (t=0x0, wait_ctx=..., w_ctx=...) at /home/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_aarch64_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/task_dispatcher.cpp:168
#52 0x0000010021db8680 in edm::EventProcessor::processLumis(std::shared_ptr<void> const&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/libFWCoreFramework.so
#53 0x0000010021dc40b0 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/week1/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-05-2300/lib/slc7_aarch64_gcc9/libFWCoreFramework.so
#54 0x000000000040afc8 in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#55 0x000001002384adb4 in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /home/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_aarch64_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/arena.cpp:696
#56 0x000000000040be48 in main::{lambda()#1}::operator()() const ()
#57 0x000000000040a864 in main ()

Current Modules:

Module: MixingModule:mix (crashed)
Module: MixingModule:mix
Module: MixingModule:mix
Module: MixingModule:mix

A fatal system signal has occurred: segmentation violation

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Sat Sep  4 07:18:09 CEST 2021
Thread 10 (Thread 0x100886e28390 (LWP 96567) "cmsRun"):
#0  0x0000100002c66df8 in poll () from /lib64/libc.so.6
#1  0x0000100009e36dac in full_read.constprop () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/pluginFWCoreServicesPlugins.so
#2  0x0000100009e37bf8 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/pluginFWCoreServicesPlugins.so
#3  0x0000100009e3bcec in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x0000100001e4d354 in ffs_lu (bitmap=<optimized out>) at include/jemalloc/internal/bit_util.h:22
#6  bitmap_sfu (binfo=<optimized out>, bitmap=<optimized out>) at include/jemalloc/internal/bitmap.h:315
#7  arena_slab_reg_alloc_batch (ptrs=<optimized out>, cnt=3, bin_info=<optimized out>, slab=0x10088b024240) at src/arena.c:296
#8  je_arena_tcache_fill_small (tsdn=0x100886e2df60, arena=0x10088b000c80, tcache=<optimized out>, tbin=0x100886e2e258, binind=<optimized out>, prof_accumbytes=<optimized out>) at src/arena.c:1402
#9  0x0000100001eae9fc in je_tcache_alloc_small_hard (tsdn=<optimized out>, arena=<optimized out>, tcache=<optimized out>, tbin=<optimized out>, binind=<optimized out>, tcache_success=0x100886e23aa0) at src/tcache.c:94
#10 0x0000100001e4045c in tcache_alloc_small (slow_path=false, zero=false, binind=<optimized out>, size=<optimized out>, tcache=0x100886e2e170, arena=0x10088b000c80, tsd=0x100886e2df60) at include/jemalloc/internal/tcache_inlines.h:60
#11 arena_malloc (slow_path=false, tcache=0x100886e2e170, zero=false, ind=<optimized out>, size=<optimized out>, arena=0x0, tsdn=0x100886e2df60) at include/jemalloc/internal/arena_inlines_b.h:165
#12 iallocztm (slow_path=false, arena=0x0, is_internal=false, tcache=0x100886e2e170, zero=false, ind=<optimized out>, size=<optimized out>, tsdn=0x100886e2df60) at include/jemalloc/internal/jemalloc_internal_inlines_c.h:53
#13 imalloc_no_sample (ind=<optimized out>, usize=160, size=<optimized out>, tsd=0x100886e2df60, dopts=<synthetic pointer>, sopts=<synthetic pointer>) at src/jemalloc.c:1949
#14 imalloc_body (tsd=0x100886e2df60, dopts=<synthetic pointer>, sopts=<synthetic pointer>) at src/jemalloc.c:2149
#15 imalloc (dopts=<synthetic pointer>, sopts=<synthetic pointer>) at src/jemalloc.c:2260
#16 je_malloc_default (size=<optimized out>) at src/jemalloc.c:2291
#17 0x0000100001eb43a0 in newImpl<false> (size=<optimized out>) at src/jemalloc_cpp.cpp:78
#18 operator new (size=<optimized out>) at src/jemalloc_cpp.cpp:87
#19 0x0000100821fabf80 in ROOT::new_FastSingleTrackerRecHit(void*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libDataFormatsTrackerRecHit2D.so
#20 0x0000100001a576e0 in TClass::NewObject(TClass::ENewType, bool) const () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libCore.so
#21 0x0000100001a57ce4 in TClass::New(TClass::ENewType, bool) const () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libCore.so
#22 0x00001000012f29f0 in TBufferFile::ReadObjectAny(TClass const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libRIO.so
#23 0x00001000013993c0 in TGenCollectionStreamer::ReadObjects(int, TBuffer&, TClass const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libRIO.so
#24 0x000010000139a214 in TGenCollectionStreamer::ReadBufferGeneric(TBuffer&, void*, TClass const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libRIO.so
#25 0x0000100001394720 in TGenCollectionStreamer::ReadBufferDefault(TBuffer&, void*, TClass const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libRIO.so
#26 0x000010000139455c in TGenCollectionStreamer::ReadBuffer(TBuffer&, void*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libRIO.so
#27 0x00001000012dd758 in TCollectionClassStreamer::Stream(TBuffer&, void*, TClass const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libRIO.so
#28 0x0000100001a4e144 in TClass::StreamerExternal(TClass const*, void*, TBuffer&, TClass const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libCore.so
#29 0x00001000012e5ef0 in TBufferFile::ReadFastArray(void*, TClass const*, int, TMemberStreamer*, TClass const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libRIO.so
#30 0x00001000014bed48 in int TStreamerInfoActions::ReadSTL<&TStreamerInfoActions::ReadSTLMemberWiseSameClass, &TStreamerInfoActions::ReadSTLObjectWiseFastArray>(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libRIO.so
#31 0x00001000012e526c in TBufferFile::ApplySequence(TStreamerInfoActions::TActionSequence const&, void*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libRIO.so
#32 0x0000100000c8a838 in TBranchElement::ReadLeavesMember(TBuffer&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libTree.so
#33 0x0000100000c7ec98 in TBranch::GetEntry(long long, int) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libTree.so
#34 0x0000100000c97918 in TBranchElement::GetEntry(long long, int) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libTree.so
#35 0x0000100000c978b8 in TBranchElement::GetEntry(long long, int) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libTree.so
#36 0x0000100000c978b8 in TBranchElement::GetEntry(long long, int) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libTree.so
#37 0x0000100877446b24 in edm::RootTree::getEntry(TBranch*, long long) const () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/pluginIOPoolInput.so
#38 0x0000100877403648 in edm::RootDelayedReader::getProduct_(edm::BranchID const&, edm::EDProductGetter const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/pluginIOPoolInput.so
#39 0x00001000001f7688 in edm::DelayedReader::getProduct(edm::BranchID const&, edm::EDProductGetter const*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libFWCoreFramework.so
#40 0x000010000034e450 in edm::DelayedReaderInputProductResolver::resolveProduct_(edm::Principal const&, bool, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libFWCoreFramework.so
#41 0x00001000003499fc in edm::SingleChoiceNoProcessProductResolver::resolveProduct_(edm::Principal const&, bool, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libFWCoreFramework.so
#42 0x0000100000332904 in edm::Principal::findProductByLabel(edm::KindOfType, edm::TypeID const&, edm::InputTag const&, edm::EDConsumerBase const*, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libFWCoreFramework.so
#43 0x0000100000332b4c in edm::Principal::findProductByTag(edm::TypeID const&, edm::InputTag const&, edm::ModuleCallingContext const*) const () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libFWCoreFramework.so
#44 0x000010087721993c in edm::Adjuster<edm::OwnVector<TrackingRecHit, edm::ClonePolicy<TrackingRecHit> > >::doOffset(int, int, edm::EventPrincipal const&, edm::ModuleCallingContext const*, unsigned int, int) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/pluginSimGeneralMixingModulePlugins.so
#45 0x000010087720da40 in edm::MixingModule::pileAllWorkers(edm::EventPrincipal const&, edm::ModuleCallingContext const*, int, int, int&, edm::EventSetup const&, edm::StreamID const&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/pluginSimGeneralMixingModulePlugins.so
#46 0x000010087721d39c in void edm::PileUp::readPileUp<std::_Bind<bool (edm::MixingModule::*(std::reference_wrapper<edm::MixingModule>, std::_Placeholder<1>, edm::ModuleCallingContext const*, int, std::_Placeholder<2>, int, std::reference_wrapper<edm::EventSetup const>, edm::StreamID))(edm::EventPrincipal const&, edm::ModuleCallingContext const*, int, int, int&, edm::EventSetup const&, edm::StreamID const&)> >(edm::EventID const&, std::vector<edm::SecondaryEventIDAndFileInfo, std::allocator<edm::SecondaryEventIDAndFileInfo> >&, std::_Bind<bool (edm::MixingModule::*(std::reference_wrapper<edm::MixingModule>, std::_Placeholder<1>, edm::ModuleCallingContext const*, int, std::_Placeholder<2>, int, std::reference_wrapper<edm::EventSetup const>, edm::StreamID))(edm::EventPrincipal const&, edm::ModuleCallingContext const*, int, int, int&, edm::EventSetup const&, edm::StreamID const&)>, int, edm::StreamID const&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/pluginSimGeneralMixingModulePlugins.so
#47 0x000010087720fc5c in edm::MixingModule::doPileUp(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/pluginSimGeneralMixingModulePlugins.so
#48 0x00001008772d7bbc in edm::BMixingModule::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libMixingBase.so
#49 0x0000100000442718 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libFWCoreFramework.so
#50 0x0000100000411524 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libFWCoreFramework.so
#51 0x000010000031d1c8 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libFWCoreFramework.so
#52 0x000010000031d450 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libFWCoreFramework.so
#53 0x000010000031d8c8 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libFWCoreFramework.so
#54 0x0000100000321150 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libFWCoreFramework.so
#55 0x0000100000abab6c in tbb::detail::d1::function_task<edm::WaitingTaskWithArenaHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}::operator()() const::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libFWCoreConcurrency.so
#56 0x00001000024aa624 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (this=0x1000038cff00, t=0x10089f12f000, waiter=...) at /scratch/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_ppc64le_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/task_dispatcher.h:321
#57 0x00001000024a4138 in tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=..., t=0x0, this=<optimized out>) at /scratch/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_ppc64le_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/task_dispatcher.h:462
#58 tbb::detail::r1::arena::process (this=0x1000038cf780, tls=...) at /scratch/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_ppc64le_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/arena.cpp:138
#59 0x00001000024bdd74 in tbb::detail::r1::market::process (this=0x1000038cab00, j=...) at /scratch/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_ppc64le_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/market.cpp:596
#60 0x00001000024c3b6c in tbb::detail::r1::rml::private_worker::run (this=0x100008e50000) at /scratch/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_ppc64le_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/private_server.cpp:267
#61 0x00001000024c3f48 in tbb::detail::r1::rml::private_worker::thread_routine (arg=<optimized out>) at /scratch/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_ppc64le_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/private_server.cpp:221
#62 0x0000100002b18cd4 in start_thread () from /lib64/libpthread.so.0
#63 0x0000100002c77f14 in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x100886418390 (LWP 96566) "cmsRun"):
#0  0x0000100002c2eb88 in nanosleep () from /lib64/libc.so.6
#1  0x0000100002c2e8bc in sleep () from /lib64/libc.so.6
#2  0x0000100009e35c8c in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00001000025828d8 in adler32_z () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libz.so.1
#5  0x0000100002582e38 in adler32 () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libz.so.1
#6  0x000010000258e5a8 in inflate () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libz.so.1
#7  0x0000100001a4a630 in R__unzip () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libCore.so
#8  0x0000100000c6de08 in TBasket::ReadBasketBuffers(long long, int, TFile*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libTree.so
#9  0x0000100000c7da0c in TBranch::GetBasketImpl(int, TBuffer*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libTree.so
#10 0x0000100000c7e248 in TBranch::GetBasketAndFirst(TBasket*&, long long&, TBuffer*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libTree.so
#11 0x0000100000c7ebe8 in TBranch::GetEntry(long long, int) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libTree.so
#12 0x0000100000c97918 in TBranchElement::GetEntry(long long, int) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libTree.so
#13 0x0000100000c978b8 in TBranchElement::GetEntry(long long, int) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libTree.so
#14 0x0000100877446b24 in edm::RootTree::getEntry(TBranch*, long long) const () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/pluginIOPoolInput.so
#15 0x0000100877403648 in edm::RootDelayedReader::getProduct_(edm::BranchID const&, edm::EDProductGetter const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/pluginIOPoolInput.so
#16 0x00001000001f7688 in edm::DelayedReader::getProduct(edm::BranchID const&, edm::EDProductGetter const*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libFWCoreFramework.so
#17 0x000010000034e450 in edm::DelayedReaderInputProductResolver::resolveProduct_(edm::Principal const&, bool, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libFWCoreFramework.so
#18 0x00001000003499fc in edm::SingleChoiceNoProcessProductResolver::resolveProduct_(edm::Principal const&, bool, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libFWCoreFramework.so
#19 0x0000100000332904 in edm::Principal::findProductByLabel(edm::KindOfType, edm::TypeID const&, edm::InputTag const&, edm::EDConsumerBase const*, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libFWCoreFramework.so
#20 0x0000100000332b4c in edm::Principal::findProductByTag(edm::TypeID const&, edm::InputTag const&, edm::ModuleCallingContext const*) const () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libFWCoreFramework.so
#21 0x0000100877218b5c in edm::Adjuster<std::vector<PSimHit, std::allocator<PSimHit> > >::doOffset(int, int, edm::EventPrincipal const&, edm::ModuleCallingContext const*, unsigned int, int) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/pluginSimGeneralMixingModulePlugins.so
#22 0x000010087720da40 in edm::MixingModule::pileAllWorkers(edm::EventPrincipal const&, edm::ModuleCallingContext const*, int, int, int&, edm::EventSetup const&, edm::StreamID const&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/pluginSimGeneralMixingModulePlugins.so
#23 0x000010087721d39c in void edm::PileUp::readPileUp<std::_Bind<bool (edm::MixingModule::*(std::reference_wrapper<edm::MixingModule>, std::_Placeholder<1>, edm::ModuleCallingContext const*, int, std::_Placeholder<2>, int, std::reference_wrapper<edm::EventSetup const>, edm::StreamID))(edm::EventPrincipal const&, edm::ModuleCallingContext const*, int, int, int&, edm::EventSetup const&, edm::StreamID const&)> >(edm::EventID const&, std::vector<edm::SecondaryEventIDAndFileInfo, std::allocator<edm::SecondaryEventIDAndFileInfo> >&, std::_Bind<bool (edm::MixingModule::*(std::reference_wrapper<edm::MixingModule>, std::_Placeholder<1>, edm::ModuleCallingContext const*, int, std::_Placeholder<2>, int, std::reference_wrapper<edm::EventSetup const>, edm::StreamID))(edm::EventPrincipal const&, edm::ModuleCallingContext const*, int, int, int&, edm::EventSetup const&, edm::StreamID const&)>, int, edm::StreamID const&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/pluginSimGeneralMixingModulePlugins.so
#24 0x000010087720fc5c in edm::MixingModule::doPileUp(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/pluginSimGeneralMixingModulePlugins.so
#25 0x00001008772d7bbc in edm::BMixingModule::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libMixingBase.so
#26 0x0000100000442718 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libFWCoreFramework.so
#27 0x0000100000411524 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libFWCoreFramework.so
#28 0x000010000031d1c8 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libFWCoreFramework.so
#29 0x000010000031d450 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libFWCoreFramework.so
#30 0x000010000031d8c8 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libFWCoreFramework.so
#31 0x0000100000321150 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libFWCoreFramework.so
#32 0x0000100000abab6c in tbb::detail::d1::function_task<edm::WaitingTaskWithArenaHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}::operator()() const::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libFWCoreConcurrency.so
#33 0x00001000024aa624 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (this=0x1000038cfe00, t=0x1000037d6c00, waiter=...) at /scratch/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_ppc64le_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/task_dispatcher.h:321
#34 0x00001000024a4138 in tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=..., t=0x0, this=<optimized out>) at /scratch/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_ppc64le_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/task_dispatcher.h:462
#35 tbb::detail::r1::arena::process (this=0x1000038cf780, tls=...) at /scratch/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_ppc64le_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/arena.cpp:138
#36 0x00001000024bdd74 in tbb::detail::r1::market::process (this=0x1000038cab00, j=...) at /scratch/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_ppc64le_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/market.cpp:596
#37 0x00001000024c3b6c in tbb::detail::r1::rml::private_worker::run (this=0x100008e50100) at /scratch/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_ppc64le_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/private_server.cpp:267
#38 0x00001000024c3f48 in tbb::detail::r1::rml::private_worker::thread_routine (arg=<optimized out>) at /scratch/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_ppc64le_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/private_server.cpp:221
#39 0x0000100002b18cd4 in start_thread () from /lib64/libpthread.so.0
#40 0x0000100002c77f14 in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x100885a08390 (LWP 96565) "cmsRun"):
#0  0x0000100002c2eb88 in nanosleep () from /lib64/libc.so.6
#1  0x0000100002c2e8bc in sleep () from /lib64/libc.so.6
#2  0x0000100009e35c8c in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000100002c6fa14 in syscall () from /lib64/libc.so.6
#5  0x000010000284e264 in std::__atomic_futex_unsigned_base::_M_futex_wait_until (this=<optimized out>, __addr=<optimized out>, __val=<optimized out>, __has_timeout=<optimized out>, __s=..., __ns=...) at ../../../../../libstdc++-v3/src/c++11/futex.cc:55
#6  0x0000100008cdd9f8 in XrdFile::readv(IOPosBuffer*, unsigned long) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libUtilitiesXrdAdaptor.so
#7  0x000010000902bb80 in StorageAccountProxy::readv(IOPosBuffer*, unsigned long) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libUtilitiesStorageFactory.so
#8  0x00001000093c48c0 in TStorageFactoryFile::ReadBuffersSync(char*, long long*, int*, int) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libIOPoolTFileAdaptor.so
#9  0x00001000093c6d8c in TStorageFactoryFile::ReadBuffers(char*, long long*, int*, int) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libIOPoolTFileAdaptor.so
#10 0x0000100001368694 in TFileCacheRead::ReadBufferExtNormal(char*, long long, int, int&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libRIO.so
#11 0x0000100001367f84 in TFileCacheRead::ReadBuffer(char*, long long, int) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libRIO.so
#12 0x0000100000d08f64 in TTreeCache::ReadBufferNormal(char*, long long, int) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libTree.so
#13 0x0000100000d093a0 in TTreeCache::ReadBuffer(char*, long long, int) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libTree.so
#14 0x0000100000c6dadc in TBasket::ReadBasketBuffers(long long, int, TFile*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libTree.so
#15 0x0000100000c7da0c in TBranch::GetBasketImpl(int, TBuffer*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libTree.so
#16 0x0000100000c7e248 in TBranch::GetBasketAndFirst(TBasket*&, long long&, TBuffer*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libTree.so
#17 0x0000100000c7ebe8 in TBranch::GetEntry(long long, int) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libTree.so
#18 0x0000100000c97918 in TBranchElement::GetEntry(long long, int) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libTree.so
#19 0x0000100877446b24 in edm::RootTree::getEntry(TBranch*, long long) const () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/pluginIOPoolInput.so
#20 0x0000100877412e44 in edm::RootFile::fillThisEventAuxiliary() () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/pluginIOPoolInput.so
#21 0x00001008774249ec in edm::RootFile::readCurrentEvent(edm::EventPrincipal&, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/pluginIOPoolInput.so
#22 0x0000100877405298 in edm::RootEmbeddedFileSequence::readOneRandom(edm::EventPrincipal&, unsigned long&, CLHEP::HepRandomEngine*, edm::EventID const*, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/pluginIOPoolInput.so
#23 0x0000100877405b08 in edm::RootEmbeddedFileSequence::readOneEvent(edm::EventPrincipal&, unsigned long&, CLHEP::HepRandomEngine*, edm::EventID const*, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/pluginIOPoolInput.so
#24 0x00001008773d988c in edm::EmbeddedRootSource::readOneEvent(edm::EventPrincipal&, unsigned long&, CLHEP::HepRandomEngine*, edm::EventID const*, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/pluginIOPoolInput.so
#25 0x000010087721d33c in void edm::PileUp::readPileUp<std::_Bind<bool (edm::MixingModule::*(std::reference_wrapper<edm::MixingModule>, std::_Placeholder<1>, edm::ModuleCallingContext const*, int, std::_Placeholder<2>, int, std::reference_wrapper<edm::EventSetup const>, edm::StreamID))(edm::EventPrincipal const&, edm::ModuleCallingContext const*, int, int, int&, edm::EventSetup const&, edm::StreamID const&)> >(edm::EventID const&, std::vector<edm::SecondaryEventIDAndFileInfo, std::allocator<edm::SecondaryEventIDAndFileInfo> >&, std::_Bind<bool (edm::MixingModule::*(std::reference_wrapper<edm::MixingModule>, std::_Placeholder<1>, edm::ModuleCallingContext const*, int, std::_Placeholder<2>, int, std::reference_wrapper<edm::EventSetup const>, edm::StreamID))(edm::EventPrincipal const&, edm::ModuleCallingContext const*, int, int, int&, edm::EventSetup const&, edm::StreamID const&)>, int, edm::StreamID const&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/pluginSimGeneralMixingModulePlugins.so
#26 0x000010087720fc5c in edm::MixingModule::doPileUp(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/pluginSimGeneralMixingModulePlugins.so
#27 0x00001008772d7bbc in edm::BMixingModule::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libMixingBase.so
#28 0x0000100000442718 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libFWCoreFramework.so
#29 0x0000100000411524 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libFWCoreFramework.so
#30 0x000010000031d1c8 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libFWCoreFramework.so
#31 0x000010000031d450 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libFWCoreFramework.so
#32 0x000010000031d8c8 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libFWCoreFramework.so
#33 0x0000100000321150 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libFWCoreFramework.so
#34 0x0000100000abab6c in tbb::detail::d1::function_task<edm::WaitingTaskWithArenaHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}::operator()() const::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libFWCoreConcurrency.so
#35 0x00001000024aa624 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (this=0x1000038cfe80, t=0x1000037c4400, waiter=...) at /scratch/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_ppc64le_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/task_dispatcher.h:321
#36 0x00001000024a4138 in tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=..., t=0x0, this=<optimized out>) at /scratch/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_ppc64le_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/task_dispatcher.h:462
#37 tbb::detail::r1::arena::process (this=0x1000038cf780, tls=...) at /scratch/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_ppc64le_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/arena.cpp:138
#38 0x00001000024bdd74 in tbb::detail::r1::market::process (this=0x1000038cab00, j=...) at /scratch/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_ppc64le_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/market.cpp:596
#39 0x00001000024c3b6c in tbb::detail::r1::rml::private_worker::run (this=0x100008e50080) at /scratch/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_ppc64le_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/private_server.cpp:267
#40 0x00001000024c3f48 in tbb::detail::r1::rml::private_worker::thread_routine (arg=<optimized out>) at /scratch/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_ppc64le_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/private_server.cpp:221
#41 0x0000100002b18cd4 in start_thread () from /lib64/libpthread.so.0
#42 0x0000100002c77f14 in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x100878918390 (LWP 96530) "cmsRun"):
#0  0x0000100002b21a54 in do_futex_wait () from /lib64/libpthread.so.0
#1  0x0000100002b21b6c in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x0000100009a3114c in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libXrdCl.so.2
#3  0x0000100009a31268 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libXrdCl.so.2
#4  0x0000100002b18cd4 in start_thread () from /lib64/libpthread.so.0
#5  0x0000100002c77f14 in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x100878508390 (LWP 96529) "cmsRun"):
#0  0x0000100002b21a54 in do_futex_wait () from /lib64/libpthread.so.0
#1  0x0000100002b21b6c in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x0000100009a3114c in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libXrdCl.so.2
#3  0x0000100009a31268 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libXrdCl.so.2
#4  0x0000100002b18cd4 in start_thread () from /lib64/libpthread.so.0
#5  0x0000100002c77f14 in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x1008780f8390 (LWP 96528) "cmsRun"):
#0  0x0000100002b21a54 in do_futex_wait () from /lib64/libpthread.so.0
#1  0x0000100002b21b6c in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x0000100009a3114c in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libXrdCl.so.2
#3  0x0000100009a31268 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libXrdCl.so.2
#4  0x0000100002b18cd4 in start_thread () from /lib64/libpthread.so.0
#5  0x0000100002c77f14 in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x100877ce8390 (LWP 96527) "cmsRun"):
#0  0x0000100002b24288 in nanosleep () from /lib64/libpthread.so.0
#1  0x000010000923d9b8 in XrdSysTimer::Wait(int) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libXrdUtils.so.2
#2  0x00001000099bf33c in XrdCl::TaskManager::RunTasks() () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libXrdCl.so.2
#3  0x00001000099bf548 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libXrdCl.so.2
#4  0x0000100002b18cd4 in start_thread () from /lib64/libpthread.so.0
#5  0x0000100002c77f14 in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x1008778d8390 (LWP 96526) "cmsRun"):
#0  0x0000100002c78aa8 in epoll_wait () from /lib64/libc.so.6
#1  0x000010000923537c in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libXrdUtils.so.2
#2  0x00001000092303e8 in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libXrdUtils.so.2
#3  0x000010000923cb84 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libXrdUtils.so.2
#4  0x0000100002b18cd4 in start_thread () from /lib64/libpthread.so.0
#5  0x0000100002c77f14 in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x100821aa8390 (LWP 95901) "cmsRun"):
#0  0x0000100002b24c88 in waitpid () from /lib64/libpthread.so.0
#1  0x0000100009e35f68 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/pluginFWCoreServicesPlugins.so
#2  0x0000100009e37b1c in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/pluginFWCoreServicesPlugins.so
#3  0x0000100009e3cad4 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)()> > >::_M_run() () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/pluginFWCoreServicesPlugins.so
#4  0x0000100002851860 in std::execute_native_thread_routine (__p=<optimized out>) at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
#5  0x0000100002b18cd4 in start_thread () from /lib64/libpthread.so.0
#6  0x0000100002c77f14 in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x1000035d0000 (LWP 95826) "cmsRun"):
#0  0x0000100002c2eb88 in nanosleep () from /lib64/libc.so.6
#1  0x0000100002c2e8bc in sleep () from /lib64/libc.so.6
#2  0x0000100009e35c8c in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000100002c6fa14 in syscall () from /lib64/libc.so.6
#5  0x000010000284e264 in std::__atomic_futex_unsigned_base::_M_futex_wait_until (this=<optimized out>, __addr=<optimized out>, __val=<optimized out>, __has_timeout=<optimized out>, __s=..., __ns=...) at ../../../../../libstdc++-v3/src/c++11/futex.cc:55
#6  0x0000100008cea318 in std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<unsigned long>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<XrdAdaptor::RequestManager::handle(std::shared_ptr<std::vector<IOPosBuffer, std::allocator<IOPosBuffer> > >)::{lambda(std::future<unsigned long>, std::future<unsigned long>)#2}, std::future<unsigned long>, std::future<unsigned long> > >, unsigned long> >::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libUtilitiesXrdAdaptor.so
#7  0x0000100008ce63d4 in std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libUtilitiesXrdAdaptor.so
#8  0x0000100009a67e54 in std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::_FUN() () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libXrdCl.so.2
#9  0x000010000284ff04 in std::__once_proxy () at ../../../../../libstdc++-v3/src/c++11/mutex.cc:90
#10 0x0000100002b160cc in __pthread_once_slow () from /lib64/libpthread.so.0
#11 0x0000100008ce8824 in std::__future_base::_Deferred_state<std::thread::_Invoker<std::tuple<XrdAdaptor::RequestManager::handle(std::shared_ptr<std::vector<IOPosBuffer, std::allocator<IOPosBuffer> > >)::{lambda(std::future<unsigned long>, std::future<unsigned long>)#2}, std::future<unsigned long>, std::future<unsigned long> > >, unsigned long>::_M_complete_async() () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libUtilitiesXrdAdaptor.so
#12 0x0000100008cdd998 in XrdFile::readv(IOPosBuffer*, unsigned long) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libUtilitiesXrdAdaptor.so
#13 0x000010000902bb80 in StorageAccountProxy::readv(IOPosBuffer*, unsigned long) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libUtilitiesStorageFactory.so
#14 0x00001000093c48c0 in TStorageFactoryFile::ReadBuffersSync(char*, long long*, int*, int) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libIOPoolTFileAdaptor.so
#15 0x00001000093c6d8c in TStorageFactoryFile::ReadBuffers(char*, long long*, int*, int) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libIOPoolTFileAdaptor.so
#16 0x0000100001368694 in TFileCacheRead::ReadBufferExtNormal(char*, long long, int, int&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libRIO.so
#17 0x0000100001367f84 in TFileCacheRead::ReadBuffer(char*, long long, int) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libRIO.so
#18 0x0000100000d08f64 in TTreeCache::ReadBufferNormal(char*, long long, int) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libTree.so
#19 0x0000100000d093a0 in TTreeCache::ReadBuffer(char*, long long, int) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libTree.so
#20 0x0000100000c6dadc in TBasket::ReadBasketBuffers(long long, int, TFile*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libTree.so
#21 0x0000100000c7da0c in TBranch::GetBasketImpl(int, TBuffer*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libTree.so
#22 0x0000100000c7e248 in TBranch::GetBasketAndFirst(TBasket*&, long long&, TBuffer*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libTree.so
#23 0x0000100000c7ebe8 in TBranch::GetEntry(long long, int) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libTree.so
#24 0x0000100000c97918 in TBranchElement::GetEntry(long long, int) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/external/slc7_ppc64le_gcc9/lib/libTree.so
#25 0x0000100877446b24 in edm::RootTree::getEntry(TBranch*, long long) const () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/pluginIOPoolInput.so
#26 0x0000100877412e44 in edm::RootFile::fillThisEventAuxiliary() () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/pluginIOPoolInput.so
#27 0x00001008774249ec in edm::RootFile::readCurrentEvent(edm::EventPrincipal&, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/pluginIOPoolInput.so
#28 0x0000100877405298 in edm::RootEmbeddedFileSequence::readOneRandom(edm::EventPrincipal&, unsigned long&, CLHEP::HepRandomEngine*, edm::EventID const*, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/pluginIOPoolInput.so
#29 0x0000100877405b08 in edm::RootEmbeddedFileSequence::readOneEvent(edm::EventPrincipal&, unsigned long&, CLHEP::HepRandomEngine*, edm::EventID const*, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/pluginIOPoolInput.so
#30 0x00001008773d988c in edm::EmbeddedRootSource::readOneEvent(edm::EventPrincipal&, unsigned long&, CLHEP::HepRandomEngine*, edm::EventID const*, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/pluginIOPoolInput.so
#31 0x000010087721d33c in void edm::PileUp::readPileUp<std::_Bind<bool (edm::MixingModule::*(std::reference_wrapper<edm::MixingModule>, std::_Placeholder<1>, edm::ModuleCallingContext const*, int, std::_Placeholder<2>, int, std::reference_wrapper<edm::EventSetup const>, edm::StreamID))(edm::EventPrincipal const&, edm::ModuleCallingContext const*, int, int, int&, edm::EventSetup const&, edm::StreamID const&)> >(edm::EventID const&, std::vector<edm::SecondaryEventIDAndFileInfo, std::allocator<edm::SecondaryEventIDAndFileInfo> >&, std::_Bind<bool (edm::MixingModule::*(std::reference_wrapper<edm::MixingModule>, std::_Placeholder<1>, edm::ModuleCallingContext const*, int, std::_Placeholder<2>, int, std::reference_wrapper<edm::EventSetup const>, edm::StreamID))(edm::EventPrincipal const&, edm::ModuleCallingContext const*, int, int, int&, edm::EventSetup const&, edm::StreamID const&)>, int, edm::StreamID const&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/pluginSimGeneralMixingModulePlugins.so
#32 0x000010087720fc5c in edm::MixingModule::doPileUp(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/pluginSimGeneralMixingModulePlugins.so
#33 0x00001008772d7bbc in edm::BMixingModule::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libMixingBase.so
#34 0x0000100000442718 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libFWCoreFramework.so
#35 0x0000100000411524 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libFWCoreFramework.so
#36 0x000010000031d1c8 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libFWCoreFramework.so
#37 0x000010000031d450 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libFWCoreFramework.so
#38 0x000010000031d8c8 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libFWCoreFramework.so
#39 0x0000100000321150 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libFWCoreFramework.so
#40 0x0000100000abab6c in tbb::detail::d1::function_task<edm::WaitingTaskWithArenaHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}::operator()() const::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libFWCoreConcurrency.so
#41 0x00001000024ce564 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (this=0x1000038cfd80, t=0x10000372eb00, waiter=...) at /scratch/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_ppc64le_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/task_dispatcher.h:321
#42 0x00001000024c98e4 in tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=<optimized out>) at /scratch/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_ppc64le_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/task_dispatcher.h:462
#43 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /scratch/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_ppc64le_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/task_dispatcher.cpp:168
#44 0x00001000024c99a4 in tbb::detail::r1::wait (wait_ctx=..., w_ctx=...) at /scratch/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_ppc64le_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/task_dispatcher.cpp:126
#45 0x000010000023375c in edm::EventProcessor::processLumis(std::shared_ptr<void> const&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libFWCoreFramework.so
#46 0x0000100000243588 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_ppc64le_gcc9/cms/cmssw/CMSSW_12_1_X_2021-09-03-2300/lib/slc7_ppc64le_gcc9/libFWCoreFramework.so
#47 0x000000001000b908 in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#48 0x00001000024a2d80 in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /scratch/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_ppc64le_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/arena.cpp:696
#49 0x00001000024a3f88 in tbb::detail::r1::execute (ta=..., d=...) at /scratch/cmsbuild/jenkins_a/workspace/build-any-ib/w/BUILD/slc7_ppc64le_gcc9/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/arena.cpp:412
#50 0x000000001000cb5c in main::{lambda()#1}::operator()() const ()
#51 0x000000001000adc0 in main ()

Current Modules:

Module: MixingModule:mix (crashed)
Module: MixingModule:mix
Module: MixingModule:mix
Module: MixingModule:mix

A fatal system signal has occurred: segmentation violation

[cmsbuild]

A new Issue was created by @dan131riley Dan Riley.

@Dr15Jones, @perrotta, @dpiparo, @makortel, @smuzaffar, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[makortel]

assign core

[cmsbuild]

New categories assigned: core

@Dr15Jones,@smuzaffar,@makortel you have been requested to review this Pull request/Issue and eventually sign? Thanks

[dan131riley]

I'm going to stick this here as possibly related, a weird crash in edm::ELextendedID::operator<():

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/raw/slc7_amd64_gcc900/CMSSW_12_1_ROOT624_X_2021-09-03-2300/pyRelValMatrixLogs/run/23434.99_TTbar_14TeV+2026D49PU_PMXS1S2+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14INPUT+PREMIX_PremixHLBeamSpot14PU+DigiTriggerPU+RecoGlobalPU+HARVESTGlobalPU/step2_TTbar_14TeV+2026D49PU_PMXS1S2+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14INPUT+PREMIX_PremixHLBeamSpot14PU+DigiTriggerPU+RecoGlobalPU+HARVESTGlobalPU.log

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Sat Sep  4 03:51:05 CEST 2021
Thread 15 (Thread 0x2b125ec00700 (LWP 23819) "cmsRun"):
#0  0x00002b120ceca8ed in nanosleep () from /lib64/libc.so.6
#1  0x00002b120ceca784 in sleep () from /lib64/libc.so.6
#2  0x00002b12130723f0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  malloc (size=size@entry=128) at include/jemalloc/internal/tsd.h:268
#5  0x00002b120b9ece49 in newImpl<false> (size=128) at src/jemalloc_cpp.cpp:77
#6  operator new (size=128) at src/jemalloc_cpp.cpp:87
#7  0x0000000000414370 in void std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_M_realloc_insert<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>(__gnu_cxx::__normal_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
#8  0x00002b120b705160 in TClassEdit::GetSplit(char const*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, int&, TClassEdit::EModType) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/external/slc7_amd64_gcc900/lib/libCore.so
#9  0x00002b120b705e71 in TClassEdit::TSplitType::TSplitType(char const*, TClassEdit::EModType) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/external/slc7_amd64_gcc900/lib/libCore.so
#10 0x00002b120b7060e1 in TClassEdit::GetNormalizedName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::basic_string_view<char, std::char_traits<char> >) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/external/slc7_amd64_gcc900/lib/libCore.so
#11 0x00002b120b72119d in TClass::GetClass(char const*, bool, bool, unsigned long, unsigned long) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/external/slc7_amd64_gcc900/lib/libCore.so
#12 0x00002b1256f769e3 in edm::RootDelayedReader::getProduct_(edm::BranchID const&, edm::EDProductGetter const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/pluginIOPoolInput.so
#13 0x00002b120a6e77f4 in edm::DelayedReader::getProduct(edm::BranchID const&, edm::EDProductGetter const*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#14 0x00002b120a7c11b4 in edm::DelayedReaderInputProductResolver::resolveProduct_(edm::Principal const&, bool, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#15 0x00002b120a7b06d9 in edm::Principal::findProductByLabel(edm::KindOfType, edm::TypeID const&, edm::InputTag const&, edm::EDConsumerBase const*, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#16 0x00002b120a7b08a5 in edm::Principal::findProductByTag(edm::TypeID const&, edm::InputTag const&, edm::ModuleCallingContext const*) const () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#17 0x00002b1256e40baa in edm::Adjuster<std::vector<PSimHit, std::allocator<PSimHit> > >::doOffset(int, int, edm::EventPrincipal const&, edm::ModuleCallingContext const*, unsigned int, int) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/pluginSimGeneralMixingModulePlugins.so
#18 0x00002b1256e3837b in edm::MixingModule::pileAllWorkers(edm::EventPrincipal const&, edm::ModuleCallingContext const*, int, int, int&, edm::EventSetup const&, edm::StreamID const&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/pluginSimGeneralMixingModulePlugins.so
#19 0x00002b1256e436cf in void edm::PileUp::readPileUp<std::_Bind<bool (edm::MixingModule::*(std::reference_wrapper<edm::MixingModule>, std::_Placeholder<1>, edm::ModuleCallingContext const*, int, std::_Placeholder<2>, int, std::reference_wrapper<edm::EventSetup const>, edm::StreamID))(edm::EventPrincipal const&, edm::ModuleCallingContext const*, int, int, int&, edm::EventSetup const&, edm::StreamID const&)> >(edm::EventID const&, std::vector<edm::SecondaryEventIDAndFileInfo, std::allocator<edm::SecondaryEventIDAndFileInfo> >&, std::_Bind<bool (edm::MixingModule::*(std::reference_wrapper<edm::MixingModule>, std::_Placeholder<1>, edm::ModuleCallingContext const*, int, std::_Placeholder<2>, int, std::reference_wrapper<edm::EventSetup const>, edm::StreamID))(edm::EventPrincipal const&, edm::ModuleCallingContext const*, int, int, int&, edm::EventSetup const&, edm::StreamID const&)>, int, edm::StreamID const&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/pluginSimGeneralMixingModulePlugins.so
#20 0x00002b1256e3a02a in edm::MixingModule::doPileUp(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/pluginSimGeneralMixingModulePlugins.so
#21 0x00002b1256ed21fa in edm::BMixingModule::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libMixingBase.so
#22 0x00002b120a8668bc in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#23 0x00002b120a847add in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#24 0x00002b120a7a25c5 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#25 0x00002b120a7a277d in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#26 0x00002b120a7a2a86 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#27 0x00002b120a7a4e16 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#28 0x00002b120a515e05 in tbb::detail::d1::function_task<edm::WaitingTaskWithArenaHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}::operator()() const::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreConcurrency.so
#29 0x00002b120bfdc3db in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (this=<optimized out>, t=0x2b120fb63c00, t@entry=0x0, waiter=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/task_dispatcher.h:321
#30 0x00002b120bfd7637 in tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=..., t=0x0, this=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/task_dispatcher.h:462
#31 tbb::detail::r1::arena::process (this=this@entry=0x2b120fc3ed00, tls=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/arena.cpp:138
#32 0x00002b120bfe914b in tbb::detail::r1::market::process (this=0x2b120fc3f580, j=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/market.cpp:596
#33 0x00002b120bfecc20 in tbb::detail::r1::rml::private_worker::run (this=0x2b1211acf100) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/private_server.cpp:267
#34 0x00002b120bfece09 in tbb::detail::r1::rml::private_worker::thread_routine (arg=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/private_server.cpp:221
#35 0x00002b120cbefea5 in start_thread () from /lib64/libpthread.so.0
#36 0x00002b120cf039fd in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x2b125dc00700 (LWP 23817) "cmsRun"):
#0  0x00002b120ceca8ed in nanosleep () from /lib64/libc.so.6
#1  0x00002b120ceca784 in sleep () from /lib64/libc.so.6
#2  0x00002b12130723f0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00002b120c05864e in ?? () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/external/slc7_amd64_gcc900/lib/libz.so.1
#5  0x00002b120c059097 in inflate () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/external/slc7_amd64_gcc900/lib/libz.so.1
#6  0x00002b120b715149 in R__unzip () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/external/slc7_amd64_gcc900/lib/libCore.so
#7  0x00002b120abfaef4 in TBasket::ReadBasketBuffers(long long, int, TFile*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/external/slc7_amd64_gcc900/lib/libTree.so
#8  0x00002b120ac04ce9 in TBranch::GetBasketImpl(int, TBuffer*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/external/slc7_amd64_gcc900/lib/libTree.so
#9  0x00002b120ac052dd in TBranch::GetBasketAndFirst(TBasket*&, long long&, TBuffer*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/external/slc7_amd64_gcc900/lib/libTree.so
#10 0x00002b120ac05bc7 in TBranch::GetEntry(long long, int) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/external/slc7_amd64_gcc900/lib/libTree.so
#11 0x00002b120ac176e8 in TBranchElement::GetEntry(long long, int) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/external/slc7_amd64_gcc900/lib/libTree.so
#12 0x00002b120ac1768d in TBranchElement::GetEntry(long long, int) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/external/slc7_amd64_gcc900/lib/libTree.so
#13 0x00002b1256fa4c4b in edm::RootTree::getEntry(TBranch*, long long) const () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/pluginIOPoolInput.so
#14 0x00002b1256f7675a in edm::RootDelayedReader::getProduct_(edm::BranchID const&, edm::EDProductGetter const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/pluginIOPoolInput.so
#15 0x00002b120a6e77f4 in edm::DelayedReader::getProduct(edm::BranchID const&, edm::EDProductGetter const*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#16 0x00002b120a7c11b4 in edm::DelayedReaderInputProductResolver::resolveProduct_(edm::Principal const&, bool, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#17 0x00002b120a7b06d9 in edm::Principal::findProductByLabel(edm::KindOfType, edm::TypeID const&, edm::InputTag const&, edm::EDConsumerBase const*, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#18 0x00002b120a7b08a5 in edm::Principal::findProductByTag(edm::TypeID const&, edm::InputTag const&, edm::ModuleCallingContext const*) const () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#19 0x00002b1256e40baa in edm::Adjuster<std::vector<PSimHit, std::allocator<PSimHit> > >::doOffset(int, int, edm::EventPrincipal const&, edm::ModuleCallingContext const*, unsigned int, int) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/pluginSimGeneralMixingModulePlugins.so
#20 0x00002b1256e3837b in edm::MixingModule::pileAllWorkers(edm::EventPrincipal const&, edm::ModuleCallingContext const*, int, int, int&, edm::EventSetup const&, edm::StreamID const&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/pluginSimGeneralMixingModulePlugins.so
#21 0x00002b1256e436cf in void edm::PileUp::readPileUp<std::_Bind<bool (edm::MixingModule::*(std::reference_wrapper<edm::MixingModule>, std::_Placeholder<1>, edm::ModuleCallingContext const*, int, std::_Placeholder<2>, int, std::reference_wrapper<edm::EventSetup const>, edm::StreamID))(edm::EventPrincipal const&, edm::ModuleCallingContext const*, int, int, int&, edm::EventSetup const&, edm::StreamID const&)> >(edm::EventID const&, std::vector<edm::SecondaryEventIDAndFileInfo, std::allocator<edm::SecondaryEventIDAndFileInfo> >&, std::_Bind<bool (edm::MixingModule::*(std::reference_wrapper<edm::MixingModule>, std::_Placeholder<1>, edm::ModuleCallingContext const*, int, std::_Placeholder<2>, int, std::reference_wrapper<edm::EventSetup const>, edm::StreamID))(edm::EventPrincipal const&, edm::ModuleCallingContext const*, int, int, int&, edm::EventSetup const&, edm::StreamID const&)>, int, edm::StreamID const&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/pluginSimGeneralMixingModulePlugins.so
#22 0x00002b1256e3a02a in edm::MixingModule::doPileUp(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/pluginSimGeneralMixingModulePlugins.so
#23 0x00002b1256ed21fa in edm::BMixingModule::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libMixingBase.so
#24 0x00002b120a8668bc in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#25 0x00002b120a847add in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#26 0x00002b120a7a25c5 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#27 0x00002b120a7a277d in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#28 0x00002b120a7a2a86 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#29 0x00002b120a7a4e16 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#30 0x00002b120a515e05 in tbb::detail::d1::function_task<edm::WaitingTaskWithArenaHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}::operator()() const::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreConcurrency.so
#31 0x00002b120bfdc3db in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (this=<optimized out>, t=0x2b120fb73e00, t@entry=0x0, waiter=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/task_dispatcher.h:321
#32 0x00002b120bfd7637 in tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=..., t=0x0, this=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/task_dispatcher.h:462
#33 tbb::detail::r1::arena::process (this=this@entry=0x2b120fc3ed00, tls=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/arena.cpp:138
#34 0x00002b120bfe914b in tbb::detail::r1::market::process (this=0x2b120fc3f580, j=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/market.cpp:596
#35 0x00002b120bfecc20 in tbb::detail::r1::rml::private_worker::run (this=0x2b1211acf000) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/private_server.cpp:267
#36 0x00002b120bfece09 in tbb::detail::r1::rml::private_worker::thread_routine (arg=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/private_server.cpp:221
#37 0x00002b120cbefea5 in start_thread () from /lib64/libpthread.so.0
#38 0x00002b120cf039fd in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x2b125cc00700 (LWP 23815) "cmsRun"):
#0  0x00002b120cef8ccd in poll () from /lib64/libc.so.6
#1  0x00002b12130729d7 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#2  0x00002b121307330c in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#3  0x00002b12130767bb in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00002b120a49b078 in edm::ELextendedID::operator<(edm::ELextendedID const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreMessageLogger.so
#6  0x00002b1210ec4f88 in std::_Rb_tree<edm::ELextendedID, std::pair<edm::ELextendedID const, edm::CountAndLimit>, std::_Select1st<std::pair<edm::ELextendedID const, edm::CountAndLimit> >, std::less<edm::ELextendedID>, std::allocator<std::pair<edm::ELextendedID const, edm::CountAndLimit> > >::find(edm::ELextendedID const&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreMessageService.so
#7  0x00002b1210ec497a in edm::service::ELlimitsTable::add(edm::ELextendedID const&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreMessageService.so
#8  0x00002b1210ec723c in edm::service::ELoutput::log(edm::ErrorObj const&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreMessageService.so
#9  0x00002b1210ec271d in edm::service::ELadministrator::log(edm::ErrorObj&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreMessageService.so
#10 0x00002b1210ed5349 in edm::service::ThreadSafeLogMessageLoggerScribe::log(edm::ErrorObj*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreMessageService.so
#11 0x00002b1210edd71c in edm::service::ThreadSafeLogMessageLoggerScribe::runCommand(edm::MessageLoggerQ::OpCode, void*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreMessageService.so
#12 0x00002b120a4af60a in edm::MessageSender::ErrorObjDeleter::operator()(edm::ErrorObj*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreMessageLogger.so
#13 0x00002b120a4b3411 in std::_Sp_counted_deleter<edm::ErrorObj*, edm::MessageSender::ErrorObjDeleter, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreMessageLogger.so
#14 0x00002b120a4ae170 in edm::MessageSender::~MessageSender() () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreMessageLogger.so
#15 0x00002b125cdb6604 in HcalDigitizer::accumulateCaloHits(edm::Handle<std::vector<PCaloHit, std::allocator<PCaloHit> > > const&, edm::Handle<std::vector<PCaloHit, std::allocator<PCaloHit> > > const&, int, CLHEP::HepRandomEngine*, HcalTopology const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libSimCalorimetryHcalSimProducers.so
#16 0x00002b125cdb7b53 in HcalDigitizer::accumulate(PileUpEventPrincipal const&, edm::EventSetup const&, CLHEP::HepRandomEngine*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libSimCalorimetryHcalSimProducers.so
#17 0x00002b1256e3824e in edm::MixingModule::accumulateEvent(PileUpEventPrincipal const&, edm::EventSetup const&, edm::StreamID const&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/pluginSimGeneralMixingModulePlugins.so
#18 0x00002b1256e383b6 in edm::MixingModule::pileAllWorkers(edm::EventPrincipal const&, edm::ModuleCallingContext const*, int, int, int&, edm::EventSetup const&, edm::StreamID const&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/pluginSimGeneralMixingModulePlugins.so
#19 0x00002b1256e436cf in void edm::PileUp::readPileUp<std::_Bind<bool (edm::MixingModule::*(std::reference_wrapper<edm::MixingModule>, std::_Placeholder<1>, edm::ModuleCallingContext const*, int, std::_Placeholder<2>, int, std::reference_wrapper<edm::EventSetup const>, edm::StreamID))(edm::EventPrincipal const&, edm::ModuleCallingContext const*, int, int, int&, edm::EventSetup const&, edm::StreamID const&)> >(edm::EventID const&, std::vector<edm::SecondaryEventIDAndFileInfo, std::allocator<edm::SecondaryEventIDAndFileInfo> >&, std::_Bind<bool (edm::MixingModule::*(std::reference_wrapper<edm::MixingModule>, std::_Placeholder<1>, edm::ModuleCallingContext const*, int, std::_Placeholder<2>, int, std::reference_wrapper<edm::EventSetup const>, edm::StreamID))(edm::EventPrincipal const&, edm::ModuleCallingContext const*, int, int, int&, edm::EventSetup const&, edm::StreamID const&)>, int, edm::StreamID const&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/pluginSimGeneralMixingModulePlugins.so
#20 0x00002b1256e3a02a in edm::MixingModule::doPileUp(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/pluginSimGeneralMixingModulePlugins.so
#21 0x00002b1256ed21fa in edm::BMixingModule::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libMixingBase.so
#22 0x00002b120a8668bc in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#23 0x00002b120a847add in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#24 0x00002b120a7a25c5 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#25 0x00002b120a7a277d in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#26 0x00002b120a7a2a86 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#27 0x00002b120a7a4e16 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#28 0x00002b120a515e05 in tbb::detail::d1::function_task<edm::WaitingTaskWithArenaHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}::operator()() const::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreConcurrency.so
#29 0x00002b120bfdc3db in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (this=<optimized out>, t=0x2b120fb82700, t@entry=0x0, waiter=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/task_dispatcher.h:321
#30 0x00002b120bfd7637 in tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=..., t=0x0, this=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/task_dispatcher.h:462
#31 tbb::detail::r1::arena::process (this=this@entry=0x2b120fc3ed00, tls=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/arena.cpp:138
#32 0x00002b120bfe914b in tbb::detail::r1::market::process (this=0x2b120fc3f580, j=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/market.cpp:596
#33 0x00002b120bfecc20 in tbb::detail::r1::rml::private_worker::run (this=0x2b1211acf080) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/private_server.cpp:267
#34 0x00002b120bfece09 in tbb::detail::r1::rml::private_worker::thread_routine (arg=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/private_server.cpp:221
#35 0x00002b120cbefea5 in start_thread () from /lib64/libpthread.so.0
#36 0x00002b120cf039fd in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x2b12595ce700 (LWP 18591) "cmsRun"):
#0  0x00002b120cbf5b3b in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x00002b120cbf5bcf in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x00002b120cbf5c6b in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3  0x00002b1212c05816 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/external/slc7_amd64_gcc900/lib/libXrdCl.so.2
#4  0x00002b1212c058c9 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/external/slc7_amd64_gcc900/lib/libXrdCl.so.2
#5  0x00002b120cbefea5 in start_thread () from /lib64/libpthread.so.0
#6  0x00002b120cf039fd in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x2b12593cd700 (LWP 18590) "cmsRun"):
#0  0x00002b120cbf5b3b in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x00002b120cbf5bcf in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x00002b120cbf5c6b in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3  0x00002b1212c05816 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/external/slc7_amd64_gcc900/lib/libXrdCl.so.2
#4  0x00002b1212c058c9 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/external/slc7_amd64_gcc900/lib/libXrdCl.so.2
#5  0x00002b120cbefea5 in start_thread () from /lib64/libpthread.so.0
#6  0x00002b120cf039fd in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x2b12591cc700 (LWP 18589) "cmsRun"):
#0  0x00002b120cbf5b3b in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x00002b120cbf5bcf in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x00002b120cbf5c6b in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3  0x00002b1212c05816 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/external/slc7_amd64_gcc900/lib/libXrdCl.so.2
#4  0x00002b1212c058c9 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/external/slc7_amd64_gcc900/lib/libXrdCl.so.2
#5  0x00002b120cbefea5 in start_thread () from /lib64/libpthread.so.0
#6  0x00002b120cf039fd in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x2b1258fcb700 (LWP 18588) "cmsRun"):
#0  0x00002b120cbf6e9d in nanosleep () from /lib64/libpthread.so.0
#1  0x00002b1211efc6ff in XrdSysTimer::Wait(int) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/external/slc7_amd64_gcc900/lib/libXrdUtils.so.2
#2  0x00002b1212ba7b0a in XrdCl::TaskManager::RunTasks() () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/external/slc7_amd64_gcc900/lib/libXrdCl.so.2
#3  0x00002b1212ba7c69 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/external/slc7_amd64_gcc900/lib/libXrdCl.so.2
#4  0x00002b120cbefea5 in start_thread () from /lib64/libpthread.so.0
#5  0x00002b120cf039fd in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x2b1258dca700 (LWP 18587) "cmsRun"):
#0  0x00002b120cf03fd3 in epoll_wait () from /lib64/libc.so.6
#1  0x00002b1211ef6ed2 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/external/slc7_amd64_gcc900/lib/libXrdUtils.so.2
#2  0x00002b1211ef378d in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/external/slc7_amd64_gcc900/lib/libXrdUtils.so.2
#3  0x00002b1211efbf68 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/external/slc7_amd64_gcc900/lib/libXrdUtils.so.2
#4  0x00002b120cbefea5 in start_thread () from /lib64/libpthread.so.0
#5  0x00002b120cf039fd in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x2b1256600700 (LWP 18056) "cmsRun"):
#0  0x00002b120cbf3a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00002b120c7bb85c in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_11_1_0_pre6-slc7_amd64_gcc900/build/CMSSW_11_1_0_pre6-build/BUILD/slc7_amd64_gcc900/external/gcc/9.3.0/gcc-9.3.0/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/x86_64-unknown-linux-gnu/bits/gthr-default.h:865
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00002b123fe60f0b in omt::ThreadHandoff::threadLoop(void*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/biglib/slc7_amd64_gcc900/pluginSimulation.so
#4  0x00002b120cbefea5 in start_thread () from /lib64/libpthread.so.0
#5  0x00002b120cf039fd in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x2b1255600700 (LWP 18055) "cmsRun"):
#0  0x00002b120cbf3a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00002b120c7bb85c in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_11_1_0_pre6-slc7_amd64_gcc900/build/CMSSW_11_1_0_pre6-build/BUILD/slc7_amd64_gcc900/external/gcc/9.3.0/gcc-9.3.0/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/x86_64-unknown-linux-gnu/bits/gthr-default.h:865
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00002b123fe60f0b in omt::ThreadHandoff::threadLoop(void*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/biglib/slc7_amd64_gcc900/pluginSimulation.so
#4  0x00002b120cbefea5 in start_thread () from /lib64/libpthread.so.0
#5  0x00002b120cf039fd in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x2b1254600700 (LWP 18054) "cmsRun"):
#0  0x00002b120cbf3a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00002b120c7bb85c in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_11_1_0_pre6-slc7_amd64_gcc900/build/CMSSW_11_1_0_pre6-build/BUILD/slc7_amd64_gcc900/external/gcc/9.3.0/gcc-9.3.0/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/x86_64-unknown-linux-gnu/bits/gthr-default.h:865
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00002b123fe60f0b in omt::ThreadHandoff::threadLoop(void*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/biglib/slc7_amd64_gcc900/pluginSimulation.so
#4  0x00002b120cbefea5 in start_thread () from /lib64/libpthread.so.0
#5  0x00002b120cf039fd in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x2b1253680700 (LWP 18053) "cmsRun"):
#0  0x00002b120cbf3a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00002b120c7bb85c in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_11_1_0_pre6-slc7_amd64_gcc900/build/CMSSW_11_1_0_pre6-build/BUILD/slc7_amd64_gcc900/external/gcc/9.3.0/gcc-9.3.0/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/x86_64-unknown-linux-gnu/bits/gthr-default.h:865
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00002b123fe60f0b in omt::ThreadHandoff::threadLoop(void*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/biglib/slc7_amd64_gcc900/pluginSimulation.so
#4  0x00002b120cbefea5 in start_thread () from /lib64/libpthread.so.0
#5  0x00002b120cf039fd in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x2b1251ff4700 (LWP 18017) "cmsRun"):
#0  0x00002b120cbf3a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00002b120c7bb85c in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_11_1_0_pre6-slc7_amd64_gcc900/build/CMSSW_11_1_0_pre6-build/BUILD/slc7_amd64_gcc900/external/gcc/9.3.0/gcc-9.3.0/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/x86_64-unknown-linux-gnu/bits/gthr-default.h:865
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00002b123fe441ab in OscarMTMasterThread::OscarMTMasterThread(edm::ParameterSet const&)::{lambda()#1}::operator()() const [clone .isra.0] () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/biglib/slc7_amd64_gcc900/pluginSimulation.so
#4  0x00002b120c7c0af0 in std::execute_native_thread_routine (__p=0x2b123ed55060) at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
#5  0x00002b120cbefea5 in start_thread () from /lib64/libpthread.so.0
#6  0x00002b120cf039fd in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x2b122f625700 (LWP 16724) "cmsRun"):
#0  0x00002b120cbf71d9 in waitpid () from /lib64/libpthread.so.0
#1  0x00002b12130725a7 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#2  0x00002b121307323a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#3  0x00002b120c7c0af0 in std::execute_native_thread_routine (__p=0x2b121eda7580) at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
#4  0x00002b120cbefea5 in start_thread () from /lib64/libpthread.so.0
#5  0x00002b120cf039fd in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x2b120ef8e200 (LWP 14403) "cmsRun"):
#0  0x00002b120ceca8ed in nanosleep () from /lib64/libc.so.6
#1  0x00002b120ceca784 in sleep () from /lib64/libc.so.6
#2  0x00002b12130723f0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00002b120cefdd19 in syscall () from /lib64/libc.so.6
#5  0x00002b120c7beb69 in std::__atomic_futex_unsigned_base::_M_futex_wait_until (this=<optimized out>, __addr=0x2b1258048570, __val=2147483648, __has_timeout=<optimized out>, __s=..., __ns=...) at ../../../../../libstdc++-v3/src/c++11/futex.cc:55
#6  0x00002b1211be6610 in XrdFile::readv(IOPosBuffer*, unsigned long) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libUtilitiesXrdAdaptor.so
#7  0x00002b1211dacbc4 in StorageAccountProxy::readv(IOPosBuffer*, unsigned long) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libUtilitiesStorageFactory.so
#8  0x00002b1213862408 in TStorageFactoryFile::ReadBuffersSync(char*, long long*, int*, int) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libIOPoolTFileAdaptor.so
#9  0x00002b1213863cc9 in TStorageFactoryFile::ReadBuffers(char*, long long*, int*, int) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libIOPoolTFileAdaptor.so
#10 0x00002b120b18a231 in TFileCacheRead::ReadBufferExtNormal(char*, long long, int, int&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/external/slc7_amd64_gcc900/lib/libRIO.so
#11 0x00002b120b189cff in TFileCacheRead::ReadBuffer(char*, long long, int) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/external/slc7_amd64_gcc900/lib/libRIO.so
#12 0x00002b120ac64e25 in TTreeCache::ReadBufferNormal(char*, long long, int) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/external/slc7_amd64_gcc900/lib/libTree.so
#13 0x00002b120abfac42 in TBasket::ReadBasketBuffers(long long, int, TFile*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/external/slc7_amd64_gcc900/lib/libTree.so
#14 0x00002b120ac04ce9 in TBranch::GetBasketImpl(int, TBuffer*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/external/slc7_amd64_gcc900/lib/libTree.so
#15 0x00002b120ac052dd in TBranch::GetBasketAndFirst(TBasket*&, long long&, TBuffer*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/external/slc7_amd64_gcc900/lib/libTree.so
#16 0x00002b120ac05bc7 in TBranch::GetEntry(long long, int) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/external/slc7_amd64_gcc900/lib/libTree.so
#17 0x00002b120ac176e8 in TBranchElement::GetEntry(long long, int) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/external/slc7_amd64_gcc900/lib/libTree.so
#18 0x00002b1256fa4c4b in edm::RootTree::getEntry(TBranch*, long long) const () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/pluginIOPoolInput.so
#19 0x00002b1256f8138d in edm::RootFile::fillThisEventAuxiliary() () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/pluginIOPoolInput.so
#20 0x00002b1256f8ea15 in edm::RootFile::readCurrentEvent(edm::EventPrincipal&, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/pluginIOPoolInput.so
#21 0x00002b1256f77792 in edm::RootEmbeddedFileSequence::readOneRandom(edm::EventPrincipal&, unsigned long&, CLHEP::HepRandomEngine*, edm::EventID const*, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/pluginIOPoolInput.so
#22 0x00002b1256e4368a in void edm::PileUp::readPileUp<std::_Bind<bool (edm::MixingModule::*(std::reference_wrapper<edm::MixingModule>, std::_Placeholder<1>, edm::ModuleCallingContext const*, int, std::_Placeholder<2>, int, std::reference_wrapper<edm::EventSetup const>, edm::StreamID))(edm::EventPrincipal const&, edm::ModuleCallingContext const*, int, int, int&, edm::EventSetup const&, edm::StreamID const&)> >(edm::EventID const&, std::vector<edm::SecondaryEventIDAndFileInfo, std::allocator<edm::SecondaryEventIDAndFileInfo> >&, std::_Bind<bool (edm::MixingModule::*(std::reference_wrapper<edm::MixingModule>, std::_Placeholder<1>, edm::ModuleCallingContext const*, int, std::_Placeholder<2>, int, std::reference_wrapper<edm::EventSetup const>, edm::StreamID))(edm::EventPrincipal const&, edm::ModuleCallingContext const*, int, int, int&, edm::EventSetup const&, edm::StreamID const&)>, int, edm::StreamID const&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/pluginSimGeneralMixingModulePlugins.so
#23 0x00002b1256e3a02a in edm::MixingModule::doPileUp(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/pluginSimGeneralMixingModulePlugins.so
#24 0x00002b1256ed21fa in edm::BMixingModule::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libMixingBase.so
#25 0x00002b120a8668bc in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#26 0x00002b120a847add in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#27 0x00002b120a7a25c5 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#28 0x00002b120a7a277d in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#29 0x00002b120a7a2a86 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#30 0x00002b120a7a4e16 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#31 0x00002b120a515e05 in tbb::detail::d1::function_task<edm::WaitingTaskWithArenaHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}::operator()() const::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreConcurrency.so
#32 0x00002b120bff4723 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (this=0x2b120fc3f300, t=0x2b120faf8a00, waiter=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/task_dispatcher.h:321
#33 0x00002b120bff0a45 in tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=0x0, this=0x2b120fc3f300) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/task_dispatcher.h:462
#34 tbb::detail::r1::task_dispatcher::execute_and_wait (t=0x0, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/task_dispatcher.cpp:168
#35 0x00002b120a70c4cf in edm::EventProcessor::processLumis(std::shared_ptr<void> const&) () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#36 0x00002b120a718465 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-03-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#37 0x000000000040bae6 in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#38 0x00002b120bfd6302 in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.3.0-13eaf94bcafc2deaec6244d3257cd1bc/tbb-v2021.3.0/src/tbb/arena.cpp:696
#39 0x000000000040ca31 in main::{lambda()#1}::operator()() const ()
#40 0x000000000040b62c in main ()

Current Modules:

Module: MixingModule:mix (crashed)
Module: MixingModule:mix
Module: MixingModule:mix
Module: MixingModule:mix

A fatal system signal has occurred: segmentation violation