search

cms-sw / cmssw

CMS Offline Software
http://cms-sw.github.io/
Apache License 2.0
1.09k stars 4.32k forks source link

Exception during begin Luminosity block not being properly handled #33567

Open Dr15Jones opened 3 years ago

Dr15Jones commented 3 years ago

An exception which went off during delayed read from the source during the begin LuminosityBlock caused the assert

cmsRun: /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-04-26-2300/src/FWCore/Concurrency/src/WaitingTaskList.cc:57: void edm::WaitingTaskList::reset(): Assertion `m_head == nullptr' failed.

when the end LuminosityBlock was called to gracefully shut down the program.

This implies that the exception caused the begin LuminosityBlock processing to abort before it ran all the required tasks.

Dr15Jones commented 3 years ago

assign core

cmsbuild commented 3 years ago

New categories assigned: core

@Dr15Jones,@smuzaffar,@makortel you have been requested to review this Pull request/Issue and eventually sign? Thanks

cmsbuild commented 3 years ago

A new Issue was created by @Dr15Jones Chris Jones.

@Dr15Jones, @dpiparo, @silviodonato, @smuzaffar, @makortel, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

Dr15Jones commented 3 years ago

The tracebacks were

Assert

Thread 1 (Thread 0x2b6072894a80 (LWP 14347)):
#0  0x00002b6070c8dccd in poll () from /lib64/libc.so.6
#1  0x00002b6079bcdcd7 in full_read.constprop () from /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/6841/14611/CMSSW_12_0_X_2021-04-26-2300/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#2  0x00002b6079bce56c in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/6841/14611/CMSSW_12_0_X_2021-04-26-2300/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#3  0x00002b6079bcf922 in sig_dostack_then_abort () from /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/6841/14611/CMSSW_12_0_X_2021-04-26-2300/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00002b6070bd03d7 in raise () from /lib64/libc.so.6
#6  0x00002b6070bd1ac8 in abort () from /lib64/libc.so.6
#7  0x00002b6070bc91a6 in __assert_fail_base () from /lib64/libc.so.6
#8  0x00002b6070bc9252 in __assert_fail () from /lib64/libc.so.6
#9  0x00002b606e159d19 in edm::WaitingTaskList::reset() () from /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/6841/14611/CMSSW_12_0_X_2021-04-26-2300/lib/slc7_amd64_gcc900/libFWCoreConcurrency.so
#10 0x00002b606e4b13bb in edm::Worker::reset() () from /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/6841/14611/CMSSW_12_0_X_2021-04-26-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#11 0x00002b606e4afd64 in edm::WorkerManager::resetAll() () from /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/6841/14611/CMSSW_12_0_X_2021-04-26-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#12 0x00002b606e3b6240 in edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/6841/14611/CMSSW_12_0_X_2021-04-26-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#13 0x00002b606e3b6574 in tbb::internal::function_task<edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#3}>::execute() () from /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/6841/14611/CMSSW_12_0_X_2021-04-26-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#14 0x00002b606fb7d26d in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::process_bypass_loop (this=this@entry=0x2b60735eb200, context_guard=..., t=0x2b60735e3340, isolation=isolation@entry=0) at ../../include/tbb/task.h:1003
#15 0x00002b606fb7d756 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x2b60735eb200, parent=..., child=<optimized out>) at ../../src/tbb/scheduler.h:650
#16 0x00002b606e39e1f3 in tbb::internal::task_group_base::wait() () from /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/6841/14611/CMSSW_12_0_X_2021-04-26-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#17 0x00002b606e38e760 in edm::EventProcessor::endUnfinishedLumi() () from /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/6841/14611/CMSSW_12_0_X_2021-04-26-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#18 0x00002b606e352b51 in edm::EventProcessor::runToCompletion() [clone .cold] () from /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/6841/14611/CMSSW_12_0_X_2021-04-26-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#19 0x000000000040bae6 in tbb::interface7::internal::delegated_function<main::{lambda()#1}::operator()() const::{lambda()#1} const, void>::operator()() const ()
#20 0x00002b606fb76552 in tbb::interface7::internal::task_arena_base::internal_execute (this=0x7fff13e841c0, d=...) at ../../src/tbb/arena.cpp:1105
#21 0x000000000040ca13 in main::{lambda()#1}::operator()() const ()
#22 0x000000000040b62c in main ()

Exception

Thread 1 "cmsRun" hit Catchpoint 1 (exception thrown), 0x00007ffff5a0532e in __cxxabiv1::__cxa_throw (obj=obj@entry=0x7fff3a43b820, 
    tinfo=0x7ffff5b2b270 <typeinfo for std::length_error>, dest=0x7ffff5a1a4f0 <std::length_error::~length_error()>) at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:78
78  ../../../../libstdc++-v3/libsupc++/eh_throw.cc: No such file or directory.
(gdb) where
#0  0x00007ffff5a0532e in __cxxabiv1::__cxa_throw (obj=obj@entry=0x7fff3a43b820, tinfo=0x7ffff5b2b270 <typeinfo for std::length_error>, 
    dest=0x7ffff5a1a4f0 <std::length_error::~length_error()>) at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:78
#1  0x00007ffff59fbcd4 in std::__throw_length_error (__s=0x7ffff7f129d1 "vector::_M_fill_insert") at ../../../../../libstdc++-v3/src/c++11/functexcept.cc:78
#2  0x00007ffff7f0405c in std::vector<void const*, std::allocator<void const*> >::_M_check_len(unsigned long, char const*) const ()
   from /build/mrodozov/root/622_install/CMSSW_12_0_X_2021-04-26-2300/lib/slc7_amd64_gcc900/libDataFormatsCommon.so
#3  0x00007ffff7f04245 in std::vector<void const*, std::allocator<void const*> >::_M_fill_insert(__gnu_cxx::__normal_iterator<void const**, std::vector<void const*, std::allocator<void const*> > >, unsigned long, void const* const&) ()
   from /build/mrodozov/root/622_install/CMSSW_12_0_X_2021-04-26-2300/lib/slc7_amd64_gcc900/libDataFormatsCommon.so
#4  0x00007fffbfdc6a6d in ROOT::read_recocLcLPFCandidate_2(char*, TVirtualObject*) ()
   from /build/mrodozov/root/622_install/CMSSW_12_0_X_2021-04-26-2300/lib/slc7_amd64_gcc900/libDataFormatsParticleFlowCandidate.so
#5  0x00007ffff71b2c25 in int TStreamerInfo::ReadBufferArtificial<char**>(TBuffer&, char** const&, TStreamerElement*, int, int) ()
   from /build/mrodozov/root/622_install/CMSSW_12_0_X_2021-04-26-2300/external/slc7_amd64_gcc900/lib/libRIO.so
#6  0x00007ffff727af7c in int TStreamerInfo::ReadBuffer<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) ()
   from /build/mrodozov/root/622_install/CMSSW_12_0_X_2021-04-26-2300/external/slc7_amd64_gcc900/lib/libRIO.so
#7  0x00007ffff712d4e4 in TStreamerInfoActions::VectorLooper::GenericRead(TBuffer&, void*, void const*, TStreamerInfoActions::TLoopConfiguration const*, TStreamerInfoActions::TConfiguration const*) () from /build/mrodozov/root/622_install/CMSSW_12_0_X_2021-04-26-2300/external/slc7_amd64_gcc900/lib/libRIO.so
#8  0x00007ffff7027c64 in TBufferFile::ApplySequence(TStreamerInfoActions::TActionSequence const&, void*, void*) ()
   from /build/mrodozov/root/622_install/CMSSW_12_0_X_2021-04-26-2300/external/slc7_amd64_gcc900/lib/libRIO.so
#9  0x00007ffff7730202 in TBranchElement::GetEntry(long long, int) ()
   from /build/mrodozov/root/622_install/CMSSW_12_0_X_2021-04-26-2300/external/slc7_amd64_gcc900/lib/libTree.so
#10 0x00007ffff772fd0d in TBranchElement::GetEntry(long long, int) ()
   from /build/mrodozov/root/622_install/CMSSW_12_0_X_2021-04-26-2300/external/slc7_amd64_gcc900/lib/libTree.so
#11 0x00007fffb56be568 in edm::RootTree::getEntry(TBranch*, long long) const ()
   from /build/mrodozov/root/622_install/CMSSW_12_0_X_2021-04-26-2300/lib/slc7_amd64_gcc900/pluginIOPoolInput.so
#12 0x00007fffb5693f23 in edm::RootDelayedReader::getProduct_(edm::BranchID const&, edm::EDProductGetter const*) ()
   from /build/mrodozov/root/622_install/CMSSW_12_0_X_2021-04-26-2300/lib/slc7_amd64_gcc900/pluginIOPoolInput.so
#13 0x00007ffff7ba1f04 in edm::DelayedReader::getProduct(edm::BranchID const&, edm::EDProductGetter const*, edm::ModuleCallingContext const*) ()
   from /build/mrodozov/root/622_install/CMSSW_12_0_X_2021-04-26-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#14 0x00007ffff7c6c885 in edm::DelayedReaderInputProductResolver::prefetchAsync_(edm::WaitingTaskHolder, edm::Principal const&, bool, edm::ServiceToken const&, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const::{lambda()#1}::operator()() const ()
   from /build/mrodozov/root/622_install/CMSSW_12_0_X_2021-04-26-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#15 0x00007ffff7c6ca61 in edm::SerialTaskQueue::QueuedTask<edm::SerialTaskQueueChain::push<edm::DelayedReaderInputProductResolver::prefetchAsync_(edm::WaitingTaskHolder, edm::Principal const&, bool, edm::ServiceToken const&, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const::{lambda()#1}&>(tbb::task_group&, edm::DelayedReaderInputProductResolver::prefetchAsync_(edm::WaitingTaskHolder, edm::Principal const&, bool, edm::ServiceToken const&, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const::{lambda()#1}&)::{lambda()#1}>::execute() ()
   from /build/mrodozov/root/622_install/CMSSW_12_0_X_2021-04-26-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#16 0x00007ffff7e8cf49 in tbb::internal::function_task<edm::SerialTaskQueue::spawn(edm::SerialTaskQueue::TaskBase&)::{lambda()#1}>::execute() ()
   from /build/mrodozov/root/622_install/CMSSW_12_0_X_2021-04-26-2300/lib/slc7_amd64_gcc900/libFWCoreConcurrency.so
#17 0x00007ffff645d26d in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::process_bypass_loop (this=this@entry=0x7ffff2147900, context_guard=..., 
--Type <RET> for more, q to quit, c to continue without paging--
    7fff4204ee40, isolation=isolation@entry=0) at ../../include/tbb/task.h:1003
#18 0x00007ffff645d756 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7ffff2147900, parent=..., child=<optimized out>)
    at ../../src/tbb/scheduler.h:650
#19 0x00007ffff7bc1058 in edm::EventProcessor::processLumis(std::shared_ptr<void> const&) ()
   from /build/mrodozov/root/622_install/CMSSW_12_0_X_2021-04-26-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#20 0x00007ffff7bca035 in edm::EventProcessor::runToCompletion() () from /build/mrodozov/root/622_install/CMSSW_12_0_X_2021-04-26-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#21 0x000000000040bae6 in tbb::interface7::internal::delegated_function<main::{lambda()#1}::operator()() const::{lambda()#1} const, void>::operator()() const ()
#22 0x00007ffff6456552 in tbb::interface7::internal::task_arena_base::internal_execute (this=0x7fffffff31b0, d=...) at ../../src/tbb/arena.cpp:1105
#23 0x000000000040ca13 in main::{lambda()#1}::operator()() const ()
#24 0x000000000040b62c in main ()
dan131riley commented 3 years ago

There's a lot of these in the ROOT6_X IB, also a couple of unit test failures that look like they could be related, one in runtestTqafTopKinFitter:

30-Apr-2021 03:45:25 CEST  Closed file root://eoscms.cern.ch//eos/cms/store/user/cmsbuild/store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root
----- Begin Fatal Exception 30-Apr-2021 03:45:26 CEST-----------------------
An exception of category 'StdException' occurred while
   [0] Calling EventProcessor::runToCompletion (which does almost everything after beginJob and before endJob)
Exception Message:
A std::exception was thrown.
vector::_M_fill_insert
----- End Fatal Exception -------------------------------------------------

and another in runtestTqafTopTools:

Begin processing the 1st record. Run 1, Event 2, LumiSection 1 on stream 0 at 30-Apr-2021 03:46:40.330 CEST
----- Begin Fatal Exception 30-Apr-2021 03:46:42 CEST-----------------------
An exception of category 'StdException' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 2 stream: 0
   [1] Running path 'outpath'
   [2] Prefetching for module PoolOutputModule/'out'
   [3] Prefetching for module TtFullHadJetPartonMatch/'ttFullHadJetPartonMatch'
   [4] Prefetching for module PATJetSelector/'selectedPatJets'
   [5] Prefetching for module PATJetProducer/'patJets'
   [6] Calling method for module JetFlavourClustering/'patJetFlavourAssociation'
Exception Message:
A std::exception was thrown.
vector::_M_fill_insert
----- End Fatal Exception -------------------------------------------------