Segfault when shutting down after exception in read

makortel commented 2 years ago

Experienced in https://github.com/cms-sw/cmssw/pull/36335#issuecomment-1022507325

#--------------------------------------------------------------------------
----- Begin Fatal Exception 26-Jan-2022 19:09:56 CET-----------------------
An exception of category 'FileReadError' occurred while
   [0] Processing  Event run: 320822 lumi: 17 event: 26886180 stream: 0
   [1] Running path 'MINIAODoutput_step'
   [2] Prefetching for module PoolOutputModule/'MINIAODoutput'
   [3] While reading from source std::vector<reco::ForwardProton> ctppsProtons 'multiRP' RECO
   [4] Rethrowing an exception that happened on a different read request.
   [5] Processing  Event run: 320822 lumi: 17 event: 26886180 stream: 0
   [6] Running path 'MINIAODoutput_step'
   [7] Prefetching for module PoolOutputModule/'MINIAODoutput'
   [8] Prefetching for module JPTJetSlimmer/'slimmedJPTJets'
   [9] While reading from source std::vector<reco::JPTJet> JetPlusTrackZSPCorJetAntiKt4 '' RECO
   [10] Reading branch recoJPTJets_JetPlusTrackZSPCorJetAntiKt4__RECO.
Exception Message:
attempting to cast a null RefToBase;
You should check for nullity before casting. 
----- End Fatal Exception -------------------------------------------------

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thread 1 (Thread 0x2af30bf40b00 (LWP 23344) "cmsRun"):
#3  0x00002af30f9763eb in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00002af334b96d68 in std::_Vector_base<void const*, std::allocator<void const*> >::~_Vector_base() () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libDataFormatsTrackReco.so
#6  0x00002af334b98f8b in edm::RefVectorBase<unsigned int>::~RefVectorBase() () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libDataFormatsTrackReco.so
#7  0x00002af3081befab in TStreamerInfo::DestructorImpl(void*, bool) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_3_X_2022-01-26-1100/external/slc7_amd64_gcc10/lib/libRIO.so
#8  0x00002af3081bfba5 in TStreamerInfo::DeleteArray(void*, bool) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_3_X_2022-01-26-1100/external/slc7_amd64_gcc10/lib/libRIO.so
#9  0x00002af307be2f1b in TBranchElement::~TBranchElement() () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_3_X_2022-01-26-1100/external/slc7_amd64_gcc10/lib/libTree.so
#10 0x00002af307be3029 in TBranchElement::~TBranchElement() () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_3_X_2022-01-26-1100/external/slc7_amd64_gcc10/lib/libTree.so
#11 0x00002af3086c576c in TObjArray::Delete(char const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_3_X_2022-01-26-1100/external/slc7_amd64_gcc10/lib/libCore.so
#12 0x00002af307bd69f0 in TBranch::~TBranch() () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_3_X_2022-01-26-1100/external/slc7_amd64_gcc10/lib/libTree.so
#13 0x00002af307be3029 in TBranchElement::~TBranchElement() () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_3_X_2022-01-26-1100/external/slc7_amd64_gcc10/lib/libTree.so
#14 0x00002af3086c576c in TObjArray::Delete(char const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_3_X_2022-01-26-1100/external/slc7_amd64_gcc10/lib/libCore.so
#15 0x00002af307bd69f0 in TBranch::~TBranch() () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_3_X_2022-01-26-1100/external/slc7_amd64_gcc10/lib/libTree.so
#16 0x00002af307be3029 in TBranchElement::~TBranchElement() () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_3_X_2022-01-26-1100/external/slc7_amd64_gcc10/lib/libTree.so
#17 0x00002af3086c576c in TObjArray::Delete(char const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_3_X_2022-01-26-1100/external/slc7_amd64_gcc10/lib/libCore.so
#18 0x00002af307c4cc50 in TTree::~TTree() () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_3_X_2022-01-26-1100/external/slc7_amd64_gcc10/lib/libTree.so
#19 0x00002af307c4d259 in TTree::~TTree() () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_3_X_2022-01-26-1100/external/slc7_amd64_gcc10/lib/libTree.so
#20 0x00002af3086bf960 in TList::Delete(char const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_3_X_2022-01-26-1100/external/slc7_amd64_gcc10/lib/libCore.so
#21 0x00002af3086b5e8b in THashList::Delete(char const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_3_X_2022-01-26-1100/external/slc7_amd64_gcc10/lib/libCore.so
#22 0x00002af308164b05 in TDirectoryFile::Close(char const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_3_X_2022-01-26-1100/external/slc7_amd64_gcc10/lib/libRIO.so
#23 0x00002af3081818ec in TFile::Close(char const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_3_X_2022-01-26-1100/external/slc7_amd64_gcc10/lib/libRIO.so
#24 0x00002af34dae266c in edm::InputFile::Close() () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/pluginIOPoolInput.so
#25 0x00002af34db0a35b in edm::RootFile::close() () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/pluginIOPoolInput.so
#26 0x00002af34db2303e in edm::RootPrimaryFileSequence::closeFile_() () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/pluginIOPoolInput.so
#27 0x00002af30777992b in decltype ({parm#1}()) edm::convertException::wrap<edm::callWithTryCatchAndPrint<void>(std::function<void ()>, char const*, bool)::{lambda()#1}>(edm::callWithTryCatchAndPrint<void>(std::function<void ()>, char const*, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#28 0x00002af307779a3c in void edm::callWithTryCatchAndPrint<void>(std::function<void ()>, char const*, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#29 0x00002af307776544 in edm::InputSource::closeFile(edm::FileBlock*, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#30 0x00002af3076ffab0 in edm::EventProcessor::closeInputFile(bool) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#31 0x00002af3076c61b7 in edm::EventProcessor::runToCompletion() [clone .cold] () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#32 0x000000000040a266 in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#33 0x00002af308fa415b in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/arena.cpp:698
#34 0x000000000040b094 in main::{lambda()#1}::operator()() const ()
#35 0x000000000040971c in main ()

Current Modules:
Module: none (crashed)

https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8761bd/22027/runTheMatrix-results/136.88811_RunJetHT2018D_reminiaodUL+RunJetHT2018D_reminiaodUL+REMINIAOD_data2018UL+HARVEST2018_REMINIAOD_data2018UL/step2_RunJetHT2018D_reminiaodUL+RunJetHT2018D_reminiaodUL+REMINIAOD_data2018UL+HARVEST2018_REMINIAOD_data2018UL.log

makortel commented 2 years ago

assign core

cmsbuild commented 2 years ago

New categories assigned: core

@Dr15Jones,@smuzaffar,@makortel you have been requested to review this Pull request/Issue and eventually sign? Thanks

cmsbuild commented 2 years ago

A new Issue was created by @makortel Matti Kortelainen.

@Dr15Jones, @perrotta, @dpiparo, @makortel, @smuzaffar, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

makortel commented 2 years ago

@pcanal Would you have any insights? (on low priority)

pcanal commented 2 years ago

Extracting from the log file:

#3  0x00002af30f9763eb in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00002af334b96d68 in std::_Vector_base<void const*, std::allocator<void const*> >::~_Vector_base() () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libDataFormatsTrackReco.so
#6  0x00002af334b98f8b in edm::RefVectorBase<unsigned int>::~RefVectorBase() () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libDataFormatsTrackReco.so
#7  0x00002af3081befab in TStreamerInfo::DestructorImpl(void*, bool) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_3_X_2022-01-26-1100/external/slc7_amd64_gcc10/lib/libRIO.so
#8  0x00002af3081bfba5 in TStreamerInfo::DeleteArray(void*, bool) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_3_X_2022-01-26-1100/external/slc7_amd64_gcc10/lib/libRIO.so
#9  0x00002af307be2f1b in TBranchElement::~TBranchElement() () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_3_X_2022-01-26-1100/external/slc7_amd64_gcc10/lib/libTree.so
#10 0x00002af307be3029 in TBranchElement::~TBranchElement() () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_3_X_2022-01-26-1100/external/slc7_amd64_gcc10/lib/libTree.so
#11 0x00002af3086c576c in TObjArray::Delete(char const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_3_X_2022-01-26-1100/external/slc7_amd64_gcc10/lib/libCore.so
#12 0x00002af307bd69f0 in TBranch::~TBranch() () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_3_X_2022-01-26-1100/external/slc7_amd64_gcc10/lib/libTree.so
#13 0x00002af307be3029 in TBranchElement::~TBranchElement() () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_3_X_2022-01-26-1100/external/slc7_amd64_gcc10/lib/libTree.so
#14 0x00002af3086c576c in TObjArray::Delete(char const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_3_X_2022-01-26-1100/external/slc7_amd64_gcc10/lib/libCore.so
#15 0x00002af307bd69f0 in TBranch::~TBranch() () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_3_X_2022-01-26-1100/external/slc7_amd64_gcc10/lib/libTree.so
#16 0x00002af307be3029 in TBranchElement::~TBranchElement() () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_3_X_2022-01-26-1100/external/slc7_amd64_gcc10/lib/libTree.so
#17 0x00002af3086c576c in TObjArray::Delete(char const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_3_X_2022-01-26-1100/external/slc7_amd64_gcc10/lib/libCore.so
#18 0x00002af307c4cc50 in TTree::~TTree() () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_3_X_2022-01-26-1100/external/slc7_amd64_gcc10/lib/libTree.so
#19 0x00002af307c4d259 in TTree::~TTree() () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_3_X_2022-01-26-1100/external/slc7_amd64_gcc10/lib/libTree.so
#20 0x00002af3086bf960 in TList::Delete(char const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_3_X_2022-01-26-1100/external/slc7_amd64_gcc10/lib/libCore.so
#21 0x00002af3086b5e8b in THashList::Delete(char const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_3_X_2022-01-26-1100/external/slc7_amd64_gcc10/lib/libCore.so
#22 0x00002af308164b05 in TDirectoryFile::Close(char const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_3_X_2022-01-26-1100/external/slc7_amd64_gcc10/lib/libRIO.so
#23 0x00002af3081818ec in TFile::Close(char const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_3_X_2022-01-26-1100/external/slc7_amd64_gcc10/lib/libRIO.so
#24 0x00002af34dae266c in edm::InputFile::Close() () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/pluginIOPoolInput.so
#25 0x00002af34db0a35b in edm::RootFile::close() () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/pluginIOPoolInput.so
#26 0x00002af34db2303e in edm::RootPrimaryFileSequence::closeFile_() () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/pluginIOPoolInput.so
#27 0x00002af30777992b in decltype ({parm#1}()) edm::convertException::wrap<edm::callWithTryCatchAndPrint<void>(std::function<void ()>, char const*, bool)::{lambda()#1}>(edm::callWithTryCatchAndPrint<void>(std::function<void ()>, char const*, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#28 0x00002af307779a3c in void edm::callWithTryCatchAndPrint<void>(std::function<void ()>, char const*, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#29 0x00002af307776544 in edm::InputSource::closeFile(edm::FileBlock*, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#30 0x00002af3076ffab0 in edm::EventProcessor::closeInputFile(bool) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#31 0x00002af3076c61b7 in edm::EventProcessor::runToCompletion() [clone .cold] () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#32 0x000000000040a266 in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#33 0x00002af308fa415b in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/arena.cpp:698
#34 0x000000000040b094 in main::{lambda()#1}::operator()() const ()
#35 0x000000000040971c in main ()

pcanal commented 2 years ago

If it is reproducible, a valgrind output would be helpful in tracking this down.

makortel commented 2 years ago

It didn't seem to be fully reproducible (some jobs shut down properly after the exception). I could try if I'd get it fail with valgrind.

makortel commented 2 years ago

I had run valgrind on that job 5 times. All of those reported this one before the exception

==9174== Invalid free() / delete / delete[] / realloc()
==9174==    at 0x4032447: operator delete[](void*) (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/external/valgrind/3.17.0-12ba960b086b024cce391ad228855798/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==9174==    by 0x5B9D792: TClass::Destructor(void*, bool) (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/lcg/root/6.24.07-a648249d43cd7f1eed4a1ce20f339e57/lib/libCore.so)
==9174==    by 0x5739BBF: TVirtualObject::~TVirtualObject() (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/lcg/root/6.24.07-a648249d43cd7f1eed4a1ce20f339e57/lib/libRIO.so)
==9174==    by 0x5739E25: int TStreamerInfo::ReadBufferArtificial<char**>(TBuffer&, char** const&, TStreamerElement*, int, int) (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/lcg/root/6.24.07-a648249d43cd7f1eed4a1ce20f339e57/lib/libRIO.so)
==9174==    by 0x57FDE03: int TStreamerInfo::ReadBuffer<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/lcg/root/6.24.07-a648249d43cd7f1eed4a1ce20f339e57/lib/libRIO.so)
==9174==    by 0x56B2C9B: TStreamerInfoActions::VectorLooper::GenericRead(TBuffer&, void*, void const*, TStreamerInfoActions::TLoopConfiguration const*, TStreamerInfoActions::TConfiguration const*) (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/lcg/root/6.24.07-a648249d43cd7f1eed4a1ce20f339e57/lib/libRIO.so)
==9174==    by 0x55AC4C3: TBufferFile::ApplySequence(TStreamerInfoActions::TActionSequence const&, void*, void*) (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/lcg/root/6.24.07-a648249d43cd7f1eed4a1ce20f339e57/lib/libRIO.so)
==9174==    by 0x5087FE1: TBranchElement::GetEntry(long long, int) (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/lcg/root/6.24.07-a648249d43cd7f1eed4a1ce20f339e57/lib/libTree.so)
==9174==    by 0x5087AFF: TBranchElement::GetEntry(long long, int) (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/lcg/root/6.24.07-a648249d43cd7f1eed4a1ce20f339e57/lib/libTree.so)
==9174==    by 0x506A74A0: edm::RootTree::getEntry(TBranch*, long long) const (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/pluginIOPoolInput.so)
==9174==    by 0x5067823C: edm::RootDelayedReader::getProduct_(edm::BranchID const&, edm::EDProductGetter const*) (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/pluginIOPoolInput.so)
==9174==    by 0x4B83BC6: edm::DelayedReader::getProduct(edm::BranchID const&, edm::EDProductGetter const*, edm::ModuleCallingContext const*) (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so)
==9174==    by 0x4C5A86A: edm::DelayedReaderInputProductResolver::prefetchAsync_(edm::WaitingTaskHolder, edm::Principal const&, bool, edm::ServiceToken const&, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const::{lambda()#1}::operator()() const::{lambda()#1}::operator()() const (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so)
==9174==    by 0x4C5AA8F: edm::DelayedReaderInputProductResolver::prefetchAsync_(edm::WaitingTaskHolder, edm::Principal const&, bool, edm::ServiceToken const&, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const::{lambda()#1}::operator()() const (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so)
==9174==    by 0x4C5AB9E: void edm::SerialTaskQueueChain::actionToRun<edm::DelayedReaderInputProductResolver::prefetchAsync_(edm::WaitingTaskHolder, edm::Principal const&, bool, edm::ServiceToken const&, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const::{lambda()#1}&>(edm::DelayedReaderInputProductResolver::prefetchAsync_(edm::WaitingTaskHolder, edm::Principal const&, bool, edm::ServiceToken const&, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const::{lambda()#1}&) (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so)
==9174==    by 0x4C5ABF0: edm::SerialTaskQueue::QueuedTask<void edm::SerialTaskQueueChain::push<edm::DelayedReaderInputProductResolver::prefetchAsync_(edm::WaitingTaskHolder, edm::Principal const&, bool, edm::ServiceToken const&, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const::{lambda()#1}&>(tbb::detail::d1::task_group&, edm::DelayedReaderInputProductResolver::prefetchAsync_(edm::WaitingTaskHolder, edm::Principal const&, bool, edm::ServiceToken const&, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const::{lambda()#1}&)::{lambda()#1}>::execute() (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so)
==9174==    by 0x41EB074: tbb::detail::d1::function_task<edm::SerialTaskQueue::spawn(edm::SerialTaskQueue::TaskBase&)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreConcurrency.so)
==9174==    by 0x6473B8B: tbb::detail::r1::task_dispatcher::execute_and_wait(tbb::detail::d1::task*, tbb::detail::d1::wait_context&, tbb::detail::d1::task_group_context&) (task_dispatcher.h:322)
==9174==    by 0x4BA9907: edm::EventProcessor::processLumis(std::shared_ptr<void> const&) (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so)
==9174==    by 0x4BB463A: edm::EventProcessor::runToCompletion() (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so)
==9174==  Address 0xff173c60 is 16 bytes inside a block of size 40,336 alloc'd
==9174==    at 0x40300F3: operator new[](unsigned long) (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/external/valgrind/3.17.0-12ba960b086b024cce391ad228855798/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==9174==    by 0x565EA52: TStreamerInfo::NewArray(long, void*) (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/lcg/root/6.24.07-a648249d43cd7f1eed4a1ce20f339e57/lib/libRIO.so)
==9174==    by 0x5BA08D8: TClass::NewObjectArray(long, TClass::ENewType) const (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/lcg/root/6.24.07-a648249d43cd7f1eed4a1ce20f339e57/lib/libCore.so)
==9174==    by 0x5083083: TBranchElement::InitInfo()::{lambda(TStreamerInfo*)#1}::operator()(TStreamerInfo*) const (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/lcg/root/6.24.07-a648249d43cd7f1eed4a1ce20f339e57/lib/libTree.so)
==9174==    by 0x508EA47: TBranchElement::InitInfo() (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/lcg/root/6.24.07-a648249d43cd7f1eed4a1ce20f339e57/lib/libTree.so)
==9174==    by 0x507E504: TBranchElement::GetInfo() const (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/lcg/root/6.24.07-a648249d43cd7f1eed4a1ce20f339e57/lib/libTree.so)
==9174==    by 0x507F852: (anonymous namespace)::GatherArtificialElements(TObjArray const&, std::vector<TStreamerInfoActions::TIDNode, std::allocator<TStreamerInfoActions::TIDNode> >&, TString, TStreamerInfo*, int) (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/lcg/root/6.24.07-a648249d43cd7f1eed4a1ce20f339e57/lib/libTree.so)
==9174==    by 0x507FD43: (anonymous namespace)::GatherArtificialElements(TObjArray const&, std::vector<TStreamerInfoActions::TIDNode, std::allocator<TStreamerInfoActions::TIDNode> >&, TString, TStreamerInfo*, int) (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/lcg/root/6.24.07-a648249d43cd7f1eed4a1ce20f339e57/lib/libTree.so)
==9174==    by 0x507FD43: (anonymous namespace)::GatherArtificialElements(TObjArray const&, std::vector<TStreamerInfoActions::TIDNode, std::allocator<TStreamerInfoActions::TIDNode> >&, TString, TStreamerInfo*, int) (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/lcg/root/6.24.07-a648249d43cd7f1eed4a1ce20f339e57/lib/libTree.so)
==9174==    by 0x508E0CC: TBranchElement::InitInfo() (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/lcg/root/6.24.07-a648249d43cd7f1eed4a1ce20f339e57/lib/libTree.so)
==9174==    by 0x507E504: TBranchElement::GetInfo() const (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/lcg/root/6.24.07-a648249d43cd7f1eed4a1ce20f339e57/lib/libTree.so)
==9174==    by 0x507F852: (anonymous namespace)::GatherArtificialElements(TObjArray const&, std::vector<TStreamerInfoActions::TIDNode, std::allocator<TStreamerInfoActions::TIDNode> >&, TString, TStreamerInfo*, int) (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/lcg/root/6.24.07-a648249d43cd7f1eed4a1ce20f339e57/lib/libTree.so)
==9174==    by 0x508E0CC: TBranchElement::InitInfo() (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/lcg/root/6.24.07-a648249d43cd7f1eed4a1ce20f339e57/lib/libTree.so)
==9174==    by 0x508F2AC: TBranchElement::SetAddressImpl(void*, bool) (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/lcg/root/6.24.07-a648249d43cd7f1eed4a1ce20f339e57/lib/libTree.so)
==9174==    by 0x5067820A: edm::RootDelayedReader::getProduct_(edm::BranchID const&, edm::EDProductGetter const*) (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/pluginIOPoolInput.so)
==9174==    by 0x4B83BC6: edm::DelayedReader::getProduct(edm::BranchID const&, edm::EDProductGetter const*, edm::ModuleCallingContext const*) (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so)
==9174==    by 0x4C5A86A: edm::DelayedReaderInputProductResolver::prefetchAsync_(edm::WaitingTaskHolder, edm::Principal const&, bool, edm::ServiceToken const&, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const::{lambda()#1}::operator()() const::{lambda()#1}::operator()() const (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so)
==9174==    by 0x4C5AA8F: edm::DelayedReaderInputProductResolver::prefetchAsync_(edm::WaitingTaskHolder, edm::Principal const&, bool, edm::ServiceToken const&, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const::{lambda()#1}::operator()() const (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so)
==9174==    by 0x4C5AB9E: void edm::SerialTaskQueueChain::actionToRun<edm::DelayedReaderInputProductResolver::prefetchAsync_(edm::WaitingTaskHolder, edm::Principal const&, bool, edm::ServiceToken const&, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const::{lambda()#1}&>(edm::DelayedReaderInputProductResolver::prefetchAsync_(edm::WaitingTaskHolder, edm::Principal const&, bool, edm::ServiceToken const&, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const::{lambda()#1}&) (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so)
==9174==    by 0x4C5ABF0: edm::SerialTaskQueue::QueuedTask<void edm::SerialTaskQueueChain::push<edm::DelayedReaderInputProductResolver::prefetchAsync_(edm::WaitingTaskHolder, edm::Principal const&, bool, edm::ServiceToken const&, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const::{lambda()#1}&>(tbb::detail::d1::task_group&, edm::DelayedReaderInputProductResolver::prefetchAsync_(edm::WaitingTaskHolder, edm::Principal const&, bool, edm::ServiceToken const&, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const::{lambda()#1}&)::{lambda()#1}>::execute() (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so)

and this after the exception

==9174== Conditional jump or move depends on uninitialised value(s)
==9174==    at 0x70DB08B9: LogErrorEventFilter::globalEndLuminosityBlock(edm::LuminosityBlock const&, edm::EventSetup const&) const (in /build/mkortela/debug/CMSSW_12_3_X_2022-01-26-1100/lib/slc7_amd64_gcc10/pluginDPGAnalysisSkims.so)
==9174==    by 0x70DB68A1: virtual thunk to edm::global::impl::LuminosityBlockCacheHolder<edm::global::EDFilterBase, leef::LumiErrors>::doEndLuminosityBlock_(edm::LuminosityBlock const&, edm::EventSetup const&) (in /build/mkortela/debug/CMSSW_12_3_X_2022-01-26-1100/lib/slc7_amd64_gcc10/pluginDPGAnalysisSkims.so)
==9174==    by 0x4CE17DC: edm::global::EDFilterBase::doEndLuminosityBlock(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so)
==9174==    by 0x4CD6E1F: edm::WorkerT<edm::global::EDFilterBase>::implDoEnd(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so)
==9174==    by 0x4BDFE33: decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3> >(edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3> >(edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3>::Context const*)::{lambda()#1}) (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so)
==9174==    by 0x4BE01E6: std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3>::Context const*) (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so)
==9174==    by 0x4BE0679: edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3> >::execute() (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so)
==9174==    by 0x4B7FA94: tbb::detail::d1::function_task<edm::WaitingTaskHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so)
==9174==    by 0x6473B8B: tbb::detail::r1::task_dispatcher::execute_and_wait(tbb::detail::d1::task*, tbb::detail::d1::wait_context&, tbb::detail::d1::task_group_context&) (task_dispatcher.h:322)
==9174==    by 0x4BA9907: edm::EventProcessor::processLumis(std::shared_ptr<void> const&) (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so)
==9174==    by 0x4BB463A: edm::EventProcessor::runToCompletion() (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so)
==9174==    by 0x40A265: tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/bin/slc7_amd64_gcc10/cmsRun)
==9174==    by 0x646215A: tbb::detail::r1::task_arena_impl::execute(tbb::detail::d1::task_arena_base&, tbb::detail::d1::delegate_base&) (arena.cpp:698)
==9174==    by 0x40B093: main::{lambda()#1}::operator()() const (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/bin/slc7_amd64_gcc10/cmsRun)
==9174==    by 0x40971B: main (in /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-2300/bin/slc7_amd64_gcc10/cmsRun)
==9174==

pcanal commented 2 years ago

The stack trace indicates this is related to the I/O customization rules. So it will be interesting to see if the latest changes to that part of the code improves the behavior. See https://github.com/root-project/root/issues/7718#issuecomment-1029362146 for the current PR testings those change on CMSSW.

cms-sw / cmssw

Segfault when shutting down after exception in read #36820