search

cms-sw / cmssw

CMS Offline Software
http://cms-sw.github.io/
Apache License 2.0
1.08k stars 4.3k forks source link

segfault in TICLLayerTileProducer::produce() #42669

Closed dan131riley closed 1 year ago

dan131riley commented 1 year ago

Seen in CMSSW_13_3_ROOT628_X_2023-08-27-2300 el8_amd64_gcc11 WF 25234.911 step2. I think we've seen this before in the ARM64 builds, but I don't find an issue for it.

Thread 5 (Thread 0x149a6dc1f700 (LWP 313210) "cmsRun"):
#2  0x0000149ac7693b10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT628_X_2023-08-27-2300/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000149ac9cb3114 in std::use_facet<std::num_get<char, std::istreambuf_iterator<char, std::char_traits<char> > > > (__loc=...) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.4.1-30ebdc301ebd200f2ae0e3d880258e65/gcc-11.4.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/locale_classes.tcc:135
#5  0x0000149ac9ca96ba in std::basic_ios<char, std::char_traits<char> >::_M_cache_locale (this=this@entry=0x149a6dc18a80, __loc=...) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.4.1-30ebdc301ebd200f2ae0e3d880258e65/gcc-11.4.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/basic_ios.tcc:170
#6  0x0000149ac9ca9ab4 in std::basic_ios<char, std::char_traits<char> >::init (this=0x149a6dc18a80, __sb=0x149a6dc18a18) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.4.1-30ebdc301ebd200f2ae0e3d880258e65/gcc-11.4.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/basic_ios.tcc:132
#7  0x0000149aa554af9f in HGCalDDDConstants::locateCell(int, int, int, int, int, int, bool, bool, bool, bool) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT628_X_2023-08-27-2300/lib/el8_amd64_gcc11/libGeometryHGCalCommonData.so
#8  0x0000149aa57a6ad6 in HGCalGeometry::getPosition(DetId const&, bool) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT628_X_2023-08-27-2300/lib/el8_amd64_gcc11/libGeometryHGCalGeometry.so
#9  0x0000149a5b1c5ada in HGCalTriggerGeometryV9Imp3::getModulePosition(unsigned int) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT628_X_2023-08-27-2300/lib/el8_amd64_gcc11/pluginL1TriggerL1THGCalPlugins_geometries.so
#10 0x0000149aa27246e2 in HGCalConcentratorTrigSumImpl::doSum(unsigned int, std::vector<l1t::HGCalTriggerCell, std::allocator<l1t::HGCalTriggerCell> > const&, std::vector<l1t::HGCalTriggerSums, std::allocator<l1t::HGCalTriggerSums> >&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT628_X_2023-08-27-2300/lib/el8_amd64_gcc11/libL1TriggerL1THGCal.so
#11 0x0000149a5dda032b in HGCalConcentratorProcessorSelection::run(edm::Handle<BXVector<l1t::HGCalTriggerCell> > const&, std::tuple<BXVector<l1t::HGCalTriggerCell>, BXVector<l1t::HGCalTriggerSums>, BXVector<l1t::HGCalConcentratorData> >&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT628_X_2023-08-27-2300/lib/el8_amd64_gcc11/pluginL1TriggerL1THGCalPlugins_fe_be.so
#12 0x0000149aa27715e2 in HGCalConcentratorProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT628_X_2023-08-27-2300/lib/el8_amd64_gcc11/pluginL1TriggerL1THGCalPlugins.so

Thread 4 (Thread 0x149a6e620700 (LWP 313208) "cmsRun"):
#2  0x0000149ac7693b10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT628_X_2023-08-27-2300/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000149ac9c40c9d in __cxxabiv1::__dynamic_cast (src_ptr=0x149ac9da2690 <(anonymous namespace)::num_put_c>, src_type=0x149ac9d95e78 <typeinfo for std::locale::facet>, dst_type=0x149ac9d9aa00 <typeinfo for std::num_put<char, std::ostreambuf_iterator<char, std::char_traits<char> > >>, src2dst=0) at ../../../../libstdc++-v3/libsupc++/dyncast.cc:76
#5  0x0000149ac9cb3232 in std::has_facet<std::num_put<char, std::ostreambuf_iterator<char, std::char_traits<char> > > > (__loc=...) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.4.1-30ebdc301ebd200f2ae0e3d880258e65/gcc-11.4.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/locale_classes.tcc:110
#6  0x0000149ac9ca9693 in std::basic_ios<char, std::char_traits<char> >::_M_cache_locale (this=this@entry=0x149a6e619a80, __loc=...) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.4.1-30ebdc301ebd200f2ae0e3d880258e65/gcc-11.4.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/basic_ios.tcc:164
#7  0x0000149ac9ca9ab4 in std::basic_ios<char, std::char_traits<char> >::init (this=0x149a6e619a80, __sb=0x149a6e619a18) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.4.1-30ebdc301ebd200f2ae0e3d880258e65/gcc-11.4.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/basic_ios.tcc:132
#8  0x0000149aa554af9f in HGCalDDDConstants::locateCell(int, int, int, int, int, int, bool, bool, bool, bool) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT628_X_2023-08-27-2300/lib/el8_amd64_gcc11/libGeometryHGCalCommonData.so
#9  0x0000149aa57a6ad6 in HGCalGeometry::getPosition(DetId const&, bool) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT628_X_2023-08-27-2300/lib/el8_amd64_gcc11/libGeometryHGCalGeometry.so
#10 0x0000149a5b1c5ada in HGCalTriggerGeometryV9Imp3::getModulePosition(unsigned int) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT628_X_2023-08-27-2300/lib/el8_amd64_gcc11/pluginL1TriggerL1THGCalPlugins_geometries.so
#11 0x0000149aa27246e2 in HGCalConcentratorTrigSumImpl::doSum(unsigned int, std::vector<l1t::HGCalTriggerCell, std::allocator<l1t::HGCalTriggerCell> > const&, std::vector<l1t::HGCalTriggerSums, std::allocator<l1t::HGCalTriggerSums> >&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT628_X_2023-08-27-2300/lib/el8_amd64_gcc11/libL1TriggerL1THGCal.so
#12 0x0000149a5dda032b in HGCalConcentratorProcessorSelection::run(edm::Handle<BXVector<l1t::HGCalTriggerCell> > const&, std::tuple<BXVector<l1t::HGCalTriggerCell>, BXVector<l1t::HGCalTriggerSums>, BXVector<l1t::HGCalConcentratorData> >&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT628_X_2023-08-27-2300/lib/el8_amd64_gcc11/pluginL1TriggerL1THGCalPlugins_fe_be.so
#13 0x0000149aa27715e2 in HGCalConcentratorProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT628_X_2023-08-27-2300/lib/el8_amd64_gcc11/pluginL1TriggerL1THGCalPlugins.so

Thread 3 (Thread 0x149a6f021700 (LWP 313206) "cmsRun"):
#2  0x0000149ac7693b10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT628_X_2023-08-27-2300/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000149a61e61f50 in void std::__introsort_loop<__gnu_cxx::__normal_iterator<HGCDataFrame<DetId, HGCSample>*, std::vector<HGCDataFrame<DetId, HGCSample>, std::allocator<HGCDataFrame<DetId, HGCSample> > > >, long, __gnu_cxx::__ops::_Iter_comp_iter<edm::StrictWeakOrdering<HGCDataFrame<DetId, HGCSample> > > >(__gnu_cxx::__normal_iterator<HGCDataFrame<DetId, HGCSample>*, std::vector<HGCDataFrame<DetId, HGCSample>, std::allocator<HGCDataFrame<DetId, HGCSample> > > >, __gnu_cxx::__normal_iterator<HGCDataFrame<DetId, HGCSample>*, std::vector<HGCDataFrame<DetId, HGCSample>, std::allocator<HGCDataFrame<DetId, HGCSample> > > >, long, __gnu_cxx::__ops::_Iter_comp_iter<edm::StrictWeakOrdering<HGCDataFrame<DetId, HGCSample> > >) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT628_X_2023-08-27-2300/lib/el8_amd64_gcc11/pluginEventFilterHGCalRawToDigiAuto.so
#5  0x0000149a61e62144 in void std::__introsort_loop<__gnu_cxx::__normal_iterator<HGCDataFrame<DetId, HGCSample>*, std::vector<HGCDataFrame<DetId, HGCSample>, std::allocator<HGCDataFrame<DetId, HGCSample> > > >, long, __gnu_cxx::__ops::_Iter_comp_iter<edm::StrictWeakOrdering<HGCDataFrame<DetId, HGCSample> > > >(__gnu_cxx::__normal_iterator<HGCDataFrame<DetId, HGCSample>*, std::vector<HGCDataFrame<DetId, HGCSample>, std::allocator<HGCDataFrame<DetId, HGCSample> > > >, __gnu_cxx::__normal_iterator<HGCDataFrame<DetId, HGCSample>*, std::vector<HGCDataFrame<DetId, HGCSample>, std::allocator<HGCDataFrame<DetId, HGCSample> > > >, long, __gnu_cxx::__ops::_Iter_comp_iter<edm::StrictWeakOrdering<HGCDataFrame<DetId, HGCSample> > >) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT628_X_2023-08-27-2300/lib/el8_amd64_gcc11/pluginEventFilterHGCalRawToDigiAuto.so
#6  0x0000149a61e62144 in void std::__introsort_loop<__gnu_cxx::__normal_iterator<HGCDataFrame<DetId, HGCSample>*, std::vector<HGCDataFrame<DetId, HGCSample>, std::allocator<HGCDataFrame<DetId, HGCSample> > > >, long, __gnu_cxx::__ops::_Iter_comp_iter<edm::StrictWeakOrdering<HGCDataFrame<DetId, HGCSample> > > >(__gnu_cxx::__normal_iterator<HGCDataFrame<DetId, HGCSample>*, std::vector<HGCDataFrame<DetId, HGCSample>, std::allocator<HGCDataFrame<DetId, HGCSample> > > >, __gnu_cxx::__normal_iterator<HGCDataFrame<DetId, HGCSample>*, std::vector<HGCDataFrame<DetId, HGCSample>, std::allocator<HGCDataFrame<DetId, HGCSample> > > >, long, __gnu_cxx::__ops::_Iter_comp_iter<edm::StrictWeakOrdering<HGCDataFrame<DetId, HGCSample> > >) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT628_X_2023-08-27-2300/lib/el8_amd64_gcc11/pluginEventFilterHGCalRawToDigiAuto.so
#7  0x0000149a61e62144 in void std::__introsort_loop<__gnu_cxx::__normal_iterator<HGCDataFrame<DetId, HGCSample>*, std::vector<HGCDataFrame<DetId, HGCSample>, std::allocator<HGCDataFrame<DetId, HGCSample> > > >, long, __gnu_cxx::__ops::_Iter_comp_iter<edm::StrictWeakOrdering<HGCDataFrame<DetId, HGCSample> > > >(__gnu_cxx::__normal_iterator<HGCDataFrame<DetId, HGCSample>*, std::vector<HGCDataFrame<DetId, HGCSample>, std::allocator<HGCDataFrame<DetId, HGCSample> > > >, __gnu_cxx::__normal_iterator<HGCDataFrame<DetId, HGCSample>*, std::vector<HGCDataFrame<DetId, HGCSample>, std::allocator<HGCDataFrame<DetId, HGCSample> > > >, long, __gnu_cxx::__ops::_Iter_comp_iter<edm::StrictWeakOrdering<HGCDataFrame<DetId, HGCSample> > >) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT628_X_2023-08-27-2300/lib/el8_amd64_gcc11/pluginEventFilterHGCalRawToDigiAuto.so
#8  0x0000149a61e62144 in void std::__introsort_loop<__gnu_cxx::__normal_iterator<HGCDataFrame<DetId, HGCSample>*, std::vector<HGCDataFrame<DetId, HGCSample>, std::allocator<HGCDataFrame<DetId, HGCSample> > > >, long, __gnu_cxx::__ops::_Iter_comp_iter<edm::StrictWeakOrdering<HGCDataFrame<DetId, HGCSample> > > >(__gnu_cxx::__normal_iterator<HGCDataFrame<DetId, HGCSample>*, std::vector<HGCDataFrame<DetId, HGCSample>, std::allocator<HGCDataFrame<DetId, HGCSample> > > >, __gnu_cxx::__normal_iterator<HGCDataFrame<DetId, HGCSample>*, std::vector<HGCDataFrame<DetId, HGCSample>, std::allocator<HGCDataFrame<DetId, HGCSample> > > >, long, __gnu_cxx::__ops::_Iter_comp_iter<edm::StrictWeakOrdering<HGCDataFrame<DetId, HGCSample> > >) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT628_X_2023-08-27-2300/lib/el8_amd64_gcc11/pluginEventFilterHGCalRawToDigiAuto.so
#9  0x0000149a61e62144 in void std::__introsort_loop<__gnu_cxx::__normal_iterator<HGCDataFrame<DetId, HGCSample>*, std::vector<HGCDataFrame<DetId, HGCSample>, std::allocator<HGCDataFrame<DetId, HGCSample> > > >, long, __gnu_cxx::__ops::_Iter_comp_iter<edm::StrictWeakOrdering<HGCDataFrame<DetId, HGCSample> > > >(__gnu_cxx::__normal_iterator<HGCDataFrame<DetId, HGCSample>*, std::vector<HGCDataFrame<DetId, HGCSample>, std::allocator<HGCDataFrame<DetId, HGCSample> > > >, __gnu_cxx::__normal_iterator<HGCDataFrame<DetId, HGCSample>*, std::vector<HGCDataFrame<DetId, HGCSample>, std::allocator<HGCDataFrame<DetId, HGCSample> > > >, long, __gnu_cxx::__ops::_Iter_comp_iter<edm::StrictWeakOrdering<HGCDataFrame<DetId, HGCSample> > >) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT628_X_2023-08-27-2300/lib/el8_amd64_gcc11/pluginEventFilterHGCalRawToDigiAuto.so
#10 0x0000149a61e62144 in void std::__introsort_loop<__gnu_cxx::__normal_iterator<HGCDataFrame<DetId, HGCSample>*, std::vector<HGCDataFrame<DetId, HGCSample>, std::allocator<HGCDataFrame<DetId, HGCSample> > > >, long, __gnu_cxx::__ops::_Iter_comp_iter<edm::StrictWeakOrdering<HGCDataFrame<DetId, HGCSample> > > >(__gnu_cxx::__normal_iterator<HGCDataFrame<DetId, HGCSample>*, std::vector<HGCDataFrame<DetId, HGCSample>, std::allocator<HGCDataFrame<DetId, HGCSample> > > >, __gnu_cxx::__normal_iterator<HGCDataFrame<DetId, HGCSample>*, std::vector<HGCDataFrame<DetId, HGCSample>, std::allocator<HGCDataFrame<DetId, HGCSample> > > >, long, __gnu_cxx::__ops::_Iter_comp_iter<edm::StrictWeakOrdering<HGCDataFrame<DetId, HGCSample> > >) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT628_X_2023-08-27-2300/lib/el8_amd64_gcc11/pluginEventFilterHGCalRawToDigiAuto.so
#11 0x0000149a61e6470c in edm::OrphanHandle<edm::SortedCollection<HGCDataFrame<DetId, HGCSample>, edm::StrictWeakOrdering<HGCDataFrame<DetId, HGCSample> > > > edm::Event::put<edm::SortedCollection<HGCDataFrame<DetId, HGCSample>, edm::StrictWeakOrdering<HGCDataFrame<DetId, HGCSample> > > >(std::unique_ptr<edm::SortedCollection<HGCDataFrame<DetId, HGCSample>, edm::StrictWeakOrdering<HGCDataFrame<DetId, HGCSample> > >, std::default_delete<edm::SortedCollection<HGCDataFrame<DetId, HGCSample>, edm::StrictWeakOrdering<HGCDataFrame<DetId, HGCSample> > > > >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT628_X_2023-08-27-2300/lib/el8_amd64_gcc11/pluginEventFilterHGCalRawToDigiAuto.so
#12 0x0000149a61e64f32 in HGCalRawToDigiFake::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT628_X_2023-08-27-2300/lib/el8_amd64_gcc11/pluginEventFilterHGCalRawToDigiAuto.so

Thread 1 (Thread 0x149ac8732c80 (LWP 309084) "cmsRun"):
#3  0x0000149ac769734b in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT628_X_2023-08-27-2300/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x0000149a62b0ed02 in TICLLayerTileProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT628_X_2023-08-27-2300/lib/el8_amd64_gcc11/pluginRecoHGCalTICLPlugins.so
#6  0x0000149acc2d14ed in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT628_X_2023-08-27-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so

Current Modules:

Module: TICLLayerTileProducer:ticlLayerTileProducer (crashed)
Module: HGCalConcentratorProducer:l1tHGCalConcentratorProducer
Module: HGCalRawToDigiFake:hgcalDigis
Module: HGCalConcentratorProducer:l1tHGCalConcentratorProducer

A fatal system signal has occurred: segmentation violation
timeout: the monitored command dumped core
cmsbuild commented 1 year ago

A new Issue was created by @dan131riley Dan Riley.

@Dr15Jones, @perrotta, @dpiparo, @rappoccio, @makortel, @smuzaffar, @antoniovilela can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

makortel commented 1 year ago

assign reconstruction, upgrade

FYI @cms-sw/hgcal-dpg-l2

cmsbuild commented 1 year ago

New categories assigned: upgrade,reconstruction

@AdrianoDee,@clacaputo,@srimanob,@mandrenguyen you have been requested to review this Pull request/Issue and eventually sign? Thanks

makortel commented 1 year ago

The segfault was in workflow 25234.911. Then this issue could be a duplicate of https://github.com/cms-sw/cmssw/issues/42470.

dan131riley commented 1 year ago

Probably so, was misled by the GEANT4 tag and not enough coffee. Will close in favor of the previous ticket.