iarspider
commented
1 year ago assign reconstruction
Open iarspider opened 1 year ago
iarspider
commented
1 year ago assign reconstruction
cmsbuild
commented
1 year ago New categories assigned: reconstruction
@mandrenguyen,@clacaputo you have been requested to review this Pull request/Issue and eventually sign? Thanks
cmsbuild
commented
1 year ago A new Issue was created by @iarspider .
@Dr15Jones, @perrotta, @dpiparo, @rappoccio, @makortel, @smuzaffar can you please review it and eventually sign/assign? Thanks.
cms-bot commands are listed here
makortel
commented
1 year ago Crash occurring in RefCore copy constructor
https://github.com/cms-sw/cmssw/blob/82334580c1a81e123e8e682b097c6ff52586e50d/DataFormats/Common/src/RefCore.cc#L36-L38
could hint to a core problem.
makortel
commented
1 year ago assign core
cmsbuild
commented
1 year ago New categories assigned: core
@Dr15Jones,@smuzaffar,@makortel you have been requested to review this Pull request/Issue and eventually sign? Thanks
makortel
commented
1 year ago Another crash in RefCore constructor in CMSSW_13_1_X_2023-04-18-2300 on ARM workflow 136.846 step 3
Thread 5 (Thread 0x400076d09250 (LWP 2403224) "cmsRun"):
#2 0x000040002b9c2f88 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02781/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-04-17-2300/lib/el8_aarch64_gcc11/pluginFWCoreServicesPlugins.so
#3 <signal handler called>
#4 0x0000400026392c40 in __memset_generic () from /lib64/libc.so.6
#5 0x000040006e8b6978 in TrackCandidate& std::vector<TrackCandidate, std::allocator<TrackCandidate> >::emplace_back<edm::OwnVector<TrackingRecHit, edm::ClonePolicy<TrackingRecHit> >&, TrajectorySeed const&, PTrajectoryStateOnDet&, edm::RefToBase<TrajectorySeed>, int, unsigned char>(edm::OwnVector<TrackingRecHit, edm::ClonePolicy<TrackingRecHit> >&, TrajectorySeed const&, PTrajectoryStateOnDet&, edm::RefToBase<TrajectorySeed>&&, int&&, unsigned char&&) [clone .constprop.0] [clone .isra.0] () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02781/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-04-17-2300/lib/el8_aarch64_gcc11/pluginRecoTrackerMkFitPlugins.so
#6 0x000040006e89c0c0 in MkFitOutputConverter::convertCandidates(MkFitOutputWrapper const&, mkfit::EventOfHits const&, MkFitClusterIndexToHit const&, MkFitClusterIndexToHit const&, edm::View<TrajectorySeed> const&, MagneticField const&, Propagator const&, Propagator const&, TkClonerImpl const&, std::vector<DetLayer const*, std::allocator<DetLayer const*> > const&, std::vector<mkfit::Track, std::allocator<mkfit::Track> > const&, reco::BeamSpot const*, std::vector<reco::Vertex, std::allocator<reco::Vertex> > const*, tensorflow::Session const*) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02781/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-04-17-2300/lib/el8_aarch64_gcc11/pluginRecoTrackerMkFitPlugins.so
#7 0x000040006e893ecc in MkFitOutputConverter::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02781/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-04-17-2300/lib/el8_aarch64_gcc11/pluginRecoTrackerMkFitPlugins.so
#8 0x00004000244d0160 in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02781/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-04-17-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#9 0x00004000244c9bc0 in edm::WorkerT<edm::global::EDProducerBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02781/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-04-17-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
Thread 4 (Thread 0x4000762f9250 (LWP 2403223) "cmsRun"):
#3 0x000040002b9c7888 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02781/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-04-17-2300/lib/el8_aarch64_gcc11/pluginFWCoreServicesPlugins.so
#4 <signal handler called>
#5 0x000040002471bc74 in edm::RefCore::RefCore(edm::RefCore const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02781/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-04-17-2300/lib/el8_aarch64_gcc11/libDataFormatsCommon.so
#6 0x000040004ee276e8 in TkClonerImpl::makeShared(SiPixelRecHit const&, TrajectoryStateOnSurface const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02781/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-04-17-2300/lib/el8_aarch64_gcc11/libRecoTrackerTransientTrackingRecHit.so
#7 0x000040004ed8268c in SiPixelRecHit::cloneSH_(TkCloner const&, TrajectoryStateOnSurface const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02781/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-04-17-2300/lib/el8_aarch64_gcc11/pluginRecoTrackerMeasurementDetPlugins.so
#8 0x000040006d13df6c in KFTrajectoryFitter::fitOne(TrajectorySeed const&, std::vector<std::shared_ptr<TrackingRecHit const>, std::allocator<std::shared_ptr<TrackingRecHit const> > > const&, TrajectoryStateOnSurface const&, TrajectoryFitter::fitType) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02781/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-04-17-2300/lib/el8_aarch64_gcc11/libTrackingToolsTrackFitters.so
#9 0x000040006d0dd124 in (anonymous namespace)::KFFittingSmoother::fitOne(TrajectorySeed const&, std::vector<std::shared_ptr<TrackingRecHit const>, std::allocator<std::shared_ptr<TrackingRecHit const> > > const&, TrajectoryStateOnSurface const&, TrajectoryFitter::fitType) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02781/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-04-17-2300/lib/el8_aarch64_gcc11/pluginTrackingToolsTrackFittersPlugins.so
#10 0x000040006d0d6ce8 in (anonymous namespace)::FlexibleKFFittingSmoother::fitOne(TrajectorySeed const&, std::vector<std::shared_ptr<TrackingRecHit const>, std::allocator<std::shared_ptr<TrackingRecHit const> > > const&, TrajectoryStateOnSurface const&, TrajectoryFitter::fitType) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02781/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-04-17-2300/lib/el8_aarch64_gcc11/pluginTrackingToolsTrackFittersPlugins.so
#11 0x000040009c847c9c in TrackProducerAlgorithm<reco::Track>::buildTrack(TrajectoryFitter const*, Propagator const*, std::vector<AlgoProductTraits<reco::Track>::AlgoProduct, std::allocator<AlgoProductTraits<reco::Track>::AlgoProduct> >&, std::vector<std::shared_ptr<TrackingRecHit const>, std::allocator<std::shared_ptr<TrackingRecHit const> > >&, TrajectoryStateOnSurface&, TrajectorySeed const&, float, reco::BeamSpot const&, edm::RefToBase<TrajectorySeed>, int, signed char) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02781/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-04-17-2300/lib/el8_aarch64_gcc11/libRecoTrackerTrackProducer.so
#12 0x00004000a2d5f59c in TrackProducerAlgorithm<reco::Track>::runWithCandidate(TrackingGeometry const*, MagneticField const*, std::vector<TrackCandidate, std::allocator<TrackCandidate> > const&, TrajectoryFitter const*, Propagator const*, TransientTrackingRecHitBuilder const*, reco::BeamSpot const&, std::vector<AlgoProductTraits<reco::Track>::AlgoProduct, std::allocator<AlgoProductTraits<reco::Track>::AlgoProduct> >&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02781/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-04-17-2300/lib/el8_aarch64_gcc11/pluginRecoTrackerTrackProducerPlugins.so
#13 0x00004000a2d5dd0c in TrackProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02781/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-04-17-2300/lib/el8_aarch64_gcc11/pluginRecoTrackerTrackProducerPlugins.so
#14 0x00004000244e444c in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02781/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-04-17-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#15 0x00004000244ca7f0 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02781/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-04-17-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
Thread 3 (Thread 0x4000758e9250 (LWP 2403222) "cmsRun"):
#2 0x000040002b9c2f88 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02781/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-04-17-2300/lib/el8_aarch64_gcc11/pluginFWCoreServicesPlugins.so
#3 <signal handler called>
#4 0x0000400026278310 in __kernel_tanf () from /lib64/libm.so.6
#5 0x000040002627a890 in tanf32 () from /lib64/libm.so.6
#6 0x000040006e996c70 in mkfit::MkFinder::selectHitIndices(mkfit::LayerOfHits const&, int) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02781/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-04-17-2300/lib/el8_aarch64_gcc11/libRecoTrackerMkFitCore.so
#7 0x000040006e99366c in mkfit::MkBuilder::find_tracks_in_layers(mkfit::CandCloner&, mkfit::MkFinder*, mkfit::SteeringParams::IterationType_e, int, int, int) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02781/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-04-17-2300/lib/el8_aarch64_gcc11/libRecoTrackerMkFitCore.so
#8 0x000040006e9939f4 in mkfit::MkBuilder::findTracksCloneEngine(mkfit::SteeringParams::IterationType_e)::{lambda(int)#1}::operator()(int) const::{lambda(tbb::detail::d1::blocked_range<int> const&)#1}::operator()(tbb::detail::d1::blocked_range<int> const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02781/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-04-17-2300/lib/el8_aarch64_gcc11/libRecoTrackerMkFitCore.so
#9 0x000040006e993e7c in tbb::detail::d1::start_for<tbb::detail::d1::blocked_range<int>, mkfit::MkBuilder::findTracksCloneEngine(mkfit::SteeringParams::IterationType_e)::{lambda(int)#1}::operator()(int) const::{lambda(tbb::detail::d1::blocked_range<int> const&)#1}, tbb::detail::d1::auto_partitioner const>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02781/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-04-17-2300/lib/el8_aarch64_gcc11/libRecoTrackerMkFitCore.so
#10 0x0000400025e28508 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x4000d11fb100, this=0x400026c53e80) at /data/cmsbuild/jenkins_c/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_aarch64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-8f30f4fc8c5b3860b0ce8f2b70736d15/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322
#11 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x400026c53e80) at /data/cmsbuild/jenkins_c/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_aarch64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-8f30f4fc8c5b3860b0ce8f2b70736d15/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458
#12 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbuild/jenkins_c/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_aarch64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-8f30f4fc8c5b3860b0ce8f2b70736d15/tbb-v2021.8.0/src/tbb/task_dispatcher.cpp:168
#13 0x000040006e98f8a0 in void tbb::detail::d1::dynamic_grainsize_mode<tbb::detail::d1::adaptive_mode<tbb::detail::d1::auto_partition_type> >::work_balance<tbb::detail::d1::start_for<tbb::detail::d1::blocked_range<unsigned long>, tbb::detail::d2::parallel_for_body_wrapper<__gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > >, mkfit::MkBuilder::findTracksCloneEngine(mkfit::SteeringParams::IterationType_e)::{lambda(int)#1}, int>, tbb::detail::d1::auto_partitioner const>, tbb::detail::d1::blocked_range<unsigned long> >(tbb::detail::d1::start_for<tbb::detail::d1::blocked_range<unsigned long>, tbb::detail::d2::parallel_for_body_wrapper<__gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > >, mkfit::MkBuilder::findTracksCloneEngine(mkfit::SteeringParams::IterationType_e)::{lambda(int)#1}, int>, tbb::detail::d1::auto_partitioner const>&, tbb::detail::d1::blocked_range<unsigned long>&, tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02781/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-04-17-2300/lib/el8_aarch64_gcc11/libRecoTrackerMkFitCore.so
#14 0x000040006e98fcfc in tbb::detail::d1::start_for<tbb::detail::d1::blocked_range<unsigned long>, tbb::detail::d2::parallel_for_body_wrapper<__gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > >, mkfit::MkBuilder::findTracksCloneEngine(mkfit::SteeringParams::IterationType_e)::{lambda(int)#1}, int>, tbb::detail::d1::auto_partitioner const>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02781/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-04-17-2300/lib/el8_aarch64_gcc11/libRecoTrackerMkFitCore.so
#15 0x0000400025e28508 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x4000d1200100, this=0x400026c53e80) at /data/cmsbuild/jenkins_c/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_aarch64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-8f30f4fc8c5b3860b0ce8f2b70736d15/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322
#16 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x400026c53e80) at /data/cmsbuild/jenkins_c/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_aarch64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-8f30f4fc8c5b3860b0ce8f2b70736d15/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458
#17 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbuild/jenkins_c/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_aarch64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-8f30f4fc8c5b3860b0ce8f2b70736d15/tbb-v2021.8.0/src/tbb/task_dispatcher.cpp:168
#18 0x000040006e98ce68 in tbb::detail::d2::for_each_root_task<__gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > >, mkfit::MkBuilder::findTracksCloneEngine(mkfit::SteeringParams::IterationType_e)::{lambda(int)#1}, int, std::random_access_iterator_tag>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02781/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-04-17-2300/lib/el8_aarch64_gcc11/libRecoTrackerMkFitCore.so
#19 0x0000400025e28508 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x4000758e7d80, this=0x400026c53e80) at /data/cmsbuild/jenkins_c/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_aarch64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-8f30f4fc8c5b3860b0ce8f2b70736d15/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322
#20 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x400026c53e80) at /data/cmsbuild/jenkins_c/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_aarch64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-8f30f4fc8c5b3860b0ce8f2b70736d15/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458
#21 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbuild/jenkins_c/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_aarch64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-8f30f4fc8c5b3860b0ce8f2b70736d15/tbb-v2021.8.0/src/tbb/task_dispatcher.cpp:168
#22 0x000040006e98e064 in mkfit::MkBuilder::findTracksCloneEngine(mkfit::SteeringParams::IterationType_e) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02781/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-04-17-2300/lib/el8_aarch64_gcc11/libRecoTrackerMkFitCore.so
#23 0x000040006e92b940 in mkfit::run_OneIteration(mkfit::TrackerInfo const&, mkfit::IterationConfig const&, mkfit::EventOfHits const&, std::vector<std::vector<bool, std::allocator<bool> > const*, std::allocator<std::vector<bool, std::allocator<bool> > const*> > const&, mkfit::MkBuilder&, std::vector<mkfit::Track, std::allocator<mkfit::Track> >&, std::vector<mkfit::Track, std::allocator<mkfit::Track> >&, bool, bool, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02781/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-04-17-2300/lib/el8_aarch64_gcc11/libRecoTrackerMkFitCMS.so
#24 0x000040006e8a7ac0 in tbb::detail::d1::task_arena_function<MkFitProducer::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const::{lambda()#1}, void>::operator()() const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02781/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-04-17-2300/lib/el8_aarch64_gcc11/pluginRecoTrackerMkFitPlugins.so
#25 0x0000400025e21554 in operator() (__closure=<optimized out>) at /data/cmsbuild/jenkins_c/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_aarch64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-8f30f4fc8c5b3860b0ce8f2b70736d15/tbb-v2021.8.0/src/tbb/arena.cpp:767
#26 tbb::detail::d0::try_call_proxy<tbb::detail::r1::isolate_within_arena(tbb::detail::d1::delegate_base&, intptr_t)::<lambda()> >::on_completion<tbb::detail::r1::isolate_within_arena(tbb::detail::d1::delegate_base&, intptr_t)::<lambda()> > (on_completion_body=..., this=<optimized out>) at /data/cmsbuild/jenkins_c/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_aarch64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-8f30f4fc8c5b3860b0ce8f2b70736d15/tbb-v2021.8.0/src/tbb/../../include/oneapi/tbb/detail/_template_helpers.h:230
#27 tbb::detail::r1::isolate_within_arena (d=..., isolation=<optimized out>) at /data/cmsbuild/jenkins_c/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_aarch64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-8f30f4fc8c5b3860b0ce8f2b70736d15/tbb-v2021.8.0/src/tbb/arena.cpp:768
#28 0x000040006e8acbc4 in MkFitProducer::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02781/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-04-17-2300/lib/el8_aarch64_gcc11/pluginRecoTrackerMkFitPlugins.so
#29 0x00004000244d0160 in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02781/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-04-17-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#30 0x00004000244c9bc0 in edm::WorkerT<edm::global::EDProducerBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02781/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-04-17-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
Thread 1 (Thread 0x400025f7d090 (LWP 2402029) "cmsRun"):
#2 0x000040002b9c2f88 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02781/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-04-17-2300/lib/el8_aarch64_gcc11/pluginFWCoreServicesPlugins.so
#3 <signal handler called>
#4 0x0000400024248d74 in do_lookup_x (undef_name=undef_name@entry=0x400050d429db "_ZNK10tensorflow6Status8ToStringB5cxx11Ev", new_hash=new_hash@entry=2442556830, old_hash=old_hash@entry=0xffffc0e2e4b8, ref=0x400050d40988, result=result@entry=0xffffc0e2e4c8, scope=<optimized out>, i=71, version=version@entry=0x40002864fc98, flags=flags@entry=5, skip=<optimized out>, skip@entry=0x0, type_class=type_class@entry=1, undef_map=undef_map@entry=0x400036dfa500) at dl-lookup.c:409
#5 0x0000400024249748 in _dl_lookup_symbol_x (undef_name=0x400050d429db "_ZNK10tensorflow6Status8ToStringB5cxx11Ev", undef_map=undef_map@entry=0x400036dfa500, ref=ref@entry=0xffffc0e2e550, symbol_scope=0x4000aed13600, version=0x40002864fc98, type_class=type_class@entry=1, flags=5, skip_map=skip_map@entry=0x0) at dl-lookup.c:855
#6 0x000040002424f130 in _dl_fixup (l=0x400036dfa500, reloc_arg=1272) at dl-runtime.c:94
#7 0x00004000242410c4 in _dl_runtime_resolve () at ../sysdeps/aarch64/dl-trampoline.S:99
#8 0x0000400050d4ceb8 in tensorflow::run(tensorflow::Session*, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor> > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*, tensorflow::thread::ThreadPoolOptions const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02781/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-04-17-2300/lib/el8_aarch64_gcc11/libPhysicsToolsTensorFlow.so
#9 0x0000400050d4ceb8 in tensorflow::run(tensorflow::Session*, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor> > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*, tensorflow::thread::ThreadPoolOptions const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02781/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-04-17-2300/lib/el8_aarch64_gcc11/libPhysicsToolsTensorFlow.so
#10 0x0000000000000000 in ?? ()
Current Modules:
Module: TrackProducer:lowPtQuadStepTracks (crashed)
Module: MkFitProducer:highPtTripletStepTrackCandidatesMkFit
Module: MkFitOutputConverter:highPtTripletStepTrackCandidates
Module: MkFitOutputConverter:highPtTripletStepTrackCandidatesHappened again in CMSSW_13_2_X_2023-05-23-2300 for el8_aarch64_gcc11: link
makortel
commented
1 year ago Happened again in CMSSW_13_2_X_2023-05-23-2300 for el8_aarch64_gcc11: link
Extracting the stack trace here (for more permanent record), workflow is 136.785 step 3
Thread 5 (Thread 0x400064a09260 (LWP 1280868) "cmsRun"):
#2 0x00004000184a2f58 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02786/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-22-2300/lib/el8_aarch64_gcc11/pluginFWCoreServicesPlugins.so
#3 <signal handler called>
#4 0x000040005707827c in tensorflow::TensorShapeBase<tensorflow::TensorShape>::UnsafeAddDim(long, long) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02786/el8_aarch64_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-23-2300/external/el8_aarch64_gcc11/lib/libtensorflow_framework.so.2
#5 0x0000400040d3004c in tensorflow::Status tensorflow::ReshapeOp::ValidateSizes<int>(tensorflow::Tensor const&, long*, int*, tensorflow::TensorShape*, bool*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02786/el8_aarch64_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-23-2300/external/el8_aarch64_gcc11/lib/libtensorflow_cc.so.2
#6 0x0000400040d30934 in tensorflow::ReshapeOp::Compute(tensorflow::OpKernelContext*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02786/el8_aarch64_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-23-2300/external/el8_aarch64_gcc11/lib/libtensorflow_cc.so.2
#7 0x0000400057131290 in tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::Process(tensorflow::SimplePropagatorState::TaggedNode, long) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02786/el8_aarch64_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-23-2300/external/el8_aarch64_gcc11/lib/libtensorflow_framework.so.2
#8 0x000040003f21719c in tensorflow::thread::ThreadPool::Schedule(std::function<void ()>) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02786/el8_aarch64_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-23-2300/external/el8_aarch64_gcc11/lib/libtensorflow_cc.so.2
#9 0x0000400045dad1a4 in std::_Function_handler<void (std::function<void ()>), tensorflow::DirectSession::RunInternal(long, tensorflow::RunOptions const&, tensorflow::CallFrameInterface*, tensorflow::DirectSession::ExecutorsAndKeys*, tensorflow::RunMetadata*, tensorflow::thread::ThreadPoolOptions const&)::{lambda(std::function<void ()>)#6}>::_M_invoke(std::_Any_data const&, std::function<void ()>&&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02786/el8_aarch64_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-23-2300/external/el8_aarch64_gcc11/lib/libtensorflow_cc.so.2
#10 0x00004000571271a0 in tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::ScheduleReady(absl::lts_20210324::InlinedVector<tensorflow::SimplePropagatorState::TaggedNode, 8ul, std::allocator<tensorflow::SimplePropagatorState::TaggedNode> >*, tensorflow::SimplePropagatorState::TaggedNodeReadyQueue*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02786/el8_aarch64_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-23-2300/external/el8_aarch64_gcc11/lib/libtensorflow_framework.so.2
#11 0x000040005712928c in tensorflow::(anonymous namespace)::ExecutorImpl::RunAsync(tensorflow::Executor::Args const&, std::function<void (tensorflow::Status const&)>) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02786/el8_aarch64_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-23-2300/external/el8_aarch64_gcc11/lib/libtensorflow_framework.so.2
#12 0x0000400045dbc62c in tensorflow::DirectSession::RunInternal(long, tensorflow::RunOptions const&, tensorflow::CallFrameInterface*, tensorflow::DirectSession::ExecutorsAndKeys*, tensorflow::RunMetadata*, tensorflow::thread::ThreadPoolOptions const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02786/el8_aarch64_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-23-2300/external/el8_aarch64_gcc11/lib/libtensorflow_cc.so.2
#13 0x0000400045dbe678 in tensorflow::DirectSession::Run(tensorflow::RunOptions const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor> > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*, tensorflow::RunMetadata*, tensorflow::thread::ThreadPoolOptions const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02786/el8_aarch64_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-23-2300/external/el8_aarch64_gcc11/lib/libtensorflow_cc.so.2
#14 0x000040003c3dcda4 in tensorflow::run(tensorflow::Session*, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor> > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*, tensorflow::thread::ThreadPoolOptions const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02786/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-22-2300/lib/el8_aarch64_gcc11/libPhysicsToolsTensorFlow.so
#15 0x000040003c3dcf9c in tensorflow::run(tensorflow::Session*, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor> > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*, tensorflow::thread::ThreadPoolInterface*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02786/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-22-2300/lib/el8_aarch64_gcc11/libPhysicsToolsTensorFlow.so
#16 0x000040005b913844 in (anonymous namespace)::TfDnn::operator()(std::vector<reco::Track, std::allocator<reco::Track> > const&, reco::BeamSpot const&, std::vector<reco::Vertex, std::allocator<reco::Vertex> > const&) const [clone .lto_priv.0] () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02786/el8_aarch64_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-23-2300/lib/el8_aarch64_gcc11/pluginRecoTrackerFinalTrackSelectorsPlugins.so
#17 0x000040005b91581c in TrackMVAClassifier<(anonymous namespace)::TfDnn, void>::computeMVA(std::vector<reco::Track, std::allocator<reco::Track> > const&, reco::BeamSpot const&, std::vector<reco::Vertex, std::allocator<reco::Vertex> > const&, std::vector<std::pair<float, bool>, std::allocator<std::pair<float, bool> > >&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02786/el8_aarch64_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-23-2300/lib/el8_aarch64_gcc11/pluginRecoTrackerFinalTrackSelectorsPlugins.so
#18 0x000040005b9b4040 in TrackMVAClassifierBase::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02786/el8_aarch64_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-23-2300/lib/el8_aarch64_gcc11/libRecoTrackerFinalTrackSelectors.so
#19 0x000040000fc84c9c in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02786/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-22-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#20 0x000040000fc6af10 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02786/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-22-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
Thread 4 (Thread 0x400061be9260 (LWP 1280867) "cmsRun"):
#2 0x00004000184a2f58 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02786/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-22-2300/lib/el8_aarch64_gcc11/pluginFWCoreServicesPlugins.so
#3 <signal handler called>
#4 0x0000400011ad13a0 in __lll_lock_wait () from /lib64/libpthread.so.0
#5 0x0000400011aca030 in pthread_mutex_lock () from /lib64/libpthread.so.0
#6 0x0000400011a9fee8 in __gthread_mutex_lock (__mutex=0x400011ab33d0 <object_mutex>) at ./gthr-default.h:749
#7 _Unwind_Find_registered_FDE (bases=0x400061be8088, pc=0x40000fc14913 <edm::FunctorWaitingTask<edm::UnscheduledProductResolver::prefetchAsync_(edm::WaitingTaskHolder, edm::Principal const&, bool, edm::ServiceToken const&, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const::{lambda(std::__exception_ptr::exception_ptr const*)#1}>::execute()+239>) at ../../../libgcc/unwind-dw2-fde.c:1049
#8 _Unwind_Find_FDE (pc=0x40000fc14913 <edm::FunctorWaitingTask<edm::UnscheduledProductResolver::prefetchAsync_(edm::WaitingTaskHolder, edm::Principal const&, bool, edm::ServiceToken const&, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const::{lambda(std::__exception_ptr::exception_ptr const*)#1}>::execute()+239>, bases=bases@entry=0x400061be8088) at ../../../libgcc/unwind-dw2-fde-dip.c:459
#9 0x0000400011a9c750 in uw_frame_state_for (context=context@entry=0x400061be7d60, fs=fs@entry=0x400061be7260) at ../../../libgcc/unwind-dw2.c:1263
#10 0x0000400011a9da0c in _Unwind_RaiseException_Phase2 (exc=exc@entry=0x4001474f24b0, context=context@entry=0x400061be7d60, frames_p=frames_p@entry=0x400061be8120) at ../../../libgcc/unwind.inc:50
#11 0x0000400011a9de60 in _Unwind_RaiseException (exc=0x4001474f24b0) at ../../../libgcc/unwind.inc:136
#12 0x0000400011851bcc in std::rethrow_exception (ep=...) at ../../../../libstdc++-v3/libsupc++/eh_ptr.cc:212
#13 0x000040000fc14914 in edm::FunctorWaitingTask<edm::UnscheduledProductResolver::prefetchAsync_(edm::WaitingTaskHolder, edm::Principal const&, bool, edm::ServiceToken const&, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const::{lambda(std::__exception_ptr::exception_ptr const*)#1}>::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02786/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-22-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#14 0x0000400010105160 in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02786/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-22-2300/lib/el8_aarch64_gcc11/libFWCoreConcurrency.so
#15 0x00004000115b2a64 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x4000cbf47900, waiter=<synthetic pointer>..., this=0x4000123e3e00) at /data/cmsbuild/jenkins_c/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_aarch64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-8f30f4fc8c5b3860b0ce8f2b70736d15/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322
#16 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=<synthetic pointer>..., t=0x0, this=0x4000123e3e00) at /data/cmsbuild/jenkins_c/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_aarch64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-8f30f4fc8c5b3860b0ce8f2b70736d15/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458
#17 tbb::detail::r1::arena::process (tls=..., this=0x4000123e3780) at /data/cmsbuild/jenkins_c/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_aarch64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-8f30f4fc8c5b3860b0ce8f2b70736d15/tbb-v2021.8.0/src/tbb/arena.cpp:137
#18 tbb::detail::r1::market::process (this=0x40001240b080, j=...) at /data/cmsbuild/jenkins_c/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_aarch64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-8f30f4fc8c5b3860b0ce8f2b70736d15/tbb-v2021.8.0/src/tbb/market.cpp:599
#19 0x00004000115bac5c in tbb::detail::r1::rml::private_worker::run (this=0x400012b4c100) at /data/cmsbuild/jenkins_c/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_aarch64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-8f30f4fc8c5b3860b0ce8f2b70736d15/tbb-v2021.8.0/src/tbb/private_server.cpp:271
#20 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x400012b4c100) at /data/cmsbuild/jenkins_c/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_aarch64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-8f30f4fc8c5b3860b0ce8f2b70736d15/tbb-v2021.8.0/src/tbb/private_server.cpp:221
#21 0x0000400011ac78b8 in start_thread () from /lib64/libpthread.so.0
#22 0x0000400011b23afc in thread_start () from /lib64/libc.so.6
Thread 3 (Thread 0x4000611d9260 (LWP 1280866) "cmsRun"):
#2 0x00004000184a2f58 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02786/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-22-2300/lib/el8_aarch64_gcc11/pluginFWCoreServicesPlugins.so
#3 <signal handler called>
#4 0x0000400011a9ff04 in _Unwind_Find_registered_FDE (bases=0x4000611d7e98, pc=0x400011851bcb <std::rethrow_exception(std::__exception_ptr::exception_ptr)+123>) at ../../../libgcc/unwind-dw2-fde.c:1055
#5 _Unwind_Find_FDE (pc=0x400011851bcb <std::rethrow_exception(std::__exception_ptr::exception_ptr)+123>, bases=bases@entry=0x4000611d7e98) at ../../../libgcc/unwind-dw2-fde-dip.c:459
#6 0x0000400011a9c750 in uw_frame_state_for (context=context@entry=0x4000611d7b70, fs=fs@entry=0x4000611d7f30) at ../../../libgcc/unwind-dw2.c:1263
#7 0x0000400011a9ddbc in _Unwind_RaiseException (exc=0x40014f690700) at ../../../libgcc/unwind.inc:104
#8 0x0000400011851bcc in std::rethrow_exception (ep=...) at ../../../../libstdc++-v3/libsupc++/eh_ptr.cc:212
#9 0x000040000fc5fcac in edm::Worker::shouldRethrowException(std::__exception_ptr::exception_ptr, edm::ParentContext const&, bool) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02786/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-22-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#10 0x000040000fbf66d8 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02786/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-22-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#11 0x000040000fbf7020 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02786/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-22-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#12 0x0000400010105160 in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02786/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-22-2300/lib/el8_aarch64_gcc11/libFWCoreConcurrency.so
#13 0x00004000115b2a64 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x4000cc285200, waiter=<synthetic pointer>..., this=0x4000123e3e80) at /data/cmsbuild/jenkins_c/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_aarch64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-8f30f4fc8c5b3860b0ce8f2b70736d15/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322
#14 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=<synthetic pointer>..., t=0x0, this=0x4000123e3e80) at /data/cmsbuild/jenkins_c/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_aarch64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-8f30f4fc8c5b3860b0ce8f2b70736d15/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458
#15 tbb::detail::r1::arena::process (tls=..., this=0x4000123e3780) at /data/cmsbuild/jenkins_c/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_aarch64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-8f30f4fc8c5b3860b0ce8f2b70736d15/tbb-v2021.8.0/src/tbb/arena.cpp:137
#16 tbb::detail::r1::market::process (this=0x40001240b080, j=...) at /data/cmsbuild/jenkins_c/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_aarch64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-8f30f4fc8c5b3860b0ce8f2b70736d15/tbb-v2021.8.0/src/tbb/market.cpp:599
#17 0x00004000115bac5c in tbb::detail::r1::rml::private_worker::run (this=0x400012b4c080) at /data/cmsbuild/jenkins_c/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_aarch64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-8f30f4fc8c5b3860b0ce8f2b70736d15/tbb-v2021.8.0/src/tbb/private_server.cpp:271
#18 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x400012b4c080) at /data/cmsbuild/jenkins_c/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_aarch64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-8f30f4fc8c5b3860b0ce8f2b70736d15/tbb-v2021.8.0/src/tbb/private_server.cpp:221
#19 0x0000400011ac78b8 in start_thread () from /lib64/libpthread.so.0
#20 0x0000400011b23afc in thread_start () from /lib64/libc.so.6
Thread 1 (Thread 0x40001170cf70 (LWP 1267755) "cmsRun"):
#3 0x00004000184a7858 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02786/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-22-2300/lib/el8_aarch64_gcc11/pluginFWCoreServicesPlugins.so
#4 <signal handler called>
#5 0x000040000feabc74 in edm::RefCore::RefCore(edm::RefCore const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02786/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-22-2300/lib/el8_aarch64_gcc11/libDataFormatsCommon.so
#6 0x000040003a4a76e8 in TkClonerImpl::makeShared(SiPixelRecHit const&, TrajectoryStateOnSurface const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02786/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-22-2300/lib/el8_aarch64_gcc11/libRecoTrackerTransientTrackingRecHit.so
#7 0x000040005a931488 in SiPixelRecHit::cloneSH_(TkCloner const&, TrajectoryStateOnSurface const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02786/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-22-2300/lib/el8_aarch64_gcc11/pluginRecoLocalTrackerSiPixelRecHitsPlugins.so
#8 0x000040005891df6c in KFTrajectoryFitter::fitOne(TrajectorySeed const&, std::vector<std::shared_ptr<TrackingRecHit const>, std::allocator<std::shared_ptr<TrackingRecHit const> > > const&, TrajectoryStateOnSurface const&, TrajectoryFitter::fitType) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02786/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-22-2300/lib/el8_aarch64_gcc11/libTrackingToolsTrackFitters.so
#9 0x00004000588bd204 in (anonymous namespace)::KFFittingSmoother::fitOne(TrajectorySeed const&, std::vector<std::shared_ptr<TrackingRecHit const>, std::allocator<std::shared_ptr<TrackingRecHit const> > > const&, TrajectoryStateOnSurface const&, TrajectoryFitter::fitType) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02786/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-22-2300/lib/el8_aarch64_gcc11/pluginTrackingToolsTrackFittersPlugins.so
#10 0x00004000588b6d78 in (anonymous namespace)::FlexibleKFFittingSmoother::fitOne(TrajectorySeed const&, std::vector<std::shared_ptr<TrackingRecHit const>, std::allocator<std::shared_ptr<TrackingRecHit const> > > const&, TrajectoryStateOnSurface const&, TrajectoryFitter::fitType) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02786/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-22-2300/lib/el8_aarch64_gcc11/pluginTrackingToolsTrackFittersPlugins.so
#11 0x0000400080227c9c in TrackProducerAlgorithm<reco::Track>::buildTrack(TrajectoryFitter const*, Propagator const*, std::vector<AlgoProductTraits<reco::Track>::AlgoProduct, std::allocator<AlgoProductTraits<reco::Track>::AlgoProduct> >&, std::vector<std::shared_ptr<TrackingRecHit const>, std::allocator<std::shared_ptr<TrackingRecHit const> > >&, TrajectoryStateOnSurface&, TrajectorySeed const&, float, reco::BeamSpot const&, edm::RefToBase<TrajectorySeed>, int, signed char) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02786/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-22-2300/lib/el8_aarch64_gcc11/libRecoTrackerTrackProducer.so
#12 0x0000400085d0f64c in TrackProducerAlgorithm<reco::Track>::runWithCandidate(TrackingGeometry const*, MagneticField const*, std::vector<TrackCandidate, std::allocator<TrackCandidate> > const&, TrajectoryFitter const*, Propagator const*, TransientTrackingRecHitBuilder const*, reco::BeamSpot const&, std::vector<AlgoProductTraits<reco::Track>::AlgoProduct, std::allocator<AlgoProductTraits<reco::Track>::AlgoProduct> >&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02786/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-22-2300/lib/el8_aarch64_gcc11/pluginRecoTrackerTrackProducerPlugins.so
#13 0x0000400085d0ddbc in TrackProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02786/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-22-2300/lib/el8_aarch64_gcc11/pluginRecoTrackerTrackProducerPlugins.so
#14 0x000040000fc84c9c in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02786/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-22-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#15 0x000040000fc6af10 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02786/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-22-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
Current Modules:
Module: TrackProducer:highPtTripletStepTracks (crashed)
Module: none
Module: TrackTfClassifier:highPtTripletStep
Module: noneInteresting that two other TBB worker threads are processing exceptions, but maybe that is just coincidence?
makortel
commented
1 year ago All the (three) cases reported so far, the RefCore has been copied in TkClonerImpl::makeShared(SiPixelRecHit const&, ...) via KFTrajectoryFitter::fitOne() from TrackProducer::produce()
https://github.com/cms-sw/cmssw/blob/012956d4c9c1500ac605b5b070fcd6b808341714/RecoTracker/TransientTrackingRecHit/src/TkClonerImpl.cc#L64-L71
Here the hit.cluster() returns edm::Ref<edmNew::DetSetVector<SiPixelCluster>, SiPixelCluster>, where edm::Ref internally uses edm::RefCore. The easiest explanation for a segfault in the RefCore copy constructor would be the copied-from object being null (or gotten corrupted).
Log:
Full log: link