search

cms-sw / cmssw

CMS Offline Software
http://cms-sw.github.io/
Apache License 2.0
1.07k stars 4.28k forks source link

[ARM][ROOT6_X] Workflow 12846.0 step 3 crashed due to SIGSEGV in EvaluateMuonMVAID:muonMVAID #45678

Open iarspider opened 1 month ago

iarspider commented 1 month ago

Full log: link

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Fri Aug  9 05:37:32 CEST 2024
Thread 8 (Thread 0x4000e0ad1220 (LWP 2393056) "cmsRun"):
#0  0x00004000354e4c60 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
#1  0x00004000719c085c in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw-patch/CMSSW_14_1_ROOT6_X_2024-08-08-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#2  0x00004000719c0f50 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw-patch/CMSSW_14_1_ROOT6_X_2024-08-08-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#3  0x00004000719be40c in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw-patch/CMSSW_14_1_ROOT6_X_2024-08-08-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#4  0x000040007c31112c in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw-patch/CMSSW_14_1_ROOT6_X_2024-08-08-2300/external/el8_aarch64_gcc12/lib/libtensorflow_framework.so.2
#5  0x00004000354de8b8 in start_thread () from /lib64/libpthread.so.0
#6  0x000040003552fafc in thread_start () from /lib64/libc.so.6
Thread 7 (Thread 0x4000e08d0220 (LWP 2393055) "cmsRun"):
#0  0x00004000354e4c60 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
#1  0x00004000719c085c in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw-patch/CMSSW_14_1_ROOT6_X_2024-08-08-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#2  0x00004000719c0f50 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw-patch/CMSSW_14_1_ROOT6_X_2024-08-08-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#3  0x00004000719be40c in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw-patch/CMSSW_14_1_ROOT6_X_2024-08-08-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#4  0x000040007c31112c in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw-patch/CMSSW_14_1_ROOT6_X_2024-08-08-2300/external/el8_aarch64_gcc12/lib/libtensorflow_framework.so.2
#5  0x00004000354de8b8 in start_thread () from /lib64/libpthread.so.0
#6  0x000040003552fafc in thread_start () from /lib64/libc.so.6
Thread 6 (Thread 0x4000e06cf220 (LWP 2393054) "cmsRun"):
#0  0x00004000354e4c60 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
#1  0x00004000719c085c in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw-patch/CMSSW_14_1_ROOT6_X_2024-08-08-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#2  0x00004000719c0f50 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw-patch/CMSSW_14_1_ROOT6_X_2024-08-08-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#3  0x00004000719be40c in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw-patch/CMSSW_14_1_ROOT6_X_2024-08-08-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#4  0x000040007c31112c in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw-patch/CMSSW_14_1_ROOT6_X_2024-08-08-2300/external/el8_aarch64_gcc12/lib/libtensorflow_framework.so.2
#5  0x00004000354de8b8 in start_thread () from /lib64/libpthread.so.0
#6  0x000040003552fafc in thread_start () from /lib64/libc.so.6
Thread 5 (Thread 0x40008a40a220 (LWP 2391247) "cmsRun"):
#0  0x00004000355b0b3c in nanosleep () from /lib64/libc.so.6
#1  0x00004000355b09e0 in sleep () from /lib64/libc.so.6
#2  0x000040003a371cb8 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-08-06-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  cache_bin_alloc_impl (adjust_low_water=false, success=<synthetic pointer>, bin=<optimized out>) at include/jemalloc/internal/cache_bin.h:369
#5  cache_bin_alloc_easy (success=<synthetic pointer>, bin=<optimized out>) at include/jemalloc/internal/cache_bin.h:399
#6  imalloc_fastpath (fallback_alloc=<optimized out>, size=416) at include/jemalloc/internal/jemalloc_internal_inlines_c.h:326
#7  newImpl<false> (size=416) at src/jemalloc_cpp.cpp:109
#8  operator new (size=416) at src/jemalloc_cpp.cpp:114
#9  0x00004000dfe528a8 in pat::PATJetSelector::filter(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-08-06-2300/lib/el8_aarch64_gcc12/pluginPhysicsToolsPatAlgos_plugins.so
#10 0x00004000338310b4 in edm::stream::EDFilterAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-08-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#11 0x0000400033818bb0 in edm::WorkerT<edm::stream::EDFilterAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-08-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#12 0x000040003379faa4 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-08-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#13 0x00004000337a0294 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-08-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#14 0x0000400033ad8b1c in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-08-06-2300/lib/el8_aarch64_gcc12/libFWCoreConcurrency.so
#15 0x0000400034cbe79c in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x400122f16f00, waiter=<synthetic pointer>..., this=0x400035e4b480) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#16 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=<synthetic pointer>..., t=0x0, this=0x400035e4b480) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#17 tbb::detail::r1::arena::process (tls=..., this=0x400035e4ad00) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/arena.cpp:137
#18 tbb::detail::r1::market::process (this=0x400035e47080, j=...) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/market.cpp:599
#19 0x0000400034cc6ffc in tbb::detail::r1::rml::private_worker::run (this=0x40003994c000) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#20 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x40003994c000) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#21 0x00004000354de8b8 in start_thread () from /lib64/libpthread.so.0
#22 0x000040003552fafc in thread_start () from /lib64/libc.so.6
Thread 4 (Thread 0x4000889fa220 (LWP 2391246) "cmsRun"):
#0  0x00004000355b0b3c in nanosleep () from /lib64/libc.so.6
#1  0x00004000355b09e0 in sleep () from /lib64/libc.so.6
#2  0x000040003a371cb8 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-08-06-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000040006906bea8 in Eigen::internal::triangular_solve_matrix<double, long, 1, 1, false, 0, 0, 1>::run(long, long, double const*, long, double*, long, long, Eigen::internal::level3_blocking<double, double>&) [clone .constprop.0] () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-08-06-2300/lib/el8_aarch64_gcc12/libRecoLocalCaloEcalRecAlgos.so
#5  0x000040006906203c in PulseChiSqSNNLS::NNLS() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-08-06-2300/lib/el8_aarch64_gcc12/libRecoLocalCaloEcalRecAlgos.so
#6  0x0000400069062c5c in PulseChiSqSNNLS::Minimize(Eigen::Matrix<double, 10, 10, 0, 10, 10> const&, Eigen::Matrix<double, 19, 19, 0, 19, 19> const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-08-06-2300/lib/el8_aarch64_gcc12/libRecoLocalCaloEcalRecAlgos.so
#7  0x0000400069063828 in PulseChiSqSNNLS::DoFit(Eigen::Matrix<double, 10, 1, 0, 10, 1> const&, Eigen::Matrix<double, 10, 10, 0, 10, 10> const&, Eigen::Matrix<char, -1, 1, 0, 12, 1> const&, Eigen::Matrix<double, 19, 1, 0, 19, 1> const&, Eigen::Matrix<double, 19, 19, 0, 19, 19> const&, Eigen::Matrix<char, 10, 1, 0, 10, 1> const&, Eigen::Matrix<char, 10, 1, 0, 10, 1> const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-08-06-2300/lib/el8_aarch64_gcc12/libRecoLocalCaloEcalRecAlgos.so
#8  0x0000400069063eb4 in EcalUncalibRecHitMultiFitAlgo::makeRecHit(EcalDataFrame const&, EcalPedestal const*, EcalMGPAGainRatio const*, std::array<Eigen::Matrix<double, 10, 10, 0, 10, 10>, 3ul> const&, Eigen::Matrix<double, 19, 1, 0, 19, 1> const&, Eigen::Matrix<double, 19, 19, 0, 19, 19> const&, Eigen::Matrix<char, -1, 1, 0, 12, 1> const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-08-06-2300/lib/el8_aarch64_gcc12/libRecoLocalCaloEcalRecAlgos.so
#9  0x000040006896cf04 in EcalUncalibRecHitWorkerMultiFit::run(edm::Event const&, EcalDigiCollection const&, edm::SortedCollection<EcalUncalibratedRecHit, edm::StrictWeakOrdering<EcalUncalibratedRecHit> >&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-08-06-2300/lib/el8_aarch64_gcc12/pluginRecoLocalCaloEcalRecProducersPlugins.so
#10 0x0000400068945b40 in EcalUncalibRecHitProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-08-06-2300/lib/el8_aarch64_gcc12/pluginRecoLocalCaloEcalRecProducersPlugins.so
#11 0x00004000338313b4 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-08-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#12 0x0000400033818950 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-08-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#13 0x000040003379faa4 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-08-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#14 0x00004000337a0294 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-08-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#15 0x0000400033ad8b1c in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-08-06-2300/lib/el8_aarch64_gcc12/libFWCoreConcurrency.so
#16 0x0000400034cbe79c in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x40012a36c500, waiter=<synthetic pointer>..., this=0x400035e4b380) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#17 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=<synthetic pointer>..., t=0x0, this=0x400035e4b380) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#18 tbb::detail::r1::arena::process (tls=..., this=0x400035e4ad00) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/arena.cpp:137
#19 tbb::detail::r1::market::process (this=0x400035e47080, j=...) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/market.cpp:599
#20 0x0000400034cc6ffc in tbb::detail::r1::rml::private_worker::run (this=0x40003994c100) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#21 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x40003994c100) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#22 0x00004000354de8b8 in start_thread () from /lib64/libpthread.so.0
#23 0x000040003552fafc in thread_start () from /lib64/libc.so.6
Thread 3 (Thread 0x400086b08220 (LWP 2391245) "cmsRun"):
#0  0x00004000355b0b3c in nanosleep () from /lib64/libc.so.6
#1  0x00004000355b09e0 in sleep () from /lib64/libc.so.6
#2  0x000040003a371cb8 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-08-06-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000040003e7632ec in BasicTrajectoryState::checkCurvilinError() const [clone .part.0] () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-08-06-2300/lib/el8_aarch64_gcc12/libTrackingToolsTrajectoryState.so
#5  0x000040007dba1540 in BasicTrajectoryState::curvilinearError() const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-08-06-2300/lib/el8_aarch64_gcc12/libTrackingToolsGsfTracking.so
#6  0x000040007dba7c80 in GsfMultiStateUpdator::update(TrajectoryStateOnSurface const&, TrackingRecHit const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-08-06-2300/lib/el8_aarch64_gcc12/libTrackingToolsGsfTracking.so
#7  0x000040007dbb2ae0 in GsfTrajectorySmoother::trajectory(Trajectory const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-08-06-2300/lib/el8_aarch64_gcc12/libTrackingToolsGsfTracking.so
#8  0x00004000f6e40f88 in LowPtGsfElectronSeedProducer::lightGsfTracking(reco::PreId&, edm::Ref<std::vector<reco::Track, std::allocator<reco::Track> >, reco::Track, edm::refhelper::FindUsingAdvance<std::vector<reco::Track, std::allocator<reco::Track> >, reco::Track> > const&, reco::ElectronSeed const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-08-06-2300/lib/el8_aarch64_gcc12/pluginRecoEgammaEgammaElectronProducersPlugins.so
#9  0x00004000f6e3a638 in void LowPtGsfElectronSeedProducer::loop<reco::PFRecTrack>(edm::Handle<std::vector<reco::PFRecTrack, std::allocator<reco::PFRecTrack> > > const&, edm::Handle<std::vector<reco::PFCluster, std::allocator<reco::PFCluster> > >&, std::vector<reco::ElectronSeed, std::allocator<reco::ElectronSeed> >&, std::vector<reco::PreId, std::allocator<reco::PreId> >&, std::vector<reco::PreId, std::allocator<reco::PreId> >&, std::unordered_map<unsigned int, unsigned long, std::hash<unsigned int>, std::equal_to<unsigned int>, std::allocator<std::pair<unsigned int const, unsigned long> > >&, edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-08-06-2300/lib/el8_aarch64_gcc12/pluginRecoEgammaEgammaElectronProducersPlugins.so
#10 0x00004000f6e3b378 in LowPtGsfElectronSeedProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-08-06-2300/lib/el8_aarch64_gcc12/pluginRecoEgammaEgammaElectronProducersPlugins.so
#11 0x00004000338313b4 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-08-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#12 0x0000400033818950 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-08-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#13 0x000040003379faa4 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-08-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#14 0x00004000337a0294 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-08-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#15 0x0000400033ad8b1c in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-08-06-2300/lib/el8_aarch64_gcc12/libFWCoreConcurrency.so
#16 0x0000400034cbe79c in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x40012a3af100, waiter=<synthetic pointer>..., this=0x400035e4b400) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#17 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=<synthetic pointer>..., t=0x0, this=0x400035e4b400) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#18 tbb::detail::r1::arena::process (tls=..., this=0x400035e4ad00) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/arena.cpp:137
#19 tbb::detail::r1::market::process (this=0x400035e47080, j=...) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/market.cpp:599
#20 0x0000400034cc6ffc in tbb::detail::r1::rml::private_worker::run (this=0x40003994c080) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#21 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x40003994c080) at /data/cmsbld/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-cd51b72bd1dc6777edf83e38784c1702/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#22 0x00004000354de8b8 in start_thread () from /lib64/libpthread.so.0
#23 0x000040003552fafc in thread_start () from /lib64/libc.so.6
Thread 2 (Thread 0x40003e689220 (LWP 2391100) "cmsRun"):
#0  0x00004000354e9580 in waitpid () from /lib64/libpthread.so.0
#1  0x000040003a371e24 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-08-06-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x000040003a374a78 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-08-06-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x00004000352ceb8c in std::execute_native_thread_routine (__p=0x4000370ab340) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x00004000354de8b8 in start_thread () from /lib64/libpthread.so.0
#5  0x000040003552fafc in thread_start () from /lib64/libc.so.6
Thread 1 (Thread 0x4000347f7010 (LWP 2387818) "cmsRun"):
#0  0x00004000355d6954 in poll () from /lib64/libc.so.6
#1  0x000040003a374c90 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-08-06-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x000040003a374ec4 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02849/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-08-06-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x006c615600000000 in ?? ()
#5  0x0000400043302158 in ?? ()
#6  0x000040023459da08 in ?? ()
#7  0x0000400109a3b6e8 in ?? ()
#8  0x00004000372014a0 in ?? ()

Current Modules:

Module: EvaluateMuonMVAID:muonMVAID (crashed)
Module: EcalUncalibRecHitProducer:ecalMultiFitUncalibRecHit@cpu
Module: LowPtGsfElectronSeedProducer:lowPtGsfElectronSeeds
Module: PATJetSelector:slimmedJets

A fatal system signal has occurred: segmentation violation
timeout: the monitored command dumped core
cmsbuild commented 1 month ago

cms-bot internal usage

cmsbuild commented 1 month ago

A new Issue was created by @iarspider.

@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

makortel commented 1 month ago

assign PhysicsTools/NanoAOD

cmsbuild commented 1 month ago

New categories assigned: xpog

@vlimant,@hqucms,@ftorrresd you have been requested to review this Pull request/Issue and eventually sign? Thanks

makortel commented 1 month ago

Looks like a memory corruption, and it is very much unclear whether EvaluateMuonMVAID is causing it or just a victim.

dan131riley commented 1 month ago

My first guess is that it's related to the other aarch64 crashes we've seen in cling, e.g. #43802. However, while BaseMVAValueMapProducer does use the expression parser, it looks like the crash happened well past where initialization should have happened, so I'm not so sure.