search

cms-sw / cmssw

CMS Offline Software
http://cms-sw.github.io/
Apache License 2.0
1.09k stars 4.32k forks source link

[AARCH64] Crashes - 2 - `DeDxEstimatorProducer` #45969

Closed AdrianoDee closed 2 months ago

AdrianoDee commented 2 months ago

While producing samples for the future ARM validation campaign I see this error when running a step3 such as:

cmsDriver.py step2 --conditions auto:phase1_2024_realistic --datatier GEN-SIM-RECO,MINIAODSIM,NANOAODSIM,DQMIO --era Run3_2024 --eventcontent RECOSIM,MINIAODSIM,NANOEDMAODSIM,DQM --filein "dbs:/RelValSinglePiFlatPt0p7To10/CMSSW_14_1_0_pre6-PU_140X_mcRun3_2024_realistic_v15_STD_2024_PU-v1/GEN-SIM-DIGI-RAW" --fileout "file:step2.root" --geometry DB:Extended --nStreams 1 --nThreads 8 --no_exec --number 10 --python_filename step_2_cfg.py --step RAW2DIGI,L1Reco,RECO,RECOSIM,PAT,NANO,VALIDATION:@standardValidation+@miniAODValidation,DQM:@standardDQM+@ExtraHLT+@miniAODDQM+@nanoAODDQM

on top of GEN-SIM-DIGI-RAW produced on x86 we get:

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Tue Sep 10 02:33:25 PM CEST 2024
Thread 26 (Thread 0xfffe6fff9160 (LWP 1471596) "cmsRun"):
#0  0x0000ffff900bedd8 in __futex_abstimed_wait_cancelable64 () from /lib64/libc.so.6
#1  0x0000ffff900c19d0 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libc.so.6
#2  0x0000fffed73ba6a0 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libtensorflow_cc.so.2
#3  0x0000fffed73bad50 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libtensorflow_cc.so.2
#4  0x0000fffed73b829c in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libtensorflow_cc.so.2
#5  0x0000ffff0d11014c in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libtensorflow_framework.so.2
#6  0x0000ffff900c2698 in start_thread () from /lib64/libc.so.6
#7  0x0000ffff9012cbdc in thread_start () from /lib64/libc.so.6
Thread 25 (Thread 0xfffe72ff9160 (LWP 1471595) "cmsRun"):
#0  0x0000ffff900bedd8 in __futex_abstimed_wait_cancelable64 () from /lib64/libc.so.6
#1  0x0000ffff900c19d0 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libc.so.6
#2  0x0000fffed73ba6a0 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libtensorflow_cc.so.2
#3  0x0000fffed73bad50 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libtensorflow_cc.so.2
#4  0x0000fffed73b829c in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libtensorflow_cc.so.2
#5  0x0000ffff0d11014c in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libtensorflow_framework.so.2
#6  0x0000ffff900c2698 in start_thread () from /lib64/libc.so.6
#7  0x0000ffff9012cbdc in thread_start () from /lib64/libc.so.6
Thread 24 (Thread 0xfffe75ad7160 (LWP 1471594) "cmsRun"):
#0  0x0000ffff900bedd8 in __futex_abstimed_wait_cancelable64 () from /lib64/libc.so.6
#1  0x0000ffff900c19d0 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libc.so.6
#2  0x0000fffed73ba6a0 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libtensorflow_cc.so.2
#3  0x0000fffed73bad50 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libtensorflow_cc.so.2
#4  0x0000fffed73b829c in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libtensorflow_cc.so.2
#5  0x0000ffff0d11014c in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libtensorflow_framework.so.2
#6  0x0000ffff900c2698 in start_thread () from /lib64/libc.so.6
#7  0x0000ffff9012cbdc in thread_start () from /lib64/libc.so.6
Thread 23 (Thread 0xfffeb7729160 (LWP 1471558) "cmsRun"):
#0  0x0000ffff900bedd8 in __futex_abstimed_wait_cancelable64 () from /lib64/libc.so.6
#1  0x0000ffff900cacd4 in __new_sem_wait_slow64.constprop.0 () from /lib64/libc.so.6
#2  0x0000ffff84b64550 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdCl.so.3
#3  0x0000ffff84b6462c in RunRunnerThread () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdCl.so.3
#4  0x0000ffff900c2698 in start_thread () from /lib64/libc.so.6
#5  0x0000ffff9012cbdc in thread_start () from /lib64/libc.so.6
Thread 22 (Thread 0xfffeb7f39160 (LWP 1471557) "cmsRun"):
#0  0x0000ffff900bedd8 in __futex_abstimed_wait_cancelable64 () from /lib64/libc.so.6
#1  0x0000ffff900cacd4 in __new_sem_wait_slow64.constprop.0 () from /lib64/libc.so.6
#2  0x0000ffff84b64550 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdCl.so.3
#3  0x0000ffff84b6462c in RunRunnerThread () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdCl.so.3
#4  0x0000ffff900c2698 in start_thread () from /lib64/libc.so.6
#5  0x0000ffff9012cbdc in thread_start () from /lib64/libc.so.6
Thread 21 (Thread 0xfffeb8749160 (LWP 1471556) "cmsRun"):
#0  0x0000ffff900bedd8 in __futex_abstimed_wait_cancelable64 () from /lib64/libc.so.6
#1  0x0000ffff900cacd4 in __new_sem_wait_slow64.constprop.0 () from /lib64/libc.so.6
#2  0x0000ffff84b64550 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdCl.so.3
#3  0x0000ffff84b6462c in RunRunnerThread () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdCl.so.3
#4  0x0000ffff900c2698 in start_thread () from /lib64/libc.so.6
#5  0x0000ffff9012cbdc in thread_start () from /lib64/libc.so.6
Thread 20 (Thread 0xfffeb8f59160 (LWP 1471555) "cmsRun"):
#0  0x0000ffff900f7514 in clock_nanosleep@@GLIBC_2.17 () from /lib64/libc.so.6
#1  0x0000ffff900fc64c in nanosleep () from /lib64/libc.so.6
#2  0x0000ffff87185b84 in XrdSysTimer::Wait(int) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x0000ffff84aef04c in XrdCl::TaskManager::RunTasks() () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdCl.so.3
#4  0x0000ffff84aef1bc in RunRunnerThread () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdCl.so.3
#5  0x0000ffff900c2698 in start_thread () from /lib64/libc.so.6
#6  0x0000ffff9012cbdc in thread_start () from /lib64/libc.so.6
Thread 19 (Thread 0xfffeb9769160 (LWP 1471554) "cmsRun"):
#0  0x0000ffff9012cd7c in epoll_pwait () from /lib64/libc.so.6
#1  0x0000ffff8717fe4c in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x0000ffff8717bf5c in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x0000ffff87185140 in XrdSysThread_Xeq () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#4  0x0000ffff900c2698 in start_thread () from /lib64/libc.so.6
#5  0x0000ffff9012cbdc in thread_start () from /lib64/libc.so.6
Thread 18 (Thread 0xfffeb9f79160 (LWP 1471553) "cmsRun"):
#0  0x0000ffff9012cd7c in epoll_pwait () from /lib64/libc.so.6
#1  0x0000ffff8717fe4c in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x0000ffff8717bf5c in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x0000ffff87185140 in XrdSysThread_Xeq () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#4  0x0000ffff900c2698 in start_thread () from /lib64/libc.so.6
#5  0x0000ffff9012cbdc in thread_start () from /lib64/libc.so.6
Thread 17 (Thread 0xfffeba789160 (LWP 1471552) "cmsRun"):
#0  0x0000ffff9012cd7c in epoll_pwait () from /lib64/libc.so.6
#1  0x0000ffff8717fe4c in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x0000ffff8717bf5c in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x0000ffff87185140 in XrdSysThread_Xeq () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#4  0x0000ffff900c2698 in start_thread () from /lib64/libc.so.6
#5  0x0000ffff9012cbdc in thread_start () from /lib64/libc.so.6
Thread 16 (Thread 0xfffebaf99160 (LWP 1471551) "cmsRun"):
#0  0x0000ffff9012cd7c in epoll_pwait () from /lib64/libc.so.6
#1  0x0000ffff8717fe4c in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x0000ffff8717bf5c in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x0000ffff87185140 in XrdSysThread_Xeq () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#4  0x0000ffff900c2698 in start_thread () from /lib64/libc.so.6
#5  0x0000ffff9012cbdc in thread_start () from /lib64/libc.so.6
Thread 15 (Thread 0xfffebb7a9160 (LWP 1471550) "cmsRun"):
#0  0x0000ffff9012cd7c in epoll_pwait () from /lib64/libc.so.6
#1  0x0000ffff8717fe4c in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x0000ffff8717bf5c in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x0000ffff87185140 in XrdSysThread_Xeq () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#4  0x0000ffff900c2698 in start_thread () from /lib64/libc.so.6
#5  0x0000ffff9012cbdc in thread_start () from /lib64/libc.so.6
Thread 14 (Thread 0xfffebbfb9160 (LWP 1471549) "cmsRun"):
#0  0x0000ffff9012cd7c in epoll_pwait () from /lib64/libc.so.6
#1  0x0000ffff8717fe4c in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x0000ffff8717bf5c in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x0000ffff87185140 in XrdSysThread_Xeq () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#4  0x0000ffff900c2698 in start_thread () from /lib64/libc.so.6
#5  0x0000ffff9012cbdc in thread_start () from /lib64/libc.so.6
Thread 13 (Thread 0xfffebc7c9160 (LWP 1471548) "cmsRun"):
#0  0x0000ffff9012cd7c in epoll_pwait () from /lib64/libc.so.6
#1  0x0000ffff8717fe4c in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x0000ffff8717bf5c in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x0000ffff87185140 in XrdSysThread_Xeq () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#4  0x0000ffff900c2698 in start_thread () from /lib64/libc.so.6
#5  0x0000ffff9012cbdc in thread_start () from /lib64/libc.so.6
Thread 12 (Thread 0xfffebcfd9160 (LWP 1471547) "cmsRun"):
#0  0x0000ffff9012cd7c in epoll_pwait () from /lib64/libc.so.6
#1  0x0000ffff8717fe4c in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x0000ffff8717bf5c in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x0000ffff87185140 in XrdSysThread_Xeq () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#4  0x0000ffff900c2698 in start_thread () from /lib64/libc.so.6
#5  0x0000ffff9012cbdc in thread_start () from /lib64/libc.so.6
Thread 11 (Thread 0xfffebd7e9160 (LWP 1471546) "cmsRun"):
#0  0x0000ffff9012cd7c in epoll_pwait () from /lib64/libc.so.6
#1  0x0000ffff8717fe4c in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x0000ffff8717bf5c in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x0000ffff87185140 in XrdSysThread_Xeq () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#4  0x0000ffff900c2698 in start_thread () from /lib64/libc.so.6
#5  0x0000ffff9012cbdc in thread_start () from /lib64/libc.so.6
Thread 10 (Thread 0xfffebdff9160 (LWP 1471545) "cmsRun"):
#0  0x0000ffff9012cd7c in epoll_pwait () from /lib64/libc.so.6
#1  0x0000ffff8717fe4c in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x0000ffff8717bf5c in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x0000ffff87185140 in XrdSysThread_Xeq () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#4  0x0000ffff900c2698 in start_thread () from /lib64/libc.so.6
#5  0x0000ffff9012cbdc in thread_start () from /lib64/libc.so.6
Thread 9 (Thread 0xfffec6ff9160 (LWP 1471544) "cmsRun"):
#0  0x0000ffff900f7514 in clock_nanosleep@@GLIBC_2.17 () from /lib64/libc.so.6
#1  0x0000ffff900fc64c in nanosleep () from /lib64/libc.so.6
#2  0x0000ffff900fc50c in sleep () from /lib64/libc.so.6
#3  0x0000ffff875c9cc8 in sig_pause_for_stacktrace () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x0000fffee0d2c9f0 in void std::__merge_sort_with_buffer<__gnu_cxx::__normal_iterator<std::pair<double, unsigned int>*, std::vector<std::pair<double, unsigned int>, std::allocator<std::pair<double, unsigned int> > > >, std::pair<double, unsigned int>*, __gnu_cxx::__ops::_Iter_comp_iter<std::less<std::pair<double, unsigned int> > > >(__gnu_cxx::__normal_iterator<std::pair<double, unsigned int>*, std::vector<std::pair<double, unsigned int>, std::allocator<std::pair<double, unsigned int> > > >, __gnu_cxx::__normal_iterator<std::pair<double, unsigned int>*, std::vector<std::pair<double, unsigned int>, std::allocator<std::pair<double, unsigned int> > > >, std::pair<double, unsigned int>*, __gnu_cxx::__ops::_Iter_comp_iter<std::less<std::pair<double, unsigned int> > >) [clone .isra.0] () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libRecoVertexPrimaryVertexProducer.so
#6  0x0000fffee0d356b0 in void std::__stable_sort_adaptive<__gnu_cxx::__normal_iterator<std::pair<double, unsigned int>*, std::vector<std::pair<double, unsigned int>, std::allocator<std::pair<double, unsigned int> > > >, std::pair<double, unsigned int>*, long, __gnu_cxx::__ops::_Iter_comp_iter<std::less<std::pair<double, unsigned int> > > >(__gnu_cxx::__normal_iterator<std::pair<double, unsigned int>*, std::vector<std::pair<double, unsigned int>, std::allocator<std::pair<double, unsigned int> > > >, __gnu_cxx::__normal_iterator<std::pair<double, unsigned int>*, std::vector<std::pair<double, unsigned int>, std::allocator<std::pair<double, unsigned int> > > >, std::pair<double, unsigned int>*, long, __gnu_cxx::__ops::_Iter_comp_iter<std::less<std::pair<double, unsigned int> > >) [clone .isra.0] () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libRecoVertexPrimaryVertexProducer.so
#7  0x0000fffee0d1c988 in DAClusterizerInZ_vect::merge(DAClusterizerInZ_vect::vertex_t&, DAClusterizerInZ_vect::track_t&, double&) const () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libRecoVertexPrimaryVertexProducer.so
#8  0x0000fffee0d216b0 in DAClusterizerInZ_vect::vertices_no_blocks(std::vector<reco::TransientTrack, std::allocator<reco::TransientTrack> > const&) const () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libRecoVertexPrimaryVertexProducer.so
#9  0x0000fffee0d219a4 in DAClusterizerInZ_vect::vertices(std::vector<reco::TransientTrack, std::allocator<reco::TransientTrack> > const&) const () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libRecoVertexPrimaryVertexProducer.so
#10 0x0000fffe72277160 in PrimaryVertexProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginRecoVertexPrimaryVertexProducerPlugins.so
#11 0x0000ffff9202ddc4 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#12 0x0000ffff92014900 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#13 0x0000ffff91fa00d8 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#14 0x0000ffff91fa1a34 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#15 0x0000ffff91be0acc in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreConcurrency.so
#16 0x0000ffff905bc70c in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0xfffd6f238400, waiter=<synthetic pointer>..., this=0xffff8f973c80) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#17 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=<synthetic pointer>..., t=0x0, this=0xffff8f973c80) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#18 tbb::detail::r1::arena::process (tls=..., this=0xffff8f972f00) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/arena.cpp:137
#19 tbb::detail::r1::market::process (this=0xffff8f98b580, j=...) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/market.cpp:599
#20 0x0000ffff905c4f5c in tbb::detail::r1::rml::private_worker::run (this=0xffff89906e00) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#21 tbb::detail::r1::rml::private_worker::thread_routine (arg=0xffff89906e00) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#22 0x0000ffff900c2698 in start_thread () from /lib64/libc.so.6
#23 0x0000ffff9012cbdc in thread_start () from /lib64/libc.so.6
Thread 8 (Thread 0xfffec9bd9160 (LWP 1471543) "cmsRun"):
#0  0x0000ffff900f7514 in clock_nanosleep@@GLIBC_2.17 () from /lib64/libc.so.6
#1  0x0000ffff900fc64c in nanosleep () from /lib64/libc.so.6
#2  0x0000ffff900fc50c in sleep () from /lib64/libc.so.6
#3  0x0000ffff875c9cc8 in sig_pause_for_stacktrace () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x0000ffff902a0cf8 in __cxxabiv1::(anonymous namespace)::adjust_pointer<void> (offset=0, base=0xfffdd2fd0800) at ../../../../libstdc++-v3/libsupc++/tinfo.h:68
#6  __cxxabiv1::__dynamic_cast (src_ptr=0xfffdd2fd0800, src_type=0xffff7da66818 <typeinfo for TrackingRecHit>, dst_type=0xffff7db5e470 <typeinfo for SiStripRecHit1D>, src2dst=0) at ../../../../libstdc++-v3/libsupc++/dyncast.cc:58
#7  0x0000fffe9d7a2e08 in std::vector<OmniClusterRef, std::allocator<OmniClusterRef> > track_associator::hitsToClusterRefs<__gnu_cxx::__normal_iterator<TrackingRecHit* const*, std::vector<TrackingRecHit*, std::allocator<TrackingRecHit*> > > >(__gnu_cxx::__normal_iterator<TrackingRecHit* const*, std::vector<TrackingRecHit*, std::allocator<TrackingRecHit*> > >, __gnu_cxx::__normal_iterator<TrackingRecHit* const*, std::vector<TrackingRecHit*, std::allocator<TrackingRecHit*> > >) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginSimTrackerTrackAssociatorProducersAuto.so
#8  0x0000fffe9d795c64 in double QuickTrackAssociatorByHitsImpl::weightedNumberOfTrackClusters<reco::Track>(reco::Track const&, ClusterTPAssociation const&) const () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginSimTrackerTrackAssociatorProducersAuto.so
#9  0x0000fffe9d795f64 in QuickTrackAssociatorByHitsImpl::associateRecoToSim(edm::RefToBaseVector<reco::Track> const&, edm::RefVector<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle, edm::refhelper::FindUsingAdvance<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle> > const&) const () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginSimTrackerTrackAssociatorProducersAuto.so
#10 0x0000fffef2e4075c in MultiTrackValidator::dqmAnalyze(edm::Event const&, edm::EventSetup const&, MultiTrackValidatorHistograms const&) const () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginValidationRecoTrackPlugins.so
#11 0x0000ffff9201ac28 in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#12 0x0000ffff92013dd0 in edm::WorkerT<edm::global::EDProducerBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#13 0x0000ffff91fa00d8 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#14 0x0000ffff91fa1a34 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#15 0x0000ffff91be0acc in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreConcurrency.so
#16 0x0000ffff905bc70c in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0xfffd6f577000, waiter=<synthetic pointer>..., this=0xffff8f973b80) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#17 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=<synthetic pointer>..., t=0x0, this=0xffff8f973b80) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#18 tbb::detail::r1::arena::process (tls=..., this=0xffff8f972f00) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/arena.cpp:137
#19 tbb::detail::r1::market::process (this=0xffff8f98b580, j=...) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/market.cpp:599
#20 0x0000ffff905c4f5c in tbb::detail::r1::rml::private_worker::run (this=0xffff89907000) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#21 tbb::detail::r1::rml::private_worker::thread_routine (arg=0xffff89907000) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#22 0x0000ffff900c2698 in start_thread () from /lib64/libc.so.6
#23 0x0000ffff9012cbdc in thread_start () from /lib64/libc.so.6
Thread 7 (Thread 0xfffeca5e9160 (LWP 1471542) "cmsRun"):
#0  0x0000ffff900f7514 in clock_nanosleep@@GLIBC_2.17 () from /lib64/libc.so.6
#1  0x0000ffff900fc64c in nanosleep () from /lib64/libc.so.6
#2  0x0000ffff900fc50c in sleep () from /lib64/libc.so.6
#3  0x0000ffff875c9cc8 in sig_pause_for_stacktrace () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x0000ffff904526e4 in _Unwind_Find_registered_FDE (bases=0xfffeca5e6c68, pc=0xffff9045077b <_Unwind_RaiseException+91>) at ../../../libgcc/unwind-dw2-fde.c:1073
#6  _Unwind_Find_FDE (pc=0xffff9045077b <_Unwind_RaiseException+91>, bases=bases@entry=0xfffeca5e6c68) at ../../../libgcc/unwind-dw2-fde-dip.c:503
#7  0x0000ffff9044f130 in uw_frame_state_for (context=context@entry=0xfffeca5e6940, fs=fs@entry=0xfffeca5e6200) at ../../../libgcc/unwind-dw2.c:1263
#8  0x0000ffff90450098 in uw_init_context_1 (context=context@entry=0xfffeca5e6940, outer_cfa=outer_cfa@entry=0xfffeca5e7740, outer_ra=0xffff902a26b8 <__cxxabiv1::__cxa_throw(void*, std::type_info*, void (*)(void*))+88>) at ../../../libgcc/unwind-dw2.c:1592
#9  0x0000ffff9045077c in _Unwind_RaiseException (exc=0xfffded8bb3e0) at ../../../libgcc/unwind.inc:93
#10 0x0000ffff902a26b8 in __cxxabiv1::__cxa_throw (obj=<optimized out>, tinfo=0xffff91b76668 <typeinfo for cms::Exception>, dest=0xffff91b28a30 <cms::Exception::~Exception()>) at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:93
#11 0x0000ffff551e7504 in TrackerGeometry::idToDet(DetId) const () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libGeometryTrackerGeometryBuilder.so
#12 0x0000fffe9624ffa8 in TrackProducerAlgorithm<reco::Track>::runWithTrack(TrackingGeometry const*, MagneticField const*, edm::View<reco::Track> const&, TrajectoryFitter const*, Propagator const*, TransientTrackingRecHitBuilder const*, reco::BeamSpot const&, std::vector<AlgoProductTraits<reco::Track>::AlgoProduct, std::allocator<AlgoProductTraits<reco::Track>::AlgoProduct> >&) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginRecoTrackerTrackProducerPlugins.so
#13 0x0000fffe96253b6c in TrackRefitter::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginRecoTrackerTrackProducerPlugins.so
#14 0x0000ffff9202ddc4 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#15 0x0000ffff92014900 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#16 0x0000ffff91fa00d8 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#17 0x0000ffff91fa1a34 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#18 0x0000ffff91be0acc in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreConcurrency.so
#19 0x0000ffff905bc70c in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0xffff8f803900, waiter=<synthetic pointer>..., this=0xffff8f973b00) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#20 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=<synthetic pointer>..., t=0x0, this=0xffff8f973b00) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#21 tbb::detail::r1::arena::process (tls=..., this=0xffff8f972f00) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/arena.cpp:137
#22 tbb::detail::r1::market::process (this=0xffff8f98b580, j=...) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/market.cpp:599
#23 0x0000ffff905c4f5c in tbb::detail::r1::rml::private_worker::run (this=0xffff89906f00) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#24 tbb::detail::r1::rml::private_worker::thread_routine (arg=0xffff89906f00) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#25 0x0000ffff900c2698 in start_thread () from /lib64/libc.so.6
#26 0x0000ffff9012cbdc in thread_start () from /lib64/libc.so.6
Thread 6 (Thread 0xfffecaff9160 (LWP 1471541) "cmsRun"):
#0  0x0000ffff90122960 in poll () from /lib64/libc.so.6
#1  0x0000ffff875ccca0 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x0000ffff875cced4 in sig_dostack_then_abort () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000fffe8bf41a64 in DeDxEstimatorProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginRecoTrackerDeDxPlugins.so
#5  0x0000ffff9202ddc4 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#6  0x0000ffff92014900 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#7  0x0000ffff91fa00d8 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#8  0x0000ffff91fa1a34 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#9  0x0000ffff91be0acc in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreConcurrency.so
#10 0x0000ffff905bc70c in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0xfffd6f238b00, waiter=<synthetic pointer>..., this=0xffff8f973980) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#11 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=<synthetic pointer>..., t=0x0, this=0xffff8f973980) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#12 tbb::detail::r1::arena::process (tls=..., this=0xffff8f972f00) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/arena.cpp:137
#13 tbb::detail::r1::market::process (this=0xffff8f98b580, j=...) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/market.cpp:599
#14 0x0000ffff905c4f5c in tbb::detail::r1::rml::private_worker::run (this=0xffff89906f80) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#15 tbb::detail::r1::rml::private_worker::thread_routine (arg=0xffff89906f80) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#16 0x0000ffff900c2698 in start_thread () from /lib64/libc.so.6
#17 0x0000ffff9012cbdc in thread_start () from /lib64/libc.so.6
Thread 5 (Thread 0xfffecedf9160 (LWP 1471540) "cmsRun"):
#0  0x0000ffff900f7514 in clock_nanosleep@@GLIBC_2.17 () from /lib64/libc.so.6
#1  0x0000ffff900fc64c in nanosleep () from /lib64/libc.so.6
#2  0x0000ffff900fc50c in sleep () from /lib64/libc.so.6
#3  0x0000ffff875c9cc8 in sig_pause_for_stacktrace () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x0000fffe8bf36c28 in DeDxEstimatorProducer::processHit(TrackingRecHit const*, float, float&, std::vector<reco::DeDxHit, std::allocator<reco::DeDxHit> >&, int&)@plt () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginRecoTrackerDeDxPlugins.so
#6  0x0000fffe8bf41b7c in DeDxEstimatorProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginRecoTrackerDeDxPlugins.so
#7  0x0000ffff9202ddc4 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#8  0x0000ffff92014900 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#9  0x0000ffff91fa00d8 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#10 0x0000ffff91fa1a34 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#11 0x0000ffff91be0acc in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreConcurrency.so
#12 0x0000ffff905bc70c in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0xfffd6f238e00, waiter=<synthetic pointer>..., this=0xffff8f973c00) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#13 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=<synthetic pointer>..., t=0x0, this=0xffff8f973c00) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#14 tbb::detail::r1::arena::process (tls=..., this=0xffff8f972f00) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/arena.cpp:137
#15 tbb::detail::r1::market::process (this=0xffff8f98b580, j=...) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/market.cpp:599
#16 0x0000ffff905c4f5c in tbb::detail::r1::rml::private_worker::run (this=0xffff89906e80) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#17 tbb::detail::r1::rml::private_worker::thread_routine (arg=0xffff89906e80) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#18 0x0000ffff900c2698 in start_thread () from /lib64/libc.so.6
#19 0x0000ffff9012cbdc in thread_start () from /lib64/libc.so.6
Thread 4 (Thread 0xffff0b337160 (LWP 1471539) "cmsRun"):
#0  0x0000ffff900f7514 in clock_nanosleep@@GLIBC_2.17 () from /lib64/libc.so.6
#1  0x0000ffff900fc64c in nanosleep () from /lib64/libc.so.6
#2  0x0000ffff900fc50c in sleep () from /lib64/libc.so.6
#3  0x0000ffff875c9cc8 in sig_pause_for_stacktrace () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x0000fffe9d7c22d0 in std::vector<std::pair<edm::Ref<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle, edm::refhelper::FindUsingAdvance<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle> >, double>, std::allocator<std::pair<edm::Ref<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle, edm::refhelper::FindUsingAdvance<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle> >, double> > > QuickTrackAssociatorByHitsImpl::associateTrack<edm::Handle<std::vector<TrackingParticle, std::allocator<TrackingParticle> > >, __gnu_cxx::__normal_iterator<TrackingRecHit* const*, std::vector<TrackingRecHit*, std::allocator<TrackingRecHit*> > > >(ClusterTPAssociation const&, edm::Handle<std::vector<TrackingParticle, std::allocator<TrackingParticle> > > const&, edm::IndexSet const*, __gnu_cxx::__normal_iterator<TrackingRecHit* const*, std::vector<TrackingRecHit*, std::allocator<TrackingRecHit*> > >, __gnu_cxx::__normal_iterator<TrackingRecHit* const*, std::vector<TrackingRecHit*, std::allocator<TrackingRecHit*> > >) const [clone .constprop.1] () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginSimTrackerTrackAssociatorProducersAuto.so
#6  0x0000fffe9d795f34 in QuickTrackAssociatorByHitsImpl::associateRecoToSim(edm::RefToBaseVector<reco::Track> const&, edm::RefVector<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle, edm::refhelper::FindUsingAdvance<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle> > const&) const () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginSimTrackerTrackAssociatorProducersAuto.so
#7  0x0000fffef2e4075c in MultiTrackValidator::dqmAnalyze(edm::Event const&, edm::EventSetup const&, MultiTrackValidatorHistograms const&) const () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginValidationRecoTrackPlugins.so
#8  0x0000ffff9201ac28 in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#9  0x0000ffff92013dd0 in edm::WorkerT<edm::global::EDProducerBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#10 0x0000ffff91fa00d8 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#11 0x0000ffff91fa1a34 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#12 0x0000ffff91be0acc in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreConcurrency.so
#13 0x0000ffff905bc70c in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0xfffd6f577400, waiter=<synthetic pointer>..., this=0xffff8f973a00) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#14 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=<synthetic pointer>..., t=0x0, this=0xffff8f973a00) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#15 tbb::detail::r1::arena::process (tls=..., this=0xffff8f972f00) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/arena.cpp:137
#16 tbb::detail::r1::market::process (this=0xffff8f98b580, j=...) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/market.cpp:599
#17 0x0000ffff905c4f5c in tbb::detail::r1::rml::private_worker::run (this=0xffff89907100) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#18 tbb::detail::r1::rml::private_worker::thread_routine (arg=0xffff89907100) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#19 0x0000ffff900c2698 in start_thread () from /lib64/libc.so.6
#20 0x0000ffff9012cbdc in thread_start () from /lib64/libc.so.6
Thread 3 (Thread 0xffff0bd47160 (LWP 1471538) "cmsRun"):
#0  0x0000ffff900f7514 in clock_nanosleep@@GLIBC_2.17 () from /lib64/libc.so.6
#1  0x0000ffff900fc64c in nanosleep () from /lib64/libc.so.6
#2  0x0000ffff900fc50c in sleep () from /lib64/libc.so.6
#3  0x0000ffff875c9cc8 in sig_pause_for_stacktrace () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x0000ffff545d04c4 in SteppingHelixPropagator::makeAtomStep(SteppingHelixStateInfo&, SteppingHelixStateInfo&, double, PropagationDirection, SteppingHelixPropagator::Fancy) const () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libTrackPropagationSteppingHelixPropagator.so
#6  0x0000ffff545d1644 in SteppingHelixPropagator::propagate(SteppingHelixStateInfo (&) [8], int&, SteppingHelixPropagator::DestType, double const*, double) const () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libTrackPropagationSteppingHelixPropagator.so
#7  0x0000ffff545d3388 in SteppingHelixPropagator::propagate(SteppingHelixStateInfo const&, Cylinder const&, SteppingHelixStateInfo&) const () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libTrackPropagationSteppingHelixPropagator.so
#8  0x0000ffff545d5f54 in SteppingHelixPropagator::propagateWithPath(FreeTrajectoryState const&, Cylinder const&) const () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libTrackPropagationSteppingHelixPropagator.so
#9  0x0000fffe7f0cd524 in TrackExtrapolator::propagateTrackToVolume(reco::Track const&, MagneticField const&, Propagator const&, FiducialVolume const&, ROOT::Math::PositionVector3D<ROOT::Math::Cartesian3D<double>, ROOT::Math::DefaultCoordinateSystemTag>&, ROOT::Math::DisplacementVector3D<ROOT::Math::Cartesian3D<double>, ROOT::Math::DefaultCoordinateSystemTag>&) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginRecoJetsJetAssociationProducers.so
#10 0x0000fffe7f0cde58 in TrackExtrapolator::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginRecoJetsJetAssociationProducers.so
#11 0x0000ffff9202ddc4 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#12 0x0000ffff92014900 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#13 0x0000ffff91fa00d8 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#14 0x0000ffff91fa1a34 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#15 0x0000ffff91be0acc in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreConcurrency.so
#16 0x0000ffff905bc70c in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0xfffd6f238600, waiter=<synthetic pointer>..., this=0xffff8f973a80) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#17 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=<synthetic pointer>..., t=0x0, this=0xffff8f973a80) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#18 tbb::detail::r1::arena::process (tls=..., this=0xffff8f972f00) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/arena.cpp:137
#19 tbb::detail::r1::market::process (this=0xffff8f98b580, j=...) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/market.cpp:599
#20 0x0000ffff905c4f5c in tbb::detail::r1::rml::private_worker::run (this=0xffff89907080) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#21 tbb::detail::r1::rml::private_worker::thread_routine (arg=0xffff89907080) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#22 0x0000ffff900c2698 in start_thread () from /lib64/libc.so.6
#23 0x0000ffff9012cbdc in thread_start () from /lib64/libc.so.6
Thread 2 (Thread 0xffff6fcb9160 (LWP 1471532) "cmsRun"):
#0  0x0000ffff900fc2c0 in wait4 () from /lib64/libc.so.6
#1  0x0000ffff875c9e34 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x0000ffff875cca88 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x0000ffff902ce86c in std::execute_native_thread_routine (__p=0xffff8d0ac6f0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x0000ffff900c2698 in start_thread () from /lib64/libc.so.6
#5  0x0000ffff9012cbdc in thread_start () from /lib64/libc.so.6
Thread 1 (Thread 0xffff91652020 (LWP 1471517) "cmsRun"):
#0  0x0000ffff900f7514 in clock_nanosleep@@GLIBC_2.17 () from /lib64/libc.so.6
#1  0x0000ffff900fc64c in nanosleep () from /lib64/libc.so.6
#2  0x0000ffff900fc50c in sleep () from /lib64/libc.so.6
#3  0x0000ffff875c9cc8 in sig_pause_for_stacktrace () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x0000ffff84f2b8c8 in TProfile2D::Fill(double, double, double) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libHist.so
#6  0x0000fffe962c7f44 in SiStripMonitorTrack::clusterInfos(SiStripClusterInfo*, unsigned int, SiStripMonitorTrack::ClusterFlags, bool, Vector3DBase<float, LocalTag>, SiStripMonitorTrack::Det2MEs const&, TrackerTopology const*, SiStripGain const*, SiStripQuality const*, edm::DetSetVector<SiStripDigi> const&, float, float) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginDQMSiStripMonitorTrack.so
#7  0x0000fffe962d0924 in void SiStripMonitorTrack::RecHitInfo<SiStripRecHit2D>(SiStripRecHit2D const*, Vector3DBase<float, LocalTag>, edm::DetSetVector<SiStripDigi> const&, edm::Event const&, bool) [clone .constprop.0] () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginDQMSiStripMonitorTrack.so
#8  0x0000fffe962c1330 in SiStripMonitorTrack::trajectoryStudy(reco::Track const&, edm::DetSetVector<SiStripDigi> const&, edm::Event const&, bool) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginDQMSiStripMonitorTrack.so
#9  0x0000fffe962c18bc in SiStripMonitorTrack::trackStudyFromTrajectory(edm::Handle<std::vector<reco::Track, std::allocator<reco::Track> > >, edm::DetSetVector<SiStripDigi> const&, edm::Event const&) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginDQMSiStripMonitorTrack.so
#10 0x0000fffe962c1cb8 in SiStripMonitorTrack::trackStudy(edm::Event const&) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginDQMSiStripMonitorTrack.so
#11 0x0000fffe962ba1dc in SiStripMonitorTrack::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginDQMSiStripMonitorTrack.so
#12 0x0000ffff9202ddc4 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#13 0x0000ffff92014900 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#14 0x0000ffff91fa00d8 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#15 0x0000ffff91fa1a34 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#16 0x0000ffff91be0acc in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreConcurrency.so
#17 0x0000ffff905c275c in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0xfffd6f577a00, this=0xffff8f973900) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#18 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0xffff8f973900) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#19 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#20 0x0000ffff91f172c4 in edm::FinalWaitingTask::wait() () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#21 0x0000ffff91f23060 in edm::EventProcessor::processRuns() () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#22 0x0000ffff91f234bc in edm::EventProcessor::runToCompletion() () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#23 0x0000000000407b1c in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#24 0x0000ffff905ba9d8 in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/arena.cpp:688
#25 0x00000000004093d0 in main::{lambda()#1}::operator()() const ()
#26 0x00000000004044c4 in main ()

Current Modules:

Module: DeDxEstimatorProducer:dedxPixelHarmonic2 (crashed)
Module: SiStripMonitorTrack:SiStripMonitorTrackCommon
Module: TrackRefitter:refittedForPixelDQM
Module: MultiTrackValidator:trackValidatorBuilding
Module: DeDxEstimatorProducer:dedxTruncated40
Module: PrimaryVertexProducer:unsortedOfflinePrimaryVertices
Module: TrackExtrapolator:trackExtrapolator
Module: MultiTrackValidator:trackValidatorBuildingPreSplitting

A fatal system signal has occurred: segmentation violation
Segmentation fault (core dumped)

on x86 the same job ends successfully. The rationale behind running an ARM job on top of an x86 one is that we want to have PU samples with exactly the same events to validate the "new" CPU architecture. Here the reproducers:

(ARM -> tested on lxplus-arm)

export SCRAM_ARCH=el9_aarch64_gcc12
scram p -n cmssw_arm CMSSW_14_1_0_pre6
cd cmssw_arm/src
cmsenv
wget https://adiflori.web.cern.ch/adiflori/arm/error_arm_DeDxEstimatorProducer_dedxPixelHarmonic2.py
cmsRun error_arm_MCTrackMatcher.py

(x86 -> tested on lxplus)

export SCRAM_ARCH=el9_amd64_gcc12
scram p -n cmssw_x86 CMSSW_14_1_0_pre6
cd cmssw_x86/src
cmsenv
wget https://adiflori.web.cern.ch/adiflori/arm/error_arm_DeDxEstimatorProducer_dedxPixelHarmonic2.py
cmsRun error_arm_MCTrackMatcher.py
cmsbuild commented 2 months ago

cms-bot internal usage

cmsbuild commented 2 months ago

A new Issue was created by @AdrianoDee.

@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

AdrianoDee commented 2 months ago

assign RecoTracker/DeDx

cmsbuild commented 2 months ago

New categories assigned: reconstruction

@jfernan2,@mandrenguyen you have been requested to review this Pull request/Issue and eventually sign? Thanks

AdrianoDee commented 2 months ago

A note: I have few of these to show. With different modules crashing. I'm not sure what's the best way to document there. If in separate issues or in a central one. Will pause for the moment.

dan131riley commented 2 months ago

Probably the same as https://github.com/cms-sw/cmssw/issues/45967#issuecomment-2340717568. I'd guess a lot of them have DAClusterizerInZ_vect in another thread, with memory corruption from there likely the ultimate culprit.

AdrianoDee commented 2 months ago

Big facepalm to myself having fixed it in pre7 with https://github.com/cms-sw/cmssw/pull/45693. And thanks @dan131riley for having a look.