search

cms-sw / cmssw

CMS Offline Software
http://cms-sw.github.io/
Apache License 2.0
1.08k stars 4.3k forks source link

[AARCH64] Crashes - 1 - `MCTrackMatcher` #45967

Closed AdrianoDee closed 1 month ago

AdrianoDee commented 1 month ago

While producing samples for the future ARM validation campaign I see this error when running a step3 such as:

cmsDriver.py step2 --conditions auto:phase1_2024_realistic --datatier GEN-SIM-RECO,MINIAODSIM,NANOAODSIM,DQMIO --era Run3_2024 --eventcontent RECOSIM,MINIAODSIM,NANOEDMAODSIM,DQM --filein "dbs:/RelValSinglePiFlatPt0p7To10/CMSSW_14_1_0_pre6-PU_140X_mcRun3_2024_realistic_v15_STD_2024_PU-v1/GEN-SIM-DIGI-RAW" --fileout "file:step2.root" --geometry DB:Extended --nStreams 1 --nThreads 8 --no_exec --number 10 --python_filename step_2_cfg.py --step RAW2DIGI,L1Reco,RECO,RECOSIM,PAT,NANO,VALIDATION:@standardValidation+@miniAODValidation,DQM:@standardDQM+@ExtraHLT+@miniAODDQM+@nanoAODDQM

on top of GEN-SIM-DIGI-RAW produced on x86 we get:

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Tue Sep 10 11:39:08 AM CEST 2024
Thread 26 (Thread 0xfffef4ff9160 (LWP 1440558) "cmsRun"):
#0  0x0000ffffb8a1ddd8 in __futex_abstimed_wait_cancelable64 () from /lib64/libc.so.6
#1  0x0000ffffb8a209d0 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libc.so.6
#2  0x0000ffff7d7ba6a0 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libtensorflow_cc.so.2
#3  0x0000ffff7d7bad50 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libtensorflow_cc.so.2
#4  0x0000ffff7d7b829c in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libtensorflow_cc.so.2
#5  0x0000ffff7491014c in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libtensorflow_framework.so.2
#6  0x0000ffffb8a21698 in start_thread () from /lib64/libc.so.6
#7  0x0000ffffb8a8bbdc in thread_start () from /lib64/libc.so.6
Thread 25 (Thread 0xfffefeff9160 (LWP 1440557) "cmsRun"):
#0  0x0000ffffb8a1ddd8 in __futex_abstimed_wait_cancelable64 () from /lib64/libc.so.6
#1  0x0000ffffb8a209d0 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libc.so.6
#2  0x0000ffff7d7ba6a0 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libtensorflow_cc.so.2
#3  0x0000ffff7d7bad50 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libtensorflow_cc.so.2
#4  0x0000ffff7d7b829c in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libtensorflow_cc.so.2
#5  0x0000ffff7491014c in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libtensorflow_framework.so.2
#6  0x0000ffffb8a21698 in start_thread () from /lib64/libc.so.6
#7  0x0000ffffb8a8bbdc in thread_start () from /lib64/libc.so.6
Thread 24 (Thread 0xffff01908160 (LWP 1440556) "cmsRun"):
#0  0x0000ffffb8a1ddd8 in __futex_abstimed_wait_cancelable64 () from /lib64/libc.so.6
#1  0x0000ffffb8a209d0 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libc.so.6
#2  0x0000ffff7d7ba6a0 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libtensorflow_cc.so.2
#3  0x0000ffff7d7bad50 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libtensorflow_cc.so.2
#4  0x0000ffff7d7b829c in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libtensorflow_cc.so.2
#5  0x0000ffff7491014c in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libtensorflow_framework.so.2
#6  0x0000ffffb8a21698 in start_thread () from /lib64/libc.so.6
#7  0x0000ffffb8a8bbdc in thread_start () from /lib64/libc.so.6
Thread 23 (Thread 0xffff54729160 (LWP 1440536) "cmsRun"):
#0  0x0000ffffb8a1ddd8 in __futex_abstimed_wait_cancelable64 () from /lib64/libc.so.6
#1  0x0000ffffb8a29cd4 in __new_sem_wait_slow64.constprop.0 () from /lib64/libc.so.6
#2  0x0000ffffb2b3c550 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdCl.so.3
#3  0x0000ffffb2b3c62c in RunRunnerThread () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdCl.so.3
#4  0x0000ffffb8a21698 in start_thread () from /lib64/libc.so.6
#5  0x0000ffffb8a8bbdc in thread_start () from /lib64/libc.so.6
Thread 22 (Thread 0xffff54f39160 (LWP 1440535) "cmsRun"):
#0  0x0000ffffb8a1ddd8 in __futex_abstimed_wait_cancelable64 () from /lib64/libc.so.6
#1  0x0000ffffb8a29cd4 in __new_sem_wait_slow64.constprop.0 () from /lib64/libc.so.6
#2  0x0000ffffb2b3c550 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdCl.so.3
#3  0x0000ffffb2b3c62c in RunRunnerThread () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdCl.so.3
#4  0x0000ffffb8a21698 in start_thread () from /lib64/libc.so.6
#5  0x0000ffffb8a8bbdc in thread_start () from /lib64/libc.so.6
Thread 21 (Thread 0xffff55749160 (LWP 1440534) "cmsRun"):
#0  0x0000ffffb8a1ddd8 in __futex_abstimed_wait_cancelable64 () from /lib64/libc.so.6
#1  0x0000ffffb8a29cd4 in __new_sem_wait_slow64.constprop.0 () from /lib64/libc.so.6
#2  0x0000ffffb2b3c550 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdCl.so.3
#3  0x0000ffffb2b3c62c in RunRunnerThread () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdCl.so.3
#4  0x0000ffffb8a21698 in start_thread () from /lib64/libc.so.6
#5  0x0000ffffb8a8bbdc in thread_start () from /lib64/libc.so.6
Thread 20 (Thread 0xffff55f59160 (LWP 1440533) "cmsRun"):
#0  0x0000ffffb8a56514 in clock_nanosleep@@GLIBC_2.17 () from /lib64/libc.so.6
#1  0x0000ffffb8a5b64c in nanosleep () from /lib64/libc.so.6
#2  0x0000ffffb2c29b84 in XrdSysTimer::Wait(int) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x0000ffffb2ac704c in XrdCl::TaskManager::RunTasks() () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdCl.so.3
#4  0x0000ffffb2ac71bc in RunRunnerThread () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdCl.so.3
#5  0x0000ffffb8a21698 in start_thread () from /lib64/libc.so.6
#6  0x0000ffffb8a8bbdc in thread_start () from /lib64/libc.so.6
Thread 19 (Thread 0xffff56769160 (LWP 1440532) "cmsRun"):
#0  0x0000ffffb8a8bd7c in epoll_pwait () from /lib64/libc.so.6
#1  0x0000ffffb2c23e4c in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x0000ffffb2c1ff5c in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x0000ffffb2c29140 in XrdSysThread_Xeq () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#4  0x0000ffffb8a21698 in start_thread () from /lib64/libc.so.6
#5  0x0000ffffb8a8bbdc in thread_start () from /lib64/libc.so.6
Thread 18 (Thread 0xffff56f79160 (LWP 1440531) "cmsRun"):
#0  0x0000ffffb8a8bd7c in epoll_pwait () from /lib64/libc.so.6
#1  0x0000ffffb2c23e4c in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x0000ffffb2c1ff5c in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x0000ffffb2c29140 in XrdSysThread_Xeq () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#4  0x0000ffffb8a21698 in start_thread () from /lib64/libc.so.6
#5  0x0000ffffb8a8bbdc in thread_start () from /lib64/libc.so.6
Thread 17 (Thread 0xffff57789160 (LWP 1440530) "cmsRun"):
#0  0x0000ffffb8a8bd7c in epoll_pwait () from /lib64/libc.so.6
#1  0x0000ffffb2c23e4c in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x0000ffffb2c1ff5c in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x0000ffffb2c29140 in XrdSysThread_Xeq () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#4  0x0000ffffb8a21698 in start_thread () from /lib64/libc.so.6
#5  0x0000ffffb8a8bbdc in thread_start () from /lib64/libc.so.6
Thread 16 (Thread 0xffff57f99160 (LWP 1440529) "cmsRun"):
#0  0x0000ffffb8a8bd7c in epoll_pwait () from /lib64/libc.so.6
#1  0x0000ffffb2c23e4c in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x0000ffffb2c1ff5c in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x0000ffffb2c29140 in XrdSysThread_Xeq () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#4  0x0000ffffb8a21698 in start_thread () from /lib64/libc.so.6
#5  0x0000ffffb8a8bbdc in thread_start () from /lib64/libc.so.6
Thread 15 (Thread 0xffff587a9160 (LWP 1440528) "cmsRun"):
#0  0x0000ffffb8a8bd7c in epoll_pwait () from /lib64/libc.so.6
#1  0x0000ffffb2c23e4c in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x0000ffffb2c1ff5c in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x0000ffffb2c29140 in XrdSysThread_Xeq () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#4  0x0000ffffb8a21698 in start_thread () from /lib64/libc.so.6
#5  0x0000ffffb8a8bbdc in thread_start () from /lib64/libc.so.6
Thread 14 (Thread 0xffff58fb9160 (LWP 1440527) "cmsRun"):
#0  0x0000ffffb8a8bd7c in epoll_pwait () from /lib64/libc.so.6
#1  0x0000ffffb2c23e4c in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x0000ffffb2c1ff5c in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x0000ffffb2c29140 in XrdSysThread_Xeq () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#4  0x0000ffffb8a21698 in start_thread () from /lib64/libc.so.6
#5  0x0000ffffb8a8bbdc in thread_start () from /lib64/libc.so.6
Thread 13 (Thread 0xffff597c9160 (LWP 1440526) "cmsRun"):
#0  0x0000ffffb8a8bd7c in epoll_pwait () from /lib64/libc.so.6
#1  0x0000ffffb2c23e4c in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x0000ffffb2c1ff5c in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x0000ffffb2c29140 in XrdSysThread_Xeq () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#4  0x0000ffffb8a21698 in start_thread () from /lib64/libc.so.6
#5  0x0000ffffb8a8bbdc in thread_start () from /lib64/libc.so.6
Thread 12 (Thread 0xffff59fd9160 (LWP 1440525) "cmsRun"):
#0  0x0000ffffb8a8bd7c in epoll_pwait () from /lib64/libc.so.6
#1  0x0000ffffb2c23e4c in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x0000ffffb2c1ff5c in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x0000ffffb2c29140 in XrdSysThread_Xeq () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#4  0x0000ffffb8a21698 in start_thread () from /lib64/libc.so.6
#5  0x0000ffffb8a8bbdc in thread_start () from /lib64/libc.so.6
Thread 11 (Thread 0xffff5a7e9160 (LWP 1440524) "cmsRun"):
#0  0x0000ffffb8a8bd7c in epoll_pwait () from /lib64/libc.so.6
#1  0x0000ffffb2c23e4c in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x0000ffffb2c1ff5c in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x0000ffffb2c29140 in XrdSysThread_Xeq () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#4  0x0000ffffb8a21698 in start_thread () from /lib64/libc.so.6
#5  0x0000ffffb8a8bbdc in thread_start () from /lib64/libc.so.6
Thread 10 (Thread 0xffff5aff9160 (LWP 1440523) "cmsRun"):
#0  0x0000ffffb8a8bd7c in epoll_pwait () from /lib64/libc.so.6
#1  0x0000ffffb2c23e4c in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x0000ffffb2c1ff5c in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x0000ffffb2c29140 in XrdSysThread_Xeq () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/external/el9_aarch64_gcc12/lib/libXrdUtils.so.3
#4  0x0000ffffb8a21698 in start_thread () from /lib64/libc.so.6
#5  0x0000ffffb8a8bbdc in thread_start () from /lib64/libc.so.6
Thread 9 (Thread 0xffff63ff9160 (LWP 1440522) "cmsRun"):
#0  0x0000ffffb8a56514 in clock_nanosleep@@GLIBC_2.17 () from /lib64/libc.so.6
#1  0x0000ffffb8a5b64c in nanosleep () from /lib64/libc.so.6
#2  0x0000ffffb8a5b50c in sleep () from /lib64/libc.so.6
#3  0x0000ffffb3574cc8 in sig_pause_for_stacktrace () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x0000ffff10d75f84 in DeDxEstimatorProducer::processHit(TrackingRecHit const*, float, float&, std::vector<reco::DeDxHit, std::allocator<reco::DeDxHit> >&, int&) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginRecoTrackerDeDxPlugins.so
#6  0x0000ffff10d76b7c in DeDxEstimatorProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginRecoTrackerDeDxPlugins.so
#7  0x0000ffffbaa2ddc4 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#8  0x0000ffffbaa14900 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#9  0x0000ffffba9a00d8 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#10 0x0000ffffba9a1a34 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#11 0x0000ffffbab29acc in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreConcurrency.so
#12 0x0000ffffb8fbc70c in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0xfffebcb81d00, waiter=<synthetic pointer>..., this=0xffffb8333c00) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#13 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=<synthetic pointer>..., t=0x0, this=0xffffb8333c00) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#14 tbb::detail::r1::arena::process (tls=..., this=0xffffb8332f00) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/arena.cpp:137
#15 tbb::detail::r1::market::process (this=0xffffb834b580, j=...) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/market.cpp:599
#16 0x0000ffffb8fc4f5c in tbb::detail::r1::rml::private_worker::run (this=0xffffb3933f00) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#17 tbb::detail::r1::rml::private_worker::thread_routine (arg=0xffffb3933f00) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#18 0x0000ffffb8a21698 in start_thread () from /lib64/libc.so.6
#19 0x0000ffffb8a8bbdc in thread_start () from /lib64/libc.so.6
Thread 8 (Thread 0xffff65bd9160 (LWP 1440521) "cmsRun"):
#0  0x0000ffffb8a56514 in clock_nanosleep@@GLIBC_2.17 () from /lib64/libc.so.6
#1  0x0000ffffb8a5b64c in nanosleep () from /lib64/libc.so.6
#2  0x0000ffffb8a5b50c in sleep () from /lib64/libc.so.6
#3  0x0000ffffb3574cc8 in sig_pause_for_stacktrace () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x0000ffff2bf2ca64 in void std::__merge_sort_with_buffer<__gnu_cxx::__normal_iterator<std::pair<double, unsigned int>*, std::vector<std::pair<double, unsigned int>, std::allocator<std::pair<double, unsigned int> > > >, std::pair<double, unsigned int>*, __gnu_cxx::__ops::_Iter_comp_iter<std::less<std::pair<double, unsigned int> > > >(__gnu_cxx::__normal_iterator<std::pair<double, unsigned int>*, std::vector<std::pair<double, unsigned int>, std::allocator<std::pair<double, unsigned int> > > >, __gnu_cxx::__normal_iterator<std::pair<double, unsigned int>*, std::vector<std::pair<double, unsigned int>, std::allocator<std::pair<double, unsigned int> > > >, std::pair<double, unsigned int>*, __gnu_cxx::__ops::_Iter_comp_iter<std::less<std::pair<double, unsigned int> > >) [clone .isra.0] () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libRecoVertexPrimaryVertexProducer.so
#6  0x0000ffff2bf356b0 in void std::__stable_sort_adaptive<__gnu_cxx::__normal_iterator<std::pair<double, unsigned int>*, std::vector<std::pair<double, unsigned int>, std::allocator<std::pair<double, unsigned int> > > >, std::pair<double, unsigned int>*, long, __gnu_cxx::__ops::_Iter_comp_iter<std::less<std::pair<double, unsigned int> > > >(__gnu_cxx::__normal_iterator<std::pair<double, unsigned int>*, std::vector<std::pair<double, unsigned int>, std::allocator<std::pair<double, unsigned int> > > >, __gnu_cxx::__normal_iterator<std::pair<double, unsigned int>*, std::vector<std::pair<double, unsigned int>, std::allocator<std::pair<double, unsigned int> > > >, std::pair<double, unsigned int>*, long, __gnu_cxx::__ops::_Iter_comp_iter<std::less<std::pair<double, unsigned int> > >) [clone .isra.0] () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libRecoVertexPrimaryVertexProducer.so
#7  0x0000ffff2bf1c988 in DAClusterizerInZ_vect::merge(DAClusterizerInZ_vect::vertex_t&, DAClusterizerInZ_vect::track_t&, double&) const () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libRecoVertexPrimaryVertexProducer.so
#8  0x0000ffff2bf216b0 in DAClusterizerInZ_vect::vertices_no_blocks(std::vector<reco::TransientTrack, std::allocator<reco::TransientTrack> > const&) const () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libRecoVertexPrimaryVertexProducer.so
#9  0x0000ffff2bf219a4 in DAClusterizerInZ_vect::vertices(std::vector<reco::TransientTrack, std::allocator<reco::TransientTrack> > const&) const () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libRecoVertexPrimaryVertexProducer.so
#10 0x0000fffefe069160 in PrimaryVertexProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginRecoVertexPrimaryVertexProducerPlugins.so
#11 0x0000ffffbaa2ddc4 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#12 0x0000ffffbaa14900 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#13 0x0000ffffba9a00d8 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#14 0x0000ffffba9a1a34 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#15 0x0000ffffbab29acc in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreConcurrency.so
#16 0x0000ffffb8fbc70c in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0xfffebcb8c500, waiter=<synthetic pointer>..., this=0xffffb8333c80) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#17 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=<synthetic pointer>..., t=0x0, this=0xffffb8333c80) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#18 tbb::detail::r1::arena::process (tls=..., this=0xffffb8332f00) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/arena.cpp:137
#19 tbb::detail::r1::market::process (this=0xffffb834b580, j=...) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/market.cpp:599
#20 0x0000ffffb8fc4f5c in tbb::detail::r1::rml::private_worker::run (this=0xffffb3933e00) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#21 tbb::detail::r1::rml::private_worker::thread_routine (arg=0xffffb3933e00) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#22 0x0000ffffb8a21698 in start_thread () from /lib64/libc.so.6
#23 0x0000ffffb8a8bbdc in thread_start () from /lib64/libc.so.6
Thread 7 (Thread 0xffff665e9160 (LWP 1440520) "cmsRun"):
#0  0x0000ffffb8a81960 in poll () from /lib64/libc.so.6
#1  0x0000ffffb3577ca0 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x0000ffffb3577ed4 in sig_dostack_then_abort () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000ffffba9b7594 in edm::UnscheduledProductResolver::resolveProduct_(edm::Principal const&, bool, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#5  0x0000ffffba9acbec in edm::Principal::getByToken(edm::KindOfType, edm::TypeID const&, unsigned int, bool, bool&, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#6  0x0000ffffba9b1324 in edm::PrincipalGetAdapter::getByToken_(edm::TypeID const&, edm::KindOfType, edm::EDGetToken, edm::ModuleCallingContext const*) const () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#7  0x0000ffff26587ca8 in bool edm::Event::getByToken<reco::TrackToTrackingParticleAssociator>(edm::EDGetTokenT<reco::TrackToTrackingParticleAssociator>, edm::Handle<reco::TrackToTrackingParticleAssociator>&) const () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginSimTrackerTrackAssociation_plugins.so
#8  0x0000ffff26589c78 in MCTrackMatcher::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginSimTrackerTrackAssociation_plugins.so
#9  0x0000ffffbaa1ac28 in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#10 0x0000ffffbaa13dd0 in edm::WorkerT<edm::global::EDProducerBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#11 0x0000ffffba9a00d8 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#12 0x0000ffffba9a1a34 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#13 0x0000ffffbab29acc in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreConcurrency.so
#14 0x0000ffffb8fbc70c in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0xfffebcb83300, waiter=<synthetic pointer>..., this=0xffffb8333a80) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#15 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=<synthetic pointer>..., t=0x0, this=0xffffb8333a80) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#16 tbb::detail::r1::arena::process (tls=..., this=0xffffb8332f00) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/arena.cpp:137
#17 tbb::detail::r1::market::process (this=0xffffb834b580, j=...) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/market.cpp:599
#18 0x0000ffffb8fc4f5c in tbb::detail::r1::rml::private_worker::run (this=0xffffb3933e80) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0xffffb3933e80) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#20 0x0000ffffb8a21698 in start_thread () from /lib64/libc.so.6
#21 0x0000ffffb8a8bbdc in thread_start () from /lib64/libc.so.6
Thread 6 (Thread 0xffff68ff9160 (LWP 1440519) "cmsRun"):
#0  0x0000ffffb8a56514 in clock_nanosleep@@GLIBC_2.17 () from /lib64/libc.so.6
#1  0x0000ffffb8a5b64c in nanosleep () from /lib64/libc.so.6
#2  0x0000ffffb8a5b50c in sleep () from /lib64/libc.so.6
#3  0x0000ffffb3574cc8 in sig_pause_for_stacktrace () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x0000ffff730becdc in TrajectoryStateCombiner::combine(TrajectoryStateOnSurface const&, TrajectoryStateOnSurface const&) const () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libTrackingToolsTrackFitters.so
#6  0x0000ffff730c18d4 in KFTrajectorySmoother::trajectory(Trajectory const&) const () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libTrackingToolsTrackFitters.so
#7  0x0000ffff730fc808 in (anonymous namespace)::KFFittingSmoother::smoothingStep(Trajectory&&) const () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginTrackingToolsTrackFittersPlugins.so
#8  0x0000ffff73100b54 in (anonymous namespace)::KFFittingSmoother::fitOne(TrajectorySeed const&, std::vector<std::shared_ptr<TrackingRecHit const>, std::allocator<std::shared_ptr<TrackingRecHit const> > > const&, TrajectoryStateOnSurface const&, TrajectoryFitter::fitType) const () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginTrackingToolsTrackFittersPlugins.so
#9  0x0000ffff730fa2a8 in (anonymous namespace)::FlexibleKFFittingSmoother::fitOne(TrajectorySeed const&, std::vector<std::shared_ptr<TrackingRecHit const>, std::allocator<std::shared_ptr<TrackingRecHit const> > > const&, TrajectoryStateOnSurface const&, TrajectoryFitter::fitType) const () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginTrackingToolsTrackFittersPlugins.so
#10 0x0000ffff15778ce0 in TrackProducerAlgorithm<reco::Track>::buildTrack(TrajectoryFitter const*, Propagator const*, std::vector<AlgoProductTraits<reco::Track>::AlgoProduct, std::allocator<AlgoProductTraits<reco::Track>::AlgoProduct> >&, std::vector<std::shared_ptr<TrackingRecHit const>, std::allocator<std::shared_ptr<TrackingRecHit const> > >&, TrajectoryStateOnSurface&, TrajectorySeed const&, float, reco::BeamSpot const&, edm::RefToBase<TrajectorySeed>, int, signed char) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libRecoTrackerTrackProducer.so
#11 0x0000ffff15882420 in TrackProducerAlgorithm<reco::Track>::runWithTrack(TrackingGeometry const*, MagneticField const*, edm::View<reco::Track> const&, TrajectoryFitter const*, Propagator const*, TransientTrackingRecHitBuilder const*, reco::BeamSpot const&, std::vector<AlgoProductTraits<reco::Track>::AlgoProduct, std::allocator<AlgoProductTraits<reco::Track>::AlgoProduct> >&) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginRecoTrackerTrackProducerPlugins.so
#12 0x0000ffff15885b6c in TrackRefitter::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginRecoTrackerTrackProducerPlugins.so
#13 0x0000ffffbaa2ddc4 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#14 0x0000ffffbaa14900 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#15 0x0000ffffba9a00d8 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#16 0x0000ffffba9a1a34 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#17 0x0000ffffbab29acc in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreConcurrency.so
#18 0x0000ffffb8fbc70c in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0xfffebcb86500, waiter=<synthetic pointer>..., this=0xffffb8333a00) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#19 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=<synthetic pointer>..., t=0x0, this=0xffffb8333a00) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#20 tbb::detail::r1::arena::process (tls=..., this=0xffffb8332f00) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/arena.cpp:137
#21 tbb::detail::r1::market::process (this=0xffffb834b580, j=...) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/market.cpp:599
#22 0x0000ffffb8fc4f5c in tbb::detail::r1::rml::private_worker::run (this=0xffffb3934000) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#23 tbb::detail::r1::rml::private_worker::thread_routine (arg=0xffffb3934000) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#24 0x0000ffffb8a21698 in start_thread () from /lib64/libc.so.6
#25 0x0000ffffb8a8bbdc in thread_start () from /lib64/libc.so.6
Thread 5 (Thread 0xffff6cdfc160 (LWP 1440518) "cmsRun"):
#0  0x0000ffffb8a56514 in clock_nanosleep@@GLIBC_2.17 () from /lib64/libc.so.6
#1  0x0000ffffb8a5b64c in nanosleep () from /lib64/libc.so.6
#2  0x0000ffffb8a5b50c in sleep () from /lib64/libc.so.6
#3  0x0000ffffb3574cc8 in sig_pause_for_stacktrace () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x0000ffff159027ec in void SiStripMonitorTrack::RecHitInfo<SiStripRecHit2D>(SiStripRecHit2D const*, Vector3DBase<float, LocalTag>, edm::DetSetVector<SiStripDigi> const&, edm::Event const&, bool) [clone .constprop.0] () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginDQMSiStripMonitorTrack.so
#6  0x0000ffff158f3330 in SiStripMonitorTrack::trajectoryStudy(reco::Track const&, edm::DetSetVector<SiStripDigi> const&, edm::Event const&, bool) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginDQMSiStripMonitorTrack.so
#7  0x0000ffff158f38bc in SiStripMonitorTrack::trackStudyFromTrajectory(edm::Handle<std::vector<reco::Track, std::allocator<reco::Track> > >, edm::DetSetVector<SiStripDigi> const&, edm::Event const&) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginDQMSiStripMonitorTrack.so
#8  0x0000ffff158f3cb8 in SiStripMonitorTrack::trackStudy(edm::Event const&) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginDQMSiStripMonitorTrack.so
#9  0x0000ffff158ec1dc in SiStripMonitorTrack::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginDQMSiStripMonitorTrack.so
#10 0x0000ffffbaa2ddc4 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#11 0x0000ffffbaa14900 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#12 0x0000ffffba9a00d8 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#13 0x0000ffffba9a1a34 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#14 0x0000ffffbab29acc in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreConcurrency.so
#15 0x0000ffffb8fbc70c in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0xfffebcb83500, waiter=<synthetic pointer>..., this=0xffffb8333b00) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#16 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=<synthetic pointer>..., t=0x0, this=0xffffb8333b00) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#17 tbb::detail::r1::arena::process (tls=..., this=0xffffb8332f00) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/arena.cpp:137
#18 tbb::detail::r1::market::process (this=0xffffb834b580, j=...) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/market.cpp:599
#19 0x0000ffffb8fc4f5c in tbb::detail::r1::rml::private_worker::run (this=0xffffb3933f80) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#20 tbb::detail::r1::rml::private_worker::thread_routine (arg=0xffffb3933f80) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#21 0x0000ffffb8a21698 in start_thread () from /lib64/libc.so.6
#22 0x0000ffffb8a8bbdc in thread_start () from /lib64/libc.so.6
Thread 4 (Thread 0xffff6d80c160 (LWP 1440517) "cmsRun"):
#0  0x0000ffffb8a56514 in clock_nanosleep@@GLIBC_2.17 () from /lib64/libc.so.6
#1  0x0000ffffb8a5b64c in nanosleep () from /lib64/libc.so.6
#2  0x0000ffffb8a5b50c in sleep () from /lib64/libc.so.6
#3  0x0000ffffb3574cc8 in sig_pause_for_stacktrace () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x0000ffff265f05d4 in ClusterTPAssociation::equal_range(OmniClusterRef const&) const () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginSimTrackerTrackAssociatorProducersAuto.so
#6  0x0000ffff2661d1c8 in std::vector<std::pair<edm::Ref<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle, edm::refhelper::FindUsingAdvance<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle> >, double>, std::allocator<std::pair<edm::Ref<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle, edm::refhelper::FindUsingAdvance<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle> >, double> > > QuickTrackAssociatorByHitsImpl::associateTrack<edm::Handle<std::vector<TrackingParticle, std::allocator<TrackingParticle> > >, __gnu_cxx::__normal_iterator<TrackingRecHit* const*, std::vector<TrackingRecHit*, std::allocator<TrackingRecHit*> > > >(ClusterTPAssociation const&, edm::Handle<std::vector<TrackingParticle, std::allocator<TrackingParticle> > > const&, edm::IndexSet const*, __gnu_cxx::__normal_iterator<TrackingRecHit* const*, std::vector<TrackingRecHit*, std::allocator<TrackingRecHit*> > >, __gnu_cxx::__normal_iterator<TrackingRecHit* const*, std::vector<TrackingRecHit*, std::allocator<TrackingRecHit*> > >) const [clone .constprop.1] () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginSimTrackerTrackAssociatorProducersAuto.so
#7  0x0000ffff265f0f34 in QuickTrackAssociatorByHitsImpl::associateRecoToSim(edm::RefToBaseVector<reco::Track> const&, edm::RefVector<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle, edm::refhelper::FindUsingAdvance<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle> > const&) const () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginSimTrackerTrackAssociatorProducersAuto.so
#8  0x0000ffff3030875c in MultiTrackValidator::dqmAnalyze(edm::Event const&, edm::EventSetup const&, MultiTrackValidatorHistograms const&) const () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginValidationRecoTrackPlugins.so
#9  0x0000ffffbaa1ac28 in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#10 0x0000ffffbaa13dd0 in edm::WorkerT<edm::global::EDProducerBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#11 0x0000ffffba9a00d8 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#12 0x0000ffffba9a1a34 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#13 0x0000ffffbab29acc in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreConcurrency.so
#14 0x0000ffffb8fbc70c in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0xfffebcb83800, waiter=<synthetic pointer>..., this=0xffffb8333b80) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#15 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=<synthetic pointer>..., t=0x0, this=0xffffb8333b80) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#16 tbb::detail::r1::arena::process (tls=..., this=0xffffb8332f00) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/arena.cpp:137
#17 tbb::detail::r1::market::process (this=0xffffb834b580, j=...) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/market.cpp:599
#18 0x0000ffffb8fc4f5c in tbb::detail::r1::rml::private_worker::run (this=0xffffb3934100) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0xffffb3934100) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#20 0x0000ffffb8a21698 in start_thread () from /lib64/libc.so.6
#21 0x0000ffffb8a8bbdc in thread_start () from /lib64/libc.so.6
Thread 3 (Thread 0xffff6e21c160 (LWP 1440516) "cmsRun"):
#0  0x0000ffffb8a56514 in clock_nanosleep@@GLIBC_2.17 () from /lib64/libc.so.6
#1  0x0000ffffb8a5b64c in nanosleep () from /lib64/libc.so.6
#2  0x0000ffffb8a5b50c in sleep () from /lib64/libc.so.6
#3  0x0000ffffb3574cc8 in sig_pause_for_stacktrace () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x0000ffff10d76b48 in DeDxEstimatorProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginRecoTrackerDeDxPlugins.so
#6  0x0000ffffbaa2ddc4 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#7  0x0000ffffbaa14900 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#8  0x0000ffffba9a00d8 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#9  0x0000ffffba9a1a34 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#10 0x0000ffffbab29acc in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreConcurrency.so
#11 0x0000ffffb8fbc70c in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0xfffebcb8ca00, waiter=<synthetic pointer>..., this=0xffffb8333980) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#12 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=<synthetic pointer>..., t=0x0, this=0xffffb8333980) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#13 tbb::detail::r1::arena::process (tls=..., this=0xffffb8332f00) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/arena.cpp:137
#14 tbb::detail::r1::market::process (this=0xffffb834b580, j=...) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/market.cpp:599
#15 0x0000ffffb8fc4f5c in tbb::detail::r1::rml::private_worker::run (this=0xffffb3934080) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#16 tbb::detail::r1::rml::private_worker::thread_routine (arg=0xffffb3934080) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#17 0x0000ffffb8a21698 in start_thread () from /lib64/libc.so.6
#18 0x0000ffffb8a8bbdc in thread_start () from /lib64/libc.so.6
Thread 2 (Thread 0xffffaf5f9160 (LWP 1440455) "cmsRun"):
#0  0x0000ffffb8a5b2c0 in wait4 () from /lib64/libc.so.6
#1  0x0000ffffb3574e34 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x0000ffffb3577a88 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x0000ffffb8cce86c in std::execute_native_thread_routine (__p=0xffffb60adee0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x0000ffffb8a21698 in start_thread () from /lib64/libc.so.6
#5  0x0000ffffb8a8bbdc in thread_start () from /lib64/libc.so.6
Thread 1 (Thread 0xffffba05c020 (LWP 1440423) "cmsRun"):
#0  0x0000ffffb8a56514 in clock_nanosleep@@GLIBC_2.17 () from /lib64/libc.so.6
#1  0x0000ffffb8a5b64c in nanosleep () from /lib64/libc.so.6
#2  0x0000ffffb8a5b50c in sleep () from /lib64/libc.so.6
#3  0x0000ffffb3574cc8 in sig_pause_for_stacktrace () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  cache_bin_full (bin=<optimized out>) at include/jemalloc/internal/cache_bin.h:427
#6  cache_bin_dalloc_easy (ptr=0xfffda86a12b0, bin=<optimized out>) at include/jemalloc/internal/cache_bin.h:435
#7  free_fastpath (size_hint=true, size=16, ptr=0xfffda86a12b0) at src/jemalloc.c:3148
#8  je_je_sdallocx_noflags (ptr=0xfffda86a12b0, size=16) at src/jemalloc.c:3950
#9  0x0000ffffb9945120 in sizedDeleteImpl (size=<optimized out>, ptr=<optimized out>) at src/jemalloc_cpp.cpp:195
#10 operator delete (ptr=<optimized out>, size=<optimized out>) at src/jemalloc_cpp.cpp:200
#11 0x0000ffff265f0c98 in double QuickTrackAssociatorByHitsImpl::weightedNumberOfTrackClusters<reco::Track>(reco::Track const&, ClusterTPAssociation const&) const () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginSimTrackerTrackAssociatorProducersAuto.so
#12 0x0000ffff265f0f64 in QuickTrackAssociatorByHitsImpl::associateRecoToSim(edm::RefToBaseVector<reco::Track> const&, edm::RefVector<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle, edm::refhelper::FindUsingAdvance<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle> > const&) const () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginSimTrackerTrackAssociatorProducersAuto.so
#13 0x0000ffff3030875c in MultiTrackValidator::dqmAnalyze(edm::Event const&, edm::EventSetup const&, MultiTrackValidatorHistograms const&) const () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/pluginValidationRecoTrackPlugins.so
#14 0x0000ffffbaa1ac28 in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#15 0x0000ffffbaa13dd0 in edm::WorkerT<edm::global::EDProducerBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#16 0x0000ffffba9a00d8 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#17 0x0000ffffba9a1a34 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#18 0x0000ffffbab29acc in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreConcurrency.so
#19 0x0000ffffb8fc275c in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0xfffebcba0f00, this=0xffffb8333900) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#20 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0xffffb8333900) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#21 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#22 0x0000ffffba9172c4 in edm::FinalWaitingTask::wait() () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#23 0x0000ffffba923060 in edm::EventProcessor::processRuns() () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#24 0x0000ffffba9234bc in edm::EventProcessor::runToCompletion() () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#25 0x0000000000407b1c in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#26 0x0000ffffb8fba9d8 in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-59c55d75b798f530d4f432fa52073ffc/tbb-v2021.9.0/src/tbb/arena.cpp:688
#27 0x00000000004093d0 in main::{lambda()#1}::operator()() const ()
#28 0x00000000004044c4 in main ()

Current Modules:

Module: MCTrackMatcher:prunedTrackMCMatch (crashed)
Module: MultiTrackValidator:trackValidatorBuilding
Module: MultiTrackValidator:trackValidatorBuildingPreSplitting
Module: DeDxEstimatorProducer:dedxPixelHarmonic2
Module: SiStripMonitorTrack:SiStripMonitorTrackCommon
Module: PrimaryVertexProducer:unsortedOfflinePrimaryVertices
Module: TrackRefitter:refittedForPixelDQM
Module: DeDxEstimatorProducer:dedxTruncated40

on x86 the same job ends successfully. The rationale behind running an ARM job on top of an x86 one is that we want to have PU samples with exactly the same events to validate the "new" CPU architecture. Here the reproducers:

(ARM -> tested on lxplus-arm)

export SCRAM_ARCH=el9_aarch64_gcc12
scram p -n cmssw_arm CMSSW_14_1_0_pre6
cd cmssw_arm/src
cmsenv
wget https://adiflori.web.cern.ch/adiflori/arm/error_1/error_arm_MCTrackMatcher.py
cmsRun error_arm_MCTrackMatcher.py

(x86 -> tested on lxplus)

export SCRAM_ARCH=el9_amd64_gcc12
scram p -n cmssw_x86 CMSSW_14_1_0_pre6
cd cmssw_x86/src
cmsenv
wget https://adiflori.web.cern.ch/adiflori/arm/error_1/error_arm_MCTrackMatcher.py
cmsRun error_arm_MCTrackMatcher.py
cmsbuild commented 1 month ago

cms-bot internal usage

cmsbuild commented 1 month ago

A new Issue was created by @AdrianoDee.

@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

AdrianoDee commented 1 month ago

Not sure who to assign this.

mandrenguyen commented 1 month ago

assign SimTracker/TrackAssociation I think we can just assign by package

cmsbuild commented 1 month ago

New categories assigned: simulation

@civanch,@mdhildreth you have been requested to review this Pull request/Issue and eventually sign? Thanks

mmusich commented 1 month ago

@elusian FYI

Module: MCTrackMatcher:prunedTrackMCMatch was added (3 years ago!) in https://github.com/cms-sw/cmssw/pull/33774

dan131riley commented 1 month ago

Your stack trace has DAClusterizerInZ_vect running at the same time in another thread. This routine has a known memory corruption issue, see #45693 and #45464. Can you switch to CMSSW_14_1_0_pre7, which has the fix in #45693?

#5  0x0000ffff2bf2ca64 in void std::__merge_sort_with_buffer<__gnu_cxx::__normal_iterator<std::pair<double, unsigned int>*, std::vector<std::pair<double, unsigned int>, std::allocator<std::pair<double, unsigned int> > > >, std::pair<double, unsigned int>*, __gnu_cxx::__ops::_Iter_comp_iter<std::less<std::pair<double, unsigned int> > > >(__gnu_cxx::__normal_iterator<std::pair<double, unsigned int>*, std::vector<std::pair<double, unsigned int>, std::allocator<std::pair<double, unsigned int> > > >, __gnu_cxx::__normal_iterator<std::pair<double, unsigned int>*, std::vector<std::pair<double, unsigned int>, std::allocator<std::pair<double, unsigned int> > > >, std::pair<double, unsigned int>*, __gnu_cxx::__ops::_Iter_comp_iter<std::less<std::pair<double, unsigned int> > >) [clone .isra.0] () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libRecoVertexPrimaryVertexProducer.so
#6  0x0000ffff2bf356b0 in void std::__stable_sort_adaptive<__gnu_cxx::__normal_iterator<std::pair<double, unsigned int>*, std::vector<std::pair<double, unsigned int>, std::allocator<std::pair<double, unsigned int> > > >, std::pair<double, unsigned int>*, long, __gnu_cxx::__ops::_Iter_comp_iter<std::less<std::pair<double, unsigned int> > > >(__gnu_cxx::__normal_iterator<std::pair<double, unsigned int>*, std::vector<std::pair<double, unsigned int>, std::allocator<std::pair<double, unsigned int> > > >, __gnu_cxx::__normal_iterator<std::pair<double, unsigned int>*, std::vector<std::pair<double, unsigned int>, std::allocator<std::pair<double, unsigned int> > > >, std::pair<double, unsigned int>*, long, __gnu_cxx::__ops::_Iter_comp_iter<std::less<std::pair<double, unsigned int> > >) [clone .isra.0] () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libRecoVertexPrimaryVertexProducer.so
#7  0x0000ffff2bf1c988 in DAClusterizerInZ_vect::merge(DAClusterizerInZ_vect::vertex_t&, DAClusterizerInZ_vect::track_t&, double&) const () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libRecoVertexPrimaryVertexProducer.so
#8  0x0000ffff2bf216b0 in DAClusterizerInZ_vect::vertices_no_blocks(std::vector<reco::TransientTrack, std::allocator<reco::TransientTrack> > const&) const () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libRecoVertexPrimaryVertexProducer.so
#9  0x0000ffff2bf219a4 in DAClusterizerInZ_vect::vertices(std::vector<reco::TransientTrack, std::allocator<reco::TransientTrack> > const&) const () from /cvmfs/cms.cern.ch/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_0_pre6/lib/el9_aarch64_gcc12/libRecoVertexPrimaryVertexProducer.so
AdrianoDee commented 1 month ago

Thanks @dan131riley. That was the culprit. Also ... I should know it. Sorry for the noise!