search

cms-sw / cmssw

CMS Offline Software
http://cms-sw.github.io/
Apache License 2.0
1.07k stars 4.28k forks source link

Crash in Phase-2 L1TrackJetEmulatorProducer:l1tTrackJetsEmulation #43735

Closed srimanob closed 7 months ago

srimanob commented 7 months ago

I see that we have a crash in L1TrackJetEmulatorProducer:l1tTrackJetsEmulation when I try to run D99 to debug another issue. The problematic workflow is the default D99 with DDD, 25234.0.

The release I used is CMSSW_14_0_X_2024-01-16-2300

Here is the recipe on how to create the crash:

Input: /eos/cms/store/group/offcomp_upgrade-sw/srimanob/D99/HGCAL-DEBUG/GENSIM_D99_20004.root cmsDriver: cmsDriver.py step2 -s DIGI:pdigi_valid,L1TrackTrigger,L1,DIGI2RAW,HLT:@relval2026 --conditions auto:phase2_realistic_T25 --datatier GEN-SIM-DIGI-RAW -n 10 --eventcontent FEVTDEBUGHLT --geometry Extended2026D99 --era Phase2C17I13M9 --python step2_D99_DIGIHLT.py --no_exec --filein file:step1_D99.root --fileout file:step2_D99.root --nThreads 8 --nStreams 1 --customise SLHCUpgradeSimulations/Configuration/aging.customise_aging_1000 --customise_commands "process.source.firstEvent = cms.untracked.uint32(155)"

Note that

So I am not so sure it is new to us (what come between 13_3_0_pre4 and now), because D99 is not a baseline yet, so we don't run it regularly. Also not sure if it comes from update of HGCal (change of detector between D98 and D99) or L1T.

cmsbuild commented 7 months ago

cms-bot internal usage

cmsbuild commented 7 months ago

A new Issue was created by @srimanob Phat Srimanobhas.

@makortel, @sextonkennedy, @smuzaffar, @antoniovilela, @Dr15Jones, @rappoccio can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

makortel commented 7 months ago

assign upgrade, l1

makortel commented 7 months ago

@srimanob Could you share the symptoms of the crash?

cmsbuild commented 7 months ago

New categories assigned: upgrade,l1

@epalencia,@aloeliger,@srimanob,@subirsarkar you have been requested to review this Pull request/Issue and eventually sign? Thanks

srimanob commented 7 months ago 
Thread 26 (Thread 0x7fe278dff700 (LWP 10212) "cmsRun"):
#0  0x00007fe315d8fa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe2db4d9c4e in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libtensorflow_cc.so.2
#2  0x00007fe2db4da223 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libtensorflow_cc.so.2
#3  0x00007fe2db4d7978 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libtensorflow_cc.so.2
#4  0x00007fe2ce2492a2 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libtensorflow_framework.so.2
#5  0x00007fe315d8bea5 in start_thread () from /lib64/libpthread.so.0
#6  0x00007fe315ab3b0d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7fe279bff700 (LWP 10211) "cmsRun"):
#0  0x00007fe315d8fa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe2db4d9c4e in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libtensorflow_cc.so.2
#2  0x00007fe2db4da223 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libtensorflow_cc.so.2
#3  0x00007fe2db4d7978 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libtensorflow_cc.so.2
#4  0x00007fe2ce2492a2 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libtensorflow_framework.so.2
#5  0x00007fe315d8bea5 in start_thread () from /lib64/libpthread.so.0
#6  0x00007fe315ab3b0d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7fe27a9a1700 (LWP 10210) "cmsRun"):
#0  0x00007fe315d8fa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe2db4d9c4e in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libtensorflow_cc.so.2
#2  0x00007fe2db4da223 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libtensorflow_cc.so.2
#3  0x00007fe2db4d7978 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libtensorflow_cc.so.2
#4  0x00007fe2ce2492a2 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libtensorflow_framework.so.2
#5  0x00007fe315d8bea5 in start_thread () from /lib64/libpthread.so.0
#6  0x00007fe315ab3b0d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7fe2b5ff2700 (LWP 10196) "cmsRun"):
#0  0x00007fe315d91b3b in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x00007fe315d91bcf in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x00007fe315d91c6b in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3  0x00007fe30e2f77a6 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libXrdCl.so.3
#4  0x00007fe30e2f7859 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libXrdCl.so.3
#5  0x00007fe315d8bea5 in start_thread () from /lib64/libpthread.so.0
#6  0x00007fe315ab3b0d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7fe2b67f3700 (LWP 10195) "cmsRun"):
#0  0x00007fe315d91b3b in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x00007fe315d91bcf in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x00007fe315d91c6b in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3  0x00007fe30e2f77a6 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libXrdCl.so.3
#4  0x00007fe30e2f7859 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libXrdCl.so.3
#5  0x00007fe315d8bea5 in start_thread () from /lib64/libpthread.so.0
#6  0x00007fe315ab3b0d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7fe2b6ff4700 (LWP 10194) "cmsRun"):
#0  0x00007fe315d91b3b in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x00007fe315d91bcf in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x00007fe315d91c6b in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3  0x00007fe30e2f77a6 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libXrdCl.so.3
#4  0x00007fe30e2f7859 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libXrdCl.so.3
#5  0x00007fe315d8bea5 in start_thread () from /lib64/libpthread.so.0
#6  0x00007fe315ab3b0d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7fe2b77f5700 (LWP 10193) "cmsRun"):
#0  0x00007fe315d92e9d in nanosleep () from /lib64/libpthread.so.0
#1  0x00007fe30f2a4df8 in XrdSysTimer::Wait(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00007fe30e26c0ac in XrdCl::TaskManager::RunTasks() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libXrdCl.so.3
#3  0x00007fe30e26c1e9 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libXrdCl.so.3
#4  0x00007fe315d8bea5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fe315ab3b0d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7fe2b7ff6700 (LWP 10192) "cmsRun"):
#0  0x00007fe315ab40e3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fe30f29f172 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00007fe30f29b0dd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x00007fe30f2a4507 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x00007fe315d8bea5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fe315ab3b0d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7fe2b87f7700 (LWP 10191) "cmsRun"):
#0  0x00007fe315ab40e3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fe30f29f172 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00007fe30f29b0dd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x00007fe30f2a4507 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x00007fe315d8bea5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fe315ab3b0d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7fe2b8ff8700 (LWP 10190) "cmsRun"):
#0  0x00007fe315ab40e3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fe30f29f172 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00007fe30f29b0dd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x00007fe30f2a4507 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x00007fe315d8bea5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fe315ab3b0d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7fe2b97f9700 (LWP 10189) "cmsRun"):
#0  0x00007fe315ab40e3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fe30f29f172 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00007fe30f29b0dd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x00007fe30f2a4507 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x00007fe315d8bea5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fe315ab3b0d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7fe2b9ffa700 (LWP 10188) "cmsRun"):
#0  0x00007fe315ab40e3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fe30f29f172 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00007fe30f29b0dd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x00007fe30f2a4507 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x00007fe315d8bea5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fe315ab3b0d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7fe2ba7fb700 (LWP 10187) "cmsRun"):
#0  0x00007fe315ab40e3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fe30f29f172 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00007fe30f29b0dd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x00007fe30f2a4507 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x00007fe315d8bea5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fe315ab3b0d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7fe2baffc700 (LWP 10186) "cmsRun"):
#0  0x00007fe315ab40e3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fe30f29f172 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00007fe30f29b0dd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x00007fe30f2a4507 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x00007fe315d8bea5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fe315ab3b0d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7fe2bb7fd700 (LWP 10185) "cmsRun"):
#0  0x00007fe315ab40e3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fe30f29f172 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00007fe30f29b0dd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x00007fe30f2a4507 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x00007fe315d8bea5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fe315ab3b0d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7fe2bbffe700 (LWP 10184) "cmsRun"):
#0  0x00007fe315ab40e3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fe30f29f172 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00007fe30f29b0dd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x00007fe30f2a4507 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x00007fe315d8bea5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fe315ab3b0d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fe2bc7ff700 (LWP 10183) "cmsRun"):
#0  0x00007fe315ab40e3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fe30f29f172 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00007fe30f29b0dd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x00007fe30f2a4507 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/external/slc7_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x00007fe315d8bea5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fe315ab3b0d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fe2bd7fc700 (LWP 10182) "cmsRun"):
#0  0x00007fe315a7a9fd in nanosleep () from /lib64/libc.so.6
#1  0x00007fe315a7a894 in sleep () from /lib64/libc.so.6
#2  0x00007fe30db24d30 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007fe3163d64f5 in std::basic_ios<char, std::char_traits<char> >::init (this=0x7fe2bd7f4dd0, __sb=0x7fe2bd7f4d68) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/bootstrap/BUILD/slc7_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/gcc-12.3.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/basic_ios.tcc:153
#5  0x00007fe2e525153e in HGCalDDDConstants::locateCell(int, int, int, int, int, int, bool, bool, bool, bool) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libGeometryHGCalCommonData.so
#6  0x00007fe2e548a14d in HGCalGeometry::getPosition(DetId const&, bool) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libGeometryHGCalGeometry.so
#7  0x00007fe21a5ce03a in HGCalTriggerGeometryV9Imp3::getModulePosition(unsigned int) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/pluginL1TriggerL1THGCalPlugins_geometries.so
#8  0x00007fe2e0d167d2 in HGCalConcentratorTrigSumImpl::doSum(unsigned int, std::vector<l1t::HGCalTriggerCell, std::allocator<l1t::HGCalTriggerCell> > const&, std::vector<l1t::HGCalTriggerSums, std::allocator<l1t::HGCalTriggerSums> >&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libL1TriggerL1THGCal.so
#9  0x00007fe27a05bfcb in HGCalConcentratorProcessorSelection::run(edm::Handle<BXVector<l1t::HGCalTriggerCell> > const&, std::tuple<BXVector<l1t::HGCalTriggerCell>, BXVector<l1t::HGCalTriggerSums>, BXVector<l1t::HGCalConcentratorData> >&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/pluginL1TriggerL1THGCalPlugins_fe_be.so
#10 0x00007fe2e0d65bd2 in HGCalConcentratorProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/pluginL1TriggerL1THGCalPlugins.so
#11 0x00007fe3184d8e11 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#12 0x00007fe3184bd71e in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#13 0x00007fe3184485d9 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#14 0x00007fe318448b44 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#15 0x00007fe3185f1f28 in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libFWCoreConcurrency.so
#16 0x00007fe316bfe96b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x7fe247673600, waiter=..., this=0x7fe31253fc80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#17 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x7fe31253fc80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#18 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/arena.cpp:137
#19 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/market.cpp:599
#20 0x00007fe316c00b1e in tbb::detail::r1::rml::private_worker::run (this=0x7fe30feffe00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#21 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7fe30feffe00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#22 0x00007fe315d8bea5 in start_thread () from /lib64/libpthread.so.0
#23 0x00007fe315ab3b0d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fe2be3fd700 (LWP 10181) "cmsRun"):
#0  0x00007fe315a7a9fd in nanosleep () from /lib64/libc.so.6
#1  0x00007fe315a7a894 in sleep () from /lib64/libc.so.6
#2  0x00007fe30db24d30 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007fe2e17c1b10 in SteppingHelixPropagator::cIndex_(int) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libTrackPropagationSteppingHelixPropagator.so
#5  0x00007fe2e17c7110 in SteppingHelixPropagator::propagate(SteppingHelixStateInfo (&) [8], int&, SteppingHelixPropagator::DestType, double const*, double) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libTrackPropagationSteppingHelixPropagator.so
#6  0x00007fe2e17cb120 in SteppingHelixPropagator::propagate(SteppingHelixStateInfo const&, Point3DBase<float, GlobalTag> const&, Point3DBase<float, GlobalTag> const&, SteppingHelixStateInfo&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libTrackPropagationSteppingHelixPropagator.so
#7  0x00007fe2e17cc740 in SteppingHelixPropagator::propagateWithPath(FreeTrajectoryState const&, Point3DBase<float, GlobalTag> const&, Point3DBase<float, GlobalTag> const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libTrackPropagationSteppingHelixPropagator.so
#8  0x00007fe2e17cca39 in SteppingHelixPropagator::propagateWithPath(FreeTrajectoryState const&, reco::BeamSpot const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libTrackPropagationSteppingHelixPropagator.so
#9  0x00007fe2a5714783 in MuonUpdatorAtVertex::propagate(TrajectoryStateOnSurface const&, reco::BeamSpot const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libRecoMuonTrackingTools.so
#10 0x00007fe2a56fe47c in MuonTrackLoader::buildTrackAtPCA(Trajectory const&, reco::BeamSpot const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libRecoMuonTrackingTools.so
#11 0x00007fe2a570041f in MuonTrackLoader::loadTracks(std::vector<std::unique_ptr<Trajectory, std::default_delete<Trajectory> >, std::allocator<std::unique_ptr<Trajectory, std::default_delete<Trajectory> > > >&, edm::Event&, std::vector<bool, std::allocator<bool> >&, TrackerTopology const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libRecoMuonTrackingTools.so
#12 0x00007fe2a5701ec5 in MuonTrackLoader::loadTracks(std::vector<std::unique_ptr<Trajectory, std::default_delete<Trajectory> >, std::allocator<std::unique_ptr<Trajectory, std::default_delete<Trajectory> > > >&, edm::Event&, TrackerTopology const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libRecoMuonTrackingTools.so
#13 0x00007fe2a56f8ee6 in MuonTrackFinder::load(std::vector<std::unique_ptr<Trajectory, std::default_delete<Trajectory> >, std::allocator<std::unique_ptr<Trajectory, std::default_delete<Trajectory> > > >&, edm::Event&, TrackerTopology const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libRecoMuonTrackingTools.so
#14 0x00007fe2a56f935e in MuonTrackFinder::reconstruct(edm::Handle<edm::View<TrajectorySeed> > const&, edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libRecoMuonTrackingTools.so
#15 0x00007fe27c019a16 in L2MuonProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/pluginRecoMuonL2MuonProducer.so
#16 0x00007fe3184d8e11 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#17 0x00007fe3184bd71e in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#18 0x00007fe3184485d9 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#19 0x00007fe318448b44 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#20 0x00007fe3185f1f28 in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libFWCoreConcurrency.so
#21 0x00007fe316bfe96b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x7fe24763d800, waiter=..., this=0x7fe31253fb80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#22 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x7fe31253fb80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#23 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/arena.cpp:137
#24 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/market.cpp:599
#25 0x00007fe316c00b1e in tbb::detail::r1::rml::private_worker::run (this=0x7fe30fefff00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#26 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7fe30fefff00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#27 0x00007fe315d8bea5 in start_thread () from /lib64/libpthread.so.0
#28 0x00007fe315ab3b0d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fe2bfbfe700 (LWP 10180) "cmsRun"):
#0  0x00007fe315a7a9fd in nanosleep () from /lib64/libc.so.6
#1  0x00007fe315a7a894 in sleep () from /lib64/libc.so.6
#2  0x00007fe30db24d30 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007fe2e7f74ed0 in Point3DBase<float, GlobalTag>& std::vector<Point3DBase<float, GlobalTag>, std::allocator<Point3DBase<float, GlobalTag> > >::emplace_back<Point3DBase<float, GlobalTag> >(Point3DBase<float, GlobalTag>&&) [clone .isra.0] () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libDataFormatsGeometrySurface.so
#5  0x00007fe2e7f75bfb in BoundingBox::corners(Plane const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libDataFormatsGeometrySurface.so
#6  0x00007fe30f4940e2 in RodPlaneBuilderFromDet::computeBounds(std::vector<GeomDet const*, std::allocator<GeomDet const*> > const&, Plane const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libTrackingToolsDetLayers.so
#7  0x00007fe30f49487d in RodPlaneBuilderFromDet::operator()(std::vector<GeomDet const*, std::allocator<GeomDet const*> > const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libTrackingToolsDetLayers.so
#8  0x00007fe2e25cb5dd in Phase2OTBarrelRodBuilder::build(GeometricDet const*, TrackerGeometry const*, bool) [clone .constprop.0] () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libRecoTrackerTkDetLayers.so
#9  0x00007fe2e25c57e0 in GeometricSearchTrackerBuilder::build(GeometricDet const*, TrackerGeometry const*, TrackerTopology const*, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libRecoTrackerTkDetLayers.so
#10 0x00007fe27c071f03 in MuonHLTSeedMVAClassifierPhase2::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/pluginRecoMuonTrackerSeedGeneratorPlugins.so
#11 0x00007fe3184d8e11 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#12 0x00007fe3184bd71e in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#13 0x00007fe3184485d9 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#14 0x00007fe318448b44 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#15 0x00007fe3185f1f28 in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libFWCoreConcurrency.so
#16 0x00007fe316bfe96b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x7fe2476baf00, waiter=..., this=0x7fe31253fa80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#17 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x7fe31253fa80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#18 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/arena.cpp:137
#19 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/market.cpp:599
#20 0x00007fe316c00b1e in tbb::detail::r1::rml::private_worker::run (this=0x7fe30feffe80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#21 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7fe30feffe80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#22 0x00007fe315d8bea5 in start_thread () from /lib64/libpthread.so.0
#23 0x00007fe315ab3b0d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fe2bf1fd700 (LWP 10179) "cmsRun"):
#0  0x00007fe315a7a9fd in nanosleep () from /lib64/libc.so.6
#1  0x00007fe315a7a894 in sleep () from /lib64/libc.so.6
#2  0x00007fe30db24d30 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007fe315a98917 in sched_yield () from /lib64/libc.so.6
#5  0x00007fe316bf2c93 in __gthread_yield () at /data/cmsbld/jenkins/workspace/build-any-ib/w/slc7_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/x86_64-redhat-linux-gnu/bits/gthr-default.h:693
#6  std::this_thread::yield () at /data/cmsbld/jenkins/workspace/build-any-ib/w/slc7_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/std_thread.h:353
#7  tbb::detail::r1::stealing_loop_backoff::pause (this=0x7fe2bf1f6e18) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/scheduler_common.h:266
#8  tbb::detail::r1::waiter_base::pause (this=0x7fe2bf1f6e10) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/waiters.h:35
#9  tbb::detail::r1::outermost_worker_waiter::pause (this=0x7fe2bf1f6e10) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/waiters.h:69
#10 tbb::detail::r1::task_dispatcher::receive_or_steal_task<true, tbb::detail::r1::outermost_worker_waiter> (this=<optimized out>, tls=..., ed=..., waiter=..., isolation=<optimized out>, fifo_allowed=<optimized out>, critical_allowed=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/task_dispatcher.h:231
#11 0x00007fe316bfeae2 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x7fe31253fa00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/task_dispatcher.h:350
#12 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x7fe31253fa00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#13 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/arena.cpp:137
#14 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/market.cpp:599
#15 0x00007fe316c00b1e in tbb::detail::r1::rml::private_worker::run (this=0x7fe30ff00000) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#16 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7fe30ff00000) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#17 0x00007fe315d8bea5 in start_thread () from /lib64/libpthread.so.0
#18 0x00007fe315ab3b0d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fe2c07ff700 (LWP 10178) "cmsRun"):
#0  0x00007fe315a7a9fd in nanosleep () from /lib64/libc.so.6
#1  0x00007fe315a7a894 in sleep () from /lib64/libc.so.6
#2  0x00007fe30db24d30 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007fe315aade29 in syscall () from /lib64/libc.so.6
#5  0x00007fe316c00e02 in tbb::detail::r1::futex_wait (comparand=2, futex=0x7fe30fefffa4) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x7fe30fefffa4) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x7fe30fefffa0) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x7fe30fefff80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7fe30fefff80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x00007fe315d8bea5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007fe315ab3b0d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fe2c19a5700 (LWP 10177) "cmsRun"):
#0  0x00007fe315a7a9fd in nanosleep () from /lib64/libc.so.6
#1  0x00007fe315a7a894 in sleep () from /lib64/libc.so.6
#2  0x00007fe30db24d30 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007fe279387b7d in ap_private<96, false, false>::shl(unsigned int) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/pluginL1TrackTriggerPlugins.so
#5  0x00007fe27937d2b1 in TTTrack_TrackWord::getTrackWord() const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/pluginL1TrackTriggerPlugins.so
#6  0x00007fe2793b638b in L1TrackVertexAssociationProducer::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/pluginL1TrackTriggerPlugins.so
#7  0x00007fe3184c6376 in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#8  0x00007fe3184b6ffe in edm::WorkerT<edm::global::EDProducerBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#9  0x00007fe3184485d9 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#10 0x00007fe318448b44 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#11 0x00007fe3185f1f28 in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libFWCoreConcurrency.so
#12 0x00007fe316bfe96b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x7fe312549400, waiter=..., this=0x7fe31253fb00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#13 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x7fe31253fb00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#14 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/arena.cpp:137
#15 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/market.cpp:599
#16 0x00007fe316c00b1e in tbb::detail::r1::rml::private_worker::run (this=0x7fe30ff00100) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#17 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7fe30ff00100) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#18 0x00007fe315d8bea5 in start_thread () from /lib64/libpthread.so.0
#19 0x00007fe315ab3b0d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fe2c23a6700 (LWP 10176) "cmsRun"):
#0  0x00007fe315aa8ddd in poll () from /lib64/libc.so.6
#1  0x00007fe30db7384f in full_read.constprop () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x00007fe30db27e2c in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x00007fe30db28790 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007fe279399eb3 in std::vector<unsigned int, std::allocator<unsigned int> >::push_back(unsigned int const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/pluginL1TrackTriggerPlugins.so
#6  0x00007fe2793a2f27 in L1TrackJetEmulatorProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/pluginL1TrackTriggerPlugins.so
#7  0x00007fe3184d8e11 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#8  0x00007fe3184bd71e in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#9  0x00007fe3184485d9 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#10 0x00007fe318448b44 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#11 0x00007fe3185f1f28 in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libFWCoreConcurrency.so
#12 0x00007fe316bfe96b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x7fe31246d700, waiter=..., this=0x7fe31253f980) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#13 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x7fe31253f980) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#14 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/arena.cpp:137
#15 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/market.cpp:599
#16 0x00007fe316c00b1e in tbb::detail::r1::rml::private_worker::run (this=0x7fe30ff00080) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#17 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7fe30ff00080) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#18 0x00007fe315d8bea5 in start_thread () from /lib64/libpthread.so.0
#19 0x00007fe315ab3b0d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fe2e8c57700 (LWP 10169) "cmsRun"):
#0  0x00007fe315d931d9 in waitpid () from /lib64/libpthread.so.0
#1  0x00007fe30db24e87 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x00007fe30db27d5a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x00007fe316396b83 in std::execute_native_thread_routine (__p=0x7fe2fc477440) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x00007fe315d8bea5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fe315ab3b0d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fe313bef780 (LWP 10119) "cmsRun"):
#0  0x00007fe315a7a9fd in nanosleep () from /lib64/libc.so.6
#1  0x00007fe315a7a894 in sleep () from /lib64/libc.so.6
#2  0x00007fe30db24d30 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007fe27937d2b6 in TTTrack_TrackWord::getTrackWord() const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/pluginL1TrackTriggerPlugins.so
#5  0x00007fe2793b71e8 in L1TrackVertexAssociationProducer::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/pluginL1TrackTriggerPlugins.so
#6  0x00007fe3184c6376 in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#7  0x00007fe3184b6ffe in edm::WorkerT<edm::global::EDProducerBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#8  0x00007fe3184485d9 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#9  0x00007fe318448b44 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#10 0x00007fe3185f1f28 in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libFWCoreConcurrency.so
#11 0x00007fe316c07291 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x7fe31253f900) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#12 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x7fe31253f900) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#13 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#14 0x00007fe3183cbccb in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#15 0x00007fe3183d563a in edm::EventProcessor::processRuns() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#16 0x00007fe3183d5b91 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-16-2300/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#17 0x00000000004074ef in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#18 0x00007fe316bf39bd in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/arena.cpp:688
#19 0x0000000000408ed2 in main::{lambda()#1}::operator()() const ()
#20 0x000000000040518c in main ()

Current Modules:

Module: L1TrackJetEmulatorProducer:l1tTrackJetsEmulation (crashed)
mmusich commented 7 months ago

related to https://github.com/cms-sw/cmssw/issues/43723 ?

dan131riley commented 7 months ago

related to #43723 ?

25234.0 does get an assertion failure with the test in #43723, so it's likely the same issue.

cmsRun: /mnt/data1/dsr/CMSSW_14_0_0_pre2/src/L1Trigger/L1TTrackMatch/plugins/L1TrackJetEmulatorProducer.cc:304: virtual void L1TrackJetEmulatorProducer::produce(edm::Event&, const edm::EventSetup&): Assertion `i < phiBins_ && i >= 0 && j < etaBins_ && j >= 0' failed.
srimanob commented 7 months ago

Close and follow up in https://github.com/cms-sw/cmssw/issues/43723