search

cms-sw / cmssw

CMS Offline Software
http://cms-sw.github.io/
Apache License 2.0
1.08k stars 4.31k forks source link

Segfault in TestDQMOnlineClient-visualization #42093

Open makortel opened 1 year ago

makortel commented 1 year ago

The test TestDQMOnlineClient-visualization crashed in CMSSW_13_2_NONLTO_X_2023-06-24-1100 on el8_amd64_gcc11 with

Begin processing the 100th record. Run 355380, Event 26785259, LumiSection 19 on stream 4 at 24-Jun-2023 18:59:42.015 CEST

Thread 9 (Thread 0x14b8941fe700 (LWP 1590063) "cmsRun"):
#2  0x000014b8eb4b5590 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02790/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_2_NONLTO_X_2023-06-24-1100/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014b8f34ff9bd in syscall () from /lib64/libc.so.6
#5  0x000014b8f472e2c4 in tbb::detail::r1::futex_wait (comparand=2, futex=0x14b8941f7cb0) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14b8941f7cb0) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::sleep_node<tbb::detail::r1::address_context>::wait (this=0x14b8941f7c80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:170
#8  tbb::detail::r1::concurrent_monitor_base<tbb::detail::r1::address_context>::commit_wait (this=<optimized out>, node=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:232
#9  tbb::detail::r1::concurrent_monitor_base<tbb::detail::r1::address_context>::commit_wait (node=..., this=0x14b8f4761460 <_ZN3tbb6detail2r1L20address_waiter_tableE.lto_priv.0+52320>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:228
#10 tbb::detail::r1::concurrent_monitor_base<tbb::detail::r1::address_context>::wait<tbb::detail::r1::sleep_node<tbb::detail::r1::address_context>, tbb::detail::d1::delegate_base&> (node=..., pred=..., this=0x14b8f4761460 <_ZN3tbb6detail2r1L20address_waiter_tableE.lto_priv.0+52320>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:262
#11 tbb::detail::r1::wait_on_address (address=<optimized out>, predicate=..., context=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/address_waiter.cpp:71
#12 0x000014b8f473e95c in tbb::detail::d1::waitable_atomic<int>::wait_until (order=<optimized out>, context=787, expected=<optimized out>, this=0x14b8f1dbb098) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/../../include/oneapi/tbb/detail/_waitable_atomic.h:79
#13 tbb::detail::r1::market::adjust_demand (this=0x14b8f1dcb580, a=..., delta=-7, mandatory=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/market.cpp:586
#14 0x000014b8f4740934 in tbb::detail::r1::arena::is_out_of_work() [clone .isra.0] (this=0x14b8f1dbaf00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/arena.cpp:333
#15 0x000014b8f4730bad in tbb::detail::r1::waiter_base::pause (this=0x14b8941f7f30) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/waiters.h:36
#16 tbb::detail::r1::outermost_worker_waiter::pause (this=0x14b8941f7f30) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/waiters.h:69
#17 tbb::detail::r1::task_dispatcher::receive_or_steal_task<true, tbb::detail::r1::outermost_worker_waiter> (this=0x14b8f1dbbc00, tls=..., ed=..., waiter=..., isolation=0, fifo_allowed=<optimized out>, critical_allowed=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/task_dispatcher.h:231
#18 0x000014b8f473cac0 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x14b8f1dbbc00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/task_dispatcher.h:350
#19 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x14b8f1dbbc00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#20 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/arena.cpp:137
#21 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/market.cpp:599
#22 0x000014b8f473ebb5 in tbb::detail::r1::rml::private_worker::run (this=0x14b8ec24e000) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#23 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14b8ec24e000) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#24 0x000014b8f38931ca in start_thread () from /lib64/libpthread.so.0
#25 0x000014b8f34ffe73 in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x14b8965fe700 (LWP 1590062) "cmsRun"):
#2  0x000014b8eb4b5590 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02790/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_2_NONLTO_X_2023-06-24-1100/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014b8f34ff12b in sched_yield () from /lib64/libc.so.6
#5  0x000014b8f4730b8a in __gthread_yield () at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc11/external/gcc/11.4.1-30ebdc301ebd200f2ae0e3d880258e65/include/c++/11.4.1/x86_64-redhat-linux-gnu/bits/gthr-default.h:693
#6  std::this_thread::yield () at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc11/external/gcc/11.4.1-30ebdc301ebd200f2ae0e3d880258e65/include/c++/11.4.1/bits/std_thread.h:329
#7  tbb::detail::r1::stealing_loop_backoff::pause (this=0x14b8965f7f38) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/scheduler_common.h:266
#8  tbb::detail::r1::waiter_base::pause (this=0x14b8965f7f30) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/waiters.h:35
#9  tbb::detail::r1::outermost_worker_waiter::pause (this=0x14b8965f7f30) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/waiters.h:69
#10 tbb::detail::r1::task_dispatcher::receive_or_steal_task<true, tbb::detail::r1::outermost_worker_waiter> (this=0x14b8f1dbbc80, tls=..., ed=..., waiter=..., isolation=0, fifo_allowed=<optimized out>, critical_allowed=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/task_dispatcher.h:231
#11 0x000014b8f473cac0 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x14b8f1dbbc80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/task_dispatcher.h:350
#12 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x14b8f1dbbc80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#13 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/arena.cpp:137
#14 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/market.cpp:599
#15 0x000014b8f473ebb5 in tbb::detail::r1::rml::private_worker::run (this=0x14b8ec24df80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#16 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14b8ec24df80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#17 0x000014b8f38931ca in start_thread () from /lib64/libpthread.so.0
#18 0x000014b8f34ffe73 in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x14b894bff700 (LWP 1590061) "cmsRun"):
#2  0x000014b8eb4b5590 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02790/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_2_NONLTO_X_2023-06-24-1100/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014b8f34ff9bd in syscall () from /lib64/libc.so.6
#5  0x000014b8f473ee9e in tbb::detail::r1::futex_wait (comparand=2, futex=0x14b8ec24de24) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14b8ec24de24) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x14b8ec24de20) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x14b8ec24de00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14b8ec24de00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000014b8f38931ca in start_thread () from /lib64/libpthread.so.0
#11 0x000014b8f34ffe73 in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x14b895bfd700 (LWP 1590060) "cmsRun"):
#2  0x000014b8eb4b5590 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02790/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_2_NONLTO_X_2023-06-24-1100/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014b8f34ff9bd in syscall () from /lib64/libc.so.6
#5  0x000014b8f473ee9e in tbb::detail::r1::futex_wait (comparand=2, futex=0x14b8ec24df24) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14b8ec24df24) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x14b8ec24df20) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x14b8ec24df00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14b8ec24df00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000014b8f38931ca in start_thread () from /lib64/libpthread.so.0
#11 0x000014b8f34ffe73 in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x14b8971ff700 (LWP 1590059) "cmsRun"):
#2  0x000014b8eb4b5590 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02790/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_2_NONLTO_X_2023-06-24-1100/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014b8f34ff9bd in syscall () from /lib64/libc.so.6
#5  0x000014b8f473ee9e in tbb::detail::r1::futex_wait (comparand=2, futex=0x14b8ec24dea4) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14b8ec24dea4) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x14b8ec24dea0) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x14b8ec24de80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14b8ec24de80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000014b8f38931ca in start_thread () from /lib64/libpthread.so.0
#11 0x000014b8f34ffe73 in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x14b89839d700 (LWP 1590057) "cmsRun"):
#0  0x000014b8f35bf868 in nanosleep () from /lib64/libc.so.6
#1  0x000014b8f35bf76e in sleep () from /lib64/libc.so.6
#2  0x000014b8eb4b5590 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02790/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_2_NONLTO_X_2023-06-24-1100/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014b8f34ff9bd in syscall () from /lib64/libc.so.6
#5  0x000014b8f473ee9e in tbb::detail::r1::futex_wait (comparand=2, futex=0x14b8ec24e124) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14b8ec24e124) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x14b8ec24e120) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x14b8ec24e100) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14b8ec24e100) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000014b8f38931ca in start_thread () from /lib64/libpthread.so.0
#11 0x000014b8f34ffe73 in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x14b898d9e700 (LWP 1590056) "cmsRun"):
#3  0x000014b8eb4b901b in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02790/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_2_NONLTO_X_2023-06-24-1100/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  tbb::detail::r1::spawn (t=..., ctx=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:36
#6  0x000014b8f5f9b01a in tbb::detail::d1::function_task<edm::WaitingTaskHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02790/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_2_NONLTO_X_2023-06-24-1100/lib/el8_amd64_gcc11/libFWCoreFramework.so
#7  0x000014b8f473c96f in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x14b87f42bc00, waiter=..., this=0x14b8f1dbbb80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#8  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x14b8f1dbbb80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#9  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/arena.cpp:137
#10 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/market.cpp:599
#11 0x000014b8f473ebb5 in tbb::detail::r1::rml::private_worker::run (this=0x14b8ec24e080) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#12 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14b8ec24e080) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#13 0x000014b8f38931ca in start_thread () from /lib64/libpthread.so.0
#14 0x000014b8f34ffe73 in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x14b8f2a23640 (LWP 1589014) "cmsRun"):
#2  0x000014b8eb4b5590 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02790/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_2_NONLTO_X_2023-06-24-1100/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014b8f34ff12b in sched_yield () from /lib64/libc.so.6
#5  0x000014b8f473e917 in __gthread_yield () at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc11/external/gcc/11.4.1-30ebdc301ebd200f2ae0e3d880258e65/include/c++/11.4.1/x86_64-redhat-linux-gnu/bits/gthr-default.h:693
#6  std::this_thread::yield () at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc11/external/gcc/11.4.1-30ebdc301ebd200f2ae0e3d880258e65/include/c++/11.4.1/bits/std_thread.h:329
#7  tbb::detail::d0::timed_spin_wait_until<tbb::detail::d1::waitable_atomic<int>::wait_until(int, unsigned long, std::memory_order)::{lambda()#1}>(tbb::detail::d1::waitable_atomic<int>::wait_until(int, unsigned long, std::memory_order)::{lambda()#1}) (condition=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/../../include/oneapi/tbb/detail/_utils.h:129
#8  tbb::detail::d1::waitable_atomic<int>::wait_until (order=<optimized out>, context=788, expected=<optimized out>, this=0x14b8f1dbb098) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/../../include/oneapi/tbb/detail/_waitable_atomic.h:74
#9  tbb::detail::r1::market::adjust_demand (this=0x14b8f1dcb580, a=..., delta=7, mandatory=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/market.cpp:586
#10 0x000014b8f474273b in tbb::detail::r1::arena::advertise_new_work<(tbb::detail::r1::arena::new_work_type)0> (this=0x14b8f1dbaf00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/arena.h:547
#11 0x000014b8f5fcd8c8 in edm::EventProcessor::endUnfinishedLumi(bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02790/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_2_NONLTO_X_2023-06-24-1100/lib/el8_amd64_gcc11/libFWCoreFramework.so
#12 0x000014b8f5fcd987 in edm::(anonymous namespace)::RunResources::~RunResources() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02790/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_2_NONLTO_X_2023-06-24-1100/lib/el8_amd64_gcc11/libFWCoreFramework.so
#13 0x000014b8f5fd26bd in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02790/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_2_NONLTO_X_2023-06-24-1100/lib/el8_amd64_gcc11/libFWCoreFramework.so
#14 0x000000000040a1bd in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#15 0x000014b8f4731877 in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/arena.cpp:688
#16 0x000000000040b009 in main::{lambda()#1}::operator()() const ()
#17 0x000000000040971c in main ()

Current Modules:

Module: none (crashed)/data/cmsbld/jenkins/workspace/ib-run-qa/CMSSW_13_2_NONLTO_X_2023-06-24-1100/src/DQM/Integration/test/runtest.sh: line 19: 1589014 Segmentation fault      (core dumped) cmsRun $CLIENTS_DIR/$1 unitTest=True

---> test TestDQMOnlineClient-visualization had ERRORS

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/el8_amd64_gcc11/CMSSW_13_2_NONLTO_X_2023-06-24-1100/unitTestLogs/DQM/Integration#/

makortel commented 1 year ago

assign core

FYI @cms-sw/dqm-l2

cmsbuild commented 1 year ago

A new Issue was created by @makortel Matti Kortelainen.

@Dr15Jones, @perrotta, @dpiparo, @rappoccio, @makortel, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

cmsbuild commented 1 year ago

New categories assigned: core

@Dr15Jones,@smuzaffar,@makortel you have been requested to review this Pull request/Issue and eventually sign? Thanks

makortel commented 1 year ago

We updated TBB to 2021.9.0 in CMSSW_13_2_X_2023-06-20-2300 (with https://github.com/cms-sw/cmsdist/pull/8553), it may be related.

makortel commented 1 year ago

Another one in CMSSW_13_3_X_2023-09-05-1100 on slc7_amd64_gcc11

Begin processing the 100th record. Run 355380, Event 26785259, LumiSection 19 on stream 4 at 05-Sep-2023 14:24:24.917 CEST

Thread 10 (Thread 0x14d71e3fc700 (LWP 1862305) "cmsRun"):
#2  0x000014d7802d0b10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014d788646e29 in syscall () from /lib64/libc.so.6
#5  0x000014d789759eae in tbb::detail::r1::futex_wait (comparand=2, futex=0x14d78115df24) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14d78115df24) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x14d78115df20) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x14d78115df00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14d78115df00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000014d788923ea5 in start_thread () from /lib64/libpthread.so.0
#11 0x000014d78864cb0d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x14d71edfd700 (LWP 1862304) "cmsRun"):
#2  0x000014d7802d0b10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014d788646e29 in syscall () from /lib64/libc.so.6
#5  0x000014d789759eae in tbb::detail::r1::futex_wait (comparand=2, futex=0x14d78115de24) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14d78115de24) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x14d78115de20) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x14d78115de00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14d78115de00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000014d788923ea5 in start_thread () from /lib64/libpthread.so.0
#11 0x000014d78864cb0d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x14d71f9fe700 (LWP 1862303) "cmsRun"):
#2  0x000014d7802d0b10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014d788646e29 in syscall () from /lib64/libc.so.6
#5  0x000014d789759eae in tbb::detail::r1::futex_wait (comparand=2, futex=0x14d78115dea4) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14d78115dea4) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x14d78115dea0) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x14d78115de80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14d78115de80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000014d788923ea5 in start_thread () from /lib64/libpthread.so.0
#11 0x000014d78864cb0d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x14d7203ff700 (LWP 1862302) "cmsRun"):
#2  0x000014d7802d0b10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014d788646e29 in syscall () from /lib64/libc.so.6
#5  0x000014d789759eae in tbb::detail::r1::futex_wait (comparand=2, futex=0x14d78115e024) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14d78115e024) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x14d78115e020) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x14d78115e000) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14d78115e000) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000014d788923ea5 in start_thread () from /lib64/libpthread.so.0
#11 0x000014d78864cb0d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x14d7215ff700 (LWP 1862301) "cmsRun"):
#2  0x000014d7802d0b10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014d788646e29 in syscall () from /lib64/libc.so.6
#5  0x000014d789759eae in tbb::detail::r1::futex_wait (comparand=2, futex=0x14d78115dfa4) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14d78115dfa4) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x14d78115dfa0) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x14d78115df80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14d78115df80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000014d788923ea5 in start_thread () from /lib64/libpthread.so.0
#11 0x000014d78864cb0d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x14d7225f6700 (LWP 1862300) "cmsRun"):
#2  0x000014d7802d0b10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014d788646e29 in syscall () from /lib64/libc.so.6
#5  0x000014d789759eae in tbb::detail::r1::futex_wait (comparand=2, futex=0x14d78115e124) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14d78115e124) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x14d78115e120) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x14d78115e100) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14d78115e100) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000014d788923ea5 in start_thread () from /lib64/libpthread.so.0
#11 0x000014d78864cb0d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x14d722ff7700 (LWP 1862299) "cmsRun"):
#2  0x000014d7802d0b10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014d788646e29 in syscall () from /lib64/libc.so.6
#5  0x000014d789759eae in tbb::detail::r1::futex_wait (comparand=2, futex=0x14d78115e0a4) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14d78115e0a4) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x14d78115e0a0) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x14d78115e080) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14d78115e080) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000014d788923ea5 in start_thread () from /lib64/libpthread.so.0
#11 0x000014d78864cb0d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x14d76065a700 (LWP 1862126) "cmsRun"):
#0  0x000014d78892b1d9 in waitpid () from /lib64/libpthread.so.0
#1  0x000014d7802d0e87 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/pluginFWCoreServicesPlugins.so
#2  0x000014d7802d381a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  0x000014d788f28b04 in std::execute_native_thread_routine (__p=0x14d76d992530) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x000014d788923ea5 in start_thread () from /lib64/libpthread.so.0
#5  0x000014d78864cb0d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x14d78678c780 (LWP 1861056) "cmsRun"):
#3  0x000014d7802d434b in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  tbb::detail::r1::spawn (t=..., ctx=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:36
#6  0x000014d78aed69d9 in tbb::detail::d1::function_task<edm::WaitingTaskHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/libFWCoreFramework.so
#7  0x000014d78975fa7d in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x14d6e1939d00, this=0x14d785bbb900) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#8  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x14d785bbb900) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#9  tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#10 0x000014d78aeea42d in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/libFWCoreFramework.so
#11 0x000014d78aef6153 in edm::EventProcessor::endUnfinishedRun(bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/libFWCoreFramework.so
#12 0x000014d78aef6213 in edm::(anonymous namespace)::RunResources::~RunResources() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/libFWCoreFramework.so
#13 0x000014d78aef83fd in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/libFWCoreFramework.so
#14 0x000000000040c17d in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#15 0x000014d78974c887 in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/arena.cpp:688
#16 0x00000000004101e0 in main::{lambda()#1}::operator()() const ()
#17 0x0000000000407d7c in main ()

Current Modules:
Module: none

---> test TestDQMOnlineClient-visualization_secondInstance had ERRORS

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc11/CMSSW_13_3_X_2023-09-05-1100/unitTestLogs/DQM/Integration#/

aandvalenzuela commented 8 months ago

Another one in CMSSW_14_1_X_2024-02-21-2300 on el8_amd64_gcc12:

Thread 3 (Thread 0x148615bcb700 (LWP 2694561) "cmsRun"):
#0  0x00001486631d3918 in nanosleep () from /lib64/libc.so.6
#1  0x00001486631d381e in sleep () from /lib64/libc.so.6
#2  0x000014865cd15cc0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02825/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-21-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00001486631139bd in syscall () from /lib64/libc.so.6
#5  0x000014866436adf2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x14865e26d0a4) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14865e26d0a4) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x14865e26d0a0) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x14865e26d080) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14865e26d080) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x00001486634a71ca in start_thread () from /lib64/libpthread.so.0
#11 0x0000148663113e73 in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x14863a3ff700 (LWP 2694273) "cmsRun"):
#0  0x00001486634b1672 in waitpid () from /lib64/libpthread.so.0
#1  0x000014865cd15e17 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02825/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-21-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x000014865cd18cea in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02825/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-21-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x0000148663b3ba73 in std::execute_native_thread_routine (__p=0x14863d4504e0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x00001486634a71ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148663113e73 in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x148662631640 (LWP 2692422) "cmsRun"):
#0  0x00001486631fe301 in poll () from /lib64/libc.so.6
#1  0x000014865cd646af in full_read.constprop () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02825/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-21-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x000014865cd18dbc in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02825/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-21-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x000014865cd19720 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02825/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-21-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  tbb::detail::r1::spawn (t=..., ctx=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:36
#6  0x0000148665b20928 in tbb::detail::d1::function_task<edm::WaitingTaskHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02825/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-21-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#7  0x0000148664371281 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x148661b2b900) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#8  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x148661b2b900) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#9  tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#10 0x0000148665b30cdb in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02825/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-21-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#11 0x0000148665b40428 in edm::EventProcessor::endUnfinishedRun(bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02825/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-21-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#12 0x0000148665b34f63 in edm::(anonymous namespace)::RunResources::~RunResources() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02825/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-21-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#13 0x0000148665b3aaf6 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02825/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-21-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#14 0x00000000004074ef in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#15 0x000014866435d9ad in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/arena.cpp:688
#16 0x0000000000408ed2 in main::{lambda()#1}::operator()() const ()
#17 0x000000000040517c in main ()

Current Modules:

Module: none (crashed)/data/cmsbld/jenkins/workspace/ib-run-qa/CMSSW_14_1_X_2024-02-21-2300/src/DQM/Integration/test/runtest.sh: line 28: 2692422 Segmentation fault      (core dumped) cmsRun $CLIENTS_DIR/$1 unitTest=True

---> test TestDQMOnlineClient-visualization had ERRORS
aandvalenzuela commented 3 months ago

Hello, new segfault occurrence in CMSSW_14_1_X_2024-07-17-2300:

%MSG
Begin processing the 98th record. Run 355380, Event 26705277, LumiSection 19 on stream 4 at 18-Jul-2024 02:49:28.143 CEST
Begin processing the 99th record. Run 355380, Event 26732456, LumiSection 19 on stream 1 at 18-Jul-2024 02:49:28.672 CEST
Begin processing the 100th record. Run 355380, Event 26785259, LumiSection 19 on stream 0 at 18-Jul-2024 02:49:29.123 CEST

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thu Jul 18 02:49:36 CEST 2024
Thread 26 (Thread 0x1506d83ff700 (LWP 1304338) "cmsRun"):
#0  0x000015075f66c48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00001507264dfba6 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#2  0x00001507264e01a5 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#3  0x00001507264dd968 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#4  0x0000150715878472 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libtensorflow_framework.so.2
#5  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#6  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x1506d8dff700 (LWP 1304337) "cmsRun"):
#0  0x000015075f66c48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00001507264dfba6 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#2  0x00001507264e01a5 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#3  0x00001507264dd968 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#4  0x0000150715878472 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libtensorflow_framework.so.2
#5  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#6  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x1506d92de700 (LWP 1304336) "cmsRun"):
#0  0x000015075f66c48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00001507264dfba6 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#2  0x00001507264e01a5 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#3  0x00001507264dd968 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#4  0x0000150715878472 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libtensorflow_framework.so.2
#5  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#6  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x150705bf1700 (LWP 1304251) "cmsRun"):
#0  0x000015075f66edd6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x000015075f66eec8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x000015075779d706 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x000015075779d7b9 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x150705df2700 (LWP 1304250) "cmsRun"):
#0  0x000015075f66edd6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x000015075f66eec8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x000015075779d706 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x000015075779d7b9 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x150705ff3700 (LWP 1304249) "cmsRun"):
#0  0x000015075f66edd6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x000015075f66eec8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x000015075779d706 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x000015075779d7b9 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x1507061f4700 (LWP 1304248) "cmsRun"):
#0  0x000015075f6701b0 in nanosleep () from /lib64/libpthread.so.0
#1  0x0000150759019f08 in XrdSysTimer::Wait(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x000015075771203c in XrdCl::TaskManager::RunTasks() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x0000150757712179 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x1507063f5700 (LWP 1304247) "cmsRun"):
#0  0x000015075f3c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000150759014282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00001507590101ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000150759019617 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x1507065f6700 (LWP 1304246) "cmsRun"):
#0  0x000015075f3c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000150759014282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00001507590101ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000150759019617 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x1507067f7700 (LWP 1304245) "cmsRun"):
#0  0x000015075f3c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000150759014282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00001507590101ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000150759019617 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x1507069f8700 (LWP 1304244) "cmsRun"):
#0  0x000015075f3c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000150759014282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00001507590101ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000150759019617 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x150706bf9700 (LWP 1304243) "cmsRun"):
#0  0x000015075f3c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000150759014282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00001507590101ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000150759019617 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x150706dfa700 (LWP 1304242) "cmsRun"):
#0  0x000015075f3c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000150759014282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00001507590101ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000150759019617 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x150706ffb700 (LWP 1304241) "cmsRun"):
#0  0x000015075f3c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000150759014282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00001507590101ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000150759019617 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x1507071fc700 (LWP 1304240) "cmsRun"):
#0  0x000015075f3c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000150759014282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00001507590101ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000150759019617 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x1507073fd700 (LWP 1304239) "cmsRun"):
#0  0x000015075f3c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000150759014282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00001507590101ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000150759019617 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x1507075fe700 (LWP 1304238) "cmsRun"):
#0  0x000015075f3c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000150759014282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00001507590101ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000150759019617 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x1507081ff700 (LWP 1304237) "cmsRun"):
#0  0x000015075f390098 in nanosleep () from /lib64/libc.so.6
#1  0x000015075f38ff9e in sleep () from /lib64/libc.so.6
#2  0x00001507599dfa10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000015075f2c141d in syscall () from /lib64/libc.so.6
#5  0x00001507613d3fd2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x15075a0d4e24) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x15075a0d4e24) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x15075a0d4e20) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x15075a0d4e00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x15075a0d4e00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#11 0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x1507095fd700 (LWP 1304236) "cmsRun"):
#0  0x000015075f390098 in nanosleep () from /lib64/libc.so.6
#1  0x000015075f38ff9e in sleep () from /lib64/libc.so.6
#2  0x00001507599dfa10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000015075f2c141d in syscall () from /lib64/libc.so.6
#5  0x00001507613d3fd2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x15075a0d4f24) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x15075a0d4f24) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x15075a0d4f20) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x15075a0d4f00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x15075a0d4f00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#11 0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x150709ffe700 (LWP 1304235) "cmsRun"):
#0  0x000015075f390098 in nanosleep () from /lib64/libc.so.6
#1  0x000015075f38ff9e in sleep () from /lib64/libc.so.6
#2  0x00001507599dfa10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000015075f2c141d in syscall () from /lib64/libc.so.6
#5  0x00001507613d3fd2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x15075a0d5024) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x15075a0d5024) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x15075a0d5020) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x15075a0d5000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x15075a0d5000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#11 0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x15070a9ff700 (LWP 1304234) "cmsRun"):
#0  0x000015075f390098 in nanosleep () from /lib64/libc.so.6
#1  0x000015075f38ff9e in sleep () from /lib64/libc.so.6
#2  0x00001507599dfa10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000015075f2c141d in syscall () from /lib64/libc.so.6
#5  0x00001507613d3fd2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x15075a0d4ea4) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x15075a0d4ea4) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x15075a0d4ea0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x15075a0d4e80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x15075a0d4e80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#11 0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x15070b76b700 (LWP 1304233) "cmsRun"):
#0  0x000015075f390098 in nanosleep () from /lib64/libc.so.6
#1  0x000015075f38ff9e in sleep () from /lib64/libc.so.6
#2  0x00001507599dfa10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000015075f2c141d in syscall () from /lib64/libc.so.6
#5  0x00001507613d3fd2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x15075a0d4fa4) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x15075a0d4fa4) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x15075a0d4fa0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x15075a0d4f80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x15075a0d4f80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#11 0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x15070c890700 (LWP 1304232) "cmsRun"):
#0  0x000015075f390098 in nanosleep () from /lib64/libc.so.6
#1  0x000015075f38ff9e in sleep () from /lib64/libc.so.6
#2  0x00001507599dfa10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000015075f2c140d in syscall () from /lib64/libc.so.6
#5  0x00001507613d346b in tbb::detail::r1::futex_wakeup_one (futex=0x15075a0d4fa4) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/semaphore.h:109
#6  tbb::detail::r1::binary_semaphore::V (this=0x15075a0d4fa4) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/semaphore.h:262
#7  tbb::detail::r1::rml::internal::thread_monitor::notify (this=0x15075a0d4fa0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:230
#8  tbb::detail::r1::rml::private_worker::wake_or_launch (this=0x15075a0d4f80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:292
#9  tbb::detail::r1::rml::private_server::wake_some (this=<optimized out>, additional_slack=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:412
#10 0x00001507613d3e49 in tbb::detail::r1::rml::private_server::propagate_chain_reaction (this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:159
#11 tbb::detail::r1::rml::private_worker::run (this=0x15075a0d5100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:275
#12 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x15075a0d5100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#13 0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#14 0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x15070d291700 (LWP 1304231) "cmsRun"):
#0  0x000015075f390098 in nanosleep () from /lib64/libc.so.6
#1  0x000015075f38ff9e in sleep () from /lib64/libc.so.6
#2  0x00001507599dfa10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000150762032377 in edm::Worker::reset() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/libFWCoreFramework.so
#5  0x0000150762032dc4 in edm::WorkerManager::resetAll() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/libFWCoreFramework.so
#6  0x0000150761f5d10d in edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/libFWCoreFramework.so
#7  0x0000150761f6dedd in tbb::detail::d1::function_task<edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#3}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/libFWCoreFramework.so
#8  0x00001507613d1b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x150708769100, waiter=..., this=0x15075ddbba00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#9  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x15075ddbba00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#10 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/arena.cpp:137
#11 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/market.cpp:599
#12 0x00001507613d3cee in tbb::detail::r1::rml::private_worker::run (this=0x15075a0d5080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#13 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x15075a0d5080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#14 0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#15 0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x1507358cd700 (LWP 1304112) "cmsRun"):
#0  0x000015075f6706a2 in waitpid () from /lib64/libpthread.so.0
#1  0x00001507599dfb67 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x00001507599e299a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x000015075fcd8a73 in std::execute_native_thread_routine (__p=0x150738c0cdb0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x1507612e1680 (LWP 1303317) "cmsRun"):
#0  0x000015075f3baac1 in poll () from /lib64/libc.so.6
#1  0x00001507599e2bb7 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x00001507599e2db4 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  tbb::detail::r1::spawn (t=..., ctx=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:36
#5  0x0000150761f2d438 in tbb::detail::d1::function_task<edm::WaitingTaskHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/libFWCoreFramework.so
#6  0x00001507613da3e1 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x15075ddbb900) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#7  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x15075ddbb900) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#8  tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#9  0x0000150761f433ab in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/libFWCoreFramework.so
#10 0x0000150761f502f6 in edm::EventProcessor::endUnfinishedRun(bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/libFWCoreFramework.so
#11 0x0000150761f5164d in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/libFWCoreFramework.so
#12 0x0000000000408410 in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#13 0x00001507613c69ad in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/arena.cpp:688
#14 0x000000000040a102 in main::{lambda()#1}::operator()() const ()
#15 0x0000000000405100 in main ()

Current Modules:

Module: none (crashed)/data/cmsbld/jenkins/workspace/ib-run-qa/CMSSW_14_1_X_2024-07-17-2300/src/DQM/Integration/test/runtest.sh: line 28: 1303317 Segmentation fault      (core dumped) cmsRun $CLIENTS_DIR/$1 unitTest=True

---> test TestDQMOnlineClient-visualization_secondInstance had ERRORS
iarspider commented 2 months ago

New occurance: CMSSW_14_1_X_2024-08-12-1100

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Mon Aug 12 18:43:19 CEST 2024
Thread 26 (Thread 0x148ea05ff700 (LWP 3757517) "cmsRun"):
#0  0x0000148f2766c48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000148eee2dfba6 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#2  0x0000148eee2e01a5 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#3  0x0000148eee2dd968 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#4  0x0000148edd678472 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libtensorflow_framework.so.2
#5  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#6  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x148ea0fff700 (LWP 3757516) "cmsRun"):
#0  0x0000148f2766c48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000148eee2dfba6 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#2  0x0000148eee2e01a5 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#3  0x0000148eee2dd968 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#4  0x0000148edd678472 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libtensorflow_framework.so.2
#5  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#6  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x148ea1525700 (LWP 3757515) "cmsRun"):
#0  0x0000148f2766c48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000148eee2dfba6 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#2  0x0000148eee2e01a5 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#3  0x0000148eee2dd968 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#4  0x0000148edd678472 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libtensorflow_framework.so.2
#5  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#6  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x148ecd9f2700 (LWP 3757385) "cmsRun"):
#0  0x0000148f2766edd6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x0000148f2766eec8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x0000148f1f5ce7c6 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x0000148f1f5ce879 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x148ecdbf3700 (LWP 3757384) "cmsRun"):
#0  0x0000148f2766edd6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x0000148f2766eec8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x0000148f1f5ce7c6 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x0000148f1f5ce879 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x148ecddf4700 (LWP 3757383) "cmsRun"):
#0  0x0000148f2766edd6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x0000148f2766eec8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x0000148f1f5ce7c6 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x0000148f1f5ce879 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x148ecdff5700 (LWP 3757382) "cmsRun"):
#0  0x0000148f276701b0 in nanosleep () from /lib64/libpthread.so.0
#1  0x0000148f218e47e8 in XrdSysTimer::Wait(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000148f1f53f37c in XrdCl::TaskManager::RunTasks() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x0000148f1f53f4b9 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x148ece1f6700 (LWP 3757381) "cmsRun"):
#0  0x0000148f273c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000148f218deb62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000148f218daacd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000148f218e3ef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x148ece3f7700 (LWP 3757380) "cmsRun"):
#0  0x0000148f273c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000148f218deb62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000148f218daacd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000148f218e3ef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x148ece5f8700 (LWP 3757379) "cmsRun"):
#0  0x0000148f273c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000148f218deb62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000148f218daacd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000148f218e3ef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x148ece7f9700 (LWP 3757378) "cmsRun"):
#0  0x0000148f273c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000148f218deb62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000148f218daacd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000148f218e3ef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x148ece9fa700 (LWP 3757377) "cmsRun"):
#0  0x0000148f273c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000148f218deb62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000148f218daacd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000148f218e3ef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x148ecebfb700 (LWP 3757376) "cmsRun"):
#0  0x0000148f273c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000148f218deb62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000148f218daacd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000148f218e3ef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x148ecedfc700 (LWP 3757375) "cmsRun"):
#0  0x0000148f273c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000148f218deb62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000148f218daacd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000148f218e3ef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x148eceffd700 (LWP 3757374) "cmsRun"):
#0  0x0000148f273c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000148f218deb62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000148f218daacd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000148f218e3ef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x148ecf1fe700 (LWP 3757373) "cmsRun"):
#0  0x0000148f273c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000148f218deb62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000148f218daacd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000148f218e3ef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x148ecf3ff700 (LWP 3757372) "cmsRun"):
#0  0x0000148f273c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000148f218deb62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000148f218daacd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000148f218e3ef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x148ed03ff700 (LWP 3757371) "cmsRun"):
#0  0x0000148f27390098 in nanosleep () from /lib64/libc.so.6
#1  0x0000148f2738ff9e in sleep () from /lib64/libc.so.6
#2  0x0000148f21a3c1d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000148f272c141d in syscall () from /lib64/libc.so.6
#5  0x0000148f2936b46b in tbb::detail::r1::futex_wakeup_one (futex=0x148f22030024) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:109
#6  tbb::detail::r1::binary_semaphore::V (this=0x148f22030024) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:262
#7  tbb::detail::r1::rml::internal::thread_monitor::notify (this=0x148f22030020) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:230
#8  tbb::detail::r1::rml::private_worker::wake_or_launch (this=0x148f22030000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:292
#9  tbb::detail::r1::rml::private_server::wake_some (this=<optimized out>, additional_slack=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:412
#10 0x0000148f2936be49 in tbb::detail::r1::rml::private_server::propagate_chain_reaction (this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:159
#11 tbb::detail::r1::rml::private_worker::run (this=0x148f2202fe00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:275
#12 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x148f2202fe00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#13 0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#14 0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x148ed13fd700 (LWP 3757370) "cmsRun"):
#0  0x0000148f27390098 in nanosleep () from /lib64/libc.so.6
#1  0x0000148f2738ff9e in sleep () from /lib64/libc.so.6
#2  0x0000148f21a3c1d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000148f29f5f956 in edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#5  0x0000148f29f6ee37 in tbb::detail::d1::enqueue_task<edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#4}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#6  0x0000148f29369b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x148eb6b88c00, waiter=..., this=0x148f25dbbb80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#7  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x148f25dbbb80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#8  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#9  tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#10 0x0000148f2936bcee in tbb::detail::r1::rml::private_worker::run (this=0x148f2202ff00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#11 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x148f2202ff00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#12 0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#13 0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x148ed1dfe700 (LWP 3757369) "cmsRun"):
#0  0x0000148f273baac1 in poll () from /lib64/libc.so.6
#1  0x0000148f21a3f857 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x0000148f21a3fa54 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  tbb::detail::r1::spawn (t=..., ctx=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:36
#5  0x0000148f29f2e5e8 in tbb::detail::d1::function_task<edm::WaitingTaskHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#6  0x0000148f29369b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x148eb6b88900, waiter=..., this=0x148f25dbba80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#7  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x148f25dbba80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#8  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#9  tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#10 0x0000148f2936bcee in tbb::detail::r1::rml::private_worker::run (this=0x148f22030000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#11 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x148f22030000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#12 0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#13 0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x148ed27ff700 (LWP 3757368) "cmsRun"):
#0  0x0000148f27390098 in nanosleep () from /lib64/libc.so.6
#1  0x0000148f2738ff9e in sleep () from /lib64/libc.so.6
#2  0x0000148f21a3c1d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000148f29f5f33f in edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#5  0x0000148f29f6ee37 in tbb::detail::d1::enqueue_task<edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#4}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#6  0x0000148f29369b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x148eb6b89100, waiter=..., this=0x148f25dbb980) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#7  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x148f25dbb980) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#8  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#9  tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#10 0x0000148f2936bcee in tbb::detail::r1::rml::private_worker::run (this=0x148f2202fe80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#11 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x148f2202fe80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#12 0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#13 0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x148ed37ff700 (LWP 3757367) "cmsRun"):
#0  0x0000148f27390098 in nanosleep () from /lib64/libc.so.6
#1  0x0000148f2738ff9e in sleep () from /lib64/libc.so.6
#2  0x0000148f21a3c1d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000148f29f5f33f in edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#5  0x0000148f29f6ee37 in tbb::detail::d1::enqueue_task<edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#4}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#6  0x0000148f29369b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x148eb6b88300, waiter=..., this=0x148f25dbbc00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#7  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x148f25dbbc00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#8  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#9  tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#10 0x0000148f2936bcee in tbb::detail::r1::rml::private_worker::run (this=0x148f2202ff80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#11 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x148f2202ff80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#12 0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#13 0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x148ed474c700 (LWP 3757366) "cmsRun"):
#0  0x0000148f27390098 in nanosleep () from /lib64/libc.so.6
#1  0x0000148f2738ff9e in sleep () from /lib64/libc.so.6
#2  0x0000148f21a3c1d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000148f29f5cf4d in std::__weak_ptr<edm::serviceregistry::ServicesManager, (__gnu_cxx::_Lock_policy)2>::__weak_ptr(std::__weak_ptr<edm::serviceregistry::ServicesManager, (__gnu_cxx::_Lock_policy)2> const&) [clone .lto_priv.0] () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#5  0x0000148f29f5f9f6 in edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#6  0x0000148f29f6eead in tbb::detail::d1::function_task<edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#3}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#7  0x0000148f29369b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x148e8c7d4600, waiter=..., this=0x148f25dbbb00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#8  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x148f25dbbb00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#9  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#10 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#11 0x0000148f2936bcee in tbb::detail::r1::rml::private_worker::run (this=0x148f22030100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#12 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x148f22030100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#13 0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#14 0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x148ed514d700 (LWP 3757365) "cmsRun"):
#0  0x0000148f27390098 in nanosleep () from /lib64/libc.so.6
#1  0x0000148f2738ff9e in sleep () from /lib64/libc.so.6
#2  0x0000148f21a3c1d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000148f272c0b8b in sched_yield () from /lib64/libc.so.6
#5  0x0000148f2935dc73 in __gthread_yield () at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/x86_64-redhat-linux-gnu/bits/gthr-default.h:693
#6  std::this_thread::yield () at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/std_thread.h:353
#7  tbb::detail::r1::stealing_loop_backoff::pause (this=0x148ed5146ef8) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/scheduler_common.h:266
#8  tbb::detail::r1::waiter_base::pause (this=0x148ed5146ef0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/waiters.h:35
#9  tbb::detail::r1::outermost_worker_waiter::pause (this=0x148ed5146ef0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/waiters.h:69
#10 tbb::detail::r1::task_dispatcher::receive_or_steal_task<true, tbb::detail::r1::outermost_worker_waiter> (this=<optimized out>, tls=..., ed=..., waiter=..., isolation=<optimized out>, fifo_allowed=<optimized out>, critical_allowed=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:231
#11 0x0000148f29369cb2 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x148f25dbba00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:350
#12 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x148f25dbba00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#13 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#14 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#15 0x0000148f2936bcee in tbb::detail::r1::rml::private_worker::run (this=0x148f22030080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#16 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x148f22030080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#17 0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#18 0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x148efd5ff700 (LWP 3757295) "cmsRun"):
#0  0x0000148f276706a2 in waitpid () from /lib64/libpthread.so.0
#1  0x0000148f21a3c327 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x0000148f21a3f63a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x0000148f27cd8a73 in std::execute_native_thread_routine (__p=0x148f00f2be70) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x148f289fe680 (LWP 3756508) "cmsRun"):
#0  0x0000148f27390098 in nanosleep () from /lib64/libc.so.6
#1  0x0000148f2738ff9e in sleep () from /lib64/libc.so.6
#2  0x0000148f21a3c1d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000148f272c0b8b in sched_yield () from /lib64/libc.so.6
#5  0x0000148f29370686 in __gthread_yield () at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/x86_64-redhat-linux-gnu/bits/gthr-default.h:693
#6  std::this_thread::yield () at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/std_thread.h:353
#7  tbb::detail::r1::stealing_loop_backoff::pause (this=0x7ffeeb9dc0e8) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/scheduler_common.h:266
#8  tbb::detail::r1::waiter_base::pause (this=0x7ffeeb9dc0e0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/waiters.h:35
#9  tbb::detail::r1::external_waiter::pause (this=0x7ffeeb9dc0e0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/waiters.h:138
#10 tbb::detail::r1::task_dispatcher::receive_or_steal_task<true, tbb::detail::r1::external_waiter> (this=<optimized out>, tls=..., ed=..., waiter=..., isolation=<optimized out>, fifo_allowed=<optimized out>, critical_allowed=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:231
#11 0x0000148f293724e2 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x0, this=0x148f25dbb900) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:350
#12 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x148f25dbb900) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#13 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#14 0x0000148f29f445fb in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#15 0x0000148f29f51296 in edm::EventProcessor::endUnfinishedRun(bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#16 0x0000148f29f525ed in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#17 0x000000000040840c in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#18 0x0000148f2935e9ad in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:688
#19 0x000000000040a0f2 in main::{lambda()#1}::operator()() const ()
#20 0x0000000000405100 in main ()

Current Modules:

Module: none (crashed)
Module: none
Module: none
Module: none
Module: none
Module: none
Module: none
Module: none
iarspider commented 2 months ago

Another occurance: el8_amd64_gcc12/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Sun Aug 18 13:15:09 CEST 2024
Thread 26 (Thread 0x1539aa5ff700 (LWP 3435629) "cmsRun"):
#0  0x0000153a3206c48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00001539f688d9a6 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/scram_x86-64-v3/libtensorflow_cc.so.2
#2  0x00001539f688dfa5 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/scram_x86-64-v3/libtensorflow_cc.so.2
#3  0x00001539f688b728 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/scram_x86-64-v3/libtensorflow_cc.so.2
#4  0x00001539e569cbf2 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/scram_x86-64-v3/libtensorflow_framework.so.2
#5  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#6  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x1539aafff700 (LWP 3435628) "cmsRun"):
#0  0x0000153a3206c48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00001539f688d9a6 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/scram_x86-64-v3/libtensorflow_cc.so.2
#2  0x00001539f688dfa5 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/scram_x86-64-v3/libtensorflow_cc.so.2
#3  0x00001539f688b728 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/scram_x86-64-v3/libtensorflow_cc.so.2
#4  0x00001539e569cbf2 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/scram_x86-64-v3/libtensorflow_framework.so.2
#5  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#6  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x1539ab524700 (LWP 3435627) "cmsRun"):
#0  0x0000153a3206c48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00001539f688d9a6 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/scram_x86-64-v3/libtensorflow_cc.so.2
#2  0x00001539f688dfa5 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/scram_x86-64-v3/libtensorflow_cc.so.2
#3  0x00001539f688b728 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/scram_x86-64-v3/libtensorflow_cc.so.2
#4  0x00001539e569cbf2 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/scram_x86-64-v3/libtensorflow_framework.so.2
#5  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#6  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x1539d7bf2700 (LWP 3435618) "cmsRun"):
#0  0x0000153a3206edd6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x0000153a3206eec8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x0000153a285ce7c6 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x0000153a285ce879 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x1539d7df3700 (LWP 3435617) "cmsRun"):
#0  0x0000153a3206edd6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x0000153a3206eec8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x0000153a285ce7c6 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x0000153a285ce879 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x1539d7ff4700 (LWP 3435616) "cmsRun"):
#0  0x0000153a3206edd6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x0000153a3206eec8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x0000153a285ce7c6 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x0000153a285ce879 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x1539d81f5700 (LWP 3435615) "cmsRun"):
#0  0x0000153a320701b0 in nanosleep () from /lib64/libpthread.so.0
#1  0x0000153a2a7bd7e8 in XrdSysTimer::Wait(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000153a2853f37c in XrdCl::TaskManager::RunTasks() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x0000153a2853f4b9 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x1539d83f6700 (LWP 3435614) "cmsRun"):
#0  0x0000153a31dc7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000153a2a7b7b62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000153a2a7b3acd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000153a2a7bcef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x1539d85f7700 (LWP 3435613) "cmsRun"):
#0  0x0000153a31dc7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000153a2a7b7b62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000153a2a7b3acd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000153a2a7bcef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x1539d87f8700 (LWP 3435612) "cmsRun"):
#0  0x0000153a31dc7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000153a2a7b7b62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000153a2a7b3acd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000153a2a7bcef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x1539d89f9700 (LWP 3435611) "cmsRun"):
#0  0x0000153a31dc7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000153a2a7b7b62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000153a2a7b3acd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000153a2a7bcef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x1539d8bfa700 (LWP 3435610) "cmsRun"):
#0  0x0000153a31dc7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000153a2a7b7b62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000153a2a7b3acd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000153a2a7bcef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x1539d8dfb700 (LWP 3435609) "cmsRun"):
#0  0x0000153a31dc7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000153a2a7b7b62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000153a2a7b3acd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000153a2a7bcef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x1539d8ffc700 (LWP 3435608) "cmsRun"):
#0  0x0000153a31dc7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000153a2a7b7b62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000153a2a7b3acd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000153a2a7bcef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x1539d91fd700 (LWP 3435607) "cmsRun"):
#0  0x0000153a31dc7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000153a2a7b7b62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000153a2a7b3acd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000153a2a7bcef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x1539d93fe700 (LWP 3435606) "cmsRun"):
#0  0x0000153a31dc7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000153a2a7b7b62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000153a2a7b3acd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000153a2a7bcef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x1539d95ff700 (LWP 3435605) "cmsRun"):
#0  0x0000153a31dc7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000153a2a7b7b62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000153a2a7b3acd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000153a2a7bcef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x1539da5ff700 (LWP 3435604) "cmsRun"):
#0  0x0000153a31d90098 in nanosleep () from /lib64/libc.so.6
#1  0x0000153a31d8ff9e in sleep () from /lib64/libc.so.6
#2  0x0000153a2b171f90 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000153a34964a0e in edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#5  0x0000153a34974127 in tbb::detail::d1::enqueue_task<edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#4}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#6  0x0000153a33d07b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x153996721d00, waiter=..., this=0x153a307bbc80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#7  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x153a307bbc80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#8  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#9  tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#10 0x0000153a33d09cee in tbb::detail::r1::rml::private_worker::run (this=0x153a2b766e00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#11 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x153a2b766e00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#12 0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#13 0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x1539db5fc700 (LWP 3435603) "cmsRun"):
#0  0x0000153a31d90098 in nanosleep () from /lib64/libc.so.6
#1  0x0000153a31d8ff9e in sleep () from /lib64/libc.so.6
#2  0x0000153a2b171f90 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000153a34964285 in edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#5  0x0000153a3497419d in tbb::detail::d1::function_task<edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#3}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#6  0x0000153a33d07b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x1539966e3200, waiter=..., this=0x153a307bbb80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#7  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x153a307bbb80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#8  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#9  tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#10 0x0000153a33d09cee in tbb::detail::r1::rml::private_worker::run (this=0x153a2b766f00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#11 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x153a2b766f00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#12 0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#13 0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x1539dbffd700 (LWP 3435602) "cmsRun"):
#0  0x0000153a31d90098 in nanosleep () from /lib64/libc.so.6
#1  0x0000153a31d8ff9e in sleep () from /lib64/libc.so.6
#2  0x0000153a2b171f90 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000153a349642ae in edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#5  0x0000153a34974127 in tbb::detail::d1::enqueue_task<edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#4}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#6  0x0000153a33d07b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x1539968b0a00, waiter=..., this=0x153a307bba80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#7  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x153a307bba80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#8  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#9  tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#10 0x0000153a33d09cee in tbb::detail::r1::rml::private_worker::run (this=0x153a2b767000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#11 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x153a2b767000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#12 0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#13 0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x1539dd3ff700 (LWP 3435601) "cmsRun"):
#0  0x0000153a31d90098 in nanosleep () from /lib64/libc.so.6
#1  0x0000153a31d8ff9e in sleep () from /lib64/libc.so.6
#2  0x0000153a2b171f90 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000153a34961fed in std::__weak_ptr<edm::serviceregistry::ServicesManager, (__gnu_cxx::_Lock_policy)2>::__weak_ptr(std::__weak_ptr<edm::serviceregistry::ServicesManager, (__gnu_cxx::_Lock_policy)2> const&) [clone .lto_priv.0] () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#5  0x0000153a34964ab5 in edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#6  0x0000153a34974127 in tbb::detail::d1::enqueue_task<edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#4}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#7  0x0000153a33d07b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x153996860c00, waiter=..., this=0x153a307bb980) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#8  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x153a307bb980) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#9  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#10 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#11 0x0000153a33d09cee in tbb::detail::r1::rml::private_worker::run (this=0x153a2b766e80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#12 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x153a2b766e80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#13 0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#14 0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x1539dc9fe700 (LWP 3435600) "cmsRun"):
#0  0x0000153a31d90098 in nanosleep () from /lib64/libc.so.6
#1  0x0000153a31d8ff9e in sleep () from /lib64/libc.so.6
#2  0x0000153a2b171f90 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000153a31cc141d in syscall () from /lib64/libc.so.6
#5  0x0000153a33d09fd2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x153a2b766fa4) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x153a2b766fa4) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x153a2b766fa0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x153a2b766f80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x153a2b766f80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#11 0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x1539de7c7700 (LWP 3435599) "cmsRun"):
#0  0x0000153a31d90098 in nanosleep () from /lib64/libc.so.6
#1  0x0000153a31d8ff9e in sleep () from /lib64/libc.so.6
#2  0x0000153a2b171f90 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000153a348f1c10 in tbb::detail::r1::spawn(tbb::detail::d1::task&, tbb::detail::d1::task_group_context&)@plt () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#5  0x0000153a34964ade in edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#6  0x0000153a34974127 in tbb::detail::d1::enqueue_task<edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#4}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#7  0x0000153a33d07b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x153996862100, waiter=..., this=0x153a307bbb00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#8  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x153a307bbb00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#9  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#10 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#11 0x0000153a33d09cee in tbb::detail::r1::rml::private_worker::run (this=0x153a2b767100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#12 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x153a2b767100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#13 0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#14 0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x1539df1c8700 (LWP 3435598) "cmsRun"):
#0  0x0000153a31d90098 in nanosleep () from /lib64/libc.so.6
#1  0x0000153a31d8ff9e in sleep () from /lib64/libc.so.6
#2  0x0000153a2b171f90 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000153a34964285 in edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#5  0x0000153a34974127 in tbb::detail::d1::enqueue_task<edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#4}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#6  0x0000153a33d07b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x153996746a00, waiter=..., this=0x153a307bba00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#7  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x153a307bba00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#8  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#9  tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#10 0x0000153a33d09cee in tbb::detail::r1::rml::private_worker::run (this=0x153a2b767080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#11 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x153a2b767080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#12 0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#13 0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x153a085a9700 (LWP 3435567) "cmsRun"):
#0  0x0000153a320706a2 in waitpid () from /lib64/libpthread.so.0
#1  0x0000153a2b1720e7 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/pluginFWCoreServicesPlugins.so
#2  0x0000153a2b1754ca in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/pluginFWCoreServicesPlugins.so
#3  0x0000153a326d8a73 in std::execute_native_thread_routine (__p=0x153a0a5cfe80) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x153a3371c680 (LWP 3434931) "cmsRun"):
#0  0x0000153a31dbaac1 in poll () from /lib64/libc.so.6
#1  0x0000153a2b1756e7 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/pluginFWCoreServicesPlugins.so
#2  0x0000153a2b1758e4 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  tbb::detail::r1::spawn (t=..., ctx=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:36
#5  0x0000153a349329f8 in tbb::detail::d1::function_task<edm::WaitingTaskHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#6  0x0000153a33d103e1 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x153a307bb900) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#7  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x153a307bb900) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#8  tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#9  0x0000153a34948f7b in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#10 0x0000153a34955f96 in edm::EventProcessor::endUnfinishedRun(bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#11 0x0000153a3495734d in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#12 0x000000000040840c in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#13 0x0000153a33cfc9ad in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:688
#14 0x000000000040a0f2 in main::{lambda()#1}::operator()() const ()
#15 0x0000000000405100 in main ()

Current Modules:

Module: none (crashed)
Module: none
Module: none
Module: none
Module: none
Module: none
Module: none
Module: none

Doesn't reproduce when ran locally

smuzaffar commented 2 months ago

https://github.com/cms-sw/cmssw/issues/42093#issuecomment-2296284425 was run on a cmsbuild926 (16 cores AlmaLinux9 node). Any idea why it is using 26 threads? I see that when this tests was running/failed then system log contains message [a]. Note that scram was running 16 unit tests in parallel and then if some unit tests starting using multiple theads then that can overload system and can hit system limits.

Is there any way to configure this unit test to not run multi-threaded ?

[a]

Aug 18 13:15:38 cmsbuild926.cern.ch kernel: cmsRun[3435603]: segfault at 10 ip 0000153a33d0de7d sp 00001539db5f5dd0 error 6 in libtbb.so.12.9[153a33cf3000+1f000] likely on CPU 7 (core 0, socket 7)
Aug 18 13:15:38 cmsbuild926.cern.ch kernel: Code: 19 01 00 e8 b5 56 fe ff 48 89 c3 48 85 c0 74 3d 48 89 de 4c 89 e7 e8 82 eb ff ff 48 8b 43 18 4c 8b 6b 20 48 89 ee 48 8b 7b 28 <4c> 89 65 10 48 8b 40 20 48 89 45 20 e8 32 e1 ff ff 48 83 c4 08 4c
Aug 18 13:15:38 cmsbuild926.cern.ch systemd[1]: Started Process Core Dump (PID 3436434/UID 0).
Aug 18 13:15:38 cmsbuild926.cern.ch systemd-coredump[3436435]: Resource limits disable core dumping for process 3434931 (cmsRun).
Aug 18 13:15:38 cmsbuild926.cern.ch systemd-coredump[3436435]: Process 3434931 (cmsRun) of user 501 dumped core.
Dr15Jones commented 2 months ago

Looking at the traceback, the threads are

iarspider commented 4 weeks ago

Two new occurances: CMSSW_14_2_DEVEL_X_2024-10-03-2300, CMSSW_14_2_GEANT4_X_2024-10-02-2300.

iarspider commented 2 weeks ago

Still happens: slc7_amd64_gcc12/CMSSW_14_2_X_2024-10-17-1100, el9_amd64_gcc12/CMSSW_14_2_X_2024-10-17-2300, el8_amd64_gcc12/CMSSW_14_2_DBG_X_2024-10-17-2300

iarspider commented 2 weeks ago

On an unrelated note: the DBG_X log file for DQM/Integration unit tests is over 11M lines long (compared to 26k+ lines for non-DBG builds), seems a bit too verbose to me.

nothingface0 commented 2 weeks ago

On an unrelated note: the DBG_X log file for DQM/Integration unit tests is over 11M lines long (compared to 26k+ lines for non-DBG builds), seems a bit too verbose to me.

I saw too many ChanStatusForCaloTower in those logs you linked (probably printed from here).