WF 280.0 failures on aarch64

dan131riley commented 3 years ago

We're seeing occasional failures of WF 280.0 on aarch64. There's a segfault in TBB during exiting, but the relevant bit is why it's exiting--AMPT is printing the message:

 A��Fwrong or unavailable proj name

and calling exit:

#14 0x0000ffff870b8914 in exit () from /lib64/libc.so.6
#15 0x0000ffff82cae5a4 in _gfortran_stop_string (string=<optimized out>, len=<optimized out>, quiet=<optimized out>) at ../../../libgfortran/runtime/stop.c:150
#16 0x0000fffefe46ce58 in hijset_ () from /cvmfs/cms-ib.cern.ch/nweek-02679/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_0_X_2021-05-07-2300/lib/slc7_aarch64_gcc9/libGeneratorInterfaceAMPTInterface.so
#17 0x0000fffefe4b787c in amptset_ () from /cvmfs/cms-ib.cern.ch/nweek-02679/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_0_X_2021-05-07-2300/lib/slc7_aarch64_gcc9/libGeneratorInterfaceAMPTInterface.so
#18 0x0000fffefe40f940 in gen::AMPTHadronizer::call_amptset(double, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int, int, int, int) () from /cvmfs/cms-ib.cern.ch/nweek-02679/slc7_aarch64_gcc9/cms/cmssw/CMSSW_12_0_X_2021-05-07-2300/lib/slc7_aarch64_gcc9/libGeneratorInterfaceAMPTInterface.so

Looking at the code for AMPTHadronizer::call_amptset() there are a couple of obvious problems: https://github.com/cms-sw/cmssw/blob/4252a8a664ef8178526a5924c8bf0a2bbb625f21/GeneratorInterface/AMPTInterface/src/AMPTHadronizer.cc#L230-L245 There's a minor efficiency issue that strings are being passed by value, not by reference, but the main problem is the calls to strlen(frame.data()). std::string::data() is not guaranteed to return a null-terminated string, so it isn't safe to use strlen() there. It's also unnecessary, better to just call frame.length()

cmsbuild commented 3 years ago

A new Issue was created by @dan131riley Dan Riley.

@Dr15Jones, @dpiparo, @silviodonato, @smuzaffar, @makortel, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

makortel commented 3 years ago

Looks like a duplicate of https://github.com/cms-sw/cmssw/issues/33452. @dan131riley Could you copy the new details there?

dan131riley commented 3 years ago

Duplicate.

mrodozov commented 3 years ago

280.0 is running steps 2, 3 and 4 so I think you did it agan Dan :)

dan131riley commented 3 years ago

Still not fixed. I can't reopen #33452 (which was unexpectedly automatically close), so I'll reopen this one...

iarspider commented 2 years ago

@dan131riley gentle ping

iarspider commented 2 years ago

The error seems to be the same, but the stack trace is shorter: (full log):

Thread 1 (Thread 0x400005c99ba0 (LWP 3564543) "cmsRun"):
#0  0x0000400006082b58 in poll () from /lib64/libc.so.6
#1  0x00004000083e0d58 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/nweek-02732/el9_aarch64_gcc11/cms/cmssw/CMSSW_12_4_X_2022-05-09-2300/lib/el9_aarch64_gcc11/pluginFWCoreServicesPlugins.so
#2  0x00004000083e17c4 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/nweek-02732/el9_aarch64_gcc11/cms/cmssw/CMSSW_12_4_X_2022-05-09-2300/lib/el9_aarch64_gcc11/pluginFWCoreServicesPlugins.so
#3  0x00004000083e4064 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/nweek-02732/el9_aarch64_gcc11/cms/cmssw/CMSSW_12_4_X_2022-05-09-2300/lib/el9_aarch64_gcc11/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x0000400005fccad0 in __memcpy_thunderx2 () from /lib64/libc.so.6
#6  0x00004000a1807f4c in amptset_ () from /cvmfs/cms-ib.cern.ch/nweek-02732/el9_aarch64_gcc11/cms/cmssw/CMSSW_12_4_X_2022-05-09-2300/lib/el9_aarch64_gcc11/libGeneratorInterfaceAMPTInterface.so
#7  0x00004000a175fd54 in gen::AMPTHadronizer::call_amptset(double, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int, int, int, int) () from /cvmfs/cms-ib.cern.ch/nweek-02732/el9_aarch64_gcc11/cms/cmssw/CMSSW_12_4_X_2022-05-09-2300/lib/el9_aarch64_gcc11/libGeneratorInterfaceAMPTInterface.so

dan131riley commented 2 years ago

This is after #37840 merged, so it looks like yet another attempt to fix this has failed...

aandvalenzuela commented 1 year ago

It seems this issue is back in CMSSW_13_3_X_2023-08-13-2300 for el8_aarch64_gcc11:


**************************************************************
 Geant4 version Name: geant4-11-01-patch-01 [MT]   (10-February-2023)
  << in Multi-threaded mode >> 
                       Copyright : Geant4 Collaboration
                      References : NIM A 506 (2003), 250-303
                                 : IEEE-TNS 53 (2006), 270-278
                                 : NIM A 835 (2016), 186-225
                             WWW : http://geant4.org/
**************************************************************

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Mon Aug 14 09:13:11 CEST 2023
Thread 10 (Thread 0x400098709c80 (LWP 2863461) "cmsRun"):
#0  0x000040001bcadc58 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
#1  0x000040001ba46fb0 in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /data/cmsbld/jenkins_b/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_aarch64_gcc11/external/gcc/11.4.1-30ebdc301ebd200f2ae0e3d880258e65/gcc-11.4.1/obj/aarch64-redhat-linux-gnu/libstdc++-v3/include/aarch64-redhat-linux-gnu/bits/gthr-default.h:865
#2  std::__condvar::wait (__m=..., this=<optimized out>) at /data/cmsbld/jenkins_b/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_aarch64_gcc11/external/gcc/11.4.1-30ebdc301ebd200f2ae0e3d880258e65/gcc-11.4.1/obj/aarch64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155
#3  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41
#4  0x000040006c9202e4 in omt::ThreadHandoff::threadLoop(void*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/biglib/el8_aarch64_gcc11/pluginSimulation.so
#5  0x000040001bca78b8 in start_thread () from /lib64/libpthread.so.0
#6  0x000040001bd03afc in thread_start () from /lib64/libc.so.6
Thread 9 (Thread 0x400097cf9c80 (LWP 2863460) "cmsRun"):
#0  0x000040001bcadc58 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
#1  0x000040001ba46fb0 in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /data/cmsbld/jenkins_b/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_aarch64_gcc11/external/gcc/11.4.1-30ebdc301ebd200f2ae0e3d880258e65/gcc-11.4.1/obj/aarch64-redhat-linux-gnu/libstdc++-v3/include/aarch64-redhat-linux-gnu/bits/gthr-default.h:865
#2  std::__condvar::wait (__m=..., this=<optimized out>) at /data/cmsbld/jenkins_b/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_aarch64_gcc11/external/gcc/11.4.1-30ebdc301ebd200f2ae0e3d880258e65/gcc-11.4.1/obj/aarch64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155
#3  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41
#4  0x000040006c9202e4 in omt::ThreadHandoff::threadLoop(void*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/biglib/el8_aarch64_gcc11/pluginSimulation.so
#5  0x000040001bca78b8 in start_thread () from /lib64/libpthread.so.0
#6  0x000040001bd03afc in thread_start () from /lib64/libc.so.6
Thread 8 (Thread 0x4000972e9c80 (LWP 2863459) "cmsRun"):
#0  0x000040001bcadc58 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
#1  0x000040001ba46fb0 in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /data/cmsbld/jenkins_b/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_aarch64_gcc11/external/gcc/11.4.1-30ebdc301ebd200f2ae0e3d880258e65/gcc-11.4.1/obj/aarch64-redhat-linux-gnu/libstdc++-v3/include/aarch64-redhat-linux-gnu/bits/gthr-default.h:865
#2  std::__condvar::wait (__m=..., this=<optimized out>) at /data/cmsbld/jenkins_b/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_aarch64_gcc11/external/gcc/11.4.1-30ebdc301ebd200f2ae0e3d880258e65/gcc-11.4.1/obj/aarch64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155
#3  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41
#4  0x000040006c9202e4 in omt::ThreadHandoff::threadLoop(void*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/biglib/el8_aarch64_gcc11/pluginSimulation.so
#5  0x000040001bca78b8 in start_thread () from /lib64/libpthread.so.0
#6  0x000040001bd03afc in thread_start () from /lib64/libc.so.6
Thread 7 (Thread 0x4000968d9c80 (LWP 2863458) "cmsRun"):
#0  0x000040001bcadc58 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
#1  0x000040001ba46fb0 in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /data/cmsbld/jenkins_b/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_aarch64_gcc11/external/gcc/11.4.1-30ebdc301ebd200f2ae0e3d880258e65/gcc-11.4.1/obj/aarch64-redhat-linux-gnu/libstdc++-v3/include/aarch64-redhat-linux-gnu/bits/gthr-default.h:865
#2  std::__condvar::wait (__m=..., this=<optimized out>) at /data/cmsbld/jenkins_b/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_aarch64_gcc11/external/gcc/11.4.1-30ebdc301ebd200f2ae0e3d880258e65/gcc-11.4.1/obj/aarch64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155
#3  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41
#4  0x000040006c9202e4 in omt::ThreadHandoff::threadLoop(void*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/biglib/el8_aarch64_gcc11/pluginSimulation.so
#5  0x000040001bca78b8 in start_thread () from /lib64/libpthread.so.0
#6  0x000040001bd03afc in thread_start () from /lib64/libc.so.6
Thread 6 (Thread 0x400095ec9c80 (LWP 2863457) "cmsRun"):
#0  0x000040001bcadc58 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
#1  0x000040001ba46fb0 in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /data/cmsbld/jenkins_b/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_aarch64_gcc11/external/gcc/11.4.1-30ebdc301ebd200f2ae0e3d880258e65/gcc-11.4.1/obj/aarch64-redhat-linux-gnu/libstdc++-v3/include/aarch64-redhat-linux-gnu/bits/gthr-default.h:865
#2  std::__condvar::wait (__m=..., this=<optimized out>) at /data/cmsbld/jenkins_b/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_aarch64_gcc11/external/gcc/11.4.1-30ebdc301ebd200f2ae0e3d880258e65/gcc-11.4.1/obj/aarch64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155
#3  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41
#4  0x000040006c91b7cc in OscarMTMasterThread::OscarMTMasterThread(edm::ParameterSet const&)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/biglib/el8_aarch64_gcc11/pluginSimulation.so
#5  0x000040001ba4d1ac in std::execute_native_thread_routine (__p=0x46a55760) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#6  0x000040001bca78b8 in start_thread () from /lib64/libpthread.so.0
#7  0x000040001bd03afc in thread_start () from /lib64/libc.so.6
Thread 5 (Thread 0x40003a089c80 (LWP 2862972) "cmsRun"):
#0  0x000040001bdaa76c in poll () from /lib64/libc.so.6
#1  0x000040001dc3e0ec in full_read.constprop () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/lib/el8_aarch64_gcc11/pluginFWCoreServicesPlugins.so
#2  0x000040001dc0c574 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/lib/el8_aarch64_gcc11/pluginFWCoreServicesPlugins.so
#3  0x000040001dc07d58 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/lib/el8_aarch64_gcc11/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x000040001bd023c4 in __memcpy_thunderx2 () from /lib64/libc.so.6
#6  0x000040003a187ccc in amptset_ () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/lib/el8_aarch64_gcc11/libGeneratorInterfaceAMPTInterface.so
#7  0x000040003a0e1184 in gen::AMPTHadronizer::call_amptset(double, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int, int, int, int) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/lib/el8_aarch64_gcc11/libGeneratorInterfaceAMPTInterface.so
#8  0x000040003a0e15c0 in gen::AMPTHadronizer::initializeForInternalPartons() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/lib/el8_aarch64_gcc11/libGeneratorInterfaceAMPTInterface.so
#9  0x000040003a09faa8 in edm::GeneratorFilter<gen::AMPTHadronizer, gen::ExternalDecayDriver>::beginLuminosityBlockProduce(edm::LuminosityBlock&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/lib/el8_aarch64_gcc11/pluginGeneratorInterfaceAMPTInterfacePlugins.so
#10 0x0000400019c09ce4 in edm::one::EDFilterBase::doBeginLuminosityBlock(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#11 0x0000400019bf1c40 in edm::WorkerT<edm::one::EDFilterBase>::implDoBegin(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#12 0x0000400019b47958 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#13 0x0000400019b4acac in edm::SerialTaskQueue::QueuedTask<edm::SerialTaskQueueChain::push<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0> >::execute()::{lambda()#1}&>(tbb::detail::d1::task_group&, edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0> >::execute()::{lambda()#1}&)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#14 0x000040001a1264f0 in tbb::detail::d1::function_task<edm::SerialTaskQueue::spawn(edm::SerialTaskQueue::TaskBase&)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/lib/el8_aarch64_gcc11/libFWCoreConcurrency.so
#15 0x000040001b522c18 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x40001c573600, waiter=<synthetic pointer>..., this=0x40001c603f00) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.9.0-a460e1ec4c4aa5a9523aaf1bd8c1ee76/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#16 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=<synthetic pointer>..., t=0x0, this=0x40001c603f00) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.9.0-a460e1ec4c4aa5a9523aaf1bd8c1ee76/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#17 tbb::detail::r1::arena::process (tls=..., this=0x40001c603780) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.9.0-a460e1ec4c4aa5a9523aaf1bd8c1ee76/tbb-v2021.9.0/src/tbb/arena.cpp:137
#18 tbb::detail::r1::market::process (this=0x40001c62ab00, j=...) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.9.0-a460e1ec4c4aa5a9523aaf1bd8c1ee76/tbb-v2021.9.0/src/tbb/market.cpp:599
#19 0x000040001b52af58 in tbb::detail::r1::rml::private_worker::run (this=0x40001ccf0000) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.9.0-a460e1ec4c4aa5a9523aaf1bd8c1ee76/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#20 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x40001ccf0000) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.9.0-a460e1ec4c4aa5a9523aaf1bd8c1ee76/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#21 0x000040001bca78b8 in start_thread () from /lib64/libpthread.so.0
#22 0x000040001bd03afc in thread_start () from /lib64/libc.so.6
Thread 4 (Thread 0x400039679c80 (LWP 2862971) "cmsRun"):
#0  0x000040001bd84b3c in nanosleep () from /lib64/libc.so.6
#1  0x000040001bd849e0 in sleep () from /lib64/libc.so.6
#2  0x000040001dc03448 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/lib/el8_aarch64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000040001bcadc5c in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
#5  0x000040001ba46fb0 in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /data/cmsbld/jenkins_b/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_aarch64_gcc11/external/gcc/11.4.1-30ebdc301ebd200f2ae0e3d880258e65/gcc-11.4.1/obj/aarch64-redhat-linux-gnu/libstdc++-v3/include/aarch64-redhat-linux-gnu/bits/gthr-default.h:865
#6  std::__condvar::wait (__m=..., this=<optimized out>) at /data/cmsbld/jenkins_b/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_aarch64_gcc11/external/gcc/11.4.1-30ebdc301ebd200f2ae0e3d880258e65/gcc-11.4.1/obj/aarch64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155
#7  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41
#8  0x000040006c9215b4 in OscarMTProducer::beginRun(edm::Run const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/biglib/el8_aarch64_gcc11/pluginSimulation.so
#9  0x0000400019c14048 in edm::stream::ProducingModuleAdaptorBase<edm::stream::EDProducerBase>::doStreamBeginRun(edm::StreamID, edm::RunTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#10 0x0000400019bf9e10 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDoStreamBegin(edm::StreamID, edm::RunTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#11 0x0000400019b234ec in edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)1> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#12 0x0000400019b2cab0 in tbb::detail::d1::function_task<edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)1> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#13 0x000040001b522c18 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x40001c58b900, waiter=<synthetic pointer>..., this=0x40001c603e00) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.9.0-a460e1ec4c4aa5a9523aaf1bd8c1ee76/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#14 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=<synthetic pointer>..., t=0x0, this=0x40001c603e00) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.9.0-a460e1ec4c4aa5a9523aaf1bd8c1ee76/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#15 tbb::detail::r1::arena::process (tls=..., this=0x40001c603780) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.9.0-a460e1ec4c4aa5a9523aaf1bd8c1ee76/tbb-v2021.9.0/src/tbb/arena.cpp:137
#16 tbb::detail::r1::market::process (this=0x40001c62ab00, j=...) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.9.0-a460e1ec4c4aa5a9523aaf1bd8c1ee76/tbb-v2021.9.0/src/tbb/market.cpp:599
#17 0x000040001b52af58 in tbb::detail::r1::rml::private_worker::run (this=0x40001ccf0100) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.9.0-a460e1ec4c4aa5a9523aaf1bd8c1ee76/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#18 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x40001ccf0100) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.9.0-a460e1ec4c4aa5a9523aaf1bd8c1ee76/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#19 0x000040001bca78b8 in start_thread () from /lib64/libpthread.so.0
#20 0x000040001bd03afc in thread_start () from /lib64/libc.so.6
Thread 3 (Thread 0x400038c69c80 (LWP 2862970) "cmsRun"):
#0  0x000040001bd84b3c in nanosleep () from /lib64/libc.so.6
#1  0x000040001bd849e0 in sleep () from /lib64/libc.so.6
#2  0x000040001dc03448 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/lib/el8_aarch64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000040001bcadc5c in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
#5  0x000040001ba46fb0 in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /data/cmsbld/jenkins_b/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_aarch64_gcc11/external/gcc/11.4.1-30ebdc301ebd200f2ae0e3d880258e65/gcc-11.4.1/obj/aarch64-redhat-linux-gnu/libstdc++-v3/include/aarch64-redhat-linux-gnu/bits/gthr-default.h:865
#6  std::__condvar::wait (__m=..., this=<optimized out>) at /data/cmsbld/jenkins_b/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_aarch64_gcc11/external/gcc/11.4.1-30ebdc301ebd200f2ae0e3d880258e65/gcc-11.4.1/obj/aarch64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155
#7  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41
#8  0x000040006c9215b4 in OscarMTProducer::beginRun(edm::Run const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/biglib/el8_aarch64_gcc11/pluginSimulation.so
#9  0x0000400019c14048 in edm::stream::ProducingModuleAdaptorBase<edm::stream::EDProducerBase>::doStreamBeginRun(edm::StreamID, edm::RunTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#10 0x0000400019bf9e10 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDoStreamBegin(edm::StreamID, edm::RunTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#11 0x0000400019b234ec in edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)1> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#12 0x0000400019b2cab0 in tbb::detail::d1::function_task<edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)1> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#13 0x000040001b522c18 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x40001c57b700, waiter=<synthetic pointer>..., this=0x40001c603e80) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.9.0-a460e1ec4c4aa5a9523aaf1bd8c1ee76/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#14 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=<synthetic pointer>..., t=0x0, this=0x40001c603e80) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.9.0-a460e1ec4c4aa5a9523aaf1bd8c1ee76/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#15 tbb::detail::r1::arena::process (tls=..., this=0x40001c603780) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.9.0-a460e1ec4c4aa5a9523aaf1bd8c1ee76/tbb-v2021.9.0/src/tbb/arena.cpp:137
#16 tbb::detail::r1::market::process (this=0x40001c62ab00, j=...) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.9.0-a460e1ec4c4aa5a9523aaf1bd8c1ee76/tbb-v2021.9.0/src/tbb/market.cpp:599
#17 0x000040001b52af58 in tbb::detail::r1::rml::private_worker::run (this=0x40001ccf0080) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.9.0-a460e1ec4c4aa5a9523aaf1bd8c1ee76/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#18 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x40001ccf0080) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.9.0-a460e1ec4c4aa5a9523aaf1bd8c1ee76/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#19 0x000040001bca78b8 in start_thread () from /lib64/libpthread.so.0
#20 0x000040001bd03afc in thread_start () from /lib64/libc.so.6
Thread 2 (Thread 0x40002e269c80 (LWP 2862545) "cmsRun"):
#0  0x000040001bcb2578 in waitpid () from /lib64/libpthread.so.0
#1  0x000040001dc08c70 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/lib/el8_aarch64_gcc11/pluginFWCoreServicesPlugins.so
#2  0x000040001dc0fa80 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/lib/el8_aarch64_gcc11/pluginFWCoreServicesPlugins.so
#3  0x000040001ba4d1ac in std::execute_native_thread_routine (__p=0x42e41040) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x000040001bca78b8 in start_thread () from /lib64/libpthread.so.0
#5  0x000040001bd03afc in thread_start () from /lib64/libc.so.6
Thread 1 (Thread 0x40001b72ca00 (LWP 2861684) "cmsRun"):
#0  0x000040001bd84b3c in nanosleep () from /lib64/libc.so.6
#1  0x000040001bd849e0 in sleep () from /lib64/libc.so.6
#2  0x000040001dc03448 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/lib/el8_aarch64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000040001bcadc5c in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
#5  0x000040001ba46fb0 in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /data/cmsbld/jenkins_b/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_aarch64_gcc11/external/gcc/11.4.1-30ebdc301ebd200f2ae0e3d880258e65/gcc-11.4.1/obj/aarch64-redhat-linux-gnu/libstdc++-v3/include/aarch64-redhat-linux-gnu/bits/gthr-default.h:865
#6  std::__condvar::wait (__m=..., this=<optimized out>) at /data/cmsbld/jenkins_b/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_aarch64_gcc11/external/gcc/11.4.1-30ebdc301ebd200f2ae0e3d880258e65/gcc-11.4.1/obj/aarch64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155
#7  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41
#8  0x000040006c9215b4 in OscarMTProducer::beginRun(edm::Run const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/biglib/el8_aarch64_gcc11/pluginSimulation.so
#9  0x0000400019c14048 in edm::stream::ProducingModuleAdaptorBase<edm::stream::EDProducerBase>::doStreamBeginRun(edm::StreamID, edm::RunTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#10 0x0000400019bf9e10 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDoStreamBegin(edm::StreamID, edm::RunTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#11 0x0000400019b234ec in edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)1> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#12 0x0000400019b2cab0 in tbb::detail::d1::function_task<edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)1> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#13 0x000040001b528774 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x40001c60ec00, this=0x40001c603d80) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.9.0-a460e1ec4c4aa5a9523aaf1bd8c1ee76/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#14 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x40001c603d80) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.9.0-a460e1ec4c4aa5a9523aaf1bd8c1ee76/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#15 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.9.0-a460e1ec4c4aa5a9523aaf1bd8c1ee76/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#16 0x0000400019afee14 in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#17 0x0000400019b0bf48 in edm::EventProcessor::processRuns() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#18 0x0000400019b0c500 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#19 0x000000000040b9d8 in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#20 0x000040001b5210a8 in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.9.0-a460e1ec4c4aa5a9523aaf1bd8c1ee76/tbb-v2021.9.0/src/tbb/arena.cpp:688
#21 0x000000000040f75c in main::{lambda()#1}::operator()() const ()
#22 0x0000000000406fd4 in main ()

The stacktrace contains the same failure as in https://github.com/cms-sw/cmssw/issues/33682#issuecomment-1122325916:

Thread 5 (Thread 0x40003a089c80 (LWP 2862972) "cmsRun"):
#0  0x000040001bdaa76c in poll () from /lib64/libc.so.6
#1  0x000040001dc3e0ec in full_read.constprop () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/lib/el8_aarch64_gcc11/pluginFWCoreServicesPlugins.so
#2  0x000040001dc0c574 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/lib/el8_aarch64_gcc11/pluginFWCoreServicesPlugins.so
#3  0x000040001dc07d58 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/lib/el8_aarch64_gcc11/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x000040001bd023c4 in __memcpy_thunderx2 () from /lib64/libc.so.6
#6  0x000040003a187ccc in amptset_ () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-08-13-2300/lib/el8_aarch64_gcc11/libGeneratorInterfaceAMPTInterface.so
#7  0x000040003a0e1184 in gen::AMPTHadronizer::call_amptset(double, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>,

Find the full log here.

iarspider commented 4 months ago

Occured again in CMSSW_14_1_X_2024-05-09-2300 for el9_aarch64_gcc12:

Thread 1 (Thread 0x40003320b360 (LWP 1794877) "cmsRun"):
#0  0x000040003364f988 in poll () from /lib64/libc.so.6
#1  0x0000400038b4390c in full_read.constprop () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02836/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-05-09-2300/lib/el9_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x0000400038afcb04 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02836/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-05-09-2300/lib/el9_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x0000400038afcc8c in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02836/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-05-09-2300/lib/el9_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x0000400033598a2c in __memcpy_thunderx () from /lib64/libc.so.6
#6  0x00004000620b6ddc in amptset_ () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02836/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-05-09-2300/lib/el9_aarch64_gcc12/libGeneratorInterfaceAMPTInterface.so
#7  0x00004000620110d4 in gen::AMPTHadronizer::call_amptset(double, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int, int, int, int) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02836/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-05-09-2300/lib/el9_aarch64_gcc12/libGeneratorInterfaceAMPTInterface.so
#8  0x000040006201150c in gen::AMPTHadronizer::initializeForInternalPartons() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02836/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-05-09-2300/lib/el9_aarch64_gcc12/libGeneratorInterfaceAMPTInterface.so
#9  0x000040005de7fe40 in edm::GeneratorFilter<gen::AMPTHadronizer, gen::ExternalDecayDriver>::beginLuminosityBlockProduce(edm::LuminosityBlock&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02836/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-05-09-2300/lib/el9_aarch64_gcc12/pluginGeneratorInterfaceAMPTInterfacePlugins.so
#10 0x0000400031600b04 in edm::one::EDFilterBase::doBeginLuminosityBlock(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02836/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-05-09-2300/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#11 0x00004000315e8780 in edm::WorkerT<edm::one::EDFilterBase>::implDoBegin(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02836/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-05-09-2300/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#12 0x0000400031538f14 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02836/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-05-09-2300/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#13 0x00004000315401ac in edm::SerialTaskQueue::QueuedTask<edm::SerialTaskQueueChain::push<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0> >::execute()::{lambda()#1}&>(tbb::detail::d1::task_group&, edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0> >::execute()::{lambda()#1}&)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02836/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-05-09-2300/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#14 0x0000400031b065a0 in tbb::detail::d1::function_task<edm::SerialTaskQueue::spawn(edm::SerialTaskQueue::TaskBase&)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02836/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-05-09-2300/lib/el9_aarch64_gcc12/libFWCoreConcurrency.so
#15 0x00004000330586c4 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x4000361bf000, this=0x4000361b3d80) at /data/cmsbuild/jenkins_c/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-eefd55a6689af9776c07e55b933efc1e/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#16 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x4000361b3d80) at /data/cmsbuild/jenkins_c/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-eefd55a6689af9776c07e55b933efc1e/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#17 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbuild/jenkins_c/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-eefd55a6689af9776c07e55b933efc1e/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#18 0x00004000314f6bb4 in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02836/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-05-09-2300/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#19 0x0000400031505f34 in edm::EventProcessor::processRuns() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02836/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-05-09-2300/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#20 0x0000400031500168 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02836/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-05-09-2300/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#21 0x0000000000406fb0 in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#22 0x0000400033051048 in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbuild/jenkins_c/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc12/external/tbb/v2021.9.0-eefd55a6689af9776c07e55b933efc1e/tbb-v2021.9.0/src/tbb/arena.cpp:688
#23 0x0000000000408824 in main::{lambda()#1}::operator()() const ()
#24 0x0000000000404904 in main ()

full log

cmsbuild commented 4 months ago

cms-bot internal usage

cms-sw / cmssw

WF 280.0 failures on aarch64 #33682