search

cms-sw / cmssw

CMS Offline Software
http://cms-sw.github.io/
Apache License 2.0
1.07k stars 4.28k forks source link

[CLANG] Segmentation Fault in `Herwig7GeneratorFilter:generator` #45510

Open aandvalenzuela opened 1 month ago

aandvalenzuela commented 1 month ago

Hello,

RelVals 511.0, 535.0, 537.0, 538.0 and 539.0 failed with segmentation fault in CMSSW_14_1_CLANG_X_2024-07-18-2300:

%MSG
Begin processing the 1st record. Run 1, Event 1, LumiSection 1 on stream 3 at 19-Jul-2024 01:43:09.296 CEST
Begin processing the 2nd record. Run 1, Event 2, LumiSection 1 on stream 1 at 19-Jul-2024 01:43:09.306 CEST
Begin processing the 3rd record. Run 1, Event 3, LumiSection 1 on stream 2 at 19-Jul-2024 01:43:09.307 CEST
Begin processing the 4th record. Run 1, Event 4, LumiSection 1 on stream 0 at 19-Jul-2024 01:43:09.311 CEST

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Fri Jul 19 01:43:09 CEST 2024
Thread 5 (Thread 0x1542415ff700 (LWP 2757243) "cmsRun"):
#0  0x00001542782ab098 in nanosleep () from /lib64/libc.so.6
#1  0x00001542782aaf9e in sleep () from /lib64/libc.so.6
#2  0x00001542732a7544 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-07-18-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00001542781dc41d in syscall () from /lib64/libc.so.6
#5  0x0000154279445fd2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x154273be3024) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x154273be3024) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x154273be3020) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x154273be3000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x154273be3000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x00001542785811ca in start_thread () from /lib64/libpthread.so.0
#11 0x00001542781dc8d3 in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x15424243a700 (LWP 2757242) "cmsRun"):
#0  0x00001542782ab098 in nanosleep () from /lib64/libc.so.6
#1  0x00001542782aaf9e in sleep () from /lib64/libc.so.6
#2  0x00001542732a7544 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-07-18-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00001542781dc41d in syscall () from /lib64/libc.so.6
#5  0x0000154279445fd2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x154273be3124) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x154273be3124) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x154273be3120) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x154273be3100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x154273be3100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x00001542785811ca in start_thread () from /lib64/libpthread.so.0
#11 0x00001542781dc8d3 in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x154242e3b700 (LWP 2757241) "cmsRun"):
#0  0x00001542782ab098 in nanosleep () from /lib64/libc.so.6
#1  0x00001542782aaf9e in sleep () from /lib64/libc.so.6
#2  0x00001542732a7544 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-07-18-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00001542781dc41d in syscall () from /lib64/libc.so.6
#5  0x0000154279445fd2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x154273be30a4) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x154273be30a4) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x154273be30a0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x154273be3080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x154273be3080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x00001542785811ca in start_thread () from /lib64/libpthread.so.0
#11 0x00001542781dc8d3 in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x154251f9b700 (LWP 2757166) "cmsRun"):
#0  0x000015427858b6a2 in waitpid () from /lib64/libpthread.so.0
#1  0x00001542732a81f1 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-07-18-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x0000154278c15a73 in std::execute_native_thread_routine (__p=0x154253ab3650) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#3  0x00001542785811ca in start_thread () from /lib64/libpthread.so.0
#4  0x00001542781dc8d3 in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x1542776f9680 (LWP 2757122) "cmsRun"):
#0  0x00001542782d5ac1 in poll () from /lib64/libc.so.6
#1  0x00001542732a84bd in (anonymous namespace)::full_read(int, char*, unsigned long, int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-07-18-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x00001542732a7f54 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-07-18-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x00001542732a78bf in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-07-18-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x000015423ea64965 in ThePEG::Particle::parents() const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-07-18-2300/external/el8_amd64_gcc12/lib/libThePEG.so.30
#6  0x000015423eb98543 in ThePEG::Step::addSubProcess(ThePEG::Pointer::TransientRCPtr<ThePEG::SubProcess>) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-07-18-2300/external/el8_amd64_gcc12/lib/libThePEG.so.30
#7  0x000015423eb4a8ce in ThePEG::StandardEventHandler::performCollision() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-07-18-2300/external/el8_amd64_gcc12/lib/libThePEG.so.30
#8  0x000015423eb50a6d in ThePEG::StandardEventHandler::generateEvent() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-07-18-2300/external/el8_amd64_gcc12/lib/libThePEG.so.30
#9  0x000015423eabd5c2 in ThePEG::EventGenerator::doShoot() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-07-18-2300/external/el8_amd64_gcc12/lib/libThePEG.so.30
#10 0x000015423eabca4b in ThePEG::EventGenerator::shoot() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-07-18-2300/external/el8_amd64_gcc12/lib/libThePEG.so.30
#11 0x0000154241a196c9 in Herwig7Hadronizer::generatePartonsAndHadronize() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-07-17-1700/lib/el8_amd64_gcc12/pluginGeneratorInterfaceHerwig7HadronizerPlugins.so
#12 0x0000154241a1f364 in edm::GeneratorFilter<Herwig7Hadronizer, gen::ExternalDecayDriver>::filter(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-07-17-1700/lib/el8_amd64_gcc12/pluginGeneratorInterfaceHerwig7HadronizerPlugins.so
#13 0x000015427ae63410 in edm::one::EDFilterBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-07-17-1700/lib/el8_amd64_gcc12/libFWCoreFramework.so
#14 0x000015427ae4d24b in edm::WorkerT<edm::one::EDFilterBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-07-17-1700/lib/el8_amd64_gcc12/libFWCoreFramework.so
#15 0x000015427ad9b98b in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-07-17-1700/lib/el8_amd64_gcc12/libFWCoreFramework.so
#16 0x000015427ad9b795 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-07-17-1700/lib/el8_amd64_gcc12/libFWCoreFramework.so
#17 0x000015427ad9b4c3 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-07-17-1700/lib/el8_amd64_gcc12/libFWCoreFramework.so
#18 0x000015427ad9d34f in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-07-17-1700/lib/el8_amd64_gcc12/libFWCoreFramework.so
#19 0x000015427ad9d105 in edm::SerialTaskQueue::QueuedTask<edm::SerialTaskQueueChain::push<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&>(tbb::detail::d1::task_group&, edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-07-17-1700/lib/el8_amd64_gcc12/libFWCoreFramework.so
#20 0x000015427aa6f3d9 in tbb::detail::d1::function_task<edm::SerialTaskQueue::spawn(edm::SerialTaskQueue::TaskBase&)::$_0>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-07-17-1700/lib/el8_amd64_gcc12/libFWCoreConcurrency.so
#21 0x000015427944c3e1 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x154276bc9380) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#22 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x154276bc9380) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#23 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#24 0x000015427ad0eb58 in void tbb::detail::d0::try_call_proxy<tbb::detail::d1::task_group_base::wait()::{lambda()#1}>::on_completion<tbb::detail::d1::task_group_base::wait()::{lambda()#2}>(tbb::detail::d1::task_group_base::wait()::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-07-17-1700/lib/el8_amd64_gcc12/libFWCoreFramework.so
#25 0x000015427ad0ceb5 in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-07-17-1700/lib/el8_amd64_gcc12/libFWCoreFramework.so
#26 0x000015427acea1b0 in edm::EventProcessor::processRuns() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-07-17-1700/lib/el8_amd64_gcc12/libFWCoreFramework.so
#27 0x000015427ace74ab in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-07-17-1700/lib/el8_amd64_gcc12/libFWCoreFramework.so
#28 0x0000563007c82b75 in tbb::detail::d1::task_arena_function<main::$_0::operator()() const::{lambda()#1}, void>::operator()() const ()
#29 0x00001542794389ad in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/arena.cpp:688
#30 0x0000563007c81d06 in main::$_0::operator()() const ()
#31 0x0000563007c7f7ff in main ()

Current Modules:

Module: Herwig7GeneratorFilter:generator (crashed)
Module: none
Module: none
Module: none

A fatal system signal has occurred: segmentation violation
timeout: the monitored command dumped core
cmsbuild commented 1 month ago

cms-bot internal usage

cmsbuild commented 1 month ago

A new Issue was created by @aandvalenzuela.

@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

makortel commented 1 month ago

assign generators

cmsbuild commented 1 month ago

New categories assigned: generators

@bbilin,@mkirsano,@menglu21,@lviliani you have been requested to review this Pull request/Issue and eventually sign? Thanks

iarspider commented 4 weeks ago

Two more occurances: 511.0 

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Wed Aug  7 03:48:52 CEST 2024
Thread 5 (Thread 0x147c59dff700 (LWP 1817545) "cmsRun"):
#0  0x0000147c91356098 in nanosleep () from /lib64/libc.so.6
#1  0x0000147c91355f9e in sleep () from /lib64/libc.so.6
#2  0x0000147c8ca36544 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000147c9128741d in syscall () from /lib64/libc.so.6
#5  0x0000147c924f0fd2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x147c8db34024) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x147c8db34024) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x147c8db34020) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x147c8db34000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x147c8db34000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x0000147c9162c1ca in start_thread () from /lib64/libpthread.so.0
#11 0x0000147c912878d3 in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x147c5ad4b700 (LWP 1817544) "cmsRun"):
#0  0x0000147c91356098 in nanosleep () from /lib64/libc.so.6
#1  0x0000147c91355f9e in sleep () from /lib64/libc.so.6
#2  0x0000147c8ca36544 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000147c9128741d in syscall () from /lib64/libc.so.6
#5  0x0000147c924f0fd2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x147c8db34124) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x147c8db34124) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x147c8db34120) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x147c8db34100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x147c8db34100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x0000147c9162c1ca in start_thread () from /lib64/libpthread.so.0
#11 0x0000147c912878d3 in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x147c5b74c700 (LWP 1817543) "cmsRun"):
#0  0x0000147c91380ac1 in poll () from /lib64/libc.so.6
#1  0x0000147c8ca374bd in (anonymous namespace)::full_read(int, char*, unsigned long, int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x0000147c8ca36f54 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x0000147c8ca368bf in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x0000147c572b8965 in ThePEG::Particle::parents() const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libThePEG.so.30
#6  0x0000147c573ec543 in ThePEG::Step::addSubProcess(ThePEG::Pointer::TransientRCPtr<ThePEG::SubProcess>) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libThePEG.so.30
#7  0x0000147c5739e8ce in ThePEG::StandardEventHandler::performCollision() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libThePEG.so.30
#8  0x0000147c573a4a6d in ThePEG::StandardEventHandler::generateEvent() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libThePEG.so.30
#9  0x0000147c573115c2 in ThePEG::EventGenerator::doShoot() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libThePEG.so.30
#10 0x0000147c57310a4b in ThePEG::EventGenerator::shoot() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libThePEG.so.30
#11 0x0000147c5a32a6c9 in Herwig7Hadronizer::generatePartonsAndHadronize() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginGeneratorInterfaceHerwig7HadronizerPlugins.so
#12 0x0000147c5a330364 in edm::GeneratorFilter<Herwig7Hadronizer, gen::ExternalDecayDriver>::filter(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginGeneratorInterfaceHerwig7HadronizerPlugins.so
#13 0x0000147c93f0e4a0 in edm::one::EDFilterBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#14 0x0000147c93ef82db in edm::WorkerT<edm::one::EDFilterBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#15 0x0000147c93e454fb in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#16 0x0000147c93e45305 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#17 0x0000147c93e45033 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#18 0x0000147c93e46ebf in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#19 0x0000147c93e46c75 in edm::SerialTaskQueue::QueuedTask<edm::SerialTaskQueueChain::push<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&>(tbb::detail::d1::task_group&, edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#20 0x0000147c93b1a3d9 in tbb::detail::d1::function_task<edm::SerialTaskQueue::spawn(edm::SerialTaskQueue::TaskBase&)::$_0>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreConcurrency.so
#21 0x0000147c924eeb3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x147c8fb12200, waiter=..., this=0x147c8fbc9480) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#22 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x147c8fbc9480) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#23 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#24 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#25 0x0000147c924f0cee in tbb::detail::r1::rml::private_worker::run (this=0x147c8db34080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#26 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x147c8db34080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#27 0x0000147c9162c1ca in start_thread () from /lib64/libpthread.so.0
#28 0x0000147c912878d3 in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x147c6a888700 (LWP 1817492) "cmsRun"):
#0  0x0000147c916366a2 in waitpid () from /lib64/libpthread.so.0
#1  0x0000147c8ca371f1 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x0000147c91cc0a73 in std::execute_native_thread_routine (__p=0x147c6bdad6c0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#3  0x0000147c9162c1ca in start_thread () from /lib64/libpthread.so.0
#4  0x0000147c912878d3 in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x147c907a4680 (LWP 1817482) "cmsRun"):
#0  0x0000147c91356098 in nanosleep () from /lib64/libc.so.6
#1  0x0000147c91355f9e in sleep () from /lib64/libc.so.6
#2  0x0000147c8ca36544 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000147c9128741d in syscall () from /lib64/libc.so.6
#5  0x0000147c924f5ca4 in tbb::detail::r1::futex_wait (comparand=2, futex=0x7ffe5d86df10) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x7ffe5d86df10) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::sleep_node<tbb::detail::r1::market_context>::wait (this=0x7ffe5d86dee0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:170
#8  tbb::detail::r1::concurrent_monitor_base<tbb::detail::r1::market_context>::commit_wait (this=<optimized out>, node=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:232
#9  tbb::detail::r1::concurrent_monitor_base<tbb::detail::r1::market_context>::commit_wait (node=..., this=0x147c8fbcb598) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:228
#10 tbb::detail::r1::concurrent_monitor_base<tbb::detail::r1::market_context>::wait<tbb::detail::r1::sleep_node<tbb::detail::r1::market_context>, tbb::detail::r1::external_waiter::pause(tbb::detail::r1::arena_slot&)::{lambda()#1}&>(tbb::detail::r1::external_waiter::pause(tbb::detail::r1::arena_slot&)::{lambda()#1}&, tbb::detail::r1::sleep_node<tbb::detail::r1::market_context>&&) (node=..., pred=<synthetic pointer>..., this=0x147c8fbcb598) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:262
#11 tbb::detail::r1::sleep_waiter::sleep<tbb::detail::r1::external_waiter::pause(tbb::detail::r1::arena_slot&)::{lambda()#1}>(unsigned long, tbb::detail::r1::external_waiter::pause(tbb::detail::r1::arena_slot&)::{lambda()#1}) (this=<optimized out>, wakeup_condition=..., uniq_tag=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/waiters.h:118
#12 tbb::detail::r1::external_waiter::pause (this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/waiters.h:144
#13 tbb::detail::r1::external_waiter::pause (this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/waiters.h:137
#14 tbb::detail::r1::task_dispatcher::receive_or_steal_task<true, tbb::detail::r1::external_waiter> (this=<optimized out>, tls=..., ed=..., waiter=..., isolation=<optimized out>, fifo_allowed=<optimized out>, critical_allowed=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:231
#15 0x0000147c924f74e2 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x0, this=0x147c8fbc9380) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:350
#16 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x147c8fbc9380) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#17 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#18 0x0000147c93db8668 in void tbb::detail::d0::try_call_proxy<tbb::detail::d1::task_group_base::wait()::{lambda()#1}>::on_completion<tbb::detail::d1::task_group_base::wait()::{lambda()#2}>(tbb::detail::d1::task_group_base::wait()::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#19 0x0000147c93db69c5 in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#20 0x0000147c93d94af0 in edm::EventProcessor::processRuns() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#21 0x0000147c93d91deb in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#22 0x000055fee7de7b6e in tbb::detail::d1::task_arena_function<main::$_0::operator()() const::{lambda()#1}, void>::operator()() const ()
#23 0x0000147c924e39ad in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:688
#24 0x000055fee7de6d06 in main::$_0::operator()() const ()
#25 0x000055fee7de47ff in main ()

Current Modules:

Module: Herwig7GeneratorFilter:generator (crashed)
Module: none
Module: none
Module: none

A fatal system signal has occurred: segmentation violation
timeout: the monitored command dumped core

and 539.0

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Wed Aug  7 05:35:50 CEST 2024
Thread 5 (Thread 0x1476eb9ff700 (LWP 97727) "cmsRun"):
#0  0x0000147722dbaac1 in poll () from /lib64/libc.so.6
#1  0x000014771e6f74bd in (anonymous namespace)::full_read(int, char*, unsigned long, int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x000014771e6f6f54 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x000014771e6f68bf in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00001476e8ba2965 in ThePEG::Particle::parents() const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libThePEG.so.30
#6  0x00001476e8cd6543 in ThePEG::Step::addSubProcess(ThePEG::Pointer::TransientRCPtr<ThePEG::SubProcess>) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libThePEG.so.30
#7  0x00001476e8c888ce in ThePEG::StandardEventHandler::performCollision() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libThePEG.so.30
#8  0x00001476e8c8ea6d in ThePEG::StandardEventHandler::generateEvent() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libThePEG.so.30
#9  0x00001476e8bfb5c2 in ThePEG::EventGenerator::doShoot() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libThePEG.so.30
#10 0x00001476e8bfaa4b in ThePEG::EventGenerator::shoot() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libThePEG.so.30
#11 0x00001476eadde6c9 in Herwig7Hadronizer::generatePartonsAndHadronize() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginGeneratorInterfaceHerwig7HadronizerPlugins.so
#12 0x00001476eade4364 in edm::GeneratorFilter<Herwig7Hadronizer, gen::ExternalDecayDriver>::filter(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginGeneratorInterfaceHerwig7HadronizerPlugins.so
#13 0x0000147725ae54a0 in edm::one::EDFilterBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#14 0x0000147725acf2db in edm::WorkerT<edm::one::EDFilterBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#15 0x0000147725a1c4fb in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#16 0x0000147725a1c305 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#17 0x0000147725a1c033 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#18 0x0000147725a1debf in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#19 0x0000147725a1dc75 in edm::SerialTaskQueue::QueuedTask<edm::SerialTaskQueueChain::push<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&>(tbb::detail::d1::task_group&, edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#20 0x0000147725c553d9 in tbb::detail::d1::function_task<edm::SerialTaskQueue::spawn(edm::SerialTaskQueue::TaskBase&)::$_0>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreConcurrency.so
#21 0x0000147724d69b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x1477216fa000, waiter=..., this=0x1477217c9500) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#22 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x1477217c9500) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#23 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#24 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#25 0x0000147724d6bcee in tbb::detail::r1::rml::private_worker::run (this=0x14771f874000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#26 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14771f874000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#27 0x00001477230661ca in start_thread () from /lib64/libpthread.so.0
#28 0x0000147722cc18d3 in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x1476eca14700 (LWP 97726) "cmsRun"):
#0  0x0000147722d90098 in nanosleep () from /lib64/libc.so.6
#1  0x0000147722d8ff9e in sleep () from /lib64/libc.so.6
#2  0x000014771e6f6544 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000147722cc141d in syscall () from /lib64/libc.so.6
#5  0x0000147724d6bfd2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x14771f874124) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14771f874124) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x14771f874120) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x14771f874100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14771f874100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x00001477230661ca in start_thread () from /lib64/libpthread.so.0
#11 0x0000147722cc18d3 in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x1476ed415700 (LWP 97725) "cmsRun"):
#0  0x0000147722d90098 in nanosleep () from /lib64/libc.so.6
#1  0x0000147722d8ff9e in sleep () from /lib64/libc.so.6
#2  0x000014771e6f6544 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000147722cc141d in syscall () from /lib64/libc.so.6
#5  0x0000147724d6bfd2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x14771f8740a4) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14771f8740a4) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x14771f8740a0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x14771f874080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14771f874080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x00001477230661ca in start_thread () from /lib64/libpthread.so.0
#11 0x0000147722cc18d3 in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x1476fbdff700 (LWP 97703) "cmsRun"):
#0  0x00001477230706a2 in waitpid () from /lib64/libpthread.so.0
#1  0x000014771e6f71f1 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x00001477236d8a73 in std::execute_native_thread_routine (__p=0x1476fd3326b0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#3  0x00001477230661ca in start_thread () from /lib64/libpthread.so.0
#4  0x0000147722cc18d3 in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x1477246d4680 (LWP 97683) "cmsRun"):
#0  0x0000147722d90098 in nanosleep () from /lib64/libc.so.6
#1  0x0000147722d8ff9e in sleep () from /lib64/libc.so.6
#2  0x000014771e6f6544 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000147722cc141d in syscall () from /lib64/libc.so.6
#5  0x0000147724d70ca4 in tbb::detail::r1::futex_wait (comparand=2, futex=0x7fff1d7803d0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x7fff1d7803d0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::sleep_node<tbb::detail::r1::market_context>::wait (this=0x7fff1d7803a0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:170
#8  tbb::detail::r1::concurrent_monitor_base<tbb::detail::r1::market_context>::commit_wait (this=<optimized out>, node=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:232
#9  tbb::detail::r1::concurrent_monitor_base<tbb::detail::r1::market_context>::commit_wait (node=..., this=0x1477217cb598) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:228
#10 tbb::detail::r1::concurrent_monitor_base<tbb::detail::r1::market_context>::wait<tbb::detail::r1::sleep_node<tbb::detail::r1::market_context>, tbb::detail::r1::external_waiter::pause(tbb::detail::r1::arena_slot&)::{lambda()#1}&>(tbb::detail::r1::external_waiter::pause(tbb::detail::r1::arena_slot&)::{lambda()#1}&, tbb::detail::r1::sleep_node<tbb::detail::r1::market_context>&&) (node=..., pred=<synthetic pointer>..., this=0x1477217cb598) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:262
#11 tbb::detail::r1::sleep_waiter::sleep<tbb::detail::r1::external_waiter::pause(tbb::detail::r1::arena_slot&)::{lambda()#1}>(unsigned long, tbb::detail::r1::external_waiter::pause(tbb::detail::r1::arena_slot&)::{lambda()#1}) (this=<optimized out>, wakeup_condition=..., uniq_tag=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/waiters.h:118
#12 tbb::detail::r1::external_waiter::pause (this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/waiters.h:144
#13 tbb::detail::r1::external_waiter::pause (this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/waiters.h:137
#14 tbb::detail::r1::task_dispatcher::receive_or_steal_task<true, tbb::detail::r1::external_waiter> (this=<optimized out>, tls=..., ed=..., waiter=..., isolation=<optimized out>, fifo_allowed=<optimized out>, critical_allowed=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:231
#15 0x0000147724d724e2 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x0, this=0x1477217c9380) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:350
#16 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x1477217c9380) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#17 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#18 0x000014772598f668 in void tbb::detail::d0::try_call_proxy<tbb::detail::d1::task_group_base::wait()::{lambda()#1}>::on_completion<tbb::detail::d1::task_group_base::wait()::{lambda()#2}>(tbb::detail::d1::task_group_base::wait()::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#19 0x000014772598d9c5 in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#20 0x000014772596baf0 in edm::EventProcessor::processRuns() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#21 0x0000147725968deb in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#22 0x000055ebf54ebb6e in tbb::detail::d1::task_arena_function<main::$_0::operator()() const::{lambda()#1}, void>::operator()() const ()
#23 0x0000147724d5e9ad in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:688
#24 0x000055ebf54ead06 in main::$_0::operator()() const ()
#25 0x000055ebf54e87ff in main ()

Current Modules:

Module: Herwig7GeneratorFilter:generator (crashed)
Module: none
Module: none
Module: none

A fatal system signal has occurred: segmentation violation
timeout: the monitored command dumped core
lviliani commented 23 hours ago

Is this the same problem as https://github.com/cms-sw/cmssw/issues/45872?

Dominic-Stafford commented 22 hours ago

From first glance it looks like these issues are happening at different parts of the Herwig code, but it may well be a set of different errors all coming from a CLANG version change or something similar, I'll look into this.