search

cms-sw / cmssw

CMS Offline Software
http://cms-sw.github.io/
Apache License 2.0
1.07k stars 4.28k forks source link

[ASNEEDED_X] Multiple RelVals failing with SIGSEGV #45747

Open iarspider opened 3 weeks ago

iarspider commented 3 weeks ago

For example Relval 132.0 step 1

Not sure if the error is in Geant4 or Pythia6:

Thread 4 (Thread 0x1544f5f1d700 (LWP 474508) "cmsRun"):
#0  0x000015452848fac1 in poll () from /lib64/libc.so.6
#1  0x00001545231ea857 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x00001545231eaa54 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00001544ed71f303 in G4PVPlacement::~G4PVPlacement() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/biglib/el8_amd64_gcc12/pluginSimulation.so
#5  0x00001544ed78af98 in G4PhysicalVolumeStore::Clean() [clone .part.0] () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/biglib/el8_amd64_gcc12/pluginSimulation.so
#6  0x00001544ed78b0b5 in G4PhysicalVolumeStore::~G4PhysicalVolumeStore() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/biglib/el8_amd64_gcc12/pluginSimulation.so
#7  0x00001545283adccc in __run_exit_handlers () from /lib64/libc.so.6
#8  0x00001545283ade00 in exit () from /lib64/libc.so.6
#9  0x00001544fb87e675 in _gfortran_stop_string (string=string@entry=0x0, len=len@entry=0, quiet=quiet@entry=false) at ../../../libgfortran/runtime/stop.c:150
#10 0x00001544f3cd8b08 in pystop (mcod=5) at pystop.f:20
#11 0x00001544f3dbdaae in pdfset (parm=..., value=..., _parm=_parm@entry=20) at pdfset.f:22
#12 0x00001544f3c46266 in pyinit (frame=..., beam=..., target=..., win=8000, _frame=3, _beam=1, _target=1) at pyinit.f:115
#13 0x00001544f4460caf in gen::Pythia6Hadronizer::initializeForInternalPartons() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/pluginGeneratorInterfacePythia6Filters.so
Full stacktrace ``` Thread 10 (Thread 0x1544e8d7f700 (LWP 474533) "cmsRun"): #0 0x0000154527bd048c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00001544ed3c706b in omt::ThreadHandoff::threadLoop(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/biglib/el8_amd64_gcc12/pluginSimulation.so #2 0x0000154527bca1ca in start_thread () from /lib64/libpthread.so.0 #3 0x00001545283968d3 in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x1544e9f7f700 (LWP 474532) "cmsRun"): #0 0x0000154527bd048c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00001544ed3c706b in omt::ThreadHandoff::threadLoop(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/biglib/el8_amd64_gcc12/pluginSimulation.so #2 0x0000154527bca1ca in start_thread () from /lib64/libpthread.so.0 #3 0x00001545283968d3 in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x1544eb17f700 (LWP 474531) "cmsRun"): #0 0x0000154527bd048c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00001544ed3c706b in omt::ThreadHandoff::threadLoop(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/biglib/el8_amd64_gcc12/pluginSimulation.so #2 0x0000154527bca1ca in start_thread () from /lib64/libpthread.so.0 #3 0x00001545283968d3 in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x1544ec37f700 (LWP 474530) "cmsRun"): #0 0x0000154527bd048c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00001544ed3c706b in omt::ThreadHandoff::threadLoop(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/biglib/el8_amd64_gcc12/pluginSimulation.so #2 0x0000154527bca1ca in start_thread () from /lib64/libpthread.so.0 #3 0x00001545283968d3 in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x1544ecddd700 (LWP 474529) "cmsRun"): #0 0x0000154527bd048c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00001544ed3ad3c3 in OscarMTMasterThread::OscarMTMasterThread(edm::ParameterSet const&)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/biglib/el8_amd64_gcc12/pluginSimulation.so #2 0x000015452882ba73 in std::execute_native_thread_routine (__p=0x1544ef3ec7c0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #3 0x0000154527bca1ca in start_thread () from /lib64/libpthread.so.0 #4 0x00001545283968d3 in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x1544f4fff700 (LWP 474509) "cmsRun"): #0 0x0000154528465098 in nanosleep () from /lib64/libc.so.6 #1 0x0000154528464f9e in sleep () from /lib64/libc.so.6 #2 0x00001545231e71d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so #3 #4 0x0000154527bd048c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #5 0x00001544ed3c94d3 in OscarMTProducer::beginRun(edm::Run const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/biglib/el8_amd64_gcc12/pluginSimulation.so #6 0x000015452938f98d in edm::WorkerT::implDoStreamBegin(edm::StreamID, edm::RunTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so #7 0x00001545292b1b40 in edm::Worker::doWorkNoPrefetchingAsync >(edm::WaitingTaskHolder, edm::OccurrenceTraits::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so #8 0x00001545292bbbd9 in tbb::detail::d1::function_task >(edm::WaitingTaskHolder, edm::OccurrenceTraits::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so #9 0x00001545294f4b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x154526283b00, waiter=..., this=0x154526339500) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322 #10 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x154526339500) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458 #11 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137 #12 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599 #13 0x00001545294f6cee in tbb::detail::r1::rml::private_worker::run (this=0x154523b6a000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271 #14 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x154523b6a000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221 #15 0x0000154527bca1ca in start_thread () from /lib64/libpthread.so.0 #16 0x00001545283968d3 in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x1544f5f1d700 (LWP 474508) "cmsRun"): #0 0x000015452848fac1 in poll () from /lib64/libc.so.6 #1 0x00001545231ea857 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so #2 0x00001545231eaa54 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so #3 #4 0x00001544ed71f303 in G4PVPlacement::~G4PVPlacement() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/biglib/el8_amd64_gcc12/pluginSimulation.so #5 0x00001544ed78af98 in G4PhysicalVolumeStore::Clean() [clone .part.0] () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/biglib/el8_amd64_gcc12/pluginSimulation.so #6 0x00001544ed78b0b5 in G4PhysicalVolumeStore::~G4PhysicalVolumeStore() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/biglib/el8_amd64_gcc12/pluginSimulation.so #7 0x00001545283adccc in __run_exit_handlers () from /lib64/libc.so.6 #8 0x00001545283ade00 in exit () from /lib64/libc.so.6 #9 0x00001544fb87e675 in _gfortran_stop_string (string=string@entry=0x0, len=len@entry=0, quiet=quiet@entry=false) at ../../../libgfortran/runtime/stop.c:150 #10 0x00001544f3cd8b08 in pystop (mcod=5) at pystop.f:20 #11 0x00001544f3dbdaae in pdfset (parm=..., value=..., _parm=_parm@entry=20) at pdfset.f:22 #12 0x00001544f3c46266 in pyinit (frame=..., beam=..., target=..., win=8000, _frame=3, _beam=1, _target=1) at pyinit.f:115 #13 0x00001544f4460caf in gen::Pythia6Hadronizer::initializeForInternalPartons() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/pluginGeneratorInterfacePythia6Filters.so #14 0x00001544f4461ddd in edm::GeneratorFilter::beginLuminosityBlockProduce(edm::LuminosityBlock&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/pluginGeneratorInterfacePythia6Filters.so #15 0x000015452939fb87 in edm::one::EDFilterBase::doBeginLuminosityBlock(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so #16 0x000015452938e010 in edm::WorkerT::implDoBegin(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so #17 0x00001545292cf1ca in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so #18 0x00001545292d114a in edm::SerialTaskQueue::QueuedTask >::execute()::{lambda()#1}&>(tbb::detail::d1::task_group&, edm::Worker::RunModuleTask >::execute()::{lambda()#1}&)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so #19 0x00001545295a0b35 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/libFWCoreConcurrency.so #20 0x00001545294f4b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x15452629b600, waiter=..., this=0x154526339400) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322 #21 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x154526339400) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458 #22 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137 #23 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599 #24 0x00001545294f6cee in tbb::detail::r1::rml::private_worker::run (this=0x154523b6a100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271 #25 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x154523b6a100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221 #26 0x0000154527bca1ca in start_thread () from /lib64/libpthread.so.0 #27 0x00001545283968d3 in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x1544f691e700 (LWP 474507) "cmsRun"): #0 0x0000154528465098 in nanosleep () from /lib64/libc.so.6 #1 0x0000154528464f9e in sleep () from /lib64/libc.so.6 #2 0x00001545231e71d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so #3 #4 0x0000154527bd048c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #5 0x00001544ed3c94d3 in OscarMTProducer::beginRun(edm::Run const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/biglib/el8_amd64_gcc12/pluginSimulation.so #6 0x000015452938f98d in edm::WorkerT::implDoStreamBegin(edm::StreamID, edm::RunTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so #7 0x00001545292b1b40 in edm::Worker::doWorkNoPrefetchingAsync >(edm::WaitingTaskHolder, edm::OccurrenceTraits::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so #8 0x00001545292bbbd9 in tbb::detail::d1::function_task >(edm::WaitingTaskHolder, edm::OccurrenceTraits::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so #9 0x00001545294f4b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x15452628b900, waiter=..., this=0x154526339480) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322 #10 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x154526339480) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458 #11 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137 #12 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599 #13 0x00001545294f6cee in tbb::detail::r1::rml::private_worker::run (this=0x154523b6a080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271 #14 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x154523b6a080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221 #15 0x0000154527bca1ca in start_thread () from /lib64/libpthread.so.0 #16 0x00001545283968d3 in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x154501c6d700 (LWP 474430) "cmsRun"): #0 0x0000154528464e42 in waitpid () from /lib64/libc.so.6 #1 0x00001545231e7327 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so #2 0x00001545231ea63a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so #3 0x000015452882ba73 in std::execute_native_thread_routine (__p=0x1545048ba440) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #4 0x0000154527bca1ca in start_thread () from /lib64/libpthread.so.0 #5 0x00001545283968d3 in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x154526e52580 (LWP 474343) "cmsRun"): #0 0x0000154528465098 in nanosleep () from /lib64/libc.so.6 #1 0x0000154528464f9e in sleep () from /lib64/libc.so.6 #2 0x00001545231e71d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so #3 #4 0x0000154527bd048c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #5 0x00001544ed3c94d3 in OscarMTProducer::beginRun(edm::Run const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/biglib/el8_amd64_gcc12/pluginSimulation.so #6 0x000015452938f98d in edm::WorkerT::implDoStreamBegin(edm::StreamID, edm::RunTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so #7 0x00001545292b1b40 in edm::Worker::doWorkNoPrefetchingAsync >(edm::WaitingTaskHolder, edm::OccurrenceTraits::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so #8 0x00001545292bbbd9 in tbb::detail::d1::function_task >(edm::WaitingTaskHolder, edm::OccurrenceTraits::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so #9 0x00001545294fd3e1 in tbb::detail::r1::task_dispatcher::local_wait_for_all (waiter=..., t=, this=0x154526339380) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322 #10 tbb::detail::r1::task_dispatcher::local_wait_for_all (waiter=..., t=, this=0x154526339380) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458 #11 tbb::detail::r1::task_dispatcher::execute_and_wait (t=, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168 #12 0x00001545292935fb in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so #13 0x00001545292a10ef in edm::EventProcessor::processRuns() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so #14 0x00001545292a15a1 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so #15 0x000000000040840c in tbb::detail::d1::task_arena_function::operator()() const () #16 0x00001545294e99ad in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:688 #17 0x000000000040a0f2 in main::{lambda()#1}::operator()() const () #18 0x0000000000405100 in main () Current Modules: Module: Pythia6GeneratorFilter:generator (crashed) ```

Geant4 also reports this exception:

-------- EEEE ------- G4Exception-START -------- EEEE -------
*** G4Exception : Run0107
      issued by : G4PhysicsListHelper::RegisterProcess
No Ordering Parameter Table 

-------- EEEE -------- G4Exception-END --------- EEEE -------

Some RelVals (e.g. 159.03) are reported to fail with SIGABRT, but log indicates that SIGSEGV also occured:

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

pure virtual method called
terminate called without an active exception
cmsbuild commented 3 weeks ago

cms-bot internal usage

cmsbuild commented 3 weeks ago

A new Issue was created by @iarspider.

@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

iarspider commented 3 weeks ago

Two RelVals - Relval 574.0 step 1 and Relval 577.0 step 1 - failed with 

 Error: you did not link PDFLIB correctly.
 Dummy routine PDFSET in PYTHIA file called instead.
 Execution stopped!
makortel commented 3 weeks ago

assign core

cmsbuild commented 3 weeks ago

New categories assigned: core

@Dr15Jones,@makortel,@smuzaffar you have been requested to review this Pull request/Issue and eventually sign? Thanks

makortel commented 4 days ago

FYI @cms-sw/simulation-l2

iarspider commented 4 days ago

@civanch latest ASNEEDED_X IB: link, example log link

civanch commented 4 days ago

@iarspider , would it be possible to test GEANT4 branch? It uses the most recent Geant4. Even in previous cases OrderingTable should not be opened, in the latest version of G4 both the table and the call to it are removed.

civanch commented 2 days ago

I am not sure but suspect that there are two problems: in generator there is no needed data and in the Geant4 initialisation data race in destruction in circomstances of incomplete initial initialisation. Proposed PR #45986 may be useful for Geant4 part but cannot fix generator problem.

civanch commented 1 day ago

It seems that #45986 removes Geant4 warnings/exceptions but generator problems remain.