search

cms-sw / cmssw

CMS Offline Software
http://cms-sw.github.io/
Apache License 2.0
1.08k stars 4.3k forks source link

SIGSEGV in (anonymous namespace)::maybeConfigPileUp #46341

Open iarspider opened 1 week ago

iarspider commented 1 week ago

RelVal 13034.412 crashed in (anonymous namespace)::maybeConfigPileUp:

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thu Oct 10 04:57:38 CEST 2024
Thread 7 (Thread 0x14619e7ff700 (LWP 1328548) "cmsRun"):
#0  0x00001462250c141d in syscall () from /lib64/libc.so.6
#1  0x0000146227125fd2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x146221574024) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/semaphore.h:100
#2  tbb::detail::r1::binary_semaphore::P (this=0x146221574024) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/semaphore.h:253
#3  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x146221574020) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#4  tbb::detail::r1::rml::private_worker::run (this=0x146221574000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#5  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x146221574000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#6  0x00001462254661ca in start_thread () from /lib64/libpthread.so.0
#7  0x00001462250c18d3 in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x14619f7a0700 (LWP 1328547) "cmsRun"):
#0  0x00001462251baac1 in poll () from /lib64/libc.so.6
#1  0x0000146220cc29f7 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02858/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_GPU_X_2024-10-08-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x0000146220cc2bf4 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02858/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_GPU_X_2024-10-08-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00001462272dfec1 in TBufferFile::WriteClassBuffer(TClass const*, void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02858/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_2_GPU_X_2024-10-09-2300/external/el8_amd64_gcc12/lib/libRIO.so
#5  0x0000146227384046 in TKey::TKey(TObject const*, char const*, int, TDirectory*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02858/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_2_GPU_X_2024-10-09-2300/external/el8_amd64_gcc12/lib/libRIO.so
#6  0x00001462273420a5 in TFile::CreateKey(TDirectory*, TObject const*, char const*, int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02858/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_2_GPU_X_2024-10-09-2300/external/el8_amd64_gcc12/lib/libRIO.so
#7  0x0000146227333286 in TDirectoryFile::WriteTObject(TObject const*, char const*, char const*, int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02858/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_2_GPU_X_2024-10-09-2300/external/el8_amd64_gcc12/lib/libRIO.so
#8  0x0000146226e40c04 in TObject::Write(char const*, int, int) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02858/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_2_GPU_X_2024-10-09-2300/external/el8_amd64_gcc12/lib/libCore.so
#9  0x000014619ec32666 in (anonymous namespace)::maybeConfigPileUp(edm::ParameterSet const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int, int, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02858/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_GPU_X_2024-10-08-2300/lib/el8_amd64_gcc12/libMixingBase.so
#10 0x000014619ec3312f in edm::MixingCache::Config::Config(edm::ParameterSet const&, unsigned int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02858/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_GPU_X_2024-10-08-2300/lib/el8_amd64_gcc12/libMixingBase.so
#11 0x000014619ec33817 in edm::BMixingModule::initializeGlobalCache(edm::ParameterSet const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02858/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_GPU_X_2024-10-08-2300/lib/el8_amd64_gcc12/libMixingBase.so
#12 0x000014619ecde586 in edm::WorkerMaker<edm::MixingModule>::makeModule(edm::ParameterSet const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02858/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_GPU_X_2024-10-08-2300/lib/el8_amd64_gcc12/pluginSimGeneralMixingModulePlugins.so
#13 0x0000146227e345a7 in edm::Maker::makeModule(edm::MakeModuleParams const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02858/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_GPU_X_2024-10-08-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#14 0x0000146227da1ed8 in edm::Factory::makeModule(edm::MakeModuleParams const&, edm::ModuleTypeResolverMaker const*, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02858/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_GPU_X_2024-10-08-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#15 0x0000146227db204f in edm::ModuleRegistry::getModule(edm::MakeModuleParams const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02858/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_GPU_X_2024-10-08-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#16 0x0000146227e34da8 in edm::WorkerRegistry::getWorker(edm::WorkerParams const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02858/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_GPU_X_2024-10-08-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#17 0x0000146227e35085 in edm::WorkerManager::getWorker(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02858/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_GPU_X_2024-10-08-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#18 0x0000146227e0f940 in edm::(anonymous namespace)::getWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, edm::ParameterSet&, edm::WorkerManager&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>) [clone .lto_priv.0] () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02858/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_GPU_X_2024-10-08-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#19 0x0000146227e1debe in edm::StreamSchedule::fillWorkers(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, std::vector<edm::WorkerInPath, std::allocator<edm::WorkerInPath> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, edm::ConditionalTaskHelper const&, std::unordered_set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02858/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_GPU_X_2024-10-08-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#20 0x0000146227e1ebe9 in edm::StreamSchedule::fillTrigPath(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<edm::HLTGlobalStatus>, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, edm::ConditionalTaskHelper const&, std::unordered_set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02858/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_GPU_X_2024-10-08-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#21 0x0000146227e13867 in edm::StreamSchedule::StreamSchedule(std::shared_ptr<edm::TriggerResultInserter>, std::vector<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> >, std::allocator<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> > > >&, std::vector<edm::propagate_const<std::shared_ptr<edm::EndPathStatusInserter> >, std::allocator<edm::propagate_const<std::shared_ptr<edm::EndPathStatusInserter> > > >&, std::shared_ptr<edm::ModuleRegistry>, edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::PreallocationConfiguration const&, edm::ProductRegistry&, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration const>, edm::StreamID, edm::ProcessContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02858/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_GPU_X_2024-10-08-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#22 0x0000146227df8f75 in edm::Schedule::Schedule(edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::ProductRegistry&, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration const>, edm::PreallocationConfiguration const&, edm::ProcessContext const*, edm::ModuleTypeResolverMaker const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02858/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_GPU_X_2024-10-08-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#23 0x0000146227e0cb00 in edm::ScheduleItems::initModules(edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::PreallocationConfiguration const&, edm::ProcessContext const*, edm::ModuleTypeResolverMaker const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02858/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_GPU_X_2024-10-08-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#24 0x0000146227d79c47 in tbb::detail::d1::function_task<edm::EventProcessor::init(std::shared_ptr<edm::ProcessDesc>&, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) [clone .lto_priv.0] () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02858/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_GPU_X_2024-10-08-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#25 0x0000146227123b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x146223bc3300, waiter=..., this=0x146223bca900) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#26 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x146223bca900) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#27 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/arena.cpp:137
#28 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/market.cpp:599
#29 0x0000146227125cee in tbb::detail::r1::rml::private_worker::run (this=0x146221574100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#30 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x146221574100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#31 0x00001462254661ca in start_thread () from /lib64/libpthread.so.0
#32 0x00001462250c18d3 in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x1461a01a1700 (LWP 1328546) "cmsRun"):
#0  0x0000146225190098 in nanosleep () from /lib64/libc.so.6
#1  0x000014622518ff9e in sleep () from /lib64/libc.so.6
#2  0x0000146220cbf3a0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02858/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_GPU_X_2024-10-08-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00001462250c141d in syscall () from /lib64/libc.so.6
#5  0x0000146227125fd2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x1462215740a4) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x1462215740a4) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x1462215740a0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x146221574080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x146221574080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x00001462254661ca in start_thread () from /lib64/libpthread.so.0
#11 0x00001462250c18d3 in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x1461f13de700 (LWP 1328438) "cuda-EvtHandlr"):
#0  0x00001462251baac1 in poll () from /lib64/libc.so.6
#1  0x000014621386889f in ?? () from /.singularity.d/libs/libcuda.so.1
#2  0x0000146213936dcf in ?? () from /.singularity.d/libs/libcuda.so.1
#3  0x0000146213863373 in ?? () from /.singularity.d/libs/libcuda.so.1
#4  0x00001462254661ca in start_thread () from /lib64/libpthread.so.0
#5  0x00001462250c18d3 in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x1461f8bfe700 (LWP 1328437) "cuda0000340000e"):
#0  0x00001462251baac1 in poll () from /lib64/libc.so.6
#1  0x000014621386889f in ?? () from /.singularity.d/libs/libcuda.so.1
#2  0x0000146213936dcf in ?? () from /.singularity.d/libs/libcuda.so.1
#3  0x0000146213863373 in ?? () from /.singularity.d/libs/libcuda.so.1
#4  0x00001462254661ca in start_thread () from /lib64/libpthread.so.0
#5  0x00001462250c18d3 in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x1461f8dff700 (LWP 1328431) "cmsRun"):
#0  0x00001462254706a2 in waitpid () from /lib64/libpthread.so.0
#1  0x0000146220cbf4f7 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02858/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_GPU_X_2024-10-08-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x0000146220cc27da in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02858/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_GPU_X_2024-10-08-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x0000146225ad8a73 in std::execute_native_thread_routine (__p=0x1462192dcb90) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x00001462254661ca in start_thread () from /lib64/libpthread.so.0
#5  0x00001462250c18d3 in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x146226b55680 (LWP 1328224) "cmsRun"):
#0  0x0000146225190098 in nanosleep () from /lib64/libc.so.6
#1  0x000014622518ff9e in sleep () from /lib64/libc.so.6
#2  0x0000146220cbf3a0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02858/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_GPU_X_2024-10-08-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014622546c48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#5  0x00001462278d4ec3 in ROOT::TReentrantRWLock<std::mutex, ROOT::Internal::RecurseCountsTBBUnique>::WriteLock() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02858/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_2_GPU_X_2024-10-09-2300/external/el8_amd64_gcc12/lib/libThread.so
#6  0x0000146227398ab5 in TStreamerInfo::BuildCheck(TFile*, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02858/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_2_GPU_X_2024-10-09-2300/external/el8_amd64_gcc12/lib/libRIO.so
#7  0x0000146227342a15 in TFile::ReadStreamerInfo() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02858/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_2_GPU_X_2024-10-09-2300/external/el8_amd64_gcc12/lib/libRIO.so
#8  0x000014622734e285 in TFile::Init(bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02858/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_2_GPU_X_2024-10-09-2300/external/el8_amd64_gcc12/lib/libRIO.so
#9  0x00001462212dfe78 in TStorageFactoryFile::Initialize(char const*, char const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02858/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_GPU_X_2024-10-08-2300/lib/el8_amd64_gcc12/libIOPoolTFileAdaptor.so
#10 0x00001462212e0494 in TStorageFactoryFile::TStorageFactoryFile(char const*, char const*, char const*, int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02858/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_GPU_X_2024-10-08-2300/lib/el8_amd64_gcc12/libIOPoolTFileAdaptor.so
#11 0x000014621e3b020d in ?? ()
#12 0x0000000000000000 in ?? ()

Current Modules:

Module: none (crashed)

A fatal system signal has occurred: segmentation violation
timeout: the monitored command dumped core

full log

iarspider commented 1 week ago

Happened in CMSSW_14_2_X_2024-10-13-0000/el8_amd64_gcc12 RelVal 12634.21 log

Thread 4 (Thread 0x14ef67bdd700 (LWP 1310764) "cmsRun"):
#0  0x000014efb49baac1 in poll () from /lib64/libc.so.6
#1  0x000014efb021b9f7 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x000014efb021bbf4 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014efb6adfec1 in TBufferFile::WriteClassBuffer(TClass const*, void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/external/el8_amd64_gcc12/lib/libRIO.so
#5  0x000014efb6b84046 in TKey::TKey(TObject const*, char const*, int, TDirectory*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/external/el8_amd64_gcc12/lib/libRIO.so
#6  0x000014efb6b420a5 in TFile::CreateKey(TDirectory*, TObject const*, char const*, int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/external/el8_amd64_gcc12/lib/libRIO.so
#7  0x000014efb6b33286 in TDirectoryFile::WriteTObject(TObject const*, char const*, char const*, int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/external/el8_amd64_gcc12/lib/libRIO.so
#8  0x000014efb6640c04 in TObject::Write(char const*, int, int) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/external/el8_amd64_gcc12/lib/libCore.so
#9  0x000014ef5ceef666 in (anonymous namespace)::maybeConfigPileUp(edm::ParameterSet const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int, int, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/lib/el8_amd64_gcc12/libMixingBase.so
#10 0x000014ef5cef012f in edm::MixingCache::Config::Config(edm::ParameterSet const&, unsigned int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/lib/el8_amd64_gcc12/libMixingBase.so
#11 0x000014ef5cef0817 in edm::BMixingModule::initializeGlobalCache(edm::ParameterSet const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/lib/el8_amd64_gcc12/libMixingBase.so
#12 0x000014ef5cf9b586 in edm::WorkerMaker<edm::MixingModule>::makeModule(edm::ParameterSet const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/lib/el8_amd64_gcc12/pluginSimGeneralMixingModulePlugins.so
#13 0x000014efb76346a7 in edm::Maker::makeModule(edm::MakeModuleParams const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#14 0x000014efb75a1ed8 in edm::Factory::makeModule(edm::MakeModuleParams const&, edm::ModuleTypeResolverMaker const*, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#15 0x000014efb75b204f in edm::ModuleRegistry::getModule(edm::MakeModuleParams const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#16 0x000014efb7634ea8 in edm::WorkerRegistry::getWorker(edm::WorkerParams const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#17 0x000014efb7635185 in edm::WorkerManager::getWorker(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#18 0x000014efb76352a0 in edm::WorkerManager::addToUnscheduledWorkers(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#19 0x000014efb7614230 in edm::StreamSchedule::StreamSchedule(std::shared_ptr<edm::TriggerResultInserter>, std::vector<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> >, std::allocator<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> > > >&, std::vector<edm::propagate_const<std::shared_ptr<edm::EndPathStatusInserter> >, std::allocator<edm::propagate_const<std::shared_ptr<edm::EndPathStatusInserter> > > >&, std::shared_ptr<edm::ModuleRegistry>, edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::PreallocationConfiguration const&, edm::ProductRegistry&, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration const>, edm::StreamID, edm::ProcessContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#20 0x000014efb75f90ad in edm::Schedule::Schedule(edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::ProductRegistry&, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration const>, edm::PreallocationConfiguration const&, edm::ProcessContext const*, edm::ModuleTypeResolverMaker const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#21 0x000014efb760cc10 in edm::ScheduleItems::initModules(edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::PreallocationConfiguration const&, edm::ProcessContext const*, edm::ModuleTypeResolverMaker const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#22 0x000014efb7579c97 in tbb::detail::d1::function_task<edm::EventProcessor::init(std::shared_ptr<edm::ProcessDesc>&, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) [clone .lto_priv.0] () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#23 0x000014efb6969b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x14efb33c7600, waiter=..., this=0x14efb33c9400) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#24 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x14efb33c9400) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#25 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/arena.cpp:137
#26 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/market.cpp:599
#27 0x000014efb696bcee in tbb::detail::r1::rml::private_worker::run (this=0x14efb14b4100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#28 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14efb14b4100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#29 0x000014efb4c661ca in start_thread () from /lib64/libpthread.so.0
#30 0x000014efb48c18d3 in clone () from /lib64/libc.so.6

Other threads are sleeping, except Thread 1, which is reading ROOT file:

Thread 1 (Thread 0x14efb62e5680 (LWP 1310648) "cmsRun"):
#0  0x000014efb4990098 in nanosleep () from /lib64/libc.so.6
#1  0x000014efb498ff9e in sleep () from /lib64/libc.so.6
#2  0x000014efb02183a0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014efb70f4def in ROOT::TReentrantRWLock<std::mutex, ROOT::Internal::RecurseCountsTBBUnique>::WriteLock() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/external/el8_amd64_gcc12/lib/libThread.so
#5  0x000014efb67021a6 in TClass::LoadClassInfo() const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/external/el8_amd64_gcc12/lib/libCore.so
#6  0x000014efb6702518 in TClass::GetListOfBases() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/external/el8_amd64_gcc12/lib/libCore.so
#7  0x000014efb67123fc in TClass::GetCheckSum(TClass::ECheckSum, bool&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/external/el8_amd64_gcc12/lib/libCore.so
#8  0x000014efb6712cee in TClass::GetCheckSum(TClass::ECheckSum) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/external/el8_amd64_gcc12/lib/libCore.so
#9  0x000014efb6b99a65 in TStreamerInfo::BuildCheck(TFile*, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/external/el8_amd64_gcc12/lib/libRIO.so
#10 0x000014efb6b42a15 in TFile::ReadStreamerInfo() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/external/el8_amd64_gcc12/lib/libRIO.so
#11 0x000014efb6b4e285 in TFile::Init(bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/external/el8_amd64_gcc12/lib/libRIO.so
#12 0x000014efb135fe78 in TStorageFactoryFile::Initialize(char const*, char const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/lib/el8_amd64_gcc12/libIOPoolTFileAdaptor.so
#13 0x000014efb1360434 in TStorageFactoryFile::TStorageFactoryFile(char const*, char const*, char const*, int, int, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/lib/el8_amd64_gcc12/libIOPoolTFileAdaptor.so
#14 0x000014efb02ed0b9 in ?? ()
#15 0x000014ef00000000 in ?? ()
#16 0x000014ef6dfbb480 in ?? ()
#17 0x000014ef8f80f7e0 in ?? ()
#18 0x00007ffc7fb78c50 in ?? ()
#19 0x000014ef671be1d1 in ?? () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/lib/el8_amd64_gcc12/pluginIOPoolInput.so
#20 0x00007ffc7fb78d89 in ?? ()
#21 0x000014ef6e1041a0 in ?? ()
#22 0x000014efa850e2d2 in TClingCallFunc::IFacePtr() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/external/el8_amd64_gcc12/lib/libCling.so
#23 0x0000006500000000 in ?? ()
#24 0x000014ef913bb3e0 in ?? ()
#25 0x000014ef6e21b980 in ?? ()
#26 0x00007ffc7fb78d50 in ?? ()
#27 0x00007ffc7fb78d58 in ?? ()
#28 0x000014ef912a6b00 in ?? ()
#29 0x000014efb6b57233 in long TPluginHandler::ExecPluginImpl<char const*, char const*, char const*, int, int>(char const* const&, char const* const&, char const* const&, int const&, int const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/external/el8_amd64_gcc12/lib/libRIO.so
#30 0x000014efb6b5369c in TFile::Open(char const*, char const*, char const*, int, int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/external/el8_amd64_gcc12/lib/libRIO.so
#31 0x000014ef6717d66e in edm::InputFile::InputFile(char const*, char const*, edm::InputType) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/lib/el8_amd64_gcc12/pluginIOPoolInput.so
#32 0x000014ef671ac904 in edm::RootInputFileSequence::initTheFile(bool, bool, edm::InputSource*, char const*, edm::InputType) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/lib/el8_amd64_gcc12/pluginIOPoolInput.so
#33 0x000014ef671adfd8 in edm::RootPrimaryFileSequence::RootPrimaryFileSequence(edm::ParameterSet const&, edm::PoolSource&, edm::InputFileCatalog const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/lib/el8_amd64_gcc12/pluginIOPoolInput.so
#34 0x000014ef6718287f in edm::PoolSource::PoolSource(edm::ParameterSet const&, edm::InputSourceDescription const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/lib/el8_amd64_gcc12/pluginIOPoolInput.so
#35 0x000014ef6718369e in edmplugin::PluginFactory<edm::InputSource* (edm::ParameterSet const&, edm::InputSourceDescription const&)>::PMaker<edm::PoolSource>::create(edm::ParameterSet const&, edm::InputSourceDescription const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/lib/el8_amd64_gcc12/pluginIOPoolInput.so
#36 0x000014efb75acfc5 in edm::InputSourceFactory::makeInputSource(edm::ParameterSet const&, edm::InputSourceDescription const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#37 0x000014efb754515c in edm::makeInput(unsigned int, edm::ParameterSet&, edm::CommonParams const&, std::shared_ptr<edm::ProductRegistry>, std::shared_ptr<edm::BranchIDListHelper>, std::shared_ptr<edm::ProcessBlockHelper> const&, std::shared_ptr<edm::ThinnedAssociationsHelper>, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration const>, edm::PreallocationConfiguration const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#38 0x000014efb7578f00 in tbb::detail::d1::function_task<edm::EventProcessor::init(std::shared_ptr<edm::ProcessDesc>&, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy)::{lambda()#2}>::execute(tbb::detail::d1::execution_data&) [clone .lto_priv.0] () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#39 0x000014efb69723e1 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x14efb33c9380) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#40 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x14efb33c9380) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#41 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#42 0x000014efb7546ff1 in edm::EventProcessor::init(std::shared_ptr<edm::ProcessDesc>&, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#43 0x000014efb754a0ee in edm::EventProcessor::EventProcessor(std::shared_ptr<edm::ProcessDesc>, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-13-0000/lib/el8_amd64_gcc12/libFWCoreFramework.so
#44 0x0000000000408227 in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#45 0x000014efb695e9ad in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-e785b749a0b6cb9c66dc1d78066210e0/tbb-v2021.9.0/src/tbb/arena.cpp:688
#46 0x000000000040a0f2 in main::{lambda()#1}::operator()() const ()
#47 0x0000000000405100 in main ()
iarspider commented 1 week ago

assign Mixing/Base

cmsbuild commented 1 week ago

New categories assigned: simulation

@civanch,@kpedro88,@mdhildreth you have been requested to review this Pull request/Issue and eventually sign? Thanks

cmsbuild commented 1 week ago

cms-bot internal usage

cmsbuild commented 1 week ago

A new Issue was created by @iarspider.

@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

makortel commented 1 week ago

type root

makortel commented 1 week ago

@pcanal Any thoughts on the stack traces? One thread is opening a TFile, and another thread is writing into another TFile.

makortel commented 1 week ago

But why are we writing into a TFile from edm::MixingCache::Config constructor?

Dr15Jones commented 1 week ago

Here is the code https://github.com/cms-sw/cmssw/blob/5c7c91f006f718a3cdb0a5b2839a114a82a5212b/Mixing/Base/src/BMixingModule.cc#L147-L156

makortel commented 1 week ago

The writing is controlled with this condition https://github.com/cms-sw/cmssw/blob/5c7c91f006f718a3cdb0a5b2839a114a82a5212b/Mixing/Base/src/BMixingModule.cc#L86-L87 and indeed that's how the MixingModule is configured in this workflow https://github.com/cms-sw/cmssw/blob/master/SimGeneral/MixingModule/python/mix_Run3_Flat55To75_PoissonOOTPU_cfi.py https://github.com/cms-sw/cmssw/blob/master/SimGeneral/MixingModule/python/mix_probFunction_25ns_PoissonOOTPU_cfi.py#L40

It seems to me the use of the binned probabilities and writing to a file have been strongly coupled. I can understand writing the resulting histogram into a file can be useful in testing, but why is it done in production?

Dr15Jones commented 1 week ago

I bet the problem is here https://github.com/cms-sw/cmssw/blob/5c7c91f006f718a3cdb0a5b2839a114a82a5212b/Mixing/Base/src/BMixingModule.cc#L153

what if the histogram attached itself to the wrong file? I.e. not to the one created by the mixing module but the one opened by the Source?

pcanal commented 1 week ago

what if the histogram attached itself to the wrong file?

That would be surprising. The line before open/create the file and change gDirectory to point to it and since gDirectory is thread local, nothing can change it in between. (That is unless threading support was not enabled and 2 thread are opening files are the same time).

pcanal commented 1 week ago 
#3  <signal handler called>
#4  0x00001462272dfec1 in TBufferFile::WriteClassBuffer(TClass const*, void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02858/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_2_GPU_X_2024-10-09-2300/external/el8_amd64_gcc12/lib/libRIO.so
#5  0x0000146227384046 in TKey::TKey(TObject const*, char const*, int, TDirectory*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02858/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_2_GPU_X_2024-10-09-2300/external/el8_amd64_gcc12/lib/libRIO.so

from this stack trace, we can not tell whether the problem is with the StreamerInfo (since the other thread is stuck while building one) or with the user objects.