search

cms-sw / cmssw

CMS Offline Software
http://cms-sw.github.io/
Apache License 2.0
1.07k stars 4.28k forks source link

Segfault in TClingCallbacks::findInGlobalModuleIndex() #44659

Open makortel opened 5 months ago

makortel commented 5 months ago

Workflow 141.035 segfaulted in CMSSW_14_1_X_2024-04-07-2300 on slc7_amd64_gcc12 with

Thread 4 (Thread 0x14cebccec700 (LWP 613465) "cmsRun"):
#2  0x000014cf0aeba4e0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02832/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-04-07-0000/lib/slc7_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014cf10ffb465 in __xstat64 () from /lib64/libc.so.6
#5  0x000014cf11989c94 in stat (__statbuf=0x14cebcce3ee0, __path=<optimized out>) at /usr/include/sys/stat.h:456
#6  std::filesystem::status (p=..., ec=...) at ../../../../../libstdc++-v3/src/c++17/fs_ops.cc:1493
#7  0x000014cf1198a24b in std::filesystem::status (p=...) at ../../../../../libstdc++-v3/src/c++17/fs_ops.cc:1558
#8  0x000014cf136bc6c8 in (anonymous namespace)::locateFile(std::filesystem::__cxx11::path, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) [clone .isra.0] () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02832/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-04-07-0000/lib/slc7_amd64_gcc12/libFWCoreUtilities.so
#9  0x000014cf1369ad8d in edm::FileInPath::initialize_() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02832/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-04-07-0000/lib/slc7_amd64_gcc12/libFWCoreUtilities.so
#10 0x000014cf1369c270 in edm::FileInPath::FileInPath(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02832/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-04-07-0000/lib/slc7_amd64_gcc12/libFWCoreUtilities.so
#11 0x000014cea3ce81d7 in TSGForOIDNN::TSGForOIDNN(edm::ParameterSet const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02832/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-04-07-0000/lib/slc7_amd64_gcc12/pluginRecoMuonTrackerSeedGeneratorPlugins.so
#12 0x000014cea3cfa66f in edm::WorkerMaker<TSGForOIDNN>::makeModule(edm::ParameterSet const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02832/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-04-07-0000/lib/slc7_amd64_gcc12/pluginRecoMuonTrackerSeedGeneratorPlugins.so
#13 0x000014cf139e2828 in edm::Maker::makeModule(edm::MakeModuleParams const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02832/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-04-07-0000/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#14 0x000014cf13959868 in edm::Factory::makeModule(edm::MakeModuleParams const&, edm::ModuleTypeResolverMaker const*, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02832/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-04-07-0000/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#15 0x000014cf13968620 in edm::ModuleRegistry::getModule(edm::MakeModuleParams const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02832/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-04-07-0000/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#16 0x000014cf139e3000 in edm::WorkerRegistry::getWorker(edm::WorkerParams const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02832/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-04-07-0000/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#17 0x000014cf139e33c2 in edm::WorkerManager::getWorker(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02832/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-04-07-0000/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#18 0x000014cf139be7fe in edm::(anonymous namespace)::getWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, edm::ParameterSet&, edm::WorkerManager&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>) [clone .lto_priv.0] () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02832/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-04-07-0000/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#19 0x000014cf139c8a22 in edm::StreamSchedule::fillWorkers(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, std::vector<edm::WorkerInPath, std::allocator<edm::WorkerInPath> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, edm::ConditionalTaskHelper const&, std::unordered_set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02832/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-04-07-0000/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#20 0x000014cf139cb2c4 in edm::StreamSchedule::fillTrigPath(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<edm::HLTGlobalStatus>, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, edm::ConditionalTaskHelper const&, std::unordered_set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02832/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-04-07-0000/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#21 0x000014cf139c15b7 in edm::StreamSchedule::StreamSchedule(std::shared_ptr<edm::TriggerResultInserter>, std::vector<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> >, std::allocator<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> > > >&, std::vector<edm::propagate_const<std::shared_ptr<edm::EndPathStatusInserter> >, std::allocator<edm::propagate_const<std::shared_ptr<edm::EndPathStatusInserter> > > >&, std::shared_ptr<edm::ModuleRegistry>, edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::PreallocationConfiguration const&, edm::ProductRegistry&, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration const>, edm::StreamID, edm::ProcessContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02832/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-04-07-0000/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#22 0x000014cf139a7c2f in edm::Schedule::Schedule(edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::ProductRegistry&, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration const>, edm::PreallocationConfiguration const&, edm::ProcessContext const*, edm::ModuleTypeResolverMaker const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02832/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-04-07-0000/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#23 0x000014cf139bc26d in edm::ScheduleItems::initModules(edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::PreallocationConfiguration const&, edm::ProcessContext const*, edm::ModuleTypeResolverMaker const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02832/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-04-07-0000/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#24 0x000014cf13939af6 in tbb::detail::d1::function_task<edm::EventProcessor::init(std::shared_ptr<edm::ProcessDesc>&, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) [clone .lto_priv.0] () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02832/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-04-07-0000/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#25 0x000014cf1212d96b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x14cf0e5c7300, waiter=..., this=0x14cf0e5c9400) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_1_0_pre2-slc7_amd64_gcc12/build/CMSSW_14_1_0_pre2-build/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-2bdfc55ee8e4ee32defbb06c66a16b5f/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#26 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x14cf0e5c9400) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_1_0_pre2-slc7_amd64_gcc12/build/CMSSW_14_1_0_pre2-build/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-2bdfc55ee8e4ee32defbb06c66a16b5f/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#27 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_1_0_pre2-slc7_amd64_gcc12/build/CMSSW_14_1_0_pre2-build/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-2bdfc55ee8e4ee32defbb06c66a16b5f/tbb-v2021.9.0/src/tbb/arena.cpp:137
#28 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_1_0_pre2-slc7_amd64_gcc12/build/CMSSW_14_1_0_pre2-build/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-2bdfc55ee8e4ee32defbb06c66a16b5f/tbb-v2021.9.0/src/tbb/market.cpp:599
#29 0x000014cf1212fb1e in tbb::detail::r1::rml::private_worker::run (this=0x14cf0be34100) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_1_0_pre2-slc7_amd64_gcc12/build/CMSSW_14_1_0_pre2-build/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-2bdfc55ee8e4ee32defbb06c66a16b5f/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#30 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14cf0be34100) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_1_0_pre2-slc7_amd64_gcc12/build/CMSSW_14_1_0_pre2-build/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-2bdfc55ee8e4ee32defbb06c66a16b5f/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#31 0x000014cf112e1ea5 in start_thread () from /lib64/libpthread.so.0
#32 0x000014cf1100ab0d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x14cf0f145780 (LWP 613234) "cmsRun"):
#3  0x000014cf0aebdf40 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02832/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-04-07-0000/lib/slc7_amd64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x000014cf10f452bd in getenv () from /lib64/libc.so.6
#6  0x000014cf03960998 in llvm::sys::Process::GetEnv[abi:cxx11](llvm::StringRef) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libCling.so
#7  0x000014cf00042fdc in TClingCallbacks::findInGlobalModuleIndex(clang::DeclarationName, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libCling.so
#8  0x000014cf00046b8b in TClingCallbacks::LookupObject(clang::DeclContext const*, clang::DeclarationName) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libCling.so
#9  0x000014cf000d3bcf in cling::MultiplexInterpreterCallbacks::LookupObject(clang::DeclContext const*, clang::DeclarationName) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libCling.so
#10 0x000014cf00c045c8 in clang::MultiplexExternalSemaSource::FindExternalVisibleDeclsByName(clang::DeclContext const*, clang::DeclarationName) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libCling.so
#11 0x000014cf02b38d36 in clang::DeclContext::lookup(clang::DeclarationName) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libCling.so
#12 0x000014cf010c09c9 in LookupDirect(clang::Sema&, clang::LookupResult&, clang::DeclContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libCling.so
#13 0x000014cf010c158e in CppNamespaceLookup(clang::Sema&, clang::LookupResult&, clang::ASTContext&, clang::DeclContext*, (anonymous namespace)::UnqualUsingDirectiveSet&) [clone .constprop.0] () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libCling.so
#14 0x000014cf010b9e03 in clang::Sema::CppLookupName(clang::LookupResult&, clang::Scope*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libCling.so
#15 0x000014cf010bab3d in clang::Sema::LookupName(clang::LookupResult&, clang::Scope*, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libCling.so
#16 0x000014cf001bd4d4 in cling::utils::Lookup::Named(clang::Sema*, clang::DeclarationName const&, clang::DeclContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libCling.so
#17 0x000014cf001bd551 in cling::utils::Lookup::Named(clang::Sema*, llvm::StringRef, clang::DeclContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libCling.so
#18 0x000014cf000da83b in cling::quickFindDecl(llvm::StringRef, clang::Decl const*&, clang::Parser&, cling::LookupHelper::DiagSetting) [clone .isra.0] () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libCling.so
#19 0x000014cf000ded03 in cling::LookupHelper::findScope(llvm::StringRef, cling::LookupHelper::DiagSetting, clang::Type const**, bool) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libCling.so
#20 0x000014cf000030f2 in TCling::CheckClassInfo(char const*, bool, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libCling.so
#21 0x000014cf12b59bf2 in TClass::Init(char const*, short, std::type_info const*, TVirtualIsAProxy*, char const*, char const*, int, int, ClassInfo_t*, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libCore.so
#22 0x000014cf12b5ad07 in TClass::TClass(char const*, short, std::type_info const&, TVirtualIsAProxy*, char const*, char const*, int, int, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libCore.so
#23 0x000014cf12b5ae0b in ROOT::CreateClass(char const*, short, std::type_info const&, TVirtualIsAProxy*, char const*, char const*, int, int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libCore.so
#24 0x000014cf12b6bac2 in ROOT::TGenericClassInfo::GetClass() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libCore.so
#25 0x000014cf129fa171 in TStreamerSTL::Dictionary() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libCore.so
#26 0x000014cf12b47840 in TClass::LoadClassDefault(char const*, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libCore.so
#27 0x000014cf12b47992 in TClass::LoadClass(char const*, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libCore.so
#28 0x000014cf12b5a63c in TClass::GetClass(ClassInfo_t*, bool, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libCore.so
#29 0x000014cf12b428bc in TBaseClass::GetClassPointer(bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libCore.so
#30 0x000014cf12b47f26 in TClass::GetBaseClass(TClass const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libCore.so
#31 0x000014cf12b47fc7 in TClass::InheritsFrom(TClass const*) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libCore.so
#32 0x000014cf12e44532 in TBufferFile::ReadClass(TClass const*, unsigned int*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libRIO.so
#33 0x000014cf12e43bfa in TBufferFile::ReadObjectAny(TClass const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libRIO.so
#34 0x000014cf12b0ec88 in TObjArray::Streamer(TBuffer&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libCore.so
#35 0x000014cf12e43cb9 in TBufferFile::ReadObjectAny(TClass const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libRIO.so
#36 0x000014cf12f055c4 in TStreamerInfo::Streamer(TBuffer&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libRIO.so
#37 0x000014cf12e43cb9 in TBufferFile::ReadObjectAny(TClass const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libRIO.so
#38 0x000014cf12b0a4b0 in TList::Streamer(TBuffer&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libCore.so
#39 0x000014cf12ee08d5 in TKey::ReadObjWithBuffer(char*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libRIO.so
#40 0x000014cf12ea333d in TFile::GetStreamerInfoListImpl(bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libRIO.so
#41 0x000014cf12eadb68 in TFile::ReadStreamerInfo() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libRIO.so
#42 0x000014cf12eae795 in TFile::Init(bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libRIO.so
#43 0x000014cf0b8618e5 in TStorageFactoryFile::Initialize(char const*, char const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02832/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-04-07-0000/lib/slc7_amd64_gcc12/libIOPoolTFileAdaptor.so
#44 0x000014cf0b8620c4 in TStorageFactoryFile::TStorageFactoryFile(char const*, char const*, char const*, int, int, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02832/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-04-07-0000/lib/slc7_amd64_gcc12/libIOPoolTFileAdaptor.so
#45 0x000014ceea2010b9 in ?? ()
#46 0x000014cf00000000 in TCling::GetMangledNameWithPrototype(TClass*, char const*, char const*, bool, ROOT::EFunctionMatchMode) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libCling.so
#47 0x000014cf12eb7bfe in long TPluginHandler::ExecPluginImpl<char const*, char const*, char const*, int, int>(char const* const&, char const* const&, char const* const&, int const&, int const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libRIO.so
#48 0x000014cf12eb3b2d in TFile::Open(char const*, char const*, char const*, int, int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libRIO.so
#49 0x000014cebc28e29e in edm::InputFile::InputFile(char const*, char const*, edm::InputType) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02832/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-04-07-0000/lib/slc7_amd64_gcc12/pluginIOPoolInput.so
#50 0x000014cebc2bff43 in edm::RootInputFileSequence::initTheFile(bool, bool, edm::InputSource*, char const*, edm::InputType) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02832/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-04-07-0000/lib/slc7_amd64_gcc12/pluginIOPoolInput.so
#51 0x000014cebc2c0b88 in edm::RootPrimaryFileSequence::RootPrimaryFileSequence(edm::ParameterSet const&, edm::PoolSource&, edm::InputFileCatalog const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02832/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-04-07-0000/lib/slc7_amd64_gcc12/pluginIOPoolInput.so
#52 0x000014cebc291082 in edm::PoolSource::PoolSource(edm::ParameterSet const&, edm::InputSourceDescription const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02832/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-04-07-0000/lib/slc7_amd64_gcc12/pluginIOPoolInput.so
#53 0x000014cebc29204e in edmplugin::PluginFactory<edm::InputSource* (edm::ParameterSet const&, edm::InputSourceDescription const&)>::PMaker<edm::PoolSource>::create(edm::ParameterSet const&, edm::InputSourceDescription const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02832/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-04-07-0000/lib/slc7_amd64_gcc12/pluginIOPoolInput.so
#54 0x000014cf13963ac5 in edm::InputSourceFactory::makeInputSource(edm::ParameterSet const&, edm::InputSourceDescription const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02832/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-04-07-0000/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#55 0x000014cf138fcd3f in edm::makeInput(unsigned int, edm::ParameterSet&, edm::CommonParams const&, std::shared_ptr<edm::ProductRegistry>, std::shared_ptr<edm::BranchIDListHelper>, std::shared_ptr<edm::ProcessBlockHelper> const&, std::shared_ptr<edm::ThinnedAssociationsHelper>, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration const>, edm::PreallocationConfiguration const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02832/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-04-07-0000/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#56 0x000014cf139373fd in tbb::detail::d1::function_task<edm::EventProcessor::init(std::shared_ptr<edm::ProcessDesc>&, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy)::{lambda()#2}>::execute(tbb::detail::d1::execution_data&) [clone .lto_priv.0] () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02832/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-04-07-0000/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#57 0x000014cf12136291 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x14cf0e5c9380) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_1_0_pre2-slc7_amd64_gcc12/build/CMSSW_14_1_0_pre2-build/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-2bdfc55ee8e4ee32defbb06c66a16b5f/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#58 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x14cf0e5c9380) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_1_0_pre2-slc7_amd64_gcc12/build/CMSSW_14_1_0_pre2-build/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-2bdfc55ee8e4ee32defbb06c66a16b5f/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#59 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_1_0_pre2-slc7_amd64_gcc12/build/CMSSW_14_1_0_pre2-build/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-2bdfc55ee8e4ee32defbb06c66a16b5f/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#60 0x000014cf139012e2 in edm::EventProcessor::init(std::shared_ptr<edm::ProcessDesc>&, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02832/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-04-07-0000/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#61 0x000014cf13903bbc in edm::EventProcessor::EventProcessor(std::shared_ptr<edm::ProcessDesc>, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02832/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-04-07-0000/lib/slc7_amd64_gcc12/libFWCoreFramework.so
#62 0x00000000004073af in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#63 0x000014cf121229bd in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_1_0_pre2-slc7_amd64_gcc12/build/CMSSW_14_1_0_pre2-build/BUILD/slc7_amd64_gcc12/external/tbb/v2021.9.0-2bdfc55ee8e4ee32defbb06c66a16b5f/tbb-v2021.9.0/src/tbb/arena.cpp:688
#64 0x0000000000408ed2 in main::{lambda()#1}::operator()() const ()
#65 0x000000000040518c in main ()

Current Modules:
Module: none (crashed)

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc12/CMSSW_14_1_X_2024-04-07-2300/pyRelValMatrixLogs/run/141.035_RunDisplacedJet2023C/step2_RunDisplacedJet2023C.log#/

makortel commented 5 months ago

assign core

makortel commented 5 months ago

type root

cmsbuild commented 5 months ago

New categories assigned: core

@Dr15Jones,@makortel,@smuzaffar you have been requested to review this Pull request/Issue and eventually sign? Thanks

cmsbuild commented 5 months ago

cms-bot internal usage

cmsbuild commented 5 months ago

A new Issue was created by @makortel.

@rappoccio, @antoniovilela, @Dr15Jones, @makortel, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

makortel commented 5 months ago

FYI @pcanal @vgvassilev

I'm not sure if this has any relation to the other random crashes we're seeing. I don't recall seeing it before.

pcanal commented 5 months ago

It says that it crashes within the getenv function itself:

Thread 1 (Thread 0x14cf0f145780 (LWP 613234) "cmsRun"):
#3  0x000014cf0aebdf40 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02832/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-04-07-0000/lib/slc7_amd64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x000014cf10f452bd in getenv () from /lib64/libc.so.6
#6  0x000014cf03960998 in llvm::sys::Process::GetEnv[abi:cxx11](llvm::StringRef) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libCling.so
#7  0x000014cf00042fdc in TClingCallbacks::findInGlobalModuleIndex(clang::DeclarationName, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libCling.so

The argument passed to Process::GetEnv is a hard-coded string in this case. There is some manipulation in there, but still I don't see anything obvious leading to the crash :(

makortel commented 5 months ago

The argument passed to Process::GetEnv is a hard-coded string in this case.

Do you know if LLVM uses the POSIX getenv() or C++ std::getenv()? (from the stack trace I'd guess the POSIX one, but I could easily be fooled here) I'm asking because the POSIX one is not required to be thread safe, whereas std::getenv() is.

pcanal commented 5 months ago

Fair point. It is ::getenv. In related note, the stack trace shows that the thread running getenv hold the ROOT global lock .. which of course does not prevent other calls to ::getenv.

makortel commented 5 months ago

Of course this observation probably means we have something else calling ::getenv() too, that should be identified and fixed.

dan131riley commented 5 months ago

maybe related, probably a clue, segfault on el8_amd64_gcc12 CMSSW_14_1_NONLTO_X_2024-04-20-1100 12634.0 step2:

Thread 4 (Thread 0x154673e9b700 (LWP 1030544) "cmsRun"):
#3  0x00001546bc8e6178 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00001546c4017651 in TBufferFile::WriteClassBuffer(TClass const*, void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/external/el8_amd64_gcc12/lib/libRIO.so
#6  0x00001546c40b8b16 in TKey::TKey(TObject const*, char const*, int, TDirectory*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/external/el8_amd64_gcc12/lib/libRIO.so
#7  0x00001546c40788b5 in TFile::CreateKey(TDirectory*, TObject const*, char const*, int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/external/el8_amd64_gcc12/lib/libRIO.so
#8  0x00001546c4069db6 in TDirectoryFile::WriteTObject(TObject const*, char const*, char const*, int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/external/el8_amd64_gcc12/lib/libRIO.so
#9  0x00001546c3c65634 in TObject::Write(char const*, int, int) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/external/el8_amd64_gcc12/lib/libCore.so
#10 0x0000154667e4bdaf in (anonymous namespace)::maybeConfigPileUp(edm::ParameterSet const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int, int, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/lib/el8_amd64_gcc12/libMixingBase.so
#11 0x0000154667e4cb58 in edm::MixingCache::Config::Config(edm::ParameterSet const&, unsigned int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/lib/el8_amd64_gcc12/libMixingBase.so
#12 0x0000154667e4cd07 in edm::BMixingModule::initializeGlobalCache(edm::ParameterSet const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/lib/el8_amd64_gcc12/libMixingBase.so
#13 0x0000154667f1d040 in edm::WorkerMaker<edm::MixingModule>::makeModule(edm::ParameterSet const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/lib/el8_amd64_gcc12/pluginSimGeneralMixingModulePlugins.so
#14 0x00001546c4d3b4f1 in edm::Maker::makeModule(edm::MakeModuleParams const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#15 0x00001546c4c6adf8 in edm::Factory::makeModule(edm::MakeModuleParams const&, edm::ModuleTypeResolverMaker const*, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#16 0x00001546c4c7f5f9 in edm::ModuleRegistry::getModule(edm::MakeModuleParams const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#17 0x00001546c4d3dcb9 in edm::WorkerRegistry::getWorker(edm::WorkerParams const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#18 0x00001546c4d3bb6b in edm::WorkerManager::getWorker(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#19 0x00001546c4d3c020 in edm::WorkerManager::addToUnscheduledWorkers(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#20 0x00001546c4d10ef4 in edm::StreamSchedule::StreamSchedule(std::shared_ptr<edm::TriggerResultInserter>, std::vector<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> >, std::allocator<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> > > >&, std::vector<edm::propagate_const<std::shared_ptr<edm::EndPathStatusInserter> >, std::allocator<edm::propagate_const<std::shared_ptr<edm::EndPathStatusInserter> > > >&, std::shared_ptr<edm::ModuleRegistry>, edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::PreallocationConfiguration const&, edm::ProductRegistry&, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration const>, edm::StreamID, edm::ProcessContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#21 0x00001546c4ce342e in edm::Schedule::Schedule(edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::ProductRegistry&, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration const>, edm::PreallocationConfiguration const&, edm::ProcessContext const*, edm::ModuleTypeResolverMaker const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#22 0x00001546c4cfbed1 in edm::ScheduleItems::initModules(edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::PreallocationConfiguration const&, edm::ProcessContext const*, edm::ModuleTypeResolverMaker const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so

Thread 1 (Thread 0x1546c15e5680 (LWP 1030482) "cmsRun"):
#2  0x00001546bc8e1f80 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00001546c3cdc350 in TList::FindObject(char const*) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/external/el8_amd64_gcc12/lib/libCore.so
#5  0x00001546c3cda2bf in THashTable::FindObject(char const*) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/external/el8_amd64_gcc12/lib/libCore.so
#6  0x00001546c3d3559c in TClass::GetClass(ClassInfo_t*, bool, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/external/el8_amd64_gcc12/lib/libCore.so
#7  0x00001546c3d1d8dc in TBaseClass::GetClassPointer(bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/external/el8_amd64_gcc12/lib/libCore.so
#8  0x00001546c3d326e2 in TClass::GetCheckSum(TClass::ECheckSum, bool&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/external/el8_amd64_gcc12/lib/libCore.so
#9  0x00001546c3d328fe in TClass::GetCheckSum(TClass::ECheckSum) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/external/el8_amd64_gcc12/lib/libCore.so
#10 0x00001546c40cb4e5 in TStreamerInfo::BuildCheck(TFile*, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/external/el8_amd64_gcc12/lib/libRIO.so
#11 0x00001546c4083dd1 in TFile::ReadStreamerInfo() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/external/el8_amd64_gcc12/lib/libRIO.so
#12 0x00001546c4084785 in TFile::Init(bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/external/el8_amd64_gcc12/lib/libRIO.so
#13 0x00001546bde332b0 in TStorageFactoryFile::Initialize(char const*, char const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/lib/el8_amd64_gcc12/libIOPoolTFileAdaptor.so
#14 0x00001546bde33764 in TStorageFactoryFile::TStorageFactoryFile(char const*, char const*, char const*, int, int, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/lib/el8_amd64_gcc12/libIOPoolTFileAdaptor.so
#15 0x000015466fe020b9 in ?? ()
#16 0x0000154600000000 in ?? ()
#17 0x000015467b1bf700 in ?? ()
#18 0x000015469d32e500 in ?? ()
#19 0x00007ffeba4a6600 in ?? ()
#20 0x00001546731dedfc in ?? () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/lib/el8_amd64_gcc12/pluginIOPoolInput.so
#21 0x00007ffeba4a6759 in ?? ()
#22 0x000015467ae52d80 in ?? ()
#23 0x00001546b5bf4b82 in TClingCallFunc::IFacePtr() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/external/el8_amd64_gcc12/lib/libCling.so
#24 0x0000006500000000 in ?? ()
#25 0x000015469ef08840 in ?? ()
#26 0x000015467b10bc00 in ?? ()
#27 0x00007ffeba4a6708 in ?? ()
#28 0x00007ffeba4a6720 in ?? ()
#29 0x000015469e6d0700 in ?? ()
#30 0x00001546c408dc0e in long TPluginHandler::ExecPluginImpl<char const*, char const*, char const*, int, int>(char const* const&, char const* const&, char const* const&, int const&, int const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/external/el8_amd64_gcc12/lib/libRIO.so
#31 0x00001546c4089b3d in TFile::Open(char const*, char const*, char const*, int, int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/external/el8_amd64_gcc12/lib/libRIO.so
#32 0x000015467318fc70 in edm::InputFile::InputFile(char const*, char const*, edm::InputType) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/lib/el8_amd64_gcc12/pluginIOPoolInput.so
#33 0x00001546731d202d in edm::RootInputFileSequence::initTheFile(bool, bool, edm::InputSource*, char const*, edm::InputType) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/lib/el8_amd64_gcc12/pluginIOPoolInput.so
#34 0x00001546731d5f8c in edm::RootPrimaryFileSequence::RootPrimaryFileSequence(edm::ParameterSet const&, edm::PoolSource&, edm::InputFileCatalog const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/lib/el8_amd64_gcc12/pluginIOPoolInput.so
#35 0x00001546731945ed in edm::PoolSource::PoolSource(edm::ParameterSet const&, edm::InputSourceDescription const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/lib/el8_amd64_gcc12/pluginIOPoolInput.so
#36 0x0000154673190a1e in edmplugin::PluginFactory<edm::InputSource* (edm::ParameterSet const&, edm::InputSourceDescription const&)>::PMaker<edm::PoolSource>::create(edm::ParameterSet const&, edm::InputSourceDescription const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/lib/el8_amd64_gcc12/pluginIOPoolInput.so
#37 0x00001546c4c792d5 in edm::InputSourceFactory::makeInputSource(edm::ParameterSet const&, edm::InputSourceDescription const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#38 0x00001546c4bfe4e3 in edm::makeInput(unsigned int, edm::ParameterSet&, edm::CommonParams const&, std::shared_ptr<edm::ProductRegistry>, std::shared_ptr<edm::BranchIDListHelper>, std::shared_ptr<edm::ProcessBlockHelper> const&, std::shared_ptr<edm::ThinnedAssociationsHelper>, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration const>, edm::PreallocationConfiguration const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#39 0x00001546c4bfe8fa in tbb::detail::d1::function_task<edm::EventProcessor::init(std::shared_ptr<edm::ProcessDesc>&, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy)::{lambda()#2}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#40 0x00001546c3321281 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x1546c09c9380) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5849be8e21b090e14f1b189539cee138/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#41 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x1546c09c9380) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5849be8e21b090e14f1b189539cee138/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#42 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5849be8e21b090e14f1b189539cee138/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#43 0x00001546c4c06452 in edm::EventProcessor::init(std::shared_ptr<edm::ProcessDesc>&, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#44 0x00001546c4c08885 in edm::EventProcessor::EventProcessor(std::shared_ptr<edm::ProcessDesc>, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-04-20-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#45 0x000000000040744f in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#46 0x00001546c330d9ad in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5849be8e21b090e14f1b189539cee138/tbb-v2021.9.0/src/tbb/arena.cpp:688
#47 0x0000000000408757 in main::{lambda()#1}::operator()() const ()
#48 0x000000000040612c in main ()
pcanal commented 4 months ago

This latest crash seems to be due to a 'corrupted' TStreamerInfo (based on the crash site TBufferFile::WriteClassBuffer). However based on the stack trace ((anonymous namespace)::maybeConfigPileUp) the object being stored is likely to be a TH1F. It is not yet clear how it would have been affected by the processing of the StreamerInfo from the file being opened by the other thread (TFile::ReadStreamerInfo). So far the code path I follow are taking the global lock before modifying the list of StreamerInfo of a class.

iarspider commented 1 month ago

Another possible occurance: RelVal 12634.0 in CMSSW_14_1_CLANG_X_2024-08-06-2300:

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Wed Aug  7 03:04:59 CEST 2024
Thread 19 (Thread 0x14911738d700 (LWP 1795616) "cmsRun"):
#0  0x0000149169275dd6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x0000149169275ec8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x0000149163aa97c6 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x0000149163aa9879 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x000014916926d1ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000149168ec88d3 in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x14911758e700 (LWP 1795615) "cmsRun"):
#0  0x0000149169275dd6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x0000149169275ec8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x0000149163aa97c6 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x0000149163aa9879 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x000014916926d1ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000149168ec88d3 in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x14911778f700 (LWP 1795614) "cmsRun"):
#0  0x0000149169275dd6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x0000149169275ec8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x0000149163aa97c6 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x0000149163aa9879 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x000014916926d1ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000149168ec88d3 in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x149117990700 (LWP 1795613) "cmsRun"):
#0  0x00001491692771b0 in nanosleep () from /lib64/libpthread.so.0
#1  0x0000149163bda7e8 in XrdSysTimer::Wait(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000149163a1a37c in XrdCl::TaskManager::RunTasks() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x0000149163a1a4b9 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x000014916926d1ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000149168ec88d3 in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x149117b91700 (LWP 1795612) "cmsRun"):
#0  0x0000149168fce307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000149163bd4b62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000149163bd0acd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000149163bd9ef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x000014916926d1ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000149168ec88d3 in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x149117d92700 (LWP 1795611) "cmsRun"):
#0  0x0000149168fce307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000149163bd4b62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000149163bd0acd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000149163bd9ef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x000014916926d1ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000149168ec88d3 in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x149117f93700 (LWP 1795610) "cmsRun"):
#0  0x0000149168fce307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000149163bd4b62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000149163bd0acd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000149163bd9ef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x000014916926d1ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000149168ec88d3 in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x149118194700 (LWP 1795609) "cmsRun"):
#0  0x0000149168fce307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000149163bd4b62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000149163bd0acd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000149163bd9ef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x000014916926d1ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000149168ec88d3 in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x149118679700 (LWP 1795608) "cmsRun"):
#0  0x0000149168fce307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000149163bd4b62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000149163bd0acd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000149163bd9ef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x000014916926d1ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000149168ec88d3 in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x149118bfa700 (LWP 1795607) "cmsRun"):
#0  0x0000149168fce307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000149163bd4b62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000149163bd0acd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000149163bd9ef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x000014916926d1ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000149168ec88d3 in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x149118dfb700 (LWP 1795606) "cmsRun"):
#0  0x0000149168fce307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000149163bd4b62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000149163bd0acd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000149163bd9ef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x000014916926d1ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000149168ec88d3 in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x149118ffc700 (LWP 1795605) "cmsRun"):
#0  0x0000149168fce307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000149163bd4b62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000149163bd0acd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000149163bd9ef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x000014916926d1ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000149168ec88d3 in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x1491191fd700 (LWP 1795604) "cmsRun"):
#0  0x0000149168fce307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000149163bd4b62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000149163bd0acd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000149163bd9ef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x000014916926d1ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000149168ec88d3 in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x1491193fe700 (LWP 1795603) "cmsRun"):
#0  0x0000149168fce307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000149163bd4b62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000149163bd0acd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000149163bd9ef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x000014916926d1ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000149168ec88d3 in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x149119dff700 (LWP 1795602) "cmsRun"):
#0  0x0000149168ec841d in syscall () from /lib64/libc.so.6
#1  0x000014916a131fd2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x1491657f4024) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:100
#2  tbb::detail::r1::binary_semaphore::P (this=0x1491657f4024) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:253
#3  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x1491657f4020) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#4  tbb::detail::r1::rml::private_worker::run (this=0x1491657f4000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#5  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x1491657f4000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#6  0x000014916926d1ca in start_thread () from /lib64/libpthread.so.0
#7  0x0000149168ec88d3 in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x14911b123700 (LWP 1795601) "cmsRun"):
#0  0x0000149168fc1ac1 in poll () from /lib64/libc.so.6
#1  0x00001491644c74bd in (anonymous namespace)::full_read(int, char*, unsigned long, int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x00001491644c6f54 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x00001491644c68bf in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x000014916ae4fec1 in TBufferFile::WriteClassBuffer(TClass const*, void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libRIO.so
#6  0x000014916aef4046 in TKey::TKey(TObject const*, char const*, int, TDirectory*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libRIO.so
#7  0x000014916aeb20a5 in TFile::CreateKey(TDirectory*, TObject const*, char const*, int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libRIO.so
#8  0x000014916aea3286 in TDirectoryFile::WriteTObject(TObject const*, char const*, char const*, int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libRIO.so
#9  0x000014916aa8fc04 in TObject::Write(char const*, int, int) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libCore.so
#10 0x00001491105f2fc3 in (anonymous namespace)::maybeConfigPileUp(edm::ParameterSet const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int, int, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libMixingBase.so
#11 0x00001491105f1377 in edm::MixingCache::Config::Config(edm::ParameterSet const&, unsigned int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libMixingBase.so
#12 0x00001491105f3d18 in edm::BMixingModule::initializeGlobalCache(edm::ParameterSet const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libMixingBase.so
#13 0x00001491106becb8 in edm::stream::ProducingModuleAdaptor<edm::MixingModule, edm::stream::EDProducerBase, edm::stream::EDProducerAdaptorBase>::ProducingModuleAdaptor(edm::ParameterSet const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginSimGeneralMixingModulePlugins.so
#14 0x00001491106be8dc in edm::WorkerMaker<edm::MixingModule>::makeModule(edm::ParameterSet const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginSimGeneralMixingModulePlugins.so
#15 0x000014916bb33df7 in edm::Maker::makeModule(edm::MakeModuleParams const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#16 0x000014916ba5dea6 in edm::Factory::makeModule(edm::MakeModuleParams const&, edm::ModuleTypeResolverMaker const*, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#17 0x000014916ba72ec8 in edm::ModuleRegistry::getModule(edm::MakeModuleParams const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#18 0x000014916bb36e93 in edm::WorkerRegistry::getWorker(edm::WorkerParams const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#19 0x000014916bb3485a in edm::WorkerManager::getWorker(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#20 0x000014916bb34a4b in edm::WorkerManager::addToUnscheduledWorkers(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#21 0x000014916baffba7 in edm::StreamSchedule::StreamSchedule(std::shared_ptr<edm::TriggerResultInserter>, std::vector<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> >, std::allocator<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> > > >&, std::vector<edm::propagate_const<std::shared_ptr<edm::EndPathStatusInserter> >, std::allocator<edm::propagate_const<std::shared_ptr<edm::EndPathStatusInserter> > > >&, std::shared_ptr<edm::ModuleRegistry>, edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::PreallocationConfiguration const&, edm::ProductRegistry&, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration const>, edm::StreamID, edm::ProcessContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#22 0x000014916baebdd4 in void std::_Construct<edm::StreamSchedule, std::shared_ptr<edm::TriggerResultInserter>&, std::vector<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> >, std::allocator<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> > > >&, std::vector<edm::propagate_const<std::shared_ptr<edm::EndPathStatusInserter> >, std::allocator<edm::propagate_const<std::shared_ptr<edm::EndPathStatusInserter> > > >&, std::shared_ptr<edm::ModuleRegistry>&, edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::PreallocationConfiguration const&, edm::ProductRegistry&, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>&, std::shared_ptr<edm::ProcessConfiguration const>&, edm::StreamID, edm::ProcessContext const*&>(edm::StreamSchedule*, std::shared_ptr<edm::TriggerResultInserter>&, std::vector<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> >, std::allocator<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> > > >&, std::vector<edm::propagate_const<std::shared_ptr<edm::EndPathStatusInserter> >, std::allocator<edm::propagate_const<std::shared_ptr<edm::EndPathStatusInserter> > > >&, std::shared_ptr<edm::ModuleRegistry>&, edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::PreallocationConfiguration const&, edm::ProductRegistry&, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>&, std::shared_ptr<edm::ProcessConfiguration const>&, edm::StreamID&&, edm::ProcessContext const*&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#23 0x000014916bad1e97 in edm::Schedule::Schedule(edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::ProductRegistry&, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration const>, edm::PreallocationConfiguration const&, edm::ProcessContext const*, edm::ModuleTypeResolverMaker const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#24 0x000014916baf473d in std::__detail::_MakeUniq<edm::Schedule>::__single_object std::make_unique<edm::Schedule, edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::SignallingProductRegistry&, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>&, std::shared_ptr<edm::ProcessConfiguration>&, edm::PreallocationConfiguration const&, edm::ProcessContext const*&, edm::ModuleTypeResolverMaker const*&>(edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::SignallingProductRegistry&, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>&, std::shared_ptr<edm::ProcessConfiguration>&, edm::PreallocationConfiguration const&, edm::ProcessContext const*&, edm::ModuleTypeResolverMaker const*&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#25 0x000014916baf3ab8 in edm::ScheduleItems::initModules(edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::PreallocationConfiguration const&, edm::ProcessContext const*, edm::ModuleTypeResolverMaker const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#26 0x000014916b9f0712 in tbb::detail::d1::function_task<edm::EventProcessor::init(std::shared_ptr<edm::ProcessDesc>&, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy)::$_0>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#27 0x000014916a12fb3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x1491677c7600, waiter=..., this=0x1491677c9400) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#28 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x1491677c9400) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#29 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#30 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#31 0x000014916a131cee in tbb::detail::r1::rml::private_worker::run (this=0x1491657f4100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#32 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x1491657f4100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#33 0x000014916926d1ca in start_thread () from /lib64/libpthread.so.0
#34 0x0000149168ec88d3 in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x14911bb24700 (LWP 1795600) "cmsRun"):
#0  0x0000149168f97098 in nanosleep () from /lib64/libc.so.6
#1  0x0000149168f96f9e in sleep () from /lib64/libc.so.6
#2  0x00001491644c6544 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000149168ec841d in syscall () from /lib64/libc.so.6
#5  0x000014916a131fd2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x1491657f40a4) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x1491657f40a4) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x1491657f40a0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x1491657f4080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x1491657f4080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000014916926d1ca in start_thread () from /lib64/libpthread.so.0
#11 0x0000149168ec88d3 in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x149142c53700 (LWP 1795511) "cmsRun"):
#0  0x00001491692776a2 in waitpid () from /lib64/libpthread.so.0
#1  0x00001491644c71f1 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x0000149169901a73 in std::execute_native_thread_routine (__p=0x149145ac1690) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#3  0x000014916926d1ca in start_thread () from /lib64/libpthread.so.0
#4  0x0000149168ec88d3 in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x1491683e5680 (LWP 1795457) "cmsRun"):
#0  0x0000149168f97098 in nanosleep () from /lib64/libc.so.6
#1  0x0000149168f96f9e in sleep () from /lib64/libc.so.6
#2  0x00001491644c6544 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014915cdcf910 in TClingClassInfo::Title() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libCling.so
#5  0x000014916ab63ff2 in TClass::Init(char const*, short, std::type_info const*, TVirtualIsAProxy*, char const*, char const*, int, int, ClassInfo_t*, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libCore.so
#6  0x000014916ab64f78 in TClass::TClass(char const*, short, std::type_info const&, TVirtualIsAProxy*, char const*, char const*, int, int, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libCore.so
#7  0x000014916ab6507b in ROOT::CreateClass(char const*, short, std::type_info const&, TVirtualIsAProxy*, char const*, char const*, int, int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libCore.so
#8  0x000014916ab76642 in ROOT::TGenericClassInfo::GetClass() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libCore.so
#9  0x0000149164142921 in TH2S::Dictionary() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libHist.so
#10 0x000014916ab590aa in TClass::GetClass(char const*, bool, bool, unsigned long, unsigned long) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libCore.so
#11 0x000014916af08aeb in TStreamerInfo::BuildCheck(TFile*, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libRIO.so
#12 0x000014916aeb2a15 in TFile::ReadStreamerInfo() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libRIO.so
#13 0x000014916aebe285 in TFile::Init(bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libRIO.so
#14 0x0000149165560a7b in TStorageFactoryFile::Initialize(char const*, char const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libIOPoolTFileAdaptor.so
#15 0x00001491655605d5 in TStorageFactoryFile::TStorageFactoryFile(char const*, char const*, char const*, int, int, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libIOPoolTFileAdaptor.so
#16 0x00001491650ad0b9 in ?? ()
#17 0x0000149100000000 in ?? ()
#18 0x0000149121a9f4c0 in ?? ()
#19 0x00001491441d3250 in ?? ()
#20 0x00007fffbd223e90 in ?? ()
#21 0x000014911a701135 in ?? () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginIOPoolInput.so
#22 0x00007fffbd223fc9 in ?? ()
#23 0x00001491219dd100 in ?? ()
#24 0x000014915cdca2d2 in TClingCallFunc::IFacePtr() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libCling.so
#25 0x0000006500000000 in ?? ()
#26 0x0000149145db8260 in ?? ()
#27 0x00001491219f3600 in ?? ()
#28 0x00007fffbd223f90 in ?? ()
#29 0x00007fffbd223f98 in ?? ()
#30 0x0000149145ac7600 in ?? ()
#31 0x000014916aec7233 in long TPluginHandler::ExecPluginImpl<char const*, char const*, char const*, int, int>(char const* const&, char const* const&, char const* const&, int const&, int const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libRIO.so
#32 0x000014916aec369c in TFile::Open(char const*, char const*, char const*, int, int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_CLANG_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libRIO.so
#33 0x000014911a6b0ef2 in edm::InputFile::InputFile(char const*, char const*, edm::InputType) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginIOPoolInput.so
#34 0x000014911a6f4cbf in edm::RootInputFileSequence::initTheFile(bool, bool, edm::InputSource*, char const*, edm::InputType) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginIOPoolInput.so
#35 0x000014911a6f6cc7 in edm::RootPrimaryFileSequence::RootPrimaryFileSequence(edm::ParameterSet const&, edm::PoolSource&, edm::InputFileCatalog const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginIOPoolInput.so
#36 0x000014911a6b2f67 in edm::PoolSource::PoolSource(edm::ParameterSet const&, edm::InputSourceDescription const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginIOPoolInput.so
#37 0x000014911a6b1c3c in edmplugin::PluginFactory<edm::InputSource* (edm::ParameterSet const&, edm::InputSourceDescription const&)>::PMaker<edm::PoolSource>::create(edm::ParameterSet const&, edm::InputSourceDescription const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginIOPoolInput.so
#38 0x000014916ba6b97d in edm::InputSourceFactory::makeInputSource(edm::ParameterSet const&, edm::InputSourceDescription const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#39 0x000014916b9c9a2a in edm::makeInput(unsigned int, edm::ParameterSet&, edm::CommonParams const&, std::shared_ptr<edm::ProductRegistry>, std::shared_ptr<edm::BranchIDListHelper>, std::shared_ptr<edm::ProcessBlockHelper> const&, std::shared_ptr<edm::ThinnedAssociationsHelper>, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration const>, edm::PreallocationConfiguration const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#40 0x000014916b9f0af1 in tbb::detail::d1::function_task<edm::EventProcessor::init(std::shared_ptr<edm::ProcessDesc>&, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy)::$_1>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#41 0x000014916a1383e1 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x1491677c9380) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#42 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x1491677c9380) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#43 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#44 0x000014916b9f9668 in void tbb::detail::d0::try_call_proxy<tbb::detail::d1::task_group_base::wait()::{lambda()#1}>::on_completion<tbb::detail::d1::task_group_base::wait()::{lambda()#2}>(tbb::detail::d1::task_group_base::wait()::{lambda()#2}) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#45 0x000014916b9cc352 in edm::EventProcessor::init(std::shared_ptr<edm::ProcessDesc>&, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#46 0x000014916b9ceb0e in edm::EventProcessor::EventProcessor(std::shared_ptr<edm::ProcessDesc>, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_CLANG_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#47 0x00005649d7d2f68d in std::__detail::_MakeUniq<edm::EventProcessor>::__single_object std::make_unique<edm::EventProcessor, std::shared_ptr<edm::ProcessDesc>&, edm::ServiceToken&, edm::serviceregistry::ServiceLegacy>(std::shared_ptr<edm::ProcessDesc>&, edm::ServiceToken&, edm::serviceregistry::ServiceLegacy&&) ()
#48 0x00005649d7d2ba57 in tbb::detail::d1::task_arena_function<main::$_0::operator()() const::{lambda()#1}, void>::operator()() const ()
#49 0x000014916a1249ad in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:688
#50 0x00005649d7d2ad06 in main::$_0::operator()() const ()
#51 0x00005649d7d287ff in main ()

Current Modules:

Module: none (crashed)
vgvassilev commented 1 month ago

It says that it crashes within the getenv function itself:

Thread 1 (Thread 0x14cf0f145780 (LWP 613234) "cmsRun"):
#3  0x000014cf0aebdf40 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02832/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-04-07-0000/lib/slc7_amd64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x000014cf10f452bd in getenv () from /lib64/libc.so.6
#6  0x000014cf03960998 in llvm::sys::Process::GetEnv[abi:cxx11](llvm::StringRef) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libCling.so
#7  0x000014cf00042fdc in TClingCallbacks::findInGlobalModuleIndex(clang::DeclarationName, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libCling.so

The argument passed to Process::GetEnv is a hard-coded string in this case. There is some manipulation in there, but still I don't see anything obvious leading to the crash :(

Are we in a multithreaded environment? I do not think getenv is threadsafe which is a bit counterintuitive...

makortel commented 1 month ago

It says that it crashes within the getenv function itself:

Thread 1 (Thread 0x14cf0f145780 (LWP 613234) "cmsRun"):
#3  0x000014cf0aebdf40 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02832/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-04-07-0000/lib/slc7_amd64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x000014cf10f452bd in getenv () from /lib64/libc.so.6
#6  0x000014cf03960998 in llvm::sys::Process::GetEnv[abi:cxx11](llvm::StringRef) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libCling.so
#7  0x000014cf00042fdc in TClingCallbacks::findInGlobalModuleIndex(clang::DeclarationName, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-07-2300/external/slc7_amd64_gcc12/lib/libCling.so

The argument passed to Process::GetEnv is a hard-coded string in this case. There is some manipulation in there, but still I don't see anything obvious leading to the crash :(

Are we in a multithreaded environment?

Yes, all of these are multithreaded.

I do not think getenv is threadsafe which is a bit counterintuitive...

Good catch! Indeed C ::getenv() is not required to be thread safe, which is why in CMSSW we migrated bunch of code to std::getenv() (https://github.com/cms-sw/cmssw/issues/28073) that is thread safe.

iarspider commented 6 days ago

@vgvassilev @makortel This still happens: slc7_amd64_gcc12/CMSSW_14_2_X_2024-09-12-1100.

vgvassilev commented 6 days ago

I've opened a PR against LLVM. Let's see how this goes and will backport the patch to ROOT. Would that work?

makortel commented 5 days ago

Following the discussion in https://github.com/llvm/llvm-project/pull/108529 (https://github.com/llvm/llvm-project/pull/108529#issuecomment-2349021404 and https://github.com/llvm/llvm-project/pull/108529#issuecomment-2349085715 in particular), and quick web searches suggesting glibc's getenv() would actually be thread safe (in the way C++ and apparently nowadays also POSIX requires).

Could the problem be the environment being modified? Already a quick git grep setenv( in CMSSW gives hits of which some could be potentially dangerous.

makortel commented 5 days ago

Could the problem be the environment being modified? Already a quick git grep setenv( in CMSSW gives hits of which some could be potentially dangerous.

Running the step2 of 12861.0 (from the latest report https://github.com/cms-sw/cmssw/issues/44659#issuecomment-2348411797) through gdb indeed shows setenv() calls from the constructors of a few modules. The module constructors are run concurrently to the Source constructor (that ends up calling the LLVM's getenv()). I think we need to follow up these (and likely find a better way to set environment variables) in a separate issue.

makortel commented 5 days ago

Could the problem be the environment being modified? Already a quick git grep setenv( in CMSSW gives hits of which some could be potentially dangerous.

Running the step2 of 12861.0 (from the latest report #44659 (comment)) through gdb indeed shows setenv() calls from the constructors of a few modules. The module constructors are run concurrently to the Source constructor (that ends up calling the LLVM's getenv()). I think we need to follow up these (and likely find a better way to set environment variables) in a separate issue.

Here is the new issue https://github.com/cms-sw/cmssw/issues/46002

vgvassilev commented 5 days ago

Can you paste both stack traces?

makortel commented 2 days ago 
Thread 3 "cmsRun" hit Breakpoint 1, 0x00007ffff516a900 in setenv () from /lib64/libc.so.6
#0  0x00007ffff516a900 in setenv () from /lib64/libc.so.6
#1  0x00007fff8b1ae7c6 in TSGForOIDNN::TSGForOIDNN(edm::ParameterSet const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_2_X_2024-09-12-1100/lib/el8_amd64_gcc12/pluginRecoMuonTrackerSeedGeneratorPlugins.so
#2  0x00007fff8b1c1bff in edm::WorkerMaker<TSGForOIDNN>::makeModule(edm::ParameterSet const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_2_X_2024-09-12-1100/lib/el8_amd64_gcc12/pluginRecoMuonTrackerSeedGeneratorPlugins.so
#3  0x00007ffff7cbf5a7 in edm::Maker::makeModule(edm::MakeModuleParams const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_2_X_2024-09-12-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#4  0x00007ffff7c2ced8 in edm::Factory::makeModule(edm::MakeModuleParams const&, edm::ModuleTypeResolverMaker const*, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_2_X_2024-09-12-1100/lib/>
#5  0x00007ffff7c3d04f in edm::ModuleRegistry::getModule(edm::MakeModuleParams const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc1>
#6  0x00007ffff7cbfda8 in edm::WorkerRegistry::getWorker(edm::WorkerParams const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_2_X_2024-09-12-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#7  0x00007ffff7cc0085 in edm::WorkerManager::getWorker(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14>
#8  0x00007ffff7c9a940 in edm::(anonymous namespace)::getWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, edm::ParameterSet&, edm::WorkerManager&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>) [clone .lto_priv.0] () from /cvmfs/cms-ib.cern.ch/sw/x86_>
#9  0x00007ffff7ca8ebe in edm::StreamSchedule::fillWorkers(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, std::vector<edm::WorkerInPath, std::allocator<edm::WorkerInPath> >&, std::vec>
#10 0x00007ffff7ca9be9 in edm::StreamSchedule::fillTrigPath(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<edm::HLTGlobalStatus>, std::vector<std::__cxx11::basic_string>
#11 0x00007ffff7c9e867 in edm::StreamSchedule::StreamSchedule(std::shared_ptr<edm::TriggerResultInserter>, std::vector<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> >, std::allocator<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> > > >&, std::vector<edm::propagate_const<std::shared_ptr<edm::EndPathStatusInserter> >, std::allocator>
#12 0x00007ffff7c83f75 in edm::Schedule::Schedule(edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::ProductRegistry&, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration const>, edm::PreallocationConfiguration const&, edm::ProcessContext const*, edm::ModuleTypeResolverMaker const>
#13 0x00007ffff7c97b00 in edm::ScheduleItems::initModules(edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::PreallocationConfiguration const&, edm::ProcessContext const*, edm::ModuleTypeResolverMaker const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_2_X_2024-09-12-1100/lib/el8_amd64_gcc12/libFWCoreFr>
#14 0x00007ffff7c04c47 in tbb::detail::d1::function_task<edm::EventProcessor::init(std::shared_ptr<edm::ProcessDesc>&, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) [clone .lto_priv.0] () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_2_X_2024-09-12-1>
#15 0x00007ffff63bcb3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x7ffff3096d00, waiter=..., this=0x7ffff308be80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#16 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x7ffff308be80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#17 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#18 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#19 0x00007ffff63becee in tbb::detail::r1::rml::private_worker::run (this=0x7fffee9df100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#20 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7fffee9df100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#21 0x00007ffff54f81ca in start_thread () from /lib64/libpthread.so.0
#22 0x00007ffff51538d3 in clone () from /lib64/libc.so.6
Thread 3 "cmsRun" hit Breakpoint 1, 0x00007ffff516a900 in setenv () from /lib64/libc.so.6
#0  0x00007ffff516a900 in setenv () from /lib64/libc.so.6
#1  0x00007fff70b0b8e3 in L2TauNNProducerAlpaka::initializeGlobalCache(edm::ParameterSet const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_2_X_2024-09-12-1100/lib/el8_amd64_gcc12/pluginRecoTauTagHLTProducers.so
#2  0x00007fff70b12415 in edm::WorkerMaker<L2TauNNProducerAlpaka>::makeModule(edm::ParameterSet const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_2_X_2024-09-12-1100/lib/el8_amd64_gcc12/pluginRecoTauTagHLTProducers.so
#3  0x00007ffff7cbf5a7 in edm::Maker::makeModule(edm::MakeModuleParams const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_2_X_2024-09-12-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#4  0x00007ffff7c2ced8 in edm::Factory::makeModule(edm::MakeModuleParams const&, edm::ModuleTypeResolverMaker const*, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_2_X_2024-09-12-1100/lib/>
#5  0x00007ffff7c3d04f in edm::ModuleRegistry::getModule(edm::MakeModuleParams const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc1>
#6  0x00007ffff7cbfda8 in edm::WorkerRegistry::getWorker(edm::WorkerParams const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_2_X_2024-09-12-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#7  0x00007ffff7cc0085 in edm::WorkerManager::getWorker(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14>
#8  0x00007ffff7c9a940 in edm::(anonymous namespace)::getWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, edm::ParameterSet&, edm::WorkerManager&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>) [clone .lto_priv.0] () from /cvmfs/cms-ib.cern.ch/sw/x86_>
#9  0x00007ffff7ca8ebe in edm::StreamSchedule::fillWorkers(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, std::vector<edm::WorkerInPath, std::allocator<edm::WorkerInPath> >&, std::vec>
#10 0x00007ffff7ca9be9 in edm::StreamSchedule::fillTrigPath(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<edm::HLTGlobalStatus>, std::vector<std::__cxx11::basic_string>
#11 0x00007ffff7c9e867 in edm::StreamSchedule::StreamSchedule(std::shared_ptr<edm::TriggerResultInserter>, std::vector<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> >, std::allocator<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> > > >&, std::vector<edm::propagate_const<std::shared_ptr<edm::EndPathStatusInserter> >, std::allocator>
#12 0x00007ffff7c83f75 in edm::Schedule::Schedule(edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::ProductRegistry&, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration const>, edm::PreallocationConfiguration const&, edm::ProcessContext const*, edm::ModuleTypeResolverMaker const>
#13 0x00007ffff7c97b00 in edm::ScheduleItems::initModules(edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::PreallocationConfiguration const&, edm::ProcessContext const*, edm::ModuleTypeResolverMaker const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_2_X_2024-09-12-1100/lib/el8_amd64_gcc12/libFWCoreFr>
#14 0x00007ffff7c04c47 in tbb::detail::d1::function_task<edm::EventProcessor::init(std::shared_ptr<edm::ProcessDesc>&, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) [clone .lto_priv.0] () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_2_X_2024-09-12-1>
#15 0x00007ffff63bcb3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x7ffff3096d00, waiter=..., this=0x7ffff308be80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#16 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x7ffff308be80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#17 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#18 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#19 0x00007ffff63becee in tbb::detail::r1::rml::private_worker::run (this=0x7fffee9df100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#20 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7fffee9df100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#21 0x00007ffff54f81ca in start_thread () from /lib64/libpthread.so.0
#22 0x00007ffff51538d3 in clone () from /lib64/libc.so.6

Being lazy I only recorded the stack trace of the thread that called the setenv(). I could re-run capturing the stack traces of all threads, but to me this was enough proof that the possibility of data race is there (since the edm::PoolSource::PoolSource() is run concurrently to these call stacks).

vgvassilev commented 2 days ago

I do not see what we can do on the ROOT side. Is avoiding setenv possible in cmssw?

makortel commented 2 days ago

Is avoiding setenv possible in cmssw?

I think avoiding setenv() in CMSSW is exactly what we should do, and I'm trying to eventually achieve that in https://github.com/cms-sw/cmssw/issues/46002.