search

cms-sw / cmssw

CMS Offline Software
http://cms-sw.github.io/
Apache License 2.0
1.09k stars 4.32k forks source link

[ARM] Crashes in Cling #43802

Open makortel opened 10 months ago

makortel commented 10 months ago

This issue is to continue to track the crashes in Cling we're seeing on ARM that were first reported in https://github.com/cms-sw/cmssw/issues/43577#issuecomment-1858008684. Repeating the stack traces:

On el8_aarch64_gcc12 CMSSW_14_0_X_2023-12-13-2300 two workflows crashed in a way that looks possibly related

11024.0 step 3

Other TBB threads are in tbb::detail::r1::futex_wait()

Thread 1 (Thread 0x4000269bd080 (LWP 63750) "cmsRun"):
#3  0x000040002bf5ce8c in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00004001acff001c in ?? ()
#6  0x0000400039d3c590 in cling::Interpreter::RunFunction(clang::FunctionDecl const*, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#7  0x0000400039d3c590 in cling::Interpreter::RunFunction(clang::FunctionDecl const*, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#8  0x0000400039d3cc3c in cling::Interpreter::EvaluateInternal(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, cling::CompilationOptions, cling::Value*, cling::Transaction**, unsigned long) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#9  0x0000400039e18844 in cling::MetaSema::actOnxCommand(llvm::StringRef, llvm::StringRef, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#10 0x0000400039e25134 in cling::MetaParser::isXCommand(cling::MetaSema::ActionResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#11 0x0000400039e26ff4 in cling::MetaParser::isCommand(cling::MetaSema::ActionResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#12 0x0000400039e122ec in cling::MetaProcessor::process(llvm::StringRef, cling::Interpreter::CompilationResult&, cling::Value*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#13 0x0000400039c602dc in HandleInterpreterException(cling::MetaProcessor*, char const*, cling::Interpreter::CompilationResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#14 0x0000400039c72d40 in TCling::ProcessLine(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#15 0x0000400039c732b4 in TCling::ProcessLineSynch(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#16 0x0000400025f398a8 in TApplication::ExecuteFile(char const*, int*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCore.so
#17 0x0000400039c71dd8 in TCling::ExecuteMacro(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#18 0x0000400025f8ec5c in TROOT::Macro(char const*, int*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCore.so
#19 0x0000400025f72a90 in TPluginManager::LoadHandlerMacros(char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCore.so
#20 0x0000400025f730ec in TPluginManager::LoadHandlersFromPluginDirs(char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCore.so
#21 0x0000400025f74164 in TPluginManager::FindHandler(char const*, char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCore.so
#22 0x000040002579899c in ROOT::Math::Factory::CreateMinimizer(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#23 0x000040002579a270 in ROOT::Fit::FitConfig::CreateMinimizer() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#24 0x00004000257b1dc0 in ROOT::Fit::Fitter::DoInitMinimizer() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#25 0x00004000257b827c in bool ROOT::Fit::Fitter::DoMinimization<ROOT::Fit::PoissonLikelihoodFCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> > >(std::unique_ptr<ROOT::Fit::PoissonLikelihoodFCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> >, std::default_delete<ROOT::Fit::PoissonLikelihoodFCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> > > >, ROOT::Math::IBaseFunctionMultiDimTempl<double> const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#26 0x00004000257b5590 in ROOT::Fit::Fitter::DoBinnedLikelihoodFit(bool, ROOT::EExecutionPolicy const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#27 0x000040002c60a3a4 in TFitResultPtr HFit::Fit<TH1>(TH1*, TF1*, Foption_t&, ROOT::Math::MinimizerOptions const&, char const*, ROOT::Fit::DataRange&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libHist.so
#28 0x000040002c5fff14 in ROOT::Fit::FitObject(TH1*, TF1*, Foption_t&, ROOT::Math::MinimizerOptions const&, char const*, ROOT::Fit::DataRange&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libHist.so
#29 0x000040002c6c97e0 in TH1::Fit(TF1*, char const*, char const*, double, double) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libHist.so
#30 0x000040006b49055c in PVFitter::runFitter() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/lib/el8_aarch64_gcc12/libRecoVertexBeamSpotProducer.so
#31 0x000040006b4871d8 in BeamFitter::runPVandTrkFitter() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/lib/el8_aarch64_gcc12/libRecoVertexBeamSpotProducer.so
#32 0x00004000c5aa0e94 in AlcaBeamMonitor::globalEndLuminosityBlock(edm::LuminosityBlock const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/lib/el8_aarch64_gcc12/pluginAlcaBeamMonitor.so
#33 0x00004000c5aad9fc in virtual thunk to edm::one::impl::LuminosityBlockCacheHolder<edm::one::EDProducerBase, alcabeammonitor::BeamSpotInfo>::doEndLuminosityBlock_(edm::LuminosityBlock const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/lib/el8_aarch64_gcc12/pluginAlcaBeamMonitor.so
#34 0x0000400024dadcc4 in edm::one::EDProducerBase::doEndLuminosityBlock(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so

Current Modules:
Module: AlcaBeamMonitor:AlcaBeamMonitor (crashed)
Module: none
Module: none
Module: none

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/el8_aarch64_gcc12/CMSSW_14_0_X_2023-12-13-2300/pyRelValMatrixLogs/run/11024.0_TTbar_13+2018PU/step3_TTbar_13+2018PU.log#/

11025.0 step 3

Other TBB threads are in tbb::detail::r1::futex_wait

Thread 3 (Thread 0x400053389250 (LWP 2727347) "cmsRun"):
#3  0x000040000fdece8c in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00004001aafe001c in ?? ()
#6  0x000040001dcac590 in cling::Interpreter::RunFunction(clang::FunctionDecl const*, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#7  0x000040001dcac590 in cling::Interpreter::RunFunction(clang::FunctionDecl const*, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#8  0x000040001dcacc3c in cling::Interpreter::EvaluateInternal(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, cling::CompilationOptions, cling::Value*, cling::Transaction**, unsigned long) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#9  0x000040001dd88844 in cling::MetaSema::actOnxCommand(llvm::StringRef, llvm::StringRef, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#10 0x000040001dd95134 in cling::MetaParser::isXCommand(cling::MetaSema::ActionResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#11 0x000040001dd96ff4 in cling::MetaParser::isCommand(cling::MetaSema::ActionResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#12 0x000040001dd822ec in cling::MetaProcessor::process(llvm::StringRef, cling::Interpreter::CompilationResult&, cling::Value*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#13 0x000040001dbd02dc in HandleInterpreterException(cling::MetaProcessor*, char const*, cling::Interpreter::CompilationResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#14 0x000040001dbe2d40 in TCling::ProcessLine(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#15 0x000040001dbe32b4 in TCling::ProcessLineSynch(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#16 0x0000400009e898a8 in TApplication::ExecuteFile(char const*, int*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCore.so
#17 0x000040001dbe1dd8 in TCling::ExecuteMacro(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCling.so
#18 0x0000400009edec5c in TROOT::Macro(char const*, int*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCore.so
#19 0x0000400009ec2a90 in TPluginManager::LoadHandlerMacros(char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCore.so
#20 0x0000400009ec30ec in TPluginManager::LoadHandlersFromPluginDirs(char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCore.so
#21 0x0000400009ec4164 in TPluginManager::FindHandler(char const*, char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libCore.so
#22 0x00004000096e899c in ROOT::Math::Factory::CreateMinimizer(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#23 0x00004000096ea270 in ROOT::Fit::FitConfig::CreateMinimizer() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#24 0x0000400009701dc0 in ROOT::Fit::Fitter::DoInitMinimizer() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#25 0x000040000970827c in bool ROOT::Fit::Fitter::DoMinimization<ROOT::Fit::PoissonLikelihoodFCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> > >(std::unique_ptr<ROOT::Fit::PoissonLikelihoodFCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> >, std::default_delete<ROOT::Fit::PoissonLikelihoodFCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> > > >, ROOT::Math::IBaseFunctionMultiDimTempl<double> const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#26 0x0000400009705590 in ROOT::Fit::Fitter::DoBinnedLikelihoodFit(bool, ROOT::EExecutionPolicy const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#27 0x000040001054a3a4 in TFitResultPtr HFit::Fit<TH1>(TH1*, TF1*, Foption_t&, ROOT::Math::MinimizerOptions const&, char const*, ROOT::Fit::DataRange&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libHist.so
#28 0x000040001053ff14 in ROOT::Fit::FitObject(TH1*, TF1*, Foption_t&, ROOT::Math::MinimizerOptions const&, char const*, ROOT::Fit::DataRange&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libHist.so
#29 0x00004000106097e0 in TH1::Fit(TF1*, char const*, char const*, double, double) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/external/el8_aarch64_gcc12/lib/libHist.so
#30 0x000040004f3c055c in PVFitter::runFitter() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/lib/el8_aarch64_gcc12/libRecoVertexBeamSpotProducer.so
#31 0x000040004f3b71d8 in BeamFitter::runPVandTrkFitter() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/lib/el8_aarch64_gcc12/libRecoVertexBeamSpotProducer.so
#32 0x00004000ab9f0e94 in AlcaBeamMonitor::globalEndLuminosityBlock(edm::LuminosityBlock const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/lib/el8_aarch64_gcc12/pluginAlcaBeamMonitor.so
#33 0x00004000ab9fd9fc in virtual thunk to edm::one::impl::LuminosityBlockCacheHolder<edm::one::EDProducerBase, alcabeammonitor::BeamSpotInfo>::doEndLuminosityBlock_(edm::LuminosityBlock const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/lib/el8_aarch64_gcc12/pluginAlcaBeamMonitor.so
#34 0x0000400008cfdcc4 in edm::one::EDProducerBase::doEndLuminosityBlock(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02815/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2023-12-13-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so

Module: AlcaBeamMonitor:AlcaBeamMonitor (crashed)
Module: none
Module: none
Module: none

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/el8_aarch64_gcc12/CMSSW_14_0_X_2023-12-13-2300/pyRelValMatrixLogs/run/11025.0_ZEE_13+2018PU/step3_ZEE_13+2018PU.log#/

cmsbuild commented 10 months ago

cms-bot internal usage

cmsbuild commented 10 months ago

A new Issue was created by @makortel Matti Kortelainen.

@rappoccio, @sextonkennedy, @Dr15Jones, @smuzaffar, @antoniovilela, @makortel can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

makortel commented 10 months ago

Some more from CMSSW_14_0_X_2024-01-25-2300 on el9_aarch64_gcc12

140.58 step 3

Thread 1 (Thread 0x40000fd3b360 (LWP 1572278) "cmsRun"):
#3  0x000040001543cf3c in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x000040018cff001c in ?? ()
#6  0x000040002052ad94 in cling::IncrementalExecutor::executeWrapper(llvm::StringRef, cling::Value*) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#7  0x000040002052ad94 in cling::IncrementalExecutor::executeWrapper(llvm::StringRef, cling::Value*) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#8  0x0000000000000000 in ?? ()

Current Modules:
Module: AlcaBeamMonitor:AlcaBeamMonitor (crashed)
Module: none
Module: none
Module: none

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/el9_aarch64_gcc12/CMSSW_14_0_X_2024-01-25-2300/pyRelValMatrixLogs/run/140.58_RunHI2018/step3_RunHI2018.log#/

11024.0 step 3

Thread 5 (Thread 0x400064e19180 (LWP 1567392) "cmsRun"):
#3  0x000040001f24cf3c in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00004001b2fe001c in ?? ()
#6  0x000040002d09bea0 in cling::Interpreter::RunFunction(clang::FunctionDecl const*, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#7  0x000040002d09bea0 in cling::Interpreter::RunFunction(clang::FunctionDecl const*, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#8  0x000040002d09c54c in cling::Interpreter::EvaluateInternal(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, cling::CompilationOptions, cling::Value*, cling::Transaction**, unsigned long) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#9  0x000040002d16cb44 in cling::MetaSema::actOnxCommand(llvm::StringRef, llvm::StringRef, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#10 0x000040002d179434 in cling::MetaParser::isXCommand(cling::MetaSema::ActionResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#11 0x000040002d17b2f4 in cling::MetaParser::isCommand(cling::MetaSema::ActionResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#12 0x000040002d1665ec in cling::MetaProcessor::process(llvm::StringRef, cling::Interpreter::CompilationResult&, cling::Value*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#13 0x000040002cfbfcfc in HandleInterpreterException(cling::MetaProcessor*, char const*, cling::Interpreter::CompilationResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#14 0x000040002cfd2790 in TCling::ProcessLine(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#15 0x000040002cfd2d04 in TCling::ProcessLineSynch(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#16 0x00004000191e9888 in TApplication::ExecuteFile(char const*, int*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCore.so
#17 0x000040002cfd1828 in TCling::ExecuteMacro(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#18 0x000040001923eb9c in TROOT::Macro(char const*, int*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCore.so
#19 0x0000400019222980 in TPluginManager::LoadHandlerMacros(char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCore.so
#20 0x0000400019222fdc in TPluginManager::LoadHandlersFromPluginDirs(char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCore.so
#21 0x0000400019224064 in TPluginManager::FindHandler(char const*, char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCore.so
#22 0x0000400018a4835c in ROOT::Math::Factory::CreateMinimizer(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libMathCore.so
#23 0x0000400018a49c30 in ROOT::Fit::FitConfig::CreateMinimizer() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libMathCore.so
#24 0x0000400018a617a0 in ROOT::Fit::Fitter::DoInitMinimizer() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libMathCore.so
#25 0x0000400018a67840 in bool ROOT::Fit::Fitter::DoMinimization<ROOT::Fit::PoissonLikelihoodFCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> > >(std::unique_ptr<ROOT::Fit::PoissonLikelihoodFCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> >, std::default_delete<ROOT::Fit::PoissonLikelihoodFCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> > > >, ROOT::Math::IBaseFunctionMultiDimTempl<double> const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libMathCore.so
#26 0x0000400018a64ddc in ROOT::Fit::Fitter::DoBinnedLikelihoodFit(bool, ROOT::EExecutionPolicy const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libMathCore.so
#27 0x000040001f8fa26c in TFitResultPtr HFit::Fit<TH1>(TH1*, TF1*, Foption_t&, ROOT::Math::MinimizerOptions const&, char const*, ROOT::Fit::DataRange&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libHist.so
#28 0x000040001f8f00e4 in ROOT::Fit::FitObject(TH1*, TF1*, Foption_t&, ROOT::Math::MinimizerOptions const&, char const*, ROOT::Fit::DataRange&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libHist.so
#29 0x000040001f9b9630 in TH1::Fit(TF1*, char const*, char const*, double, double) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libHist.so
#30 0x000040005e95058c in PVFitter::runFitter() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libRecoVertexBeamSpotProducer.so
#31 0x000040005e9471ac in BeamFitter::runPVandTrkFitter() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libRecoVertexBeamSpotProducer.so
#32 0x00004000bb0a0ee4 in AlcaBeamMonitor::globalEndLuminosityBlock(edm::LuminosityBlock const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginAlcaBeamMonitor.so
#33 0x00004000bb0ad9cc in virtual thunk to edm::one::impl::LuminosityBlockCacheHolder<edm::one::EDProducerBase, alcabeammonitor::BeamSpotInfo>::doEndLuminosityBlock_(edm::LuminosityBlock const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginAlcaBeamMonitor.so
#34 0x000040001805e6b4 in edm::one::EDProducerBase::doEndLuminosityBlock(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#35 0x00004000180456f0 in edm::WorkerT<edm::one::EDProducerBase>::implDoEnd(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libFWCoreFramework.so

Current Modules:
Module: AlcaBeamMonitor:AlcaBeamMonitor (crashed)
Module: none
Module: none
Module: none

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/el9_aarch64_gcc12/CMSSW_14_0_X_2024-01-25-2300/pyRelValMatrixLogs/run/11024.0_TTbar_13+2018PU/step3_TTbar_13+2018PU.log#/

25034.114 step 3

Thread 5 (Thread 0x400059a09180 (LWP 489105) "cmsRun"):
#3  0x0000400012adcf3c in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x000040028bfe001c in ?? ()
#6  0x000040002096bea0 in cling::Interpreter::RunFunction(clang::FunctionDecl const*, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#7  0x000040002096bea0 in cling::Interpreter::RunFunction(clang::FunctionDecl const*, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#8  0x000040002096c54c in cling::Interpreter::EvaluateInternal(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, cling::CompilationOptions, cling::Value*, cling::Transaction**, unsigned long) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#9  0x0000400020a3cb44 in cling::MetaSema::actOnxCommand(llvm::StringRef, llvm::StringRef, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#10 0x0000400020a49434 in cling::MetaParser::isXCommand(cling::MetaSema::ActionResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#11 0x0000400020a4b2f4 in cling::MetaParser::isCommand(cling::MetaSema::ActionResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#12 0x0000400020a365ec in cling::MetaProcessor::process(llvm::StringRef, cling::Interpreter::CompilationResult&, cling::Value*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#13 0x000040002088fcfc in HandleInterpreterException(cling::MetaProcessor*, char const*, cling::Interpreter::CompilationResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#14 0x00004000208a2790 in TCling::ProcessLine(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#15 0x00004000208a2d04 in TCling::ProcessLineSynch(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#16 0x000040000ca39888 in TApplication::ExecuteFile(char const*, int*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCore.so
#17 0x00004000208a1828 in TCling::ExecuteMacro(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCling.so
#18 0x000040000ca8eb9c in TROOT::Macro(char const*, int*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCore.so
#19 0x000040000ca72980 in TPluginManager::LoadHandlerMacros(char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCore.so
#20 0x000040000ca72fdc in TPluginManager::LoadHandlersFromPluginDirs(char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCore.so
#21 0x000040000ca74064 in TPluginManager::FindHandler(char const*, char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libCore.so
#22 0x000040000c29835c in ROOT::Math::Factory::CreateMinimizer(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libMathCore.so
#23 0x000040000c299c30 in ROOT::Fit::FitConfig::CreateMinimizer() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libMathCore.so
#24 0x000040000c2b17a0 in ROOT::Fit::Fitter::DoInitMinimizer() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libMathCore.so
#25 0x000040000c2b7490 in bool ROOT::Fit::Fitter::DoMinimization<ROOT::Fit::Chi2FCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> > >(std::unique_ptr<ROOT::Fit::Chi2FCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> >, std::default_delete<ROOT::Fit::Chi2FCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> > > >, ROOT::Math::IBaseFunctionMultiDimTempl<double> const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libMathCore.so
#26 0x000040000c2b448c in ROOT::Fit::Fitter::DoLeastSquareFit(ROOT::EExecutionPolicy const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libMathCore.so
#27 0x00004000131c9138 in TFitResultPtr HFit::Fit<TGraph>(TGraph*, TF1*, Foption_t&, ROOT::Math::MinimizerOptions const&, char const*, ROOT::Fit::DataRange&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libHist.so
#28 0x00004000131c006c in ROOT::Fit::FitObject(TGraph*, TF1*, Foption_t&, ROOT::Math::MinimizerOptions const&, char const*, ROOT::Fit::DataRange&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libHist.so
#29 0x000040001326d0d4 in TGraph::Fit(TF1*, char const*, char const*, double, double) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/external/el9_aarch64_gcc12/lib/libHist.so
#30 0x0000400090e150dc in CSCSimHitMatcher::fitHitsInChamber(unsigned int, float&, float&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libValidationMuonHits.so
#31 0x00004000959796e8 in CSCStubResolutionValidation::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libValidationMuonCSCDigis.so
#32 0x000040000b8b760c in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#33 0x000040000b89c310 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#34 0x000040000b82ac14 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#35 0x000040000b82b434 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#36 0x000040000bdc50cc in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libFWCoreConcurrency.so

Thread 4 (Thread 0x400056da9180 (LWP 489104) "cmsRun"):
#3  0x0000400012ad924c in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x0000400039a02100 in std::_Sp_counted_ptr_inplace<SiPixelRecHit, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginRecoTrackerMeasurementDetPlugins.so
#6  0x00004000bdd01358 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginRecoTrackerCkfPatternPlugins.so
#7  0x00004000bdd013a0 in TrajectoryMeasurement::~TrajectoryMeasurement() [clone .lto_priv.0] () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginRecoTrackerCkfPatternPlugins.so
#8  0x00004000bdd014b0 in cmsutils::bqueue<TrajectoryMeasurement>::~bqueue() [clone .lto_priv.0] () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginRecoTrackerCkfPatternPlugins.so
#9  0x00004000bdd0172c in std::vector<TempTrajectory, std::allocator<TempTrajectory> >::_M_erase(__gnu_cxx::__normal_iterator<TempTrajectory*, std::vector<TempTrajectory, std::allocator<TempTrajectory> > >, __gnu_cxx::__normal_iterator<TempTrajectory*, std::vector<TempTrajectory, std::allocator<TempTrajectory> > >) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginRecoTrackerCkfPatternPlugins.so
#10 0x00004000bdd039e0 in GroupedCkfTrajectoryBuilder::groupedLimitedCandidates(TrajectorySeed const&, TempTrajectory const&, TrajectoryFilter const*, Propagator const*, bool, std::vector<TempTrajectory, std::allocator<TempTrajectory> >&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginRecoTrackerCkfPatternPlugins.so
#11 0x00004000bdd0573c in GroupedCkfTrajectoryBuilder::rebuildTrajectories(TempTrajectory const&, TrajectorySeed const&, std::vector<Trajectory, std::allocator<Trajectory> >&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginRecoTrackerCkfPatternPlugins.so
#12 0x00004000bc7b9d18 in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&)::{lambda(unsigned long)#1}::operator()(unsigned long) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libRecoTrackerCkfPattern.so
#13 0x00004000bc7ba9fc in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libRecoTrackerCkfPattern.so
#14 0x000040000b8b760c in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#15 0x000040000b89c310 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#16 0x000040000b82ac14 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#17 0x000040000b82b434 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#18 0x000040000bdc50cc in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libFWCoreConcurrency.so

Thread 3 (Thread 0x400056399180 (LWP 489103) "cmsRun"):
#3  0x0000400012ad924c in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00004000e153a20c in std::_Hashtable<std::pair<unsigned int, EncodedEventId>, std::pair<std::pair<unsigned int, EncodedEventId> const, edm::Ref<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle, edm::refhelper::FindUsingAdvance<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle> > >, std::allocator<std::pair<std::pair<unsigned int, EncodedEventId> const, edm::Ref<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle, edm::refhelper::FindUsingAdvance<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle> > > >, std::__detail::_Select1st, std::equal_to<std::pair<unsigned int, EncodedEventId> >, UniqueSimTrackIdHash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_find_before_node(unsigned long, std::pair<unsigned int, EncodedEventId> const&, unsigned long) const [clone .constprop.0] () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginSimTrackerTrackerHitAssociationPlugins.so
#6  0x00004000e153a310 in std::_Hashtable<std::pair<unsigned int, EncodedEventId>, std::pair<std::pair<unsigned int, EncodedEventId> const, edm::Ref<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle, edm::refhelper::FindUsingAdvance<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle> > >, std::allocator<std::pair<std::pair<unsigned int, EncodedEventId> const, edm::Ref<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle, edm::refhelper::FindUsingAdvance<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle> > > >, std::__detail::_Select1st, std::equal_to<std::pair<unsigned int, EncodedEventId> >, UniqueSimTrackIdHash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::find(std::pair<unsigned int, EncodedEventId> const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginSimTrackerTrackerHitAssociationPlugins.so
#7  0x00004000e1540310 in ClusterTPAssociationProducer::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginSimTrackerTrackerHitAssociationPlugins.so
#8  0x000040000b89e638 in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#9  0x000040000b896540 in edm::WorkerT<edm::global::EDProducerBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#10 0x000040000b82ac14 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#11 0x000040000b82b434 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#12 0x000040000bdc50cc in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libFWCoreConcurrency.so

Thread 1 (Thread 0x40000d4cb360 (LWP 485190) "cmsRun"):
#3  0x0000400012ad924c in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00004000e1a1b44c in std::__detail::_Map_base<DetId, std::pair<DetId const, std::vector<std::pair<int, float>, std::allocator<std::pair<int, float> > > >, std::allocator<std::pair<DetId const, std::vector<std::pair<int, float>, std::allocator<std::pair<int, float> > > > >, std::__detail::_Select1st, std::equal_to<DetId>, std::hash<DetId>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true>, true>::operator[](DetId const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginSimCalorimetryHGCalAssociatorProducersAuto.so
#6  0x00004000e1a1c89c in TSToSimTSHitLCAssociatorByEnergyScoreImpl::makeConnections(edm::Handle<std::vector<ticl::Trackster, std::allocator<ticl::Trackster> > > const&, edm::Handle<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > > const&, edm::Handle<std::vector<SimCluster, std::allocator<SimCluster> > > const&, edm::Handle<std::vector<CaloParticle, std::allocator<CaloParticle> > > const&, edm::Handle<std::vector<ticl::Trackster, std::allocator<ticl::Trackster> > > const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginSimCalorimetryHGCalAssociatorProducersAuto.so
#7  0x00004000e1a20b18 in TSToSimTSHitLCAssociatorByEnergyScoreImpl::associateSimToReco(edm::Handle<std::vector<ticl::Trackster, std::allocator<ticl::Trackster> > > const&, edm::Handle<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > > const&, edm::Handle<std::vector<SimCluster, std::allocator<SimCluster> > > const&, edm::Handle<std::vector<CaloParticle, std::allocator<CaloParticle> > > const&, edm::Handle<std::vector<ticl::Trackster, std::allocator<ticl::Trackster> > > const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginSimCalorimetryHGCalAssociatorProducersAuto.so
#8  0x00004000e1a2046c in TSToSimTSHitLCAssociatorEDProducer::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/pluginSimCalorimetryHGCalAssociatorProducersAuto.so
#9  0x000040000b89e638 in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#10 0x000040000b896540 in edm::WorkerT<edm::global::EDProducerBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#11 0x000040000b82ac14 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libFWCoreFramework.so
#12 0x000040000b82b434 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02821/el9_aarch64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-25-2300/lib/el9_aarch64_gcc12/libFWCoreFramework.so

Current Modules:
Module: CSCDigiValidation:cscDigiValidation (crashed)
Module: ClusterTPAssociationProducer:tpClusterProducer
Module: TSToSimTSHitLCAssociatorEDProducer:tracksterSimTracksterAssociationPRbyCLUE3D
Module: CkfTrackCandidateMaker:initialStepTrackCandidates

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/el9_aarch64_gcc12/CMSSW_14_0_X_2024-01-25-2300/pyRelValMatrixLogs/run/25034.114_TTbar_14TeV+2026D98PU_OTInefficiency10PC/step3_TTbar_14TeV+2026D98PU_OTInefficiency10PC.log#/

makortel commented 10 months ago

assign core

cmsbuild commented 10 months ago

New categories assigned: core

@Dr15Jones,@makortel,@smuzaffar you have been requested to review this Pull request/Issue and eventually sign? Thanks

makortel commented 10 months ago

type root

makortel commented 10 months ago

@pcanal @vgvassilev

makortel commented 10 months ago

Responding to @vgvassilev's https://github.com/cms-sw/cmssw/issues/43577#issuecomment-1889402225

Can we run with valgrind to make sure there are no obvious memory errors?

I ran valgrind for step3 of workflow 11025.0 in CMSSW_14_0_X_2024-01-11-2300. Below is a summary of different kinds of warnings

==25280== Conditional jump or move depends on uninitialised value(s)
==25280==    at 0x46AF40E1: std::_Function_handler<void (long, long), tensorflow::functor::OneHot<Eigen::ThreadPoolDevice, float, int>::Compute(Eigen::ThreadPoolDevice const&, Eigen::TensorMap<Eigen::Tensor<int const, 2, 1, long>, 16, Eigen::MakePointer> const&, Eigen::TensorMap<Eigen::TensorFixedSize<float const, Eigen::Sizes<>, 1, long>, 16, Eigen::MakePointer> const&, Eigen::TensorMap<Eigen::TensorFixedSize<float const, Eigen::Sizes<>, 1, long>, 16, Eigen::MakePointer> const&, Eigen::TensorMap<Eigen::Tensor<float, 3, 1, long>, 16, Eigen::MakePointer>*)::{lambda(long, long)#1}>::_M_invoke(std::_Any_data const&, long&&, long&&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_cc.so.2.12.0)
==25280==    by 0x46AF7033: Eigen::ThreadPoolDevice::parallelFor(long, Eigen::TensorOpCost const&, std::function<long (long)>, std::function<void (long, long)>) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_cc.so.2.12.0)
==25280==    by 0x46B1963E: tensorflow::OneHotOp<Eigen::ThreadPoolDevice, float, int>::Compute(tensorflow::OpKernelContext*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_cc.so.2.12.0)
==25280==    by 0x5657CF6A: tensorflow::ThreadPoolDevice::Compute(tensorflow::OpKernel*, tensorflow::OpKernelContext*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_framework.so.2.12.0)
==25280==    by 0x5666AEF0: tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::ProcessInline(tensorflow::SimplePropagatorState::TaggedNodeReadyQueue*, long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_framework.so.2.12.0)
==25280==    by 0x5666BFD5: tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::Process(tensorflow::SimplePropagatorState::TaggedNode, long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_framework.so.2.12.0)
==25280==    by 0x4FDCC99B: tsl::thread::ThreadPool::Schedule(std::function<void ()>) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_cc.so.2.12.0)
==25280==    by 0x4FF0AD77: std::_Function_handler<void (std::function<void ()>), tensorflow::DirectSession::RunInternal(long, tensorflow::RunOptions const&, tensorflow::CallFrameInterface*, tensorflow::DirectSession::ExecutorsAndKeys*, tensorflow::RunMetadata*, tsl::thread::ThreadPoolOptions const&)::{lambda(std::function<void ()>)#6}>::_M_invoke(std::_Any_data const&, std::function<void ()>&&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_cc.so.2.12.0)
==25280==    by 0x56650923: void tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::RunTask<std::_Bind<void (tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::*(tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>*, tensorflow::SimplePropagatorState::TaggedNode, long))(tensorflow::SimplePropagatorState::TaggedNode, long)> >(std::_Bind<void (tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::*(tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>*, tensorflow::SimplePropagatorState::TaggedNode, long))(tensorflow::SimplePropagatorState::TaggedNode, long)>&&, int) [clone .constprop.0] (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_framework.so.2.12.0)
==25280==    by 0x5665636D: tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::ScheduleReady(absl::lts_20220623::InlinedVector<tensorflow::SimplePropagatorState::TaggedNode, 8ul, std::allocator<tensorflow::SimplePropagatorState::TaggedNode> >*, tensorflow::SimplePropagatorState::TaggedNodeReadyQueue*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_framework.so.2.12.0)
==25280==    by 0x5665FFA4: tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::NodeDone(tsl::Status const&, absl::lts_20220623::InlinedVector<tensorflow::SimplePropagatorState::TaggedNode, 8ul, std::allocator<tensorflow::SimplePropagatorState::TaggedNode> >*, tensorflow::NodeExecStatsInterface*, tensorflow::SimplePropagatorState::TaggedNodeReadyQueue*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_framework.so.2.12.0)
==25280==    by 0x5666A20B: tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::ProcessInline(tensorflow::SimplePropagatorState::TaggedNodeReadyQueue*, long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_framework.so.2.12.0)
==25280==    by 0x5666BFD5: tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::Process(tensorflow::SimplePropagatorState::TaggedNode, long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_framework.so.2.12.0)
==25280==    by 0x4FDCC99B: tsl::thread::ThreadPool::Schedule(std::function<void ()>) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_cc.so.2.12.0)
==25280==    by 0x4FF0AD77: std::_Function_handler<void (std::function<void ()>), tensorflow::DirectSession::RunInternal(long, tensorflow::RunOptions const&, tensorflow::CallFrameInterface*, tensorflow::DirectSession::ExecutorsAndKeys*, tensorflow::RunMetadata*, tsl::thread::ThreadPoolOptions const&)::{lambda(std::function<void ()>)#6}>::_M_invoke(std::_Any_data const&, std::function<void ()>&&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_cc.so.2.12.0)
==25280==    by 0x56656270: tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::ScheduleReady(absl::lts_20220623::InlinedVector<tensorflow::SimplePropagatorState::TaggedNode, 8ul, std::allocator<tensorflow::SimplePropagatorState::TaggedNode> >*, tensorflow::SimplePropagatorState::TaggedNodeReadyQueue*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_framework.so.2.12.0)
==25280==    by 0x5665AF07: tensorflow::(anonymous namespace)::ExecutorImpl::RunAsync(tensorflow::Executor::Args const&, std::function<void (tsl::Status const&)>) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_framework.so.2.12.0)
==25280==    by 0x4FF1D619: tensorflow::DirectSession::RunInternal(long, tensorflow::RunOptions const&, tensorflow::CallFrameInterface*, tensorflow::DirectSession::ExecutorsAndKeys*, tensorflow::RunMetadata*, tsl::thread::ThreadPoolOptions const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_cc.so.2.12.0)
==25280==    by 0x4FF2034D: tensorflow::DirectSession::Run(tensorflow::RunOptions const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor> > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*, tensorflow::RunMetadata*, tsl::thread::ThreadPoolOptions const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_cc.so.2.12.0)
==25280==    by 0x4342D4D4: tensorflow::run(tensorflow::Session*, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor> > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*, tsl::thread::ThreadPoolOptions const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libPhysicsToolsTensorFlow.so)
==25280==  Uninitialised value was created by a heap allocation
==25280==    at 0x4036601: posix_memalign (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/valgrind/3.22.0-e989c977f8949fc48f1bc1206f1f87c8/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==25280==    by 0x4FDF4371: tsl::port::AlignedMalloc(unsigned long, int) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_cc.so.2.12.0)
==25280==    by 0x56413E89: tsl::(anonymous namespace)::CPUAllocator::AllocateRaw(unsigned long, unsigned long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_framework.so.2.12.0)
==25280==    by 0x564BABB4: tensorflow::Tensor::Tensor(tsl::Allocator*, tensorflow::DataType, tensorflow::TensorShape const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/tensorflow/2.12.0-150b7aa8b33fac0cef686f0aee2109fb/lib/libtensorflow_framework.so.2.12.0)
==25280==    by 0x5F0BC1E5: (anonymous namespace)::TfDnn::operator()(std::vector<reco::Track, std::allocator<reco::Track> > const&, reco::BeamSpot const&, std::vector<reco::Vertex, std::allocator<reco::Vertex> > const&) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/pluginRecoTrackerFinalTrackSelectorsPlugins.so)
==25280==    by 0x5F0BE105: TrackMVAClassifier<(anonymous namespace)::TfDnn, void>::computeMVA(std::vector<reco::Track, std::allocator<reco::Track> > const&, reco::BeamSpot const&, std::vector<reco::Vertex, std::allocator<reco::Vertex> > const&, std::vector<std::pair<float, bool>, std::allocator<std::pair<float, bool> > >&) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/pluginRecoTrackerFinalTrackSelectorsPlugins.so)
==25280==    by 0x5F13CAF7: TrackMVAClassifierBase::produce(edm::Event&, edm::EventSetup const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libRecoTrackerFinalTrackSelectors.so)
==25280==    by 0x4C6EBA0: edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4C5347D: edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4BDE3E8: std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4BDE983: edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x420BF27: tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreConcurrency.so)
==25280==    by 0x6401290: UnknownInlinedFun (task_dispatcher.h:322)
==25280==    by 0x6401290: UnknownInlinedFun (task_dispatcher.h:458)
==25280==    by 0x6401290: tbb::detail::r1::task_dispatcher::execute_and_wait(tbb::detail::d1::task*, tbb::detail::d1::wait_context&, tbb::detail::d1::task_group_context&) (task_dispatcher.cpp:168)
==25280==    by 0x4B6178A: edm::FinalWaitingTask::wait() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4B715F9: edm::EventProcessor::processRuns() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4B6B690: edm::EventProcessor::runToCompletion() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4074EE: tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/bin/slc7_amd64_gcc12/cmsRun)
==25280==    by 0x63ED9BC: tbb::detail::r1::task_arena_impl::execute(tbb::detail::d1::task_arena_base&, tbb::detail::d1::delegate_base&) (arena.cpp:688)
==25280==    by 0x408ED1: main::{lambda()#1}::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/bin/slc7_amd64_gcc12/cmsRun)
==25280==    by 0x40518B: main (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/bin/slc7_amd64_gcc12/cmsRun)

Similar report repeats for Use of uninitialised value of size 8

==25280== Conditional jump or move depends on uninitialised value(s)
==25280==    at 0x138E3C3A: TrackerTopology::print[abi:cxx11](DetId) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libDataFormatsTrackerCommon.so)
==25280==    by 0x7D3348D2: MuonAssociatorByHitsHelper::getMatchedIds(std::vector<std::unique_ptr<std::pair<unsigned int, std::vector<std::pair<unsigned int, EncodedEventId>, std::allocator<std::pair<unsigned int, EncodedEventId> > > >, std::default_delete<std::pair<unsigned int, std::vector<std::pair<unsigned int, EncodedEventId>, std::allocator<std::pair<unsigned int, EncodedEventId> > > > > >, std::allocator<std::unique_ptr<std::pair<unsigned int, std::vector<std::pair<unsigned int, EncodedEventId>, std::allocator<std::pair<unsigned int, EncodedEventId> > > >, std::default_delete<std::pair<unsigned int, std::vector<std::pair<unsigned int, EncodedEventId>, std::allocator<std::pair<unsigned int, EncodedEventId> > > > > > > >&, std::vector<std::unique_ptr<std::pair<unsigned int, std::vector<std::pair<unsigned int, EncodedEventId>, std::allocator<std::pair<unsigned int, EncodedEventId> > > >, std::default_delete<std::pair<unsigned int, std::vector<std::pair<unsigned int, EncodedEventId>, std::allocator<std::pair<unsigned int, EncodedEventId> > > > > >, std::allocator<std::unique_ptr<std::pair<unsigned int, std::vector<std::pair<unsigned int, EncodedEventId>, std::allocator<std::pair<unsigned int, EncodedEventId> > > >, std::default_delete<std::pair<unsigned int, std::vector<std::pair<unsigned int, EncodedEventId>, std::allocator<std::pair<unsigned int, EncodedEventId> > > > > > > >&, std::vector<std::unique_ptr<std::pair<unsigned int, std::vector<std::pair<unsigned int, EncodedEventId>, std::allocator<std::pair<unsigned int, EncodedEventId> > > >, std::default_delete<std::pair<unsigned int, std::vector<std::pair<unsigned int, EncodedEventId>, std::allocator<std::pair<unsigned int, EncodedEventId> > > > > >, std::allocator<std::unique_ptr<std::pair<unsigned int, std::vector<std::pair<unsigned int, EncodedEventId>, std::allocator<std::pair<unsigned int, EncodedEventId> > > >, std::default_delete<std::pair<unsigned int, std::vector<std::pair<unsigned int, EncodedEventId>, std::allocator<std::pair<unsigned int, EncodedEventId> > > > > > > >&, std::vector<std::unique_ptr<std::pair<unsigned int, std::vector<std::pair<unsigned int, EncodedEventId>, std::allocator<std::pair<unsigned int, EncodedEventId> > > >, std::default_delete<std::pair<unsigned int, std::vector<std::pair<unsigned int, EncodedEventId>, std::allocator<std::pair<unsigned int, EncodedEventId> > > > > >, std::allocator<std::unique_ptr<std::pair<unsigned int, std::vector<std::pair<unsigned int, EncodedEventId>, std::allocator<std::pair<unsigned int, EncodedEventId> > > >, std::default_delete<std::pair<unsigned int, std::vector<std::pair<unsigned int, EncodedEventId>, std::allocator<std::pair<unsigned int, EncodedEventId> > > > > > > >&, int&, int&, int&, int&, int&, int&, int&, int&, int&, int&, int&, int&, int&, int&, int&, int&, int&, int&, int&, int&, __gnu_cxx::__normal_iterator<TrackingRecHit* const*, std::vector<TrackingRecHit*, std::allocator<TrackingRecHit*> > >, __gnu_cxx::__normal_iterator<TrackingRecHit* const*, std::vector<TrackingRecHit*, std::allocator<TrackingRecHit*> > >, TrackerHitAssociator const*, DTHitAssociator const&, CSCHitAssociator const&, RPCHitAssociator const&, GEMHitAssociator const&, bool, TrackerTopology const*) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libSimMuonMCTruth.so)
==25280==    by 0x7D33A4A7: MuonAssociatorByHitsHelper::associateRecoToSimIndices(std::vector<std::pair<__gnu_cxx::__normal_iterator<TrackingRecHit* const*, std::vector<TrackingRecHit*, std::allocator<TrackingRecHit*> > >, __gnu_cxx::__normal_iterator<TrackingRecHit* const*, std::vector<TrackingRecHit*, std::allocator<TrackingRecHit*> > > >, std::allocator<std::pair<__gnu_cxx::__normal_iterator<TrackingRecHit* const*, std::vector<TrackingRecHit*, std::allocator<TrackingRecHit*> > >, __gnu_cxx::__normal_iterator<TrackingRecHit* const*, std::vector<TrackingRecHit*, std::allocator<TrackingRecHit*> > > > > > const&, edm::RefVector<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle, edm::refhelper::FindUsingAdvance<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle> > const&, MuonAssociatorByHitsHelper::Resources const&) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libSimMuonMCTruth.so)
==25280==    by 0x7D32FB9B: MuonAssociatorByHits::associateRecoToSim(edm::RefToBaseVector<reco::Track> const&, edm::RefVector<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle, edm::refhelper::FindUsingAdvance<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle> > const&, edm::Event const*, edm::EventSetup const*) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libSimMuonMCTruth.so)
==25280==    by 0x7D4874C8: MuonAssociatorEDProducer::produce(edm::Event&, edm::EventSetup const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/pluginSimMuonMCTruthPlugins.so)
==25280==    by 0x4C6EBA0: edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4C5347D: edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4BDE3E8: std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4BDE983: edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x420BF27: tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreConcurrency.so)
==25280==    by 0x6401290: UnknownInlinedFun (task_dispatcher.h:322)
==25280==    by 0x6401290: UnknownInlinedFun (task_dispatcher.h:458)
==25280==    by 0x6401290: tbb::detail::r1::task_dispatcher::execute_and_wait(tbb::detail::d1::task*, tbb::detail::d1::wait_context&, tbb::detail::d1::task_group_context&) (task_dispatcher.cpp:168)
==25280==    by 0x4B6178A: edm::FinalWaitingTask::wait() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4B715F9: edm::EventProcessor::processRuns() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4B6B690: edm::EventProcessor::runToCompletion() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4074EE: tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/bin/slc7_amd64_gcc12/cmsRun)
==25280==    by 0x63ED9BC: tbb::detail::r1::task_arena_impl::execute(tbb::detail::d1::task_arena_base&, tbb::detail::d1::delegate_base&) (arena.cpp:688)
==25280==    by 0x408ED1: main::{lambda()#1}::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/bin/slc7_amd64_gcc12/cmsRun)
==25280==    by 0x40518B: main (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/bin/slc7_amd64_gcc12/cmsRun)
==25280==  Uninitialised value was created by a stack allocation
==25280==    at 0x5F1DAC06: TrackerTopologyEP::produce(TrackerTopologyRcd const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/pluginGeometryTrackerNumberingBuilderPlugins.so)
==25280== Conditional jump or move depends on uninitialised value(s)
==25280==    at 0x643878B: ??? (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/xz/5.2.5-6f3f49b07db84e10c9be594a1176c114/lib/liblzma.so.5.2.5)
==25280==    by 0x6437922: ??? (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/xz/5.2.5-6f3f49b07db84e10c9be594a1176c114/lib/liblzma.so.5.2.5)
==25280==    by 0x643BE40: ??? (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/xz/5.2.5-6f3f49b07db84e10c9be594a1176c114/lib/liblzma.so.5.2.5)
==25280==    by 0x64396D4: ??? (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/xz/5.2.5-6f3f49b07db84e10c9be594a1176c114/lib/liblzma.so.5.2.5)
==25280==    by 0x6441B80: ??? (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/xz/5.2.5-6f3f49b07db84e10c9be594a1176c114/lib/liblzma.so.5.2.5)
==25280==    by 0x6436963: ??? (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/xz/5.2.5-6f3f49b07db84e10c9be594a1176c114/lib/liblzma.so.5.2.5)
==25280==    by 0x642E14B: ??? (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/xz/5.2.5-6f3f49b07db84e10c9be594a1176c114/lib/liblzma.so.5.2.5)
==25280==    by 0x642FB46: ??? (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/xz/5.2.5-6f3f49b07db84e10c9be594a1176c114/lib/liblzma.so.5.2.5)
==25280==    by 0x642AB55: lzma_code (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/xz/5.2.5-6f3f49b07db84e10c9be594a1176c114/lib/liblzma.so.5.2.5)
==25280==    by 0x5BA1763: R__zipLZMA (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libCore.so)
==25280==    by 0x4F795FC: TBasket::WriteBuffer() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4F89237: std::_Function_handler<void (), ROOT::Internal::TBranchIMTHelper::Run<TBranch::WriteBasketImpl(TBasket*, int, ROOT::Internal::TBranchIMTHelper*)::{lambda()#1}>(TBranch::WriteBasketImpl(TBasket*, int, ROOT::Internal::TBranchIMTHelper*)::{lambda()#1} const&)::{lambda()#1}>::_M_invoke(std::_Any_data const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x763660D: tbb::detail::d1::function_task<std::function<void ()> >::execute(tbb::detail::d1::execution_data&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libImt.so)
==25280==    by 0x6401290: UnknownInlinedFun (task_dispatcher.h:322)
==25280==    by 0x6401290: UnknownInlinedFun (task_dispatcher.h:458)
==25280==    by 0x6401290: tbb::detail::r1::task_dispatcher::execute_and_wait(tbb::detail::d1::task*, tbb::detail::d1::wait_context&, tbb::detail::d1::task_group_context&) (task_dispatcher.cpp:168)
==25280==    by 0x7636398: ROOT::Experimental::TTaskGroup::Wait() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libImt.so)
==25280==    by 0x5002D35: TTree::Fill() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x8D5712C7: tbb::detail::d1::task_arena_function<edm::RootOutputTree::fillTree()::{lambda()#1}, void>::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libIOPoolOutput.so)
==25280==    by 0x63EE29D: UnknownInlinedFun (arena.cpp:757)
==25280==    by 0x63EE29D: UnknownInlinedFun (_template_helpers.h:230)
==25280==    by 0x63EE29D: tbb::detail::r1::isolate_within_arena(tbb::detail::d1::delegate_base&, long) (arena.cpp:758)
==25280==    by 0x8D571DFF: edm::RootOutputTree::fillTree() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libIOPoolOutput.so)
==25280==    by 0x8D56FC58: edm::RootOutputFile::fillBranches(edm::BranchType const&, edm::OccurrenceForOutput const&, unsigned int, std::vector<edm::StoredProductProvenance, std::allocator<edm::StoredProductProvenance> >*, edm::ProductProvenanceRetriever const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libIOPoolOutput.so)
==25280==  Uninitialised value was created by a stack allocation
==25280==    at 0x988789B0: pat::PATElectronProducer::produce(edm::Event&, edm::EventSetup const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/pluginPhysicsToolsPatAlgos_plugins.so)

Repeats for Use of uninitialised value of size 8

==25280== Conditional jump or move depends on uninitialised value(s)
==25280==    at 0x65C74B52: .LTHUNK17.lto_priv.0 (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/pluginDPGAnalysisSkims.so)
==25280==    by 0x65C751A1: virtual thunk to edm::global::impl::LuminosityBlockCacheHolder<edm::global::EDFilterBase, leef::LumiErrors>::doEndLuminosityBlock_(edm::LuminosityBlock const&, edm::EventSetup const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/pluginDPGAnalysisSkims.so)
==25280==    by 0x4C55DBA: edm::global::EDFilterBase::doEndLuminosityBlock(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4C4D46F: edm::WorkerT<edm::global::EDFilterBase>::implDoEnd(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4B9F12E: std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3>::Context const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4B8D9DA: edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3> >::execute() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4B5188D: tbb::detail::d1::function_task<edm::WaitingTaskHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x6401290: UnknownInlinedFun (task_dispatcher.h:322)
==25280==    by 0x6401290: UnknownInlinedFun (task_dispatcher.h:458)
==25280==    by 0x6401290: tbb::detail::r1::task_dispatcher::execute_and_wait(tbb::detail::d1::task*, tbb::detail::d1::wait_context&, tbb::detail::d1::task_group_context&) (task_dispatcher.cpp:168)
==25280==    by 0x4B6178A: edm::FinalWaitingTask::wait() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4B715F9: edm::EventProcessor::processRuns() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4B6B690: edm::EventProcessor::runToCompletion() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4074EE: tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/bin/slc7_amd64_gcc12/cmsRun)
==25280==    by 0x63ED9BC: tbb::detail::r1::task_arena_impl::execute(tbb::detail::d1::task_arena_base&, tbb::detail::d1::delegate_base&) (arena.cpp:688)
==25280==    by 0x408ED1: main::{lambda()#1}::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/bin/slc7_amd64_gcc12/cmsRun)
==25280==    by 0x40518B: main (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/bin/slc7_amd64_gcc12/cmsRun)
==25280==  Uninitialised value was created by a heap allocation
==25280==    at 0x402EEE1: operator new(unsigned long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/valgrind/3.22.0-e989c977f8949fc48f1bc1206f1f87c8/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==25280==    by 0x65C79BD0: edm::WorkerMaker<LogErrorEventFilter>::makeModule(edm::ParameterSet const&) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/pluginDPGAnalysisSkims.so)
==25280==    by 0x4C49B67: edm::Maker::makeModule(edm::MakeModuleParams const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4BC08C7: edm::Factory::makeModule(edm::MakeModuleParams const&, edm::ModuleTypeResolverMaker const*, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4BCF58F: edm::ModuleRegistry::getModule(edm::MakeModuleParams const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4C4A33F: edm::WorkerRegistry::getWorker(edm::WorkerParams const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4C4A701: edm::WorkerManager::getWorker(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4C257FD: edm::(anonymous namespace)::getWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, edm::ParameterSet&, edm::WorkerManager&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>) [clone .lto_priv.0] (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4C2FA11: edm::StreamSchedule::fillWorkers(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, std::vector<edm::WorkerInPath, std::allocator<edm::WorkerInPath> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, edm::ConditionalTaskHelper const&, std::unordered_set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4C322A3: edm::StreamSchedule::fillTrigPath(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<edm::HLTGlobalStatus>, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, edm::ConditionalTaskHelper const&, std::unordered_set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4C285A6: edm::StreamSchedule::StreamSchedule(std::shared_ptr<edm::TriggerResultInserter>, std::vector<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> >, std::allocator<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> > > >&, std::vector<edm::propagate_const<std::shared_ptr<edm::EndPathStatusInserter> >, std::allocator<edm::propagate_const<std::shared_ptr<edm::EndPathStatusInserter> > > >&, std::shared_ptr<edm::ModuleRegistry>, edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::PreallocationConfiguration const&, edm::ProductRegistry&, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration const>, edm::StreamID, edm::ProcessContext const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4C0EAAF: edm::Schedule::Schedule(edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::ProductRegistry&, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration const>, edm::PreallocationConfiguration const&, edm::ProcessContext const*, edm::ModuleTypeResolverMaker const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4C22D2C: edm::ScheduleItems::initModules(edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::PreallocationConfiguration const&, edm::ProcessContext const*, edm::ModuleTypeResolverMaker const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4BA0B95: tbb::detail::d1::function_task<edm::EventProcessor::init(std::shared_ptr<edm::ProcessDesc>&, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) [clone .lto_priv.0] (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x6401290: UnknownInlinedFun (task_dispatcher.h:322)
==25280==    by 0x6401290: UnknownInlinedFun (task_dispatcher.h:458)
==25280==    by 0x6401290: tbb::detail::r1::task_dispatcher::execute_and_wait(tbb::detail::d1::task*, tbb::detail::d1::wait_context&, tbb::detail::d1::task_group_context&) (task_dispatcher.cpp:168)
==25280==    by 0x4B682B1: edm::EventProcessor::init(std::shared_ptr<edm::ProcessDesc>&, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4B6AB8B: edm::EventProcessor::EventProcessor(std::shared_ptr<edm::ProcessDesc>, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4073AE: tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/bin/slc7_amd64_gcc12/cmsRun)
==25280==    by 0x63ED9BC: tbb::detail::r1::task_arena_impl::execute(tbb::detail::d1::task_arena_base&, tbb::detail::d1::delegate_base&) (arena.cpp:688)
==25280==    by 0x408ED1: main::{lambda()#1}::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/bin/slc7_amd64_gcc12/cmsRun)
makortel commented 10 months ago

More from the same valgrind run

==25280== Conditional jump or move depends on uninitialised value(s)
==25280==    at 0x7D9DDCE: ZSTD_HcFindBestMatch_noDict_4 (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/zstd/1.5.2-0530c990dd9ca27a8d966ab201ed2dfe/lib/libzstd.so.1.5.2)
==25280==    by 0x7DBAE18: ZSTD_compressBlock_lazy2 (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/zstd/1.5.2-0530c990dd9ca27a8d966ab201ed2dfe/lib/libzstd.so.1.5.2)
==25280==    by 0x7D754A2: ZSTD_compressBlock_internal (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/zstd/1.5.2-0530c990dd9ca27a8d966ab201ed2dfe/lib/libzstd.so.1.5.2)
==25280==    by 0x7D7705E: ZSTD_compress_frameChunk (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/zstd/1.5.2-0530c990dd9ca27a8d966ab201ed2dfe/lib/libzstd.so.1.5.2)
==25280==    by 0x7D79F65: ZSTD_compressEnd (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/zstd/1.5.2-0530c990dd9ca27a8d966ab201ed2dfe/lib/libzstd.so.1.5.2)
==25280==    by 0x7D7DE3F: ZSTD_compressCCtx (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/zstd/1.5.2-0530c990dd9ca27a8d966ab201ed2dfe/lib/libzstd.so.1.5.2)
==25280==    by 0x5BA1C14: R__zipZSTD (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libCore.so)
==25280==    by 0x4F795FC: TBasket::WriteBuffer() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4F8800E: TBranch::WriteBasketImpl(TBasket*, int, ROOT::Internal::TBranchIMTHelper*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4F88C15: TBranch::FlushBaskets() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4F88C5B: TBranch::FlushBaskets() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4FF76F1: TTree::FlushBasketsImpl() const::{lambda()#1}::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x7639D53: tbb::detail::d1::start_for<tbb::detail::d1::blocked_range<unsigned int>, tbb::detail::d1::parallel_for_body_wrapper<std::function<void (unsigned int)>, unsigned int>, tbb::detail::d1::auto_partitioner const>::execute(tbb::detail::d1::execution_data&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libImt.so)
==25280==    by 0x6401290: UnknownInlinedFun (task_dispatcher.h:322)
==25280==    by 0x6401290: UnknownInlinedFun (task_dispatcher.h:458)
==25280==    by 0x6401290: tbb::detail::r1::task_dispatcher::execute_and_wait(tbb::detail::d1::task*, tbb::detail::d1::wait_context&, tbb::detail::d1::task_group_context&) (task_dispatcher.cpp:168)
==25280==    by 0x7638BB2: tbb::detail::d1::task_arena_function<ROOT::TThreadExecutor::ParallelFor(unsigned int, unsigned int, unsigned int, std::function<void (unsigned int)> const&)::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libImt.so)
==25280==    by 0x63EE29D: UnknownInlinedFun (arena.cpp:757)
==25280==    by 0x63EE29D: UnknownInlinedFun (_template_helpers.h:230)
==25280==    by 0x63EE29D: tbb::detail::r1::isolate_within_arena(tbb::detail::d1::delegate_base&, long) (arena.cpp:758)
==25280==    by 0x7637492: tbb::detail::d1::task_arena_function<ROOT::TThreadExecutor::ParallelFor(unsigned int, unsigned int, unsigned int, std::function<void (unsigned int)> const&)::{lambda()#1}, void>::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libImt.so)
==25280==    by 0x63ED9BC: tbb::detail::r1::task_arena_impl::execute(tbb::detail::d1::task_arena_base&, tbb::detail::d1::delegate_base&) (arena.cpp:688)
==25280==    by 0x7639198: ROOT::TThreadExecutor::ParallelFor(unsigned int, unsigned int, unsigned int, std::function<void (unsigned int)> const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libImt.so)
==25280==    by 0x4FF8A8E: TTree::FlushBasketsImpl() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==  Uninitialised value was created by a stack allocation
==25280==    at 0x7D48F8B3: MuonSimClassifier::convertAndPush(TrackingParticle const&, std::vector<reco::GenParticle, std::allocator<reco::GenParticle> >&, edm::Ref<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle, edm::refhelper::FindUsingAdvance<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle> > const&, edm::Handle<std::vector<reco::GenParticle, std::allocator<reco::GenParticle> > > const&) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/pluginSimMuonMCTruthPlugins.so)
==25280== Conditional jump or move depends on uninitialised value(s)
==25280==    at 0x554C093: TFree::GetBestFree(TList*, int) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libRIO.so)
==25280==    by 0x558B4E3: TKey::Create(int, TFile*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libRIO.so)
==25280==    by 0x4F7997D: TBasket::WriteBuffer() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4F8800E: TBranch::WriteBasketImpl(TBasket*, int, ROOT::Internal::TBranchIMTHelper*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4F88C15: TBranch::FlushBaskets() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4F88C5B: TBranch::FlushBaskets() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4FF76F1: TTree::FlushBasketsImpl() const::{lambda()#1}::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x7639D53: tbb::detail::d1::start_for<tbb::detail::d1::blocked_range<unsigned int>, tbb::detail::d1::parallel_for_body_wrapper<std::function<void (unsigned int)>, unsigned int>, tbb::detail::d1::auto_partitioner const>::execute(tbb::detail::d1::execution_data&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libImt.so)
==25280==    by 0x6401290: UnknownInlinedFun (task_dispatcher.h:322)
==25280==    by 0x6401290: UnknownInlinedFun (task_dispatcher.h:458)
==25280==    by 0x6401290: tbb::detail::r1::task_dispatcher::execute_and_wait(tbb::detail::d1::task*, tbb::detail::d1::wait_context&, tbb::detail::d1::task_group_context&) (task_dispatcher.cpp:168)
==25280==    by 0x7638BB2: tbb::detail::d1::task_arena_function<ROOT::TThreadExecutor::ParallelFor(unsigned int, unsigned int, unsigned int, std::function<void (unsigned int)> const&)::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libImt.so)
==25280==    by 0x63EE29D: UnknownInlinedFun (arena.cpp:757)
==25280==    by 0x63EE29D: UnknownInlinedFun (_template_helpers.h:230)
==25280==    by 0x63EE29D: tbb::detail::r1::isolate_within_arena(tbb::detail::d1::delegate_base&, long) (arena.cpp:758)
==25280==    by 0x7637492: tbb::detail::d1::task_arena_function<ROOT::TThreadExecutor::ParallelFor(unsigned int, unsigned int, unsigned int, std::function<void (unsigned int)> const&)::{lambda()#1}, void>::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libImt.so)
==25280==    by 0x63ED9BC: tbb::detail::r1::task_arena_impl::execute(tbb::detail::d1::task_arena_base&, tbb::detail::d1::delegate_base&) (arena.cpp:688)
==25280==    by 0x7639198: ROOT::TThreadExecutor::ParallelFor(unsigned int, unsigned int, unsigned int, std::function<void (unsigned int)> const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libImt.so)
==25280==    by 0x4FF8A8E: TTree::FlushBasketsImpl() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4FF98EA: TTree::AutoSave(char const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x8D56F2C8: edm::RootOutputFile::finishEndFile() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libIOPoolOutput.so)
==25280==    by 0x8D55AA64: edm::PoolOutputModule::finishEndFile() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libIOPoolOutput.so)
==25280==    by 0x8D55ABB2: edm::PoolOutputModule::reallyCloseFile() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libIOPoolOutput.so)
==25280==    by 0x4C105C4: edm::Schedule::closeOutputFiles() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==  Uninitialised value was created by a heap allocation
==25280==    at 0x402EEE1: operator new(unsigned long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/valgrind/3.22.0-e989c977f8949fc48f1bc1206f1f87c8/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==25280==    by 0x4F88113: TBranch::WriteBasketImpl(TBasket*, int, ROOT::Internal::TBranchIMTHelper*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4F889C9: TBranch::FillImpl(ROOT::Internal::TBranchIMTHelper*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4F92C5C: TBranchElement::FillImpl(ROOT::Internal::TBranchIMTHelper*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x50030D6: TTree::Fill() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x8D569F70: edm::RootOutputFile::writeParentageRegistry() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libIOPoolOutput.so)
==25280==    by 0x8D55AB7E: edm::PoolOutputModule::reallyCloseFile() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libIOPoolOutput.so)
==25280==    by 0x4C105C4: edm::Schedule::closeOutputFiles() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4B6658C: edm::EventProcessor::closeOutputFiles() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4B6B779: edm::EventProcessor::runToCompletion() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4074EE: tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/bin/slc7_amd64_gcc12/cmsRun)
==25280==    by 0x63ED9BC: tbb::detail::r1::task_arena_impl::execute(tbb::detail::d1::task_arena_base&, tbb::detail::d1::delegate_base&) (arena.cpp:688)
==25280==    by 0x408ED1: main::{lambda()#1}::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/bin/slc7_amd64_gcc12/cmsRun)
==25280==    by 0x40518B: main (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/bin/slc7_amd64_gcc12/cmsRun)
==25280== Syscall param lseek(offset) contains uninitialised byte(s)
==25280==    at 0x704FDDD: __lseek64 (in /usr/lib64/libpthread-2.17.so)
==25280==    by 0xE9A456E: edm::storage::File::position(long, edm::storage::Storage::Relative) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libUtilitiesStorageFactory.so)
==25280==    by 0xE9A30F1: edm::storage::StorageAccountProxy::position(long, edm::storage::Storage::Relative) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libUtilitiesStorageFactory.so)
==25280==    by 0xFCC7332: TStorageFactoryFile::SysSeek(int, long long, int) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libIOPoolTFileAdaptor.so)
==25280==    by 0x554F841: TFile::Seek(long long, TFile::ERelativeTo) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libRIO.so)
==25280==    by 0x558B262: TKey::WriteFileKeepBuffer(TFile*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libRIO.so)
==25280==    by 0x4F7949F: TBasket::WriteBuffer() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4F8800E: TBranch::WriteBasketImpl(TBasket*, int, ROOT::Internal::TBranchIMTHelper*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4F88C15: TBranch::FlushBaskets() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4F88C5B: TBranch::FlushBaskets() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4FF76F1: TTree::FlushBasketsImpl() const::{lambda()#1}::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x7639D53: tbb::detail::d1::start_for<tbb::detail::d1::blocked_range<unsigned int>, tbb::detail::d1::parallel_for_body_wrapper<std::function<void (unsigned int)>, unsigned int>, tbb::detail::d1::auto_partitioner const>::execute(tbb::detail::d1::execution_data&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libImt.so)
==25280==    by 0x6401290: UnknownInlinedFun (task_dispatcher.h:322)
==25280==    by 0x6401290: UnknownInlinedFun (task_dispatcher.h:458)
==25280==    by 0x6401290: tbb::detail::r1::task_dispatcher::execute_and_wait(tbb::detail::d1::task*, tbb::detail::d1::wait_context&, tbb::detail::d1::task_group_context&) (task_dispatcher.cpp:168)
==25280==    by 0x7638BB2: tbb::detail::d1::task_arena_function<ROOT::TThreadExecutor::ParallelFor(unsigned int, unsigned int, unsigned int, std::function<void (unsigned int)> const&)::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libImt.so)
==25280==    by 0x63EE29D: UnknownInlinedFun (arena.cpp:757)
==25280==    by 0x63EE29D: UnknownInlinedFun (_template_helpers.h:230)
==25280==    by 0x63EE29D: tbb::detail::r1::isolate_within_arena(tbb::detail::d1::delegate_base&, long) (arena.cpp:758)
==25280==    by 0x7637492: tbb::detail::d1::task_arena_function<ROOT::TThreadExecutor::ParallelFor(unsigned int, unsigned int, unsigned int, std::function<void (unsigned int)> const&)::{lambda()#1}, void>::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libImt.so)
==25280==    by 0x63ED9BC: tbb::detail::r1::task_arena_impl::execute(tbb::detail::d1::task_arena_base&, tbb::detail::d1::delegate_base&) (arena.cpp:688)
==25280==    by 0x7639198: ROOT::TThreadExecutor::ParallelFor(unsigned int, unsigned int, unsigned int, std::function<void (unsigned int)> const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libImt.so)
==25280==    by 0x4FF8A8E: TTree::FlushBasketsImpl() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4FF98EA: TTree::AutoSave(char const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==  Uninitialised value was created by a heap allocation
==25280==    at 0x402EEE1: operator new(unsigned long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/valgrind/3.22.0-e989c977f8949fc48f1bc1206f1f87c8/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==25280==    by 0x4F88113: TBranch::WriteBasketImpl(TBasket*, int, ROOT::Internal::TBranchIMTHelper*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4F889C9: TBranch::FillImpl(ROOT::Internal::TBranchIMTHelper*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4F92C5C: TBranchElement::FillImpl(ROOT::Internal::TBranchIMTHelper*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x50030D6: TTree::Fill() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x8D569F70: edm::RootOutputFile::writeParentageRegistry() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libIOPoolOutput.so)
==25280==    by 0x8D55AB7E: edm::PoolOutputModule::reallyCloseFile() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libIOPoolOutput.so)
==25280==    by 0x4C105C4: edm::Schedule::closeOutputFiles() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4B6658C: edm::EventProcessor::closeOutputFiles() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4B6B779: edm::EventProcessor::runToCompletion() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==25280==    by 0x4074EE: tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/bin/slc7_amd64_gcc12/cmsRun)
==25280==    by 0x63ED9BC: tbb::detail::r1::task_arena_impl::execute(tbb::detail::d1::task_arena_base&, tbb::detail::d1::delegate_base&) (arena.cpp:688)
==25280==    by 0x408ED1: main::{lambda()#1}::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/bin/slc7_amd64_gcc12/cmsRun)
==25280==    by 0x40518B: main (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/bin/slc7_amd64_gcc12/cmsRun)
==25280== Conditional jump or move depends on uninitialised value(s)
==25280==    at 0x7D9DD06: ZSTD_HcFindBestMatch_noDict_4 (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/zstd/1.5.2-0530c990dd9ca27a8d966ab201ed2dfe/lib/libzstd.so.1.5.2)
==25280==    by 0x7DBAB16: ZSTD_compressBlock_lazy2 (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/zstd/1.5.2-0530c990dd9ca27a8d966ab201ed2dfe/lib/libzstd.so.1.5.2)
==25280==    by 0x7D754A2: ZSTD_compressBlock_internal (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/zstd/1.5.2-0530c990dd9ca27a8d966ab201ed2dfe/lib/libzstd.so.1.5.2)
==25280==    by 0x7D7705E: ZSTD_compress_frameChunk (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/zstd/1.5.2-0530c990dd9ca27a8d966ab201ed2dfe/lib/libzstd.so.1.5.2)
==25280==    by 0x7D79F65: ZSTD_compressEnd (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/zstd/1.5.2-0530c990dd9ca27a8d966ab201ed2dfe/lib/libzstd.so.1.5.2)
==25280==    by 0x7D7DE3F: ZSTD_compressCCtx (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/zstd/1.5.2-0530c990dd9ca27a8d966ab201ed2dfe/lib/libzstd.so.1.5.2)
==25280==    by 0x5BA1C14: R__zipZSTD (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libCore.so)
==25280==    by 0x4F795FC: TBasket::WriteBuffer() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4F8800E: TBranch::WriteBasketImpl(TBasket*, int, ROOT::Internal::TBranchIMTHelper*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4F88C15: TBranch::FlushBaskets() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4F88C5B: TBranch::FlushBaskets() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x4FF76F1: TTree::FlushBasketsImpl() const::{lambda()#1}::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==    by 0x7639D53: tbb::detail::d1::start_for<tbb::detail::d1::blocked_range<unsigned int>, tbb::detail::d1::parallel_for_body_wrapper<std::function<void (unsigned int)>, unsigned int>, tbb::detail::d1::auto_partitioner const>::execute(tbb::detail::d1::execution_data&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libImt.so)
==25280==    by 0x6401290: UnknownInlinedFun (task_dispatcher.h:322)
==25280==    by 0x6401290: UnknownInlinedFun (task_dispatcher.h:458)
==25280==    by 0x6401290: tbb::detail::r1::task_dispatcher::execute_and_wait(tbb::detail::d1::task*, tbb::detail::d1::wait_context&, tbb::detail::d1::task_group_context&) (task_dispatcher.cpp:168)
==25280==    by 0x7638BB2: tbb::detail::d1::task_arena_function<ROOT::TThreadExecutor::ParallelFor(unsigned int, unsigned int, unsigned int, std::function<void (unsigned int)> const&)::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libImt.so)
==25280==    by 0x63EE29D: UnknownInlinedFun (arena.cpp:757)
==25280==    by 0x63EE29D: UnknownInlinedFun (_template_helpers.h:230)
==25280==    by 0x63EE29D: tbb::detail::r1::isolate_within_arena(tbb::detail::d1::delegate_base&, long) (arena.cpp:758)
==25280==    by 0x7637492: tbb::detail::d1::task_arena_function<ROOT::TThreadExecutor::ParallelFor(unsigned int, unsigned int, unsigned int, std::function<void (unsigned int)> const&)::{lambda()#1}, void>::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libImt.so)
==25280==    by 0x63ED9BC: tbb::detail::r1::task_arena_impl::execute(tbb::detail::d1::task_arena_base&, tbb::detail::d1::delegate_base&) (arena.cpp:688)
==25280==    by 0x7639198: ROOT::TThreadExecutor::ParallelFor(unsigned int, unsigned int, unsigned int, std::function<void (unsigned int)> const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libImt.so)
==25280==    by 0x4FF8A8E: TTree::FlushBasketsImpl() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libTree.so)
==25280==  Uninitialised value was created by a stack allocation
==25280==    at 0xADA39931: TcdsRawToDigi::produce(edm::Event&, edm::EventSetup const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/pluginEventFilterOnlineMetaDataRawToDigiAuto.so)
makortel commented 10 months ago

So I didn't see anything obvious related to Cling, but we might want to follow up

pcanal commented 10 months ago

The following errors usually indicates uninitialised memory in the user data.

==25280== Conditional jump or move depends on uninitialised value(s)
==25280==    at 0x7D9DDCE: ZSTD_HcFindBestMatch_noDict_4 (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/zstd/1.5.2-0530c990dd9ca27a8d966ab201ed2dfe/lib/libzstd.so.1.5.2)
==25280==    by 0x7DBAE18: ZSTD_compressBlock_lazy2 (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/zstd/1.5.2-0530c990dd9ca27a8d966ab201ed2dfe/lib/libzstd.so.1.5.2)
==25280==    by 0x7D754A2: ZSTD_compressBlock_internal (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/zstd/1.5.2-0530c990dd9ca27a8d966ab201ed2dfe/lib/libzstd.so.1.5.2)
==25280==    by 0x7D7705E: ZSTD_compress_frameChunk (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/zstd/1.5.2-0530c990dd9ca27a8d966ab201ed2dfe/lib/libzstd.so.1.5.2)
==25280==    by 0x7D79F65: ZSTD_compressEnd (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/zstd/1.5.2-0530c990dd9ca27a8d966ab201ed2dfe/lib/libzstd.so.1.5.2)
==25280==    by 0x7D7DE3F: ZSTD_compressCCtx (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/external/zstd/1.5.2-0530c990dd9ca27a8d966ab201ed2dfe/lib/libzstd.so.1.5.2)
==25280==    by 0x5BA1C14: R__zipZSTD (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libCore.so)

and, at least in the first case, valgrind points to:

==25280==  Uninitialised value was created by a stack allocation
==25280==    at 0x7D48F8B3: MuonSimClassifier::convertAndPush(TrackingParticle const&, std::vector<reco::GenParticle, std::allocator<reco::GenParticle> >&, edm::Ref<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle, edm::refhelper::FindUsingAdvance<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle> > const&, edm::Handle<std::vector<reco::GenParticle, std::allocator<reco::GenParticle> > > const&) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/pluginSimMuonMCTruthPlugins.so)
pcanal commented 10 months ago 
==25280== Conditional jump or move depends on uninitialised value(s)
==25280==    at 0x554C093: TFree::GetBestFree(TList*, int) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libRIO.so)
==25280==    by 0x558B4E3: TKey::Create(int, TFile*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libRIO.so)

is a bit worrisome and might be an issue in ROOT (or valgrind)

makortel commented 10 months ago

The following errors usually indicates uninitialised memory in the user data. ... and, at least in the first case, valgrind points to:

Thanks for confirming, this was my interpretation as well.

pcanal commented 10 months ago

The following is the most unexpected of all and may indicate either that valgrind is not ready for the platform or that there is a serious memory corruption somewhere. Since the symbol mentions LTO, did you also try without LTO turned on.

==25280== Conditional jump or move depends on uninitialised value(s)
==25280==    at 0x65C74B52: .LTHUNK17.lto_priv.0 (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/pluginDPGAnalysisSkims.so)
==25280==    by 0x65C751A1: virtual thunk to edm::global::impl::LuminosityBlockCacheHolder<edm::global::EDFilterBase, leef::LumiErrors>::doEndLuminosityBlock_(edm::LuminosityBlock const&, edm::EventSetup const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-11-1100/lib/slc7_amd64_gcc12/pluginDPGAnalysisSkims.so)
makortel commented 10 months ago 
==25280== Conditional jump or move depends on uninitialised value(s)
==25280==    at 0x554C093: TFree::GetBestFree(TList*, int) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libRIO.so)
==25280==    by 0x558B4E3: TKey::Create(int, TFile*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02819/slc7_amd64_gcc12/lcg/root/6.30.03-f674aef727eff41f98138d889e8a964c/lib/libRIO.so)

is a bit worrisome and might be an issue in ROOT (or valgrind)

Ok, there were more of those. I attached a file that should contain all of them (in case that would be helpful). valgrind_root.txt

makortel commented 10 months ago

Since the symbol mentions LTO, did you also try without LTO turned on.

I did not (and at this stage I'd have to move to newer IB). @smuzaffar Is it possible to turn off LTO for select package(s) in a local build, or would that require using the NONLTO build?

smuzaffar commented 10 months ago

@makortel , currently you can not disable LTO for individual packages. But what you can do is to

this should allow you to build the checked out packages without LTO

aandvalenzuela commented 9 months ago

Hello,

I am not 100% sure it is fully related, but workflow 25034.114 is consistently failing for el8_aarch64_gcc12 in CSCDigiValidation:cscDigiValidation as reported above but with a different stacktrace:

Thread 6 (Thread 0x4000a6b89250 (LWP 92865) "cmsRun"):
#0  0x00004000372d41e8 in epoll_pwait () from /lib64/libc.so.6
#1  0x000040003bedbdec in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x000040003bed7efc in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x000040003bee10e0 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#4  0x00004000371c78b8 in start_thread () from /lib64/libpthread.so.0
#5  0x0000400037223afc in thread_start () from /lib64/libc.so.6
Thread 5 (Thread 0x40007df09250 (LWP 92716) "cmsRun"):
#0  0x00004000372a4b3c in nanosleep () from /lib64/libc.so.6
#1  0x00004000372a49e0 in sleep () from /lib64/libc.so.6
#2  0x000040003c3a927c in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000040010553cc70 in void (anonymous namespace)::getSimTrackId<PixelDigiSimLink>(std::vector<std::pair<unsigned int, EncodedEventId>, std::allocator<std::pair<unsigned int, EncodedEventId> > >&, edm::Handle<edm::DetSetVector<PixelDigiSimLink> > const&, DetId const&, unsigned int) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/pluginSimTrackerTrackerHitAssociationPlugins.so
#5  0x000040010553fb30 in ClusterTPAssociationProducer::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/pluginSimTrackerTrackerHitAssociationPlugins.so
#6  0x00004000351c64e8 in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#7  0x00004000351b8d00 in edm::WorkerT<edm::global::EDProducerBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#8  0x000040003514c2c4 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#9  0x000040003514cae4 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#10 0x00004000356e50dc in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreConcurrency.so
#11 0x0000400036c32dfc in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x40016abcb900, waiter=<synthetic pointer>..., this=0x400037b4b480) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#12 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=<synthetic pointer>..., t=0x0, this=0x400037b4b480) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#13 tbb::detail::r1::arena::process (tls=..., this=0x400037b4ad00) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/arena.cpp:137
#14 tbb::detail::r1::market::process (this=0x400037b4b580, j=...) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/market.cpp:599
#15 0x0000400036c3af2c in tbb::detail::r1::rml::private_worker::run (this=0x40003b618000) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#16 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x40003b618000) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#17 0x00004000371c78b8 in start_thread () from /lib64/libpthread.so.0
#18 0x0000400037223afc in thread_start () from /lib64/libc.so.6
Thread 4 (Thread 0x40007d4f9250 (LWP 92715) "cmsRun"):
#0  0x00004000372ca91c in poll () from /lib64/libc.so.6
#1  0x000040003c3f333c in full_read.constprop () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x000040003c3ac5e4 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x000040003c3acf5c in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x0000400213ff001c in ?? ()
#6  0x000040004a27d300 in cling::Interpreter::RunFunction(clang::FunctionDecl const*, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libCling.so
#7  0x000040004a27d300 in cling::Interpreter::RunFunction(clang::FunctionDecl const*, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libCling.so
#8  0x000040004a27d9ac in cling::Interpreter::EvaluateInternal(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, cling::CompilationOptions, cling::Value*, cling::Transaction**, unsigned long) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libCling.so
#9  0x000040004a34f744 in cling::MetaSema::actOnxCommand(llvm::StringRef, llvm::StringRef, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libCling.so
#10 0x000040004a35c034 in cling::MetaParser::isXCommand(cling::MetaSema::ActionResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libCling.so
#11 0x000040004a35def4 in cling::MetaParser::isCommand(cling::MetaSema::ActionResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libCling.so
#12 0x000040004a3491ec in cling::MetaProcessor::process(llvm::StringRef, cling::Interpreter::CompilationResult&, cling::Value*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libCling.so
#13 0x000040004a1a08ac in HandleInterpreterException(cling::MetaProcessor*, char const*, cling::Interpreter::CompilationResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libCling.so
#14 0x000040004a1b3340 in TCling::ProcessLine(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libCling.so
#15 0x000040004a1b38b4 in TCling::ProcessLineSynch(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libCling.so
#16 0x0000400036357258 in TApplication::ExecuteFile(char const*, int*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libCore.so
#17 0x000040004a1b23d8 in TCling::ExecuteMacro(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libCling.so
#18 0x00004000363ac11c in TROOT::Macro(char const*, int*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libCore.so
#19 0x00004000363900a0 in TPluginManager::LoadHandlerMacros(char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libCore.so
#20 0x00004000363906fc in TPluginManager::LoadHandlersFromPluginDirs(char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libCore.so
#21 0x0000400036391774 in TPluginManager::FindHandler(char const*, char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libCore.so
#22 0x0000400035bb7cbc in ROOT::Math::Factory::CreateMinimizer(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#23 0x0000400035bb9680 in ROOT::Fit::FitConfig::CreateMinimizer() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#24 0x0000400035bd11d0 in ROOT::Fit::Fitter::DoInitMinimizer() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#25 0x0000400035bd80fc in bool ROOT::Fit::Fitter::DoMinimization<ROOT::Fit::Chi2FCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> > >(std::unique_ptr<ROOT::Fit::Chi2FCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> >, std::default_delete<ROOT::Fit::Chi2FCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> > > >, ROOT::Math::IBaseFunctionMultiDimTempl<double> const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#26 0x0000400035bd3f08 in ROOT::Fit::Fitter::DoLeastSquareFit(ROOT::EExecutionPolicy const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#27 0x000040003ca9ae60 in TFitResultPtr HFit::Fit<TGraph>(TGraph*, TF1*, Foption_t&, ROOT::Math::MinimizerOptions const&, char const*, ROOT::Fit::DataRange&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libHist.so
#28 0x000040003ca8f340 in ROOT::Fit::FitObject(TGraph*, TF1*, Foption_t&, ROOT::Math::MinimizerOptions const&, char const*, ROOT::Fit::DataRange&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libHist.so
#29 0x000040003cb3c734 in TGraph::Fit(TF1*, char const*, char const*, double, double) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/external/el8_aarch64_gcc12/lib/libHist.so
#30 0x00004000c21a50cc in CSCSimHitMatcher::fitHitsInChamber(unsigned int, float&, float&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libValidationMuonHits.so
#31 0x00004000c44f9700 in CSCStubResolutionValidation::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libValidationMuonCSCDigis.so
#32 0x00004000351da61c in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#33 0x00004000351bf1a0 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#34 0x000040003514c2c4 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#35 0x000040003514cae4 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#36 0x00004000356e50dc in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreConcurrency.so
#37 0x0000400036c32dfc in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x40016ac69300, waiter=<synthetic pointer>..., this=0x400037b4b380) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#38 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=<synthetic pointer>..., t=0x0, this=0x400037b4b380) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#39 tbb::detail::r1::arena::process (tls=..., this=0x400037b4ad00) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/arena.cpp:137
#40 tbb::detail::r1::market::process (this=0x400037b4b580, j=...) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/market.cpp:599
#41 0x0000400036c3af2c in tbb::detail::r1::rml::private_worker::run (this=0x40003b618100) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#42 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x40003b618100) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#43 0x00004000371c78b8 in start_thread () from /lib64/libpthread.so.0
#44 0x0000400037223afc in thread_start () from /lib64/libc.so.6
Thread 3 (Thread 0x40007cae9250 (LWP 92714) "cmsRun"):
#0  0x00004000372a4b3c in nanosleep () from /lib64/libc.so.6
#1  0x00004000372a49e0 in sleep () from /lib64/libc.so.6
#2  0x000040003c3a927c in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00004000df56c868 in std::__detail::_Map_base<DetId, std::pair<DetId const, std::vector<std::pair<int, float>, std::allocator<std::pair<int, float> > > >, std::allocator<std::pair<DetId const, std::vector<std::pair<int, float>, std::allocator<std::pair<int, float> > > > >, std::__detail::_Select1st, std::equal_to<DetId>, std::hash<DetId>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true>, true>::operator[](DetId const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/pluginSimCalorimetryHGCalAssociatorProducersAuto.so
#5  0x00004000df56ff34 in TSToSimTSHitLCAssociatorByEnergyScoreImpl::makeConnections(edm::Handle<std::vector<ticl::Trackster, std::allocator<ticl::Trackster> > > const&, edm::Handle<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > > const&, edm::Handle<std::vector<SimCluster, std::allocator<SimCluster> > > const&, edm::Handle<std::vector<CaloParticle, std::allocator<CaloParticle> > > const&, edm::Handle<std::vector<ticl::Trackster, std::allocator<ticl::Trackster> > > const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/pluginSimCalorimetryHGCalAssociatorProducersAuto.so
#6  0x00004000df5729a8 in TSToSimTSHitLCAssociatorByEnergyScoreImpl::associateRecoToSim(edm::Handle<std::vector<ticl::Trackster, std::allocator<ticl::Trackster> > > const&, edm::Handle<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > > const&, edm::Handle<std::vector<SimCluster, std::allocator<SimCluster> > > const&, edm::Handle<std::vector<CaloParticle, std::allocator<CaloParticle> > > const&, edm::Handle<std::vector<ticl::Trackster, std::allocator<ticl::Trackster> > > const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/pluginSimCalorimetryHGCalAssociatorProducersAuto.so
#7  0x00004000df56e9ec in TSToSimTSHitLCAssociatorEDProducer::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/pluginSimCalorimetryHGCalAssociatorProducersAuto.so
#8  0x00004000351c64e8 in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#9  0x00004000351b8d00 in edm::WorkerT<edm::global::EDProducerBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#10 0x000040003514c2c4 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#11 0x000040003514cae4 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#12 0x00004000356e50dc in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreConcurrency.so
#13 0x0000400036c32dfc in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x40013b781e00, waiter=<synthetic pointer>..., this=0x400037b4b400) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#14 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=<synthetic pointer>..., t=0x0, this=0x400037b4b400) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#15 tbb::detail::r1::arena::process (tls=..., this=0x400037b4ad00) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/arena.cpp:137
#16 tbb::detail::r1::market::process (this=0x400037b4b580, j=...) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/market.cpp:599
#17 0x0000400036c3af2c in tbb::detail::r1::rml::private_worker::run (this=0x40003b618080) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#18 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x40003b618080) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#19 0x00004000371c78b8 in start_thread () from /lib64/libpthread.so.0
#20 0x0000400037223afc in thread_start () from /lib64/libc.so.6
Thread 2 (Thread 0x400040879250 (LWP 92602) "cmsRun"):
#0  0x00004000371d2578 in waitpid () from /lib64/libpthread.so.0
#1  0x000040003c3a93f0 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x000040003c3ac550 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x0000400036f5eb8c in std::execute_native_thread_routine (__p=0x4000390ac5d0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x00004000371c78b8 in start_thread () from /lib64/libpthread.so.0
#5  0x0000400037223afc in thread_start () from /lib64/libc.so.6
Thread 1 (Thread 0x400036ddd080 (LWP 92020) "cmsRun"):
#0  0x00004000372a4b3c in nanosleep () from /lib64/libc.so.6
#1  0x00004000372a49e0 in sleep () from /lib64/libc.so.6
#2  0x000040003c3a927c in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00004000cd40bf80 in std::_Hashtable<unsigned int, std::pair<unsigned int const, MTDHit>, std::allocator<std::pair<unsigned int const, MTDHit> >, std::__detail::_Select1st, std::equal_to<unsigned int>, std::hash<unsigned int>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::find(unsigned int const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/pluginValidationMtdValidationAuto.so
#5  0x00004000cd41f624 in EtlLocalRecoValidation::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/pluginValidationMtdValidationAuto.so
#6  0x00004000351da61c in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#7  0x00004000351bf1a0 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#8  0x000040003514c2c4 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#9  0x000040003514cae4 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#10 0x00004000356e50dc in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreConcurrency.so
#11 0x0000400036c38774 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x400118c4c400, this=0x400037b4b300) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#12 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x400037b4b300) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#13 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#14 0x00004000350c7044 in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#15 0x00004000350d0668 in edm::EventProcessor::processRuns() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#16 0x00004000350d0bf8 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#17 0x0000000000406fe4 in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#18 0x0000400036c310d8 in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/arena.cpp:688
#19 0x0000000000408854 in main::{lambda()#1}::operator()() const ()
#20 0x0000000000404954 in main ()

Current Modules:

Module: CSCDigiValidation:cscDigiValidation (crashed)
Module: EtlLocalRecoValidation:etlLocalRecoValid
Module: TSToSimTSHitLCAssociatorEDProducer:tracksterSimTracksterAssociationLinkingbyCLUE3D
Module: ClusterTPAssociationProducer:hltTPClusterProducer

A fatal system signal has occurred: segmentation violation

Log from today's IBs at CMSSW_14_1_X_2024-02-26-2300.

RelVal 140.56 is also failing (as 140.58 before) showing:

#5  0x000040003fef3afc in thread_start () from /lib64/libc.so.6
Thread 1 (Thread 0x40003faad080 (LWP 317044) "cmsRun"):
#0  0x000040003ff74b3c in nanosleep () from /lib64/libc.so.6
#1  0x000040003ff749e0 in sleep () from /lib64/libc.so.6
#2  0x00004000451f927c in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000040003fef37e4 in syscall () from /lib64/libc.so.6
#5  0x000040003f908f68 in tbb::detail::r1::futex_wait (comparand=2, futex=0xffffc4b36758) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0xffffc4b36758) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::sleep_node<tbb::detail::r1::market_context>::wait (this=0xffffc4b36728) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:170
#8  tbb::detail::r1::concurrent_monitor_base<tbb::detail::r1::market_context>::commit_wait (this=<optimized out>, node=...) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:232
#9  tbb::detail::r1::concurrent_monitor_base<tbb::detail::r1::market_context>::commit_wait (node=..., this=0x40004081ab18) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:228
#10 tbb::detail::r1::concurrent_monitor_base<tbb::detail::r1::market_context>::wait<tbb::detail::r1::sleep_node<tbb::detail::r1::market_context>, tbb::detail::r1::external_waiter::pause(tbb::detail::r1::arena_slot&)::{lambda()#1}&>(tbb::detail::r1::external_waiter::pause(tbb::detail::r1::arena_slot&)::{lambda()#1}&, tbb::detail::r1::sleep_node<tbb::detail::r1::market_context>&&) (node=..., pred=<synthetic pointer>..., this=0x40004081ab18) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:262
#11 tbb::detail::r1::sleep_waiter::sleep<tbb::detail::r1::external_waiter::pause(tbb::detail::r1::arena_slot&)::{lambda()#1}>(unsigned long, tbb::detail::r1::external_waiter::pause(tbb::detail::r1::arena_slot&)::{lambda()#1}) (this=0xffffc4b366d0, wakeup_condition=..., uniq_tag=70369926930688) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/waiters.h:118
#12 tbb::detail::r1::external_waiter::pause (this=0xffffc4b366d0) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/waiters.h:144
#13 tbb::detail::r1::external_waiter::pause (this=<optimized out>) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/waiters.h:137
#14 tbb::detail::r1::task_dispatcher::receive_or_steal_task<false, tbb::detail::r1::external_waiter> (critical_allowed=true, fifo_allowed=true, isolation=0, waiter=..., ed=..., tls=..., this=0x4000407f3d80) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:231
#15 tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x0, this=0x4000407f3d80) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:350
#16 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x4000407f3d80) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#17 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#18 0x000040003dd97044 in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#19 0x000040003dda0668 in edm::EventProcessor::processRuns() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#20 0x000040003dda0bf8 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02826/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-26-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#21 0x0000000000406fe4 in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#22 0x000040003f9010d8 in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/arena.cpp:688
#23 0x0000000000408854 in main::{lambda()#1}::operator()() const ()
#24 0x0000000000404954 in main ()

Current Modules:

Module: AlcaBeamMonitor:AlcaBeamMonitor (crashed)
Module: none
Module: none
Module: none

A fatal system signal has occurred: segmentation violation
iarspider commented 8 months ago

Two new crashes in CMSSW_14_1_ROOT6_X_2024-03-06-2300:

140.58

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thu Mar  7 05:58:42 CET 2024
Thread 8 (Thread 0x4000c3729250 (LWP 513138) "cmsRun"):
#0  0x000040003beedc58 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
#1  0x000040006da1d02c in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#2  0x000040006da1d720 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#3  0x000040006da1abdc in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#4  0x0000400077d7112c in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_framework.so.2
#5  0x000040003bee78b8 in start_thread () from /lib64/libpthread.so.0
#6  0x000040003bf43afc in thread_start () from /lib64/libc.so.6
Thread 7 (Thread 0x4000c3519250 (LWP 513137) "cmsRun"):
#0  0x000040003beedc58 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
#1  0x000040006da1d02c in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#2  0x000040006da1d720 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#3  0x000040006da1abdc in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#4  0x0000400077d7112c in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_framework.so.2
#5  0x000040003bee78b8 in start_thread () from /lib64/libpthread.so.0
#6  0x000040003bf43afc in thread_start () from /lib64/libc.so.6
Thread 6 (Thread 0x4000c2309250 (LWP 513136) "cmsRun"):
#0  0x000040003beedc58 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
#1  0x000040006da1d02c in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#2  0x000040006da1d720 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#3  0x000040006da1abdc in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#4  0x0000400077d7112c in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_framework.so.2
#5  0x000040003bee78b8 in start_thread () from /lib64/libpthread.so.0
#6  0x000040003bf43afc in thread_start () from /lib64/libc.so.6
Thread 5 (Thread 0x400086a09250 (LWP 512865) "cmsRun"):
#0  0x000040003bfc4b3c in nanosleep () from /lib64/libc.so.6
#1  0x000040003bfc49e0 in sleep () from /lib64/libc.so.6
#2  0x00004000411a927c in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000040003bf437e4 in syscall () from /lib64/libc.so.6
#5  0x000040003b949818 in tbb::detail::r1::futex_wait (comparand=2, futex=0x400040408024) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x400040408024) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  0x000040003b95b09c in tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x400040408020) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x400040408000) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x400040408000) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000040003bee78b8 in start_thread () from /lib64/libpthread.so.0
#11 0x000040003bf43afc in thread_start () from /lib64/libc.so.6
Thread 4 (Thread 0x4000839d9250 (LWP 512864) "cmsRun"):
#0  0x000040003bfc4b3c in nanosleep () from /lib64/libc.so.6
#1  0x000040003bfc49e0 in sleep () from /lib64/libc.so.6
#2  0x00004000411a927c in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000040003bf437e4 in syscall () from /lib64/libc.so.6
#5  0x000040003b949818 in tbb::detail::r1::futex_wait (comparand=2, futex=0x400040408124) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x400040408124) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  0x000040003b95b09c in tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x400040408120) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x400040408100) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x400040408100) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000040003bee78b8 in start_thread () from /lib64/libpthread.so.0
#11 0x000040003bf43afc in thread_start () from /lib64/libc.so.6
Thread 3 (Thread 0x400082fc9250 (LWP 512863) "cmsRun"):
#0  0x000040003bfea91c in poll () from /lib64/libc.so.6
#1  0x00004000411f333c in full_read.constprop () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x00004000411ac5e4 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x00004000411acf5c in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x000040003b0b16f0 in TPluginManager::RemoveHandler(char const*, char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCore.so
#6  0x000040003b0b1818 in TPluginManager::AddHandler(char const*, char const*, char const*, char const*, char const*, char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCore.so
#7  0x000040004cce89c4 in cling::IncrementalExecutor::executeWrapper(llvm::StringRef, cling::Value*) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#8  0x000040004cc75b60 in cling::Interpreter::RunFunction(clang::FunctionDecl const*, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#9  0x000040004cc7620c in cling::Interpreter::EvaluateInternal(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, cling::CompilationOptions, cling::Value*, cling::Transaction**, unsigned long) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#10 0x000040004cd54ad0 in cling::MetaSema::actOnxCommand(llvm::StringRef, llvm::StringRef, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#11 0x000040004cd61624 in cling::MetaParser::isXCommand(cling::MetaSema::ActionResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#12 0x000040004cd634d4 in cling::MetaParser::isCommand(cling::MetaSema::ActionResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#13 0x000040004cd4e55c in cling::MetaProcessor::process(llvm::StringRef, cling::Interpreter::CompilationResult&, cling::Value*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#14 0x000040004cb91d9c in HandleInterpreterException(cling::MetaProcessor*, char const*, cling::Interpreter::CompilationResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#15 0x000040004cba48d0 in TCling::ProcessLine(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#16 0x000040004cba4e44 in TCling::ProcessLineSynch(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#17 0x000040003b079048 in TApplication::ExecuteFile(char const*, int*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCore.so
#18 0x000040004cba3968 in TCling::ExecuteMacro(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#19 0x000040003b0ce8ec in TROOT::Macro(char const*, int*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCore.so
#20 0x000040003b0b27d0 in TPluginManager::LoadHandlerMacros(char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCore.so
#21 0x000040003b0b2e2c in TPluginManager::LoadHandlersFromPluginDirs(char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCore.so
#22 0x000040003b0b3ea4 in TPluginManager::FindHandler(char const*, char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCore.so
#23 0x000040003a8e7b9c in ROOT::Math::Factory::CreateMinimizer(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#24 0x000040003a8e9560 in ROOT::Fit::FitConfig::CreateMinimizer() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#25 0x000040003a900f30 in ROOT::Fit::Fitter::DoInitMinimizer() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#26 0x000040003a9071dc in bool ROOT::Fit::Fitter::DoMinimization<ROOT::Fit::PoissonLikelihoodFCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> > >(std::unique_ptr<ROOT::Fit::PoissonLikelihoodFCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> >, std::default_delete<ROOT::Fit::PoissonLikelihoodFCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> > > >, ROOT::Math::IBaseFunctionMultiDimTempl<double> const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#27 0x000040003a9046c0 in ROOT::Fit::Fitter::DoBinnedLikelihoodFit(bool, ROOT::EExecutionPolicy const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#28 0x0000400043af8a4c in TFitResultPtr HFit::Fit<TH1>(TH1*, TF1*, Foption_t&, ROOT::Math::MinimizerOptions const&, char const*, ROOT::Fit::DataRange&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libHist.so
#29 0x0000400043aee194 in ROOT::Fit::FitObject(TH1*, TF1*, Foption_t&, ROOT::Math::MinimizerOptions const&, char const*, ROOT::Fit::DataRange&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libHist.so
#30 0x0000400043bb9be0 in TH1::Fit(TF1*, char const*, char const*, double, double) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libHist.so
#31 0x000040007e96c910 in BSFitter::Fit_d0phi() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libRecoVertexBeamSpotProducer.so
#32 0x000040007e96d454 in BSFitter::Fit_ited0phi() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libRecoVertexBeamSpotProducer.so
#33 0x000040007e96d940 in BSFitter::Fit(double*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libRecoVertexBeamSpotProducer.so
#34 0x000040007e96e098 in BSFitter::Fit() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libRecoVertexBeamSpotProducer.so
#35 0x000040007e976e20 in BeamFitter::runFitterNoTxt() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libRecoVertexBeamSpotProducer.so
#36 0x000040007e977294 in BeamFitter::runPVandTrkFitter() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libRecoVertexBeamSpotProducer.so
#37 0x00004000bc3e0ee4 in AlcaBeamMonitor::globalEndLuminosityBlock(edm::LuminosityBlock const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/pluginAlcaBeamMonitor.so
#38 0x00004000bc3ed9dc in virtual thunk to edm::one::impl::LuminosityBlockCacheHolder<edm::one::EDProducerBase, alcabeammonitor::BeamSpotInfo>::doEndLuminosityBlock_(edm::LuminosityBlock const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/pluginAlcaBeamMonitor.so
#39 0x0000400039eef144 in edm::one::EDProducerBase::doEndLuminosityBlock(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#40 0x0000400039ed7d00 in edm::WorkerT<edm::one::EDProducerBase>::implDoEnd(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#41 0x0000400039e2a6c0 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#42 0x0000400039e17acc in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3> >::execute()::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#43 0x0000400039e2e430 in edm::SerialTaskQueue::QueuedTask<edm::SerialTaskQueueChain::push<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3> >::execute()::{lambda()#1}&>(tbb::detail::d1::task_group&, edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3> >::execute()::{lambda()#1}&)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#44 0x000040003a4065b0 in tbb::detail::d1::function_task<edm::SerialTaskQueue::spawn(edm::SerialTaskQueue::TaskBase&)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libFWCoreConcurrency.so
#45 0x000040003b952dfc in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x40003c718900, waiter=<synthetic pointer>..., this=0x40003c86b400) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#46 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=<synthetic pointer>..., t=0x0, this=0x40003c86b400) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#47 tbb::detail::r1::arena::process (tls=..., this=0x40003c86ad00) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/arena.cpp:137
#48 tbb::detail::r1::market::process (this=0x40003c86b580, j=...) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/market.cpp:599
#49 0x000040003b95af2c in tbb::detail::r1::rml::private_worker::run (this=0x400040408080) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#50 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x400040408080) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#51 0x000040003bee78b8 in start_thread () from /lib64/libpthread.so.0
#52 0x000040003bf43afc in thread_start () from /lib64/libc.so.6
Thread 2 (Thread 0x4000450d9250 (LWP 512780) "cmsRun"):
#0  0x000040003bef2578 in waitpid () from /lib64/libpthread.so.0
#1  0x00004000411a93f0 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x00004000411ac550 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x000040003bc7eb8c in std::execute_native_thread_routine (__p=0x40003e0ab230) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x000040003bee78b8 in start_thread () from /lib64/libpthread.so.0
#5  0x000040003bf43afc in thread_start () from /lib64/libc.so.6
Thread 1 (Thread 0x40003bafd0f0 (LWP 512617) "cmsRun"):
#0  0x000040003bfc4b3c in nanosleep () from /lib64/libc.so.6
#1  0x000040003bfc49e0 in sleep () from /lib64/libc.so.6
#2  0x00004000411a927c in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000040003bf437e4 in syscall () from /lib64/libc.so.6
#5  0x000040003b958f68 in tbb::detail::r1::futex_wait (comparand=2, futex=0xffffd9801418) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0xffffd9801418) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::sleep_node<tbb::detail::r1::market_context>::wait (this=0xffffd98013e8) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:170
#8  tbb::detail::r1::concurrent_monitor_base<tbb::detail::r1::market_context>::commit_wait (this=<optimized out>, node=...) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:232
#9  tbb::detail::r1::concurrent_monitor_base<tbb::detail::r1::market_context>::commit_wait (node=..., this=0x40003c86b598) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:228
#10 tbb::detail::r1::concurrent_monitor_base<tbb::detail::r1::market_context>::wait<tbb::detail::r1::sleep_node<tbb::detail::r1::market_context>, tbb::detail::r1::external_waiter::pause(tbb::detail::r1::arena_slot&)::{lambda()#1}&>(tbb::detail::r1::external_waiter::pause(tbb::detail::r1::arena_slot&)::{lambda()#1}&, tbb::detail::r1::sleep_node<tbb::detail::r1::market_context>&&) (node=..., pred=<synthetic pointer>..., this=0x40003c86b598) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:262
#11 tbb::detail::r1::sleep_waiter::sleep<tbb::detail::r1::external_waiter::pause(tbb::detail::r1::arena_slot&)::{lambda()#1}>(unsigned long, tbb::detail::r1::external_waiter::pause(tbb::detail::r1::arena_slot&)::{lambda()#1}) (this=0xffffd9801390, wakeup_condition=..., uniq_tag=70369862305792) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/waiters.h:118
#12 tbb::detail::r1::external_waiter::pause (this=0xffffd9801390) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/waiters.h:144
#13 tbb::detail::r1::external_waiter::pause (this=<optimized out>) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/waiters.h:137
#14 tbb::detail::r1::task_dispatcher::receive_or_steal_task<false, tbb::detail::r1::external_waiter> (critical_allowed=true, fifo_allowed=true, isolation=0, waiter=..., ed=..., tls=..., this=0x40003c86b300) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:231
#15 tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x0, this=0x40003c86b300) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:350
#16 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x40003c86b300) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#17 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#18 0x0000400039de6fc4 in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#19 0x0000400039df05e8 in edm::EventProcessor::processRuns() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#20 0x0000400039df0b78 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#21 0x0000000000406fe4 in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#22 0x000040003b9510d8 in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/arena.cpp:688
#23 0x0000000000408854 in main::{lambda()#1}::operator()() const ()
#24 0x0000000000404954 in main ()

Current Modules:

Module: AlcaBeamMonitor:AlcaBeamMonitor (crashed)
Module: none
Module: none
Module: none

and 11025.0

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thu Mar  7 06:14:29 CET 2024
Thread 22 (Thread 0x4000d7209250 (LWP 532101) "cmsRun"):
#0  0x0000400028d3dc58 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
#1  0x000040005ce3d02c in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#2  0x000040005ce3d720 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#3  0x000040005ce3abdc in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#4  0x000040006719112c in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_framework.so.2
#5  0x0000400028d378b8 in start_thread () from /lib64/libpthread.so.0
#6  0x0000400028d93afc in thread_start () from /lib64/libc.so.6
Thread 21 (Thread 0x4000d2e99250 (LWP 532100) "cmsRun"):
#0  0x0000400028d3dc58 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
#1  0x000040005ce3d02c in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#2  0x000040005ce3d720 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#3  0x000040005ce3abdc in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#4  0x000040006719112c in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_framework.so.2
#5  0x0000400028d378b8 in start_thread () from /lib64/libpthread.so.0
#6  0x0000400028d93afc in thread_start () from /lib64/libc.so.6
Thread 20 (Thread 0x4000d2c89250 (LWP 532099) "cmsRun"):
#0  0x0000400028d3dc58 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
#1  0x000040005ce3d02c in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#2  0x000040005ce3d720 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#3  0x000040005ce3abdc in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_cc.so.2
#4  0x000040006719112c in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libtensorflow_framework.so.2
#5  0x0000400028d378b8 in start_thread () from /lib64/libpthread.so.0
#6  0x0000400028d93afc in thread_start () from /lib64/libc.so.6
Thread 19 (Thread 0x400083439250 (LWP 531609) "cmsRun"):
#0  0x0000400028d406d8 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x0000400028d407f4 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x000040002dc88a70 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdCl.so.3
#3  0x000040002dc88b4c in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdCl.so.3
#4  0x0000400028d378b8 in start_thread () from /lib64/libpthread.so.0
#5  0x0000400028d93afc in thread_start () from /lib64/libc.so.6
Thread 18 (Thread 0x400083229250 (LWP 531608) "cmsRun"):
#0  0x0000400028d406d8 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x0000400028d407f4 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x000040002dc88a70 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdCl.so.3
#3  0x000040002dc88b4c in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdCl.so.3
#4  0x0000400028d378b8 in start_thread () from /lib64/libpthread.so.0
#5  0x0000400028d93afc in thread_start () from /lib64/libc.so.6
Thread 17 (Thread 0x400083019250 (LWP 531607) "cmsRun"):
#0  0x0000400028d406d8 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x0000400028d407f4 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x000040002dc88a70 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdCl.so.3
#3  0x000040002dc88b4c in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdCl.so.3
#4  0x0000400028d378b8 in start_thread () from /lib64/libpthread.so.0
#5  0x0000400028d93afc in thread_start () from /lib64/libc.so.6
Thread 16 (Thread 0x400082e09250 (LWP 531606) "cmsRun"):
#0  0x0000400028d41fb0 in nanosleep () from /lib64/libpthread.so.0
#1  0x000040002da51b24 in XrdSysTimer::Wait(int) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x000040002dc131ac in XrdCl::TaskManager::RunTasks() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdCl.so.3
#3  0x000040002dc1331c in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdCl.so.3
#4  0x0000400028d378b8 in start_thread () from /lib64/libpthread.so.0
#5  0x0000400028d93afc in thread_start () from /lib64/libc.so.6
Thread 15 (Thread 0x400082bf9250 (LWP 531605) "cmsRun"):
#0  0x0000400028e441e8 in epoll_pwait () from /lib64/libc.so.6
#1  0x000040002da4bdec in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x000040002da47efc in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x000040002da510e0 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#4  0x0000400028d378b8 in start_thread () from /lib64/libpthread.so.0
#5  0x0000400028d93afc in thread_start () from /lib64/libc.so.6
Thread 14 (Thread 0x4000829e9250 (LWP 531604) "cmsRun"):
#0  0x0000400028e441e8 in epoll_pwait () from /lib64/libc.so.6
#1  0x000040002da4bdec in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x000040002da47efc in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x000040002da510e0 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#4  0x0000400028d378b8 in start_thread () from /lib64/libpthread.so.0
#5  0x0000400028d93afc in thread_start () from /lib64/libc.so.6
Thread 13 (Thread 0x4000827d9250 (LWP 531603) "cmsRun"):
#0  0x0000400028e441e8 in epoll_pwait () from /lib64/libc.so.6
#1  0x000040002da4bdec in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x000040002da47efc in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x000040002da510e0 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#4  0x0000400028d378b8 in start_thread () from /lib64/libpthread.so.0
#5  0x0000400028d93afc in thread_start () from /lib64/libc.so.6
Thread 12 (Thread 0x4000825c9250 (LWP 531602) "cmsRun"):
#0  0x0000400028e441e8 in epoll_pwait () from /lib64/libc.so.6
#1  0x000040002da4bdec in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x000040002da47efc in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x000040002da510e0 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#4  0x0000400028d378b8 in start_thread () from /lib64/libpthread.so.0
#5  0x0000400028d93afc in thread_start () from /lib64/libc.so.6
Thread 11 (Thread 0x4000823b9250 (LWP 531601) "cmsRun"):
#0  0x0000400028e441e8 in epoll_pwait () from /lib64/libc.so.6
#1  0x000040002da4bdec in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x000040002da47efc in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x000040002da510e0 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#4  0x0000400028d378b8 in start_thread () from /lib64/libpthread.so.0
#5  0x0000400028d93afc in thread_start () from /lib64/libc.so.6
Thread 10 (Thread 0x4000821a9250 (LWP 531600) "cmsRun"):
#0  0x0000400028e441e8 in epoll_pwait () from /lib64/libc.so.6
#1  0x000040002da4bdec in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x000040002da47efc in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x000040002da510e0 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#4  0x0000400028d378b8 in start_thread () from /lib64/libpthread.so.0
#5  0x0000400028d93afc in thread_start () from /lib64/libc.so.6
Thread 9 (Thread 0x400081f99250 (LWP 531599) "cmsRun"):
#0  0x0000400028e441e8 in epoll_pwait () from /lib64/libc.so.6
#1  0x000040002da4bdec in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x000040002da47efc in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x000040002da510e0 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#4  0x0000400028d378b8 in start_thread () from /lib64/libpthread.so.0
#5  0x0000400028d93afc in thread_start () from /lib64/libc.so.6
Thread 8 (Thread 0x400081d89250 (LWP 531598) "cmsRun"):
#0  0x0000400028e441e8 in epoll_pwait () from /lib64/libc.so.6
#1  0x000040002da4bdec in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x000040002da47efc in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x000040002da510e0 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#4  0x0000400028d378b8 in start_thread () from /lib64/libpthread.so.0
#5  0x0000400028d93afc in thread_start () from /lib64/libc.so.6
Thread 7 (Thread 0x400081b79250 (LWP 531597) "cmsRun"):
#0  0x0000400028e441e8 in epoll_pwait () from /lib64/libc.so.6
#1  0x000040002da4bdec in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x000040002da47efc in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x000040002da510e0 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#4  0x0000400028d378b8 in start_thread () from /lib64/libpthread.so.0
#5  0x0000400028d93afc in thread_start () from /lib64/libc.so.6
Thread 6 (Thread 0x400081969250 (LWP 531596) "cmsRun"):
#0  0x0000400028e441e8 in epoll_pwait () from /lib64/libc.so.6
#1  0x000040002da4bdec in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#2  0x000040002da47efc in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#3  0x000040002da510e0 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libXrdUtils.so.3
#4  0x0000400028d378b8 in start_thread () from /lib64/libpthread.so.0
#5  0x0000400028d93afc in thread_start () from /lib64/libc.so.6
Thread 5 (Thread 0x400072a09250 (LWP 531583) "cmsRun"):
#0  0x0000400028e14b3c in nanosleep () from /lib64/libc.so.6
#1  0x0000400028e149e0 in sleep () from /lib64/libc.so.6
#2  0x000040002df1927c in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000400028d937e4 in syscall () from /lib64/libc.so.6
#5  0x0000400028799818 in tbb::detail::r1::futex_wait (comparand=2, futex=0x400029da8024) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x400029da8024) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  0x00004000287ab09c in tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x400029da8020) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x400029da8000) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x400029da8000) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x0000400028d378b8 in start_thread () from /lib64/libpthread.so.0
#11 0x0000400028d93afc in thread_start () from /lib64/libc.so.6
Thread 4 (Thread 0x40006f869250 (LWP 531582) "cmsRun"):
#0  0x0000400028e14b3c in nanosleep () from /lib64/libc.so.6
#1  0x0000400028e149e0 in sleep () from /lib64/libc.so.6
#2  0x000040002df1927c in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000400028d937e4 in syscall () from /lib64/libc.so.6
#5  0x0000400028799818 in tbb::detail::r1::futex_wait (comparand=2, futex=0x400029da8124) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x400029da8124) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  0x00004000287ab09c in tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x400029da8120) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x400029da8100) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x400029da8100) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x0000400028d378b8 in start_thread () from /lib64/libpthread.so.0
#11 0x0000400028d93afc in thread_start () from /lib64/libc.so.6
Thread 3 (Thread 0x40006ee59250 (LWP 531581) "cmsRun"):
#0  0x0000400028e14b3c in nanosleep () from /lib64/libc.so.6
#1  0x0000400028e149e0 in sleep () from /lib64/libc.so.6
#2  0x000040002df1927c in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000400028d937e4 in syscall () from /lib64/libc.so.6
#5  0x0000400028799818 in tbb::detail::r1::futex_wait (comparand=2, futex=0x400029da80a4) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x400029da80a4) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  0x00004000287ab09c in tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x400029da80a0) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x400029da8080) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x400029da8080) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x0000400028d378b8 in start_thread () from /lib64/libpthread.so.0
#11 0x0000400028d93afc in thread_start () from /lib64/libc.so.6
Thread 2 (Thread 0x4000323c9250 (LWP 531548) "cmsRun"):
#0  0x0000400028d42578 in waitpid () from /lib64/libpthread.so.0
#1  0x000040002df193f0 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x000040002df1c550 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x0000400028aceb8c in std::execute_native_thread_routine (__p=0x40002b0ab230) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x0000400028d378b8 in start_thread () from /lib64/libpthread.so.0
#5  0x0000400028d93afc in thread_start () from /lib64/libc.so.6
Thread 1 (Thread 0x40002894d0f0 (LWP 531083) "cmsRun"):
#0  0x0000400028e3a91c in poll () from /lib64/libc.so.6
#1  0x000040002df6333c in full_read.constprop () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x000040002df1c5e4 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x000040002df1cf5c in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00004001b1fe0024 in ?? ()
#6  0x000040003bd45b60 in cling::Interpreter::RunFunction(clang::FunctionDecl const*, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#7  0x000040003bd45b60 in cling::Interpreter::RunFunction(clang::FunctionDecl const*, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#8  0x000040003bd4620c in cling::Interpreter::EvaluateInternal(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, cling::CompilationOptions, cling::Value*, cling::Transaction**, unsigned long) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#9  0x000040003be24ad0 in cling::MetaSema::actOnxCommand(llvm::StringRef, llvm::StringRef, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#10 0x000040003be31624 in cling::MetaParser::isXCommand(cling::MetaSema::ActionResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#11 0x000040003be334d4 in cling::MetaParser::isCommand(cling::MetaSema::ActionResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#12 0x000040003be1e55c in cling::MetaProcessor::process(llvm::StringRef, cling::Interpreter::CompilationResult&, cling::Value*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#13 0x000040003bc61d9c in HandleInterpreterException(cling::MetaProcessor*, char const*, cling::Interpreter::CompilationResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#14 0x000040003bc748d0 in TCling::ProcessLine(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#15 0x000040003bc74e44 in TCling::ProcessLineSynch(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#16 0x0000400027ec9048 in TApplication::ExecuteFile(char const*, int*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCore.so
#17 0x000040003bc73968 in TCling::ExecuteMacro(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCling.so
#18 0x0000400027f1e8ec in TROOT::Macro(char const*, int*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCore.so
#19 0x0000400027f027d0 in TPluginManager::LoadHandlerMacros(char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCore.so
#20 0x0000400027f02e2c in TPluginManager::LoadHandlersFromPluginDirs(char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCore.so
#21 0x0000400027f03ea4 in TPluginManager::FindHandler(char const*, char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libCore.so
#22 0x0000400027737b9c in ROOT::Math::Factory::CreateMinimizer(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#23 0x0000400027739560 in ROOT::Fit::FitConfig::CreateMinimizer() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#24 0x0000400027750f30 in ROOT::Fit::Fitter::DoInitMinimizer() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#25 0x00004000277571dc in bool ROOT::Fit::Fitter::DoMinimization<ROOT::Fit::PoissonLikelihoodFCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> > >(std::unique_ptr<ROOT::Fit::PoissonLikelihoodFCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> >, std::default_delete<ROOT::Fit::PoissonLikelihoodFCN<ROOT::Math::IBaseFunctionMultiDimTempl<double>, ROOT::Math::IParametricFunctionMultiDimTempl<double> > > >, ROOT::Math::IBaseFunctionMultiDimTempl<double> const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#26 0x00004000277546c0 in ROOT::Fit::Fitter::DoBinnedLikelihoodFit(bool, ROOT::EExecutionPolicy const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libMathCore.so
#27 0x000040002e5c8a4c in TFitResultPtr HFit::Fit<TH1>(TH1*, TF1*, Foption_t&, ROOT::Math::MinimizerOptions const&, char const*, ROOT::Fit::DataRange&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libHist.so
#28 0x000040002e5be194 in ROOT::Fit::FitObject(TH1*, TF1*, Foption_t&, ROOT::Math::MinimizerOptions const&, char const*, ROOT::Fit::DataRange&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libHist.so
#29 0x000040002e689be0 in TH1::Fit(TF1*, char const*, char const*, double, double) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/external/el8_aarch64_gcc12/lib/libHist.so
#30 0x000040006ada05fc in PVFitter::runFitter() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libRecoVertexBeamSpotProducer.so
#31 0x000040006ad97278 in BeamFitter::runPVandTrkFitter() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libRecoVertexBeamSpotProducer.so
#32 0x00004000c6660ee4 in AlcaBeamMonitor::globalEndLuminosityBlock(edm::LuminosityBlock const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/pluginAlcaBeamMonitor.so
#33 0x00004000c666d9dc in virtual thunk to edm::one::impl::LuminosityBlockCacheHolder<edm::one::EDProducerBase, alcabeammonitor::BeamSpotInfo>::doEndLuminosityBlock_(edm::LuminosityBlock const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/pluginAlcaBeamMonitor.so
#34 0x0000400026d3f144 in edm::one::EDProducerBase::doEndLuminosityBlock(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#35 0x0000400026d27d00 in edm::WorkerT<edm::one::EDProducerBase>::implDoEnd(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#36 0x0000400026c7a6c0 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#37 0x0000400026c67acc in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3> >::execute()::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#38 0x0000400026c7e430 in edm::SerialTaskQueue::QueuedTask<edm::SerialTaskQueueChain::push<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3> >::execute()::{lambda()#1}&>(tbb::detail::d1::task_group&, edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3> >::execute()::{lambda()#1}&)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#39 0x00004000272565b0 in tbb::detail::d1::function_task<edm::SerialTaskQueue::spawn(edm::SerialTaskQueue::TaskBase&)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libFWCoreConcurrency.so
#40 0x00004000287a8774 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x400108faff00, this=0x4000296bb300) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#41 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x4000296bb300) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#42 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#43 0x0000400026c36fc4 in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#44 0x0000400026c405e8 in edm::EventProcessor::processRuns() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#45 0x0000400026c40b78 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02827/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-03-06-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#46 0x0000000000406fe4 in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#47 0x00004000287a10d8 in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/arena.cpp:688
#48 0x0000000000408854 in main::{lambda()#1}::operator()() const ()
#49 0x0000000000404954 in main ()

Current Modules:

Module: AlcaBeamMonitor:AlcaBeamMonitor (crashed)
Module: none
Module: none
Module: none
vgvassilev commented 8 months ago

Any chance to cook up a ROOT-only reproducer?

iarspider commented 5 months ago

Crash in CMSSW_14_1_X_2024-05-28-1100

Thread 5 (Thread 0x40005eb29220 (LWP 2760826) "cmsRun"):
#0  0x000040001402a91c in poll () from /lib64/libc.so.6
#1  0x000040001943f700 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02839/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-05-27-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x000040001943f934 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02839/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-05-27-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000400273fe001c in ?? ()
#5  0x0000400027420464 in cling::IncrementalExecutor::executeWrapper(llvm::StringRef, cling::Value*) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02839/el8_aarch64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-05-28-1100/external/el8_aarch64_gcc12/lib/libCling.so
#6  0x0000400027420464 in cling::IncrementalExecutor::executeWrapper(llvm::StringRef, cling::Value*) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02839/el8_aarch64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-05-28-1100/external/el8_aarch64_gcc12/lib/libCling.so
#7  0x0000000000000000 in ?? ()
aandvalenzuela commented 3 months ago

Hello,

We see multiple of those crashed in cling reporting two types of stack trace:

makortel commented 5 days ago

Just to record more stack traces, one from workflow 11834.0 step 3 CMSSW_14_2_X_2024-11-17-0000 on el8_aarch64_gcc12

Thread 5 (Thread 0x40007ea09220 (LWP 2522448) "cmsRun"):
#0  0x00004000315c4b3c in nanosleep () from /lib64/libc.so.6
#1  0x00004000315c49e0 in sleep () from /lib64/libc.so.6
#2  0x000040003635dc88 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000400031250c2c in __cxxabiv1::__class_type_info::__do_dyncast (this=<optimized out>, access_path=<optimized out>, dst_type=<optimized out>, obj_ptr=<optimized out>, src_type=<optimized out>, src_ptr=<optimized out>, result=...) at ../../../../libstdc++-v3/libsupc++/class_type_info.cc:94
#5  0x0000400031250d14 in __cxxabiv1::__dynamic_cast (src_ptr=0x4002256f4bf0, src_type=0x40003b38f848 <typeinfo for TrackingRecHit>, dst_type=0x40003b22e570 <typeinfo for SiStripMatchedRecHit2D>, src2dst=0) at ../../../../libstdc++-v3/libsupc++/dyncast.cc:74
#6  0x000040008c6d4820 in SiStripTrackingRecHitsValid::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/libValidationRecoTrack.so
#7  0x000040002f4d8f98 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#8  0x000040002f4b77c0 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#9  0x000040002f43f914 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#10 0x000040002f440104 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#11 0x000040002f965b1c in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/libFWCoreConcurrency.so
#12 0x0000400030f5279c in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x400162da4f00, waiter=<synthetic pointer>..., this=0x400031e6b480) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-7833746569d926dff19145e540ff04c3/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#13 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=<synthetic pointer>..., t=0x0, this=0x400031e6b480) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-7833746569d926dff19145e540ff04c3/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#14 tbb::detail::r1::arena::process (tls=..., this=0x400031e6ad00) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-7833746569d926dff19145e540ff04c3/tbb-v2021.9.0/src/tbb/arena.cpp:137
#15 tbb::detail::r1::market::process (this=0x400031e6b580, j=...) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-7833746569d926dff19145e540ff04c3/tbb-v2021.9.0/src/tbb/market.cpp:599
#16 0x0000400030f5affc in tbb::detail::r1::rml::private_worker::run (this=0x4000359a8000) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-7833746569d926dff19145e540ff04c3/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#17 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x4000359a8000) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-7833746569d926dff19145e540ff04c3/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#18 0x00004000314e78b8 in start_thread () from /lib64/libpthread.so.0
#19 0x0000400031543afc in thread_start () from /lib64/libc.so.6

Thread 4 (Thread 0x40007b8b9220 (LWP 2522447) "cmsRun"):
#0  0x00004000315c4b3c in nanosleep () from /lib64/libc.so.6
#1  0x00004000315c49e0 in sleep () from /lib64/libc.so.6
#2  0x000040003635dc88 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000400077376598 in SiPixelTemplateReco::PixelTempReco1D(int, float, float, float, float, SiPixelTemplateReco::ClusMatrix&, SiPixelTemplate&, float&, float&, float&, float&, float&, float&, int&, int, bool, std::vector<std::pair<int, int>, std::allocator<std::pair<int, int> > >&, float&, int&, int&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/libRecoLocalTrackerSiPixelRecHits.so
#5  0x0000400077350238 in PixelCPEClusterRepair::callTempReco1D(PixelCPEBase::DetParam const&, PixelCPEClusterRepair::ClusterParamTemplate&, SiPixelTemplateReco::ClusMatrix&, int, Point3DBase<float, LocalTag>&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/libRecoLocalTrackerSiPixelRecHits.so
#6  0x0000400077350bd0 in PixelCPEClusterRepair::localPosition(PixelCPEBase::DetParam const&, PixelCPEBase::ClusterParam&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/libRecoLocalTrackerSiPixelRecHits.so
#7  0x00004000773499b8 in PixelClusterParameterEstimator::getParameters(SiPixelCluster const&, GeomDet const&, TrajectoryStateOnSurface const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/libRecoLocalTrackerSiPixelRecHits.so
#8  0x000040005dd3879c in TkClonerImpl::makeShared(SiPixelRecHit const&, TrajectoryStateOnSurface const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/libRecoTrackerTransientTrackingRecHit.so
#9  0x00004000770291e8 in SiPixelRecHit::cloneSH_(TkCloner const&, TrajectoryStateOnSurface const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/pluginRecoLocalTrackerSiPixelRecHitsPlugins.so
#10 0x0000400074ebd2d8 in KFTrajectoryFitter::fitOne(TrajectorySeed const&, std::vector<std::shared_ptr<TrackingRecHit const>, std::allocator<std::shared_ptr<TrackingRecHit const> > > const&, TrajectoryStateOnSurface const&, TrajectoryFitter::fitType) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/libTrackingToolsTrackFitters.so
#11 0x0000400074e59190 in (anonymous namespace)::KFFittingSmoother::fitOne(TrajectorySeed const&, std::vector<std::shared_ptr<TrackingRecHit const>, std::allocator<std::shared_ptr<TrackingRecHit const> > > const&, TrajectoryStateOnSurface const&, TrajectoryFitter::fitType) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/pluginTrackingToolsTrackFittersPlugins.so
#12 0x0000400074e528a8 in (anonymous namespace)::FlexibleKFFittingSmoother::fitOne(TrajectorySeed const&, std::vector<std::shared_ptr<TrackingRecHit const>, std::allocator<std::shared_ptr<TrackingRecHit const> > > const&, TrajectoryStateOnSurface const&, TrajectoryFitter::fitType) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/pluginTrackingToolsTrackFittersPlugins.so
#13 0x00004000de2b0990 in TrackProducerAlgorithm<reco::Track>::buildTrack(TrajectoryFitter const*, Propagator const*, std::vector<AlgoProductTraits<reco::Track>::AlgoProduct, std::allocator<AlgoProductTraits<reco::Track>::AlgoProduct> >&, std::vector<std::shared_ptr<TrackingRecHit const>, std::allocator<std::shared_ptr<TrackingRecHit const> > >&, TrajectoryStateOnSurface&, TrajectorySeed const&, float, reco::BeamSpot const&, edm::RefToBase<TrajectorySeed>, int, signed char) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/libRecoTrackerTrackProducer.so
#14 0x00004000de194608 in TrackProducerAlgorithm<reco::Track>::runWithCandidate(TrackingGeometry const*, MagneticField const*, std::vector<TrackCandidate, std::allocator<TrackCandidate> > const&, TrajectoryFitter const*, Propagator const*, TransientTrackingRecHitBuilder const*, reco::BeamSpot const&, std::vector<AlgoProductTraits<reco::Track>::AlgoProduct, std::allocator<AlgoProductTraits<reco::Track>::AlgoProduct> >&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/pluginRecoTrackerTrackProducerPlugins.so
#15 0x00004000de194bb4 in TrackProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/pluginRecoTrackerTrackProducerPlugins.so
#16 0x000040002f4d8f98 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#17 0x000040002f4b77c0 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#18 0x000040002f43f914 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#19 0x000040002f440104 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#20 0x000040002f965b1c in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/libFWCoreConcurrency.so
#21 0x0000400030f5279c in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x40016301bf00, waiter=<synthetic pointer>..., this=0x400031e6b380) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-7833746569d926dff19145e540ff04c3/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#22 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=<synthetic pointer>..., t=0x0, this=0x400031e6b380) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-7833746569d926dff19145e540ff04c3/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#23 tbb::detail::r1::arena::process (tls=..., this=0x400031e6ad00) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-7833746569d926dff19145e540ff04c3/tbb-v2021.9.0/src/tbb/arena.cpp:137
#24 tbb::detail::r1::market::process (this=0x400031e6b580, j=...) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-7833746569d926dff19145e540ff04c3/tbb-v2021.9.0/src/tbb/market.cpp:599
#25 0x0000400030f5affc in tbb::detail::r1::rml::private_worker::run (this=0x4000359a8100) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-7833746569d926dff19145e540ff04c3/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#26 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x4000359a8100) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-7833746569d926dff19145e540ff04c3/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#27 0x00004000314e78b8 in start_thread () from /lib64/libpthread.so.0
#28 0x0000400031543afc in thread_start () from /lib64/libc.so.6

Thread 3 (Thread 0x40007aea9220 (LWP 2522446) "cmsRun"):
#0  0x00004000315ea954 in poll () from /lib64/libc.so.6
#1  0x0000400036360c60 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x0000400036360e94 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000400216ff001c in ?? ()
#5  0x0000400044f83cd4 in cling::IncrementalExecutor::executeWrapper(llvm::StringRef, cling::Value*) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/external/el8_aarch64_gcc12/lib/libCling.so
#6  0x0000400044f83cd4 in cling::IncrementalExecutor::executeWrapper(llvm::StringRef, cling::Value*) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/external/el8_aarch64_gcc12/lib/libCling.so
#7  0x0000000000000000 in ?? ()

Thread 2 (Thread 0x40003ae29220 (LWP 2522136) "cmsRun"):
#0  0x00004000314f2580 in waitpid () from /lib64/libpthread.so.0
#1  0x000040003635ddf4 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x0000400036360a48 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x000040003127eb8c in std::execute_native_thread_routine (__p=0x4000330ac820) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x00004000314e78b8 in start_thread () from /lib64/libpthread.so.0
#5  0x0000400031543afc in thread_start () from /lib64/libc.so.6

Thread 1 (Thread 0x40003112cbc0 (LWP 2521121) "cmsRun"):
#0  0x00004000315c4b3c in nanosleep () from /lib64/libc.so.6
#1  0x00004000315c49e0 in sleep () from /lib64/libc.so.6
#2  0x000040003635dc88 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000040007737f054 in VVIObjF::VVIObjF(float) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/libRecoLocalTrackerSiPixelRecHits.so
#5  0x000040007737752c in SiPixelTemplateReco::PixelTempReco1D(int, float, float, float, float, SiPixelTemplateReco::ClusMatrix&, SiPixelTemplate&, float&, float&, float&, float&, float&, float&, int&, int, bool, std::vector<std::pair<int, int>, std::allocator<std::pair<int, int> > >&, float&, int&, int&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/libRecoLocalTrackerSiPixelRecHits.so
#6  0x0000400077350238 in PixelCPEClusterRepair::callTempReco1D(PixelCPEBase::DetParam const&, PixelCPEClusterRepair::ClusterParamTemplate&, SiPixelTemplateReco::ClusMatrix&, int, Point3DBase<float, LocalTag>&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/libRecoLocalTrackerSiPixelRecHits.so
#7  0x0000400077350bd0 in PixelCPEClusterRepair::localPosition(PixelCPEBase::DetParam const&, PixelCPEBase::ClusterParam&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/libRecoLocalTrackerSiPixelRecHits.so
#8  0x00004000773499b8 in PixelClusterParameterEstimator::getParameters(SiPixelCluster const&, GeomDet const&, TrajectoryStateOnSurface const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/libRecoLocalTrackerSiPixelRecHits.so
#9  0x000040005dd3879c in TkClonerImpl::makeShared(SiPixelRecHit const&, TrajectoryStateOnSurface const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/libRecoTrackerTransientTrackingRecHit.so
#10 0x000040005dd35998 in SiPixelRecHit::cloneSH_(TkCloner const&, TrajectoryStateOnSurface const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/libRecoTrackerTransientTrackingRecHit.so
#11 0x0000400074ebe978 in KFTrajectorySmoother::trajectory(Trajectory const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/libTrackingToolsTrackFitters.so
#12 0x0000400074eb6460 in TrajectorySmoother::trajectories(Trajectory const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/libTrackingToolsTrackFitters.so
#13 0x00004000bff06ec4 in TrackTransformer::transform(reco::TransientTrack const&, std::vector<std::shared_ptr<TrackingRecHit const>, std::allocator<std::shared_ptr<TrackingRecHit const> > >&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/libTrackingToolsTrackRefitter.so
#14 0x00004000bff07478 in TrackTransformer::transform(reco::Track const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/libTrackingToolsTrackRefitter.so
#15 0x00004000df009bc8 in TrackingRecoMaterialAnalyser::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/pluginDQMTrackingMonitor.so
#16 0x000040002f4d8f98 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#17 0x000040002f4b77c0 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#18 0x000040002f43f914 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#19 0x000040002f440104 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#20 0x000040002f965b1c in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/libFWCoreConcurrency.so
#21 0x0000400030f587fc in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x40016300da00, this=0x400031e6b300) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-7833746569d926dff19145e540ff04c3/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#22 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x400031e6b300) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-7833746569d926dff19145e540ff04c3/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#23 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-7833746569d926dff19145e540ff04c3/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#24 0x000040002f3b6d34 in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#25 0x000040002f3c396c in edm::EventProcessor::processRuns() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#26 0x000040002f3c3e0c in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-11-17-0000/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#27 0x0000000000407afc in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#28 0x0000400030f50a68 in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-7833746569d926dff19145e540ff04c3/tbb-v2021.9.0/src/tbb/arena.cpp:688
#29 0x0000000000409398 in main::{lambda()#1}::operator()() const ()
#30 0x0000000000404524 in main ()

Current Modules:

Module: CSCDigiValidation:cscDigiValidation (crashed)
Module: TrackProducer:detachedTripletStepTracks
Module: TrackingRecoMaterialAnalyser:materialDumperAnalyzer
Module: SiStripTrackingRecHitsValid:StripTrackingRecHitsValid

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/el8_aarch64_gcc12/CMSSW_14_2_X_2024-11-17-0000/pyRelValMatrixLogs/run/11834.0_TTbar_14TeV+2021PU/step3_TTbar_14TeV+2021PU.log#/

makortel commented 2 days ago

Occurred in CMSSW_14_2_X_2024-11-20-2300 on el8_aarch64_gcc12

on el9_aarch64_gcc12