search

cms-sw / cmssw

CMS Offline Software
http://cms-sw.github.io/
Apache License 2.0
1.09k stars 4.33k forks source link

LHAPDF::_getQ2CachesMap() is not thread safe #35082

Open makortel opened 3 years ago

makortel commented 3 years ago

https://github.com/cms-sw/cmssw/pull/35073#issuecomment-908647991 experienced a rare crash

Thread 10 (Thread 0x2ba051c00700 (LWP 11137) "cmsRun"):
#2  0x00002ba009f033f0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00002ba004138a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#5  0x00002ba003d0085c in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_11_1_0_pre6-slc7_amd64_gcc900/build/CMSSW_11_1_0_pre6-build/BUILD/slc7_amd64_gcc900/external/gcc/9.3.0/gcc-9.3.0/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/x86_64-unknown-linux-gnu/bits/gthr-default.h:865
#6  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#7  0x00002ba03690968b in OscarMTProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/biglib/slc7_amd64_gcc900/pluginSimulation.so
#8  0x00002ba001dd586c in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#9  0x00002ba001db6a5d in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#10 0x00002ba001d11545 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#11 0x00002ba001d116fd in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#12 0x00002ba001d11a06 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#13 0x00002ba001d13d96 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#14 0x00002ba001a852a5 in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc900/libFWCoreConcurrency.so

Thread 9 (Thread 0x2ba050cfb700 (LWP 11136) "cmsRun"):
#2  0x00002ba009f033f0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00002ba031eb6e80 in LHAPDF::KnotArray1F::iq2below(double) const@plt () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/external/slc7_amd64_gcc900/lib/libLHAPDF.so
#5  0x00002ba031ee19dd in LHAPDF::Interpolator::interpolateXQ2(int, double, double) const () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/external/slc7_amd64_gcc900/lib/libLHAPDF.so
#6  0x00002ba031edcadf in LHAPDF::GridPDF::_xfxQ2(int, double, double) const () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/external/slc7_amd64_gcc900/lib/libLHAPDF.so
#7  0x00002ba031ecf807 in LHAPDF::PDF::xfxQ2(int, double, double) const () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/external/slc7_amd64_gcc900/lib/libLHAPDF.so
#8  0x00002ba04d3f2931 in Pythia8::LHAPDF6::xfUpdate(int, double, double) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/external/slc7_amd64_gcc900/lib/libpythia8lhapdf6.so
#9  0x00002ba0318931b7 in Pythia8::PDF::xf(int, double, double) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/external/slc7_amd64_gcc900/lib/libpythia8.so
#10 0x00002ba031a6d799 in Pythia8::SigmaProcess::sigmaPDF(bool, bool, bool, double, double) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/external/slc7_amd64_gcc900/lib/libpythia8.so
#11 0x00002ba0318db1e7 in Pythia8::PhaseSpace::setupSampling123(bool, bool) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/external/slc7_amd64_gcc900/lib/libpythia8.so
#12 0x00002ba03190f2d0 in Pythia8::ProcessContainer::init(bool, Pythia8::ResonanceDecays*, Pythia8::SLHAinterface*, Pythia8::GammaKinematics*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/external/slc7_amd64_gcc900/lib/libpythia8.so
#13 0x00002ba031928c5b in Pythia8::ProcessLevel::init(bool, Pythia8::SLHAinterface*, std::vector<Pythia8::SigmaProcess*, std::allocator<Pythia8::SigmaProcess*> >&, std::vector<Pythia8::PhaseSpace*, std::allocator<Pythia8::PhaseSpace*> >&) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/external/slc7_amd64_gcc900/lib/libpythia8.so
#14 0x00002ba03194e025 in Pythia8::Pythia::init() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/external/slc7_amd64_gcc900/lib/libpythia8.so
#15 0x00002ba02fdf20f0 in Pythia8Hadronizer::initializeForInternalPartons() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc900/pluginGeneratorInterfacePythia8Filters.so
#16 0x00002ba02fe2e034 in edm::ConcurrentGeneratorFilter<Pythia8Hadronizer, gen::ConcurrentExternalDecayDriver>::initLumi(edm::gen::GenStreamCache<Pythia8Hadronizer, gen::ConcurrentExternalDecayDriver>*, edm::LuminosityBlock const&, edm::EventSetup const&) const () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc900/pluginGeneratorInterfacePythia8Filters.so
#17 0x00002ba001dba720 in edm::global::EDFilterBase::doStreamBeginLuminosityBlock(edm::StreamID, edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#18 0x00002ba001daf280 in edm::WorkerT<edm::global::EDFilterBase>::implDoStreamBegin(edm::StreamID, edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#19 0x00002ba001cacfc2 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#20 0x00002ba001cad20d in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#21 0x00002ba001cad3f0 in edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)1> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#22 0x00002ba001cad56a in tbb::detail::d1::function_task<edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)1> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so

Thread 8 (Thread 0x2ba0502fa700 (LWP 11135) "cmsRun"):
#3  0x00002ba009f077bb in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  std::local_Rb_tree_rotate_left (__root=@0x2ba031f64710: 0x2ba080fcc440, __x=0x2ba04e84ba40) at ../../../../../libstdc++-v3/src/c++98/tree.cc:138
#6  std::_Rb_tree_insert_and_rebalance (__insert_left=<optimized out>, __x=0x2ba05ec01180, __p=<optimized out>, __header=...) at ../../../../../libstdc++-v3/src/c++98/tree.cc:278
#7  0x00002ba031ee316d in LHAPDF::_getQ2CachesMap() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/external/slc7_amd64_gcc900/lib/libLHAPDF.so
#8  0x00002ba031ee322d in LHAPDF::LogBicubicInterpolator::_getCacheQ2(LHAPDF::KnotArray1F const&, double, unsigned long) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/external/slc7_amd64_gcc900/lib/libLHAPDF.so
#9  0x00002ba031ee34a8 in LHAPDF::LogBicubicInterpolator::_interpolateXQ2(LHAPDF::KnotArray1F const&, double, unsigned long, double, unsigned long) const () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/external/slc7_amd64_gcc900/lib/libLHAPDF.so
#10 0x00002ba031ee19ff in LHAPDF::Interpolator::interpolateXQ2(int, double, double) const () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/external/slc7_amd64_gcc900/lib/libLHAPDF.so
#11 0x00002ba031edcadf in LHAPDF::GridPDF::_xfxQ2(int, double, double) const () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/external/slc7_amd64_gcc900/lib/libLHAPDF.so
#12 0x00002ba031ecf807 in LHAPDF::PDF::xfxQ2(int, double, double) const () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/external/slc7_amd64_gcc900/lib/libLHAPDF.so
#13 0x00002ba04d3f28e7 in Pythia8::LHAPDF6::xfUpdate(int, double, double) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/external/slc7_amd64_gcc900/lib/libpythia8lhapdf6.so
#14 0x00002ba0318931b7 in Pythia8::PDF::xf(int, double, double) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/external/slc7_amd64_gcc900/lib/libpythia8.so
#15 0x00002ba031a6d799 in Pythia8::SigmaProcess::sigmaPDF(bool, bool, bool, double, double) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/external/slc7_amd64_gcc900/lib/libpythia8.so
#16 0x00002ba0318db1e7 in Pythia8::PhaseSpace::setupSampling123(bool, bool) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/external/slc7_amd64_gcc900/lib/libpythia8.so
#17 0x00002ba03190f2d0 in Pythia8::ProcessContainer::init(bool, Pythia8::ResonanceDecays*, Pythia8::SLHAinterface*, Pythia8::GammaKinematics*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/external/slc7_amd64_gcc900/lib/libpythia8.so
#18 0x00002ba031928c5b in Pythia8::ProcessLevel::init(bool, Pythia8::SLHAinterface*, std::vector<Pythia8::SigmaProcess*, std::allocator<Pythia8::SigmaProcess*> >&, std::vector<Pythia8::PhaseSpace*, std::allocator<Pythia8::PhaseSpace*> >&) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/external/slc7_amd64_gcc900/lib/libpythia8.so
#19 0x00002ba03194e025 in Pythia8::Pythia::init() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/external/slc7_amd64_gcc900/lib/libpythia8.so
#20 0x00002ba02fdf20f0 in Pythia8Hadronizer::initializeForInternalPartons() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc900/pluginGeneratorInterfacePythia8Filters.so
#21 0x00002ba02fe2e034 in edm::ConcurrentGeneratorFilter<Pythia8Hadronizer, gen::ConcurrentExternalDecayDriver>::initLumi(edm::gen::GenStreamCache<Pythia8Hadronizer, gen::ConcurrentExternalDecayDriver>*, edm::LuminosityBlock const&, edm::EventSetup const&) const () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc900/pluginGeneratorInterfacePythia8Filters.so
#22 0x00002ba001dba720 in edm::global::EDFilterBase::doStreamBeginLuminosityBlock(edm::StreamID, edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#23 0x00002ba001daf280 in edm::WorkerT<edm::global::EDFilterBase>::implDoStreamBegin(edm::StreamID, edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#24 0x00002ba001cacfc2 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#25 0x00002ba001cad20d in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#26 0x00002ba001cad3f0 in edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)1> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#27 0x00002ba001cad56a in tbb::detail::d1::function_task<edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)1> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so

Thread 4 (Thread 0x2ba04a280700 (LWP 11130) "cmsRun"):
#0  CLHEP::MixMaxRng::MULWU (this=this@entry=0x2ba0359f8250, k=k@entry=1839237037704408705) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_12_0_0_pre5-slc7_amd64_gcc900/build/CMSSW_12_0_0_pre5-build/BUILD/slc7_amd64_gcc900/external/clhep/2.4.4.0-llifpc2/clhep-2.4.4.0/Random/src/MixMaxRng.cc:495
#1  0x00002ba00a5f21fb in CLHEP::MixMaxRng::iterate (this=0x2ba0359f8250) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_12_0_0_pre5-slc7_amd64_gcc900/build/CMSSW_12_0_0_pre5-build/BUILD/slc7_amd64_gcc900/external/clhep/2.4.4.0-llifpc2/clhep-2.4.4.0/Random/src/MixMaxRng.cc:312
#2  0x00002ba037133380 in G4UrbanMscModel::SampleScattering(CLHEP::Hep3Vector const&, double) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/biglib/slc7_amd64_gcc900/pluginSimulation.so
#3  0x00002ba03715e5fe in G4VMultipleScattering::AlongStepDoIt(G4Track const&, G4Step const&) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/biglib/slc7_amd64_gcc900/pluginSimulation.so
#4  0x00002ba0374cdeca in G4SteppingManager::InvokeAlongStepDoItProcs() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/biglib/slc7_amd64_gcc900/pluginSimulation.so
#5  0x00002ba0374cc3a7 in G4SteppingManager::Stepping() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/biglib/slc7_amd64_gcc900/pluginSimulation.so
#6  0x00002ba036fa8d98 in G4TrackingManager::ProcessOneTrack(G4Track*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/biglib/slc7_amd64_gcc900/pluginSimulation.so
#7  0x00002ba036c77d9b in G4EventManager::DoProcessing(G4Event*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/biglib/slc7_amd64_gcc900/pluginSimulation.so
#8  0x00002ba0368ecfbc in RunManagerMTWorker::produce(edm::Event const&, edm::EventSetup const&, RunManagerMT&) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/biglib/slc7_amd64_gcc900/pluginSimulation.so
#9  0x00002ba03690636f in omt::ThreadHandoff::Functor<OscarMTProducer::produce(edm::Event&, edm::EventSetup const&)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/biglib/slc7_amd64_gcc900/pluginSimulation.so
#10 0x00002ba0368fd6ea in omt::ThreadHandoff::threadLoop(void*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/biglib/slc7_amd64_gcc900/pluginSimulation.so

Thread 1 (Thread 0x2ba0064c3980 (LWP 11056) "cmsRun"):
#2  0x00002ba009f033f0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00002ba0043880f6 in ____strtod_l_internal () from /lib64/libc.so.6
#5  0x00002ba031edd050 in LHAPDF::GridPDF::_loadData(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/external/slc7_amd64_gcc900/lib/libLHAPDF.so
#6  0x00002ba031eef9dc in LHAPDF::mkPDF(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/external/slc7_amd64_gcc900/lib/libLHAPDF.so
#7  0x00002ba04d3f445d in Pythia8::LHAPDF6::init(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/external/slc7_amd64_gcc900/lib/libpythia8lhapdf6.so
#8  0x00002ba04d3f480f in newPDF () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/external/slc7_amd64_gcc900/lib/libpythia8lhapdf6.so
#9  0x00002ba0318aab68 in Pythia8::LHAPDF::LHAPDF(int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, Pythia8::Info*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/external/slc7_amd64_gcc900/lib/libpythia8.so
#10 0x00002ba03197081c in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<Pythia8::LHAPDF, std::allocator<Pythia8::LHAPDF>, int&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, Pythia8::Info*>(Pythia8::LHAPDF*&, std::_Sp_alloc_shared_tag<std::allocator<Pythia8::LHAPDF> >, int&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, Pythia8::Info*&&) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/external/slc7_amd64_gcc900/lib/libpythia8.so
#11 0x00002ba03193e114 in Pythia8::Pythia::getPDFPtr(int, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/external/slc7_amd64_gcc900/lib/libpythia8.so
#12 0x00002ba0319423e1 in Pythia8::Pythia::initPDFs() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/external/slc7_amd64_gcc900/lib/libpythia8.so
#13 0x00002ba03194dce2 in Pythia8::Pythia::init() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/external/slc7_amd64_gcc900/lib/libpythia8.so
#14 0x00002ba02fdf20f0 in Pythia8Hadronizer::initializeForInternalPartons() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc900/pluginGeneratorInterfacePythia8Filters.so
#15 0x00002ba02fe2e034 in edm::ConcurrentGeneratorFilter<Pythia8Hadronizer, gen::ConcurrentExternalDecayDriver>::initLumi(edm::gen::GenStreamCache<Pythia8Hadronizer, gen::ConcurrentExternalDecayDriver>*, edm::LuminosityBlock const&, edm::EventSetup const&) const () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc900/pluginGeneratorInterfacePythia8Filters.so
#16 0x00002ba001dba720 in edm::global::EDFilterBase::doStreamBeginLuminosityBlock(edm::StreamID, edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#17 0x00002ba001daf280 in edm::WorkerT<edm::global::EDFilterBase>::implDoStreamBegin(edm::StreamID, edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#18 0x00002ba001cacfc2 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#19 0x00002ba001cad20d in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#20 0x00002ba001cad3f0 in edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)1> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so

that points to a race condition in LHAPDF that is discussed in https://gitlab.com/hepcedar/lhapdf/-/issues/2.

makortel commented 3 years ago

assign core, generators

cmsbuild commented 3 years ago

New categories assigned: core,generators

@Dr15Jones,@smuzaffar,@Saptaparna,@mkirsano,@SiewYan,@alberto-sanchez,@makortel,@agrohsje,@GurpreetSinghChahal you have been requested to review this Pull request/Issue and eventually sign? Thanks

cmsbuild commented 3 years ago

A new Issue was created by @makortel Matti Kortelainen.

@Dr15Jones, @perrotta, @dpiparo, @makortel, @smuzaffar, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

makortel commented 3 years ago

Do we want to try patch https://gitlab.com/hepcedar/lhapdf/-/merge_requests/3 ? I'm not really sure how we could test it conclusively, given that it took ~1.5 months to hit to the issue after the update to LHAPDF 6.3.0.

agrohsje commented 3 years ago

Hi @mkirsano : I would agree with @makortel and apply the proposed patch.

mkirsano commented 3 years ago

 Hello, does it mean to put lhapdf code to our repo?

 If it is needed for CMS and it will work, why not.

 Then probably we have to write to authors about it.

On 9/7/21 5:12 PM, agrohsje wrote:

Hi @mkirsano https://github.com/mkirsano : I would agree with @makortel https://github.com/makortel and apply the proposed patch.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cms-sw/cmssw/issues/35082#issuecomment-914394276, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABGOJUVCW4SNU34FS3E2PALUAYTWPANCNFSM5DDBTJNA. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

makortel commented 3 years ago

Apparently the authors released a beta of 6.4.0 that eliminates the cache (I did not look into details), see https://gitlab.com/hepcedar/lhapdf/-/issues/2#note_674891431. Should we try that?

makortel commented 3 years ago

If https://github.com/cms-sw/cmssw/issues/35251 is connected, we just got a way to test rather conclusively if the beta 6.4.0 would fix the race conditions in our case.

Dr15Jones commented 3 years ago

It happened here as well

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc900/CMSSW_12_1_ROOT624_X_2021-09-22-2300/pyRelValMatrixLogs/run/39034.0_TTbar_14TeV+2026D87+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger/step1_TTbar_14TeV+2026D87+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger.log#/599-599

#3  0x00002b69b6b287bb in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/nweek-02699/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-22-2300/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  std::local_Rb_tree_rotate_left (__root=@0x2b69e142c750: 0x2b6a27010360, __x=0x2b69fd8e4100) at ../../../../../libstdc++-v3/src/c++98/tree.cc:138
#6  std::_Rb_tree_insert_and_rebalance (__insert_left=<optimized out>, __x=0x2b6a2697c660, __p=<optimized out>, __header=...) at ../../../../../libstdc++-v3/src/c++98/tree.cc:278
#7  0x00002b69e13aad9d in LHAPDF::_getXCachesMap() () from /cvmfs/cms-ib.cern.ch/nweek-02699/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-22-2300/external/slc7_amd64_gcc900/lib/libLHAPDF.so
#8  0x00002b69e13aae5d in LHAPDF::LogBicubicInterpolator::_getCacheX(LHAPDF::KnotArray1F const&, double, unsigned long) () from /cvmfs/cms-ib.cern.ch/nweek-02699/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-22-2300/external/slc7_amd64_gcc900/lib/libLHAPDF.so
#9  0x00002b69e13ab48f in LHAPDF::LogBicubicInterpolator::_interpolateXQ2(LHAPDF::KnotArray1F const&, double, unsigned long, double, unsigned long) const () from /cvmfs/cms-ib.cern.ch/nweek-02699/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-22-2300/external/slc7_amd64_gcc900/lib/libLHAPDF.so
#10 0x00002b69e13a99ff in LHAPDF::Interpolator::interpolateXQ2(int, double, double) const () from /cvmfs/cms-ib.cern.ch/nweek-02699/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-22-2300/external/slc7_amd64_gcc900/lib/libLHAPDF.so
#11 0x00002b69e13a4adf in LHAPDF::GridPDF::_xfxQ2(int, double, double) const () from /cvmfs/cms-ib.cern.ch/nweek-02699/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-22-2300/external/slc7_amd64_gcc900/lib/libLHAPDF.so
#12 0x00002b69e1397807 in LHAPDF::PDF::xfxQ2(int, double, double) const () from /cvmfs/cms-ib.cern.ch/nweek-02699/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-22-2300/external/slc7_amd64_gcc900/lib/libLHAPDF.so
#13 0x00002b69ff3ab8e7 in Pythia8::LHAPDF6::xfUpdate(int, double, double) () from /cvmfs/cms-ib.cern.ch/nweek-02699/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-22-2300/external/slc7_amd64_gcc900/lib/libpythia8lhapdf6.so
#14 0x00002b69e0d5b1b7 in Pythia8::PDF::xf(int, double, double) () from /cvmfs/cms-ib.cern.ch/nweek-02699/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-22-2300/external/slc7_amd64_gcc900/lib/libpythia8.so
#15 0x00002b69e0f35799 in Pythia8::SigmaProcess::sigmaPDF(bool, bool, bool, double, double) () from /cvmfs/cms-ib.cern.ch/nweek-02699/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-22-2300/external/slc7_amd64_gcc900/lib/libpythia8.so
#16 0x00002b69e0da31e7 in Pythia8::PhaseSpace::setupSampling123(bool, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02699/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-22-2300/external/slc7_amd64_gcc900/lib/libpythia8.so
#17 0x00002b69e0dd72d0 in Pythia8::ProcessContainer::init(bool, Pythia8::ResonanceDecays*, Pythia8::SLHAinterface*, Pythia8::GammaKinematics*) () from /cvmfs/cms-ib.cern.ch/nweek-02699/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-22-2300/external/slc7_amd64_gcc900/lib/libpythia8.so
#18 0x00002b69e0df0c5b in Pythia8::ProcessLevel::init(bool, Pythia8::SLHAinterface*, std::vector<Pythia8::SigmaProcess*, std::allocator<Pythia8::SigmaProcess*> >&, std::vector<Pythia8::PhaseSpace*, std::allocator<Pythia8::PhaseSpace*> >&) () from /cvmfs/cms-ib.cern.ch/nweek-02699/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-22-2300/external/slc7_amd64_gcc900/lib/libpythia8.so
#19 0x00002b69e0e16025 in Pythia8::Pythia::init() () from /cvmfs/cms-ib.cern.ch/nweek-02699/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-22-2300/external/slc7_amd64_gcc900/lib/libpythia8.so
#20 0x00002b69df2ba0f0 in Pythia8Hadronizer::initializeForInternalPartons() () from /cvmfs/cms-ib.cern.ch/nweek-02699/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-22-2300/lib/slc7_amd64_gcc900/pluginGeneratorInterfacePythia8Filters.so
#21 0x00002b69df2f6034 in edm::ConcurrentGeneratorFilter<Pythia8Hadronizer, gen::ConcurrentExternalDecayDriver>::initLumi(edm::gen::GenStreamCache<Pythia8Hadronizer, gen::ConcurrentExternalDecayDriver>*, edm::LuminosityBlock const&, edm::EventSetup const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02699/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-22-2300/lib/slc7_amd64_gcc900/pluginGeneratorInterfacePythia8Filters.so
#22 0x00002b69ae8757c0 in edm::global::EDFilterBase::doStreamBeginLuminosityBlock(edm::StreamID, edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02699/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_ROOT624_X_2021-09-22-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so

...

Current Modules:

Module: Pythia8ConcurrentGeneratorFilter:generator (crashed)
Module: Pythia8ConcurrentGeneratorFilter:generator
Module: OscarMTProducer:g4SimHits
Module: Pythia8ConcurrentGeneratorFilter:generator

A fatal system signal has occurred: segmentation violation
agrohsje commented 3 years ago

Dear @mkirsano, @SiewYan , lhapdf 6.4.0 including NNPDF 4.0 is now out. I would suggest to update cmsdist. Are you available? This allows us to study the potential PDFs for Run 3 as well as a possible answer to the problem of thread safety.

mkirsano commented 3 years ago

Yes, OK

makortel commented 2 months ago

Given that we have updated LHAPDF to 6.4.0 that (presumably) eliminated the cache, and that we don't have any reports from past few years (although that doesn't mean they couldn't have occurred), maybe we should close this issue?

cmsbuild commented 2 months ago

cms-bot internal usage