search

cms-sw / cmssw

CMS Offline Software
http://cms-sw.github.io/
Apache License 2.0
1.07k stars 4.28k forks source link

DQM/Phase2OuterTracker ParameterSet memory management #22182

Closed dan131riley closed 5 years ago

dan131riley commented 6 years ago

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc630/CMSSW_10_1_X_2018-02-11-1100/pyRelValMatrixLogs/run/20834.11_TTbar_14TeV_Timing+TTbar_14TeV_TuneCUETP8M1_2023D20_GenSimHLBeamSpotFull14_Timing+DigiFull_Timing_2023D20+RecoFullGlobal_Timing_2023D20+HARVESTFullGlobal_Timing_2023D20/step3_TTbar_14TeV_Timing+TTbar_14TeV_TuneCUETP8M1_2023D20_GenSimHLBeamSpotFull14_Timing+DigiFull_Timing_2023D20+RecoFullGlobal_Timing_2023D20+HARVESTFullGlobal_Timing_2023D20.log

There are a bunch of instances of

edm::ParameterSet psTTStub_Barrel_XY =  conf_.getParameter<edm::ParameterSet("TH2TTStub_Position");

https://github.com/cms-sw/cmssw/blob/master/DQM/Phase2OuterTracker/plugins/OuterTrackerMonitorTTStub.cc#L177

that I think ought to be const edm::ParameterSet&

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thread 6 (Thread 0x7f4ac40ff700 (LWP 15316)):
#0  0x000000318f20b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f4b23a5353c in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /build/cmsbld/gcc6oldabi/cms/BUILD/slc6_amd64_gcc630/external/gcc/6.3.0-cms/gcc-tags_gcc_6_3_0_release-243837/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/x86_64-unknown-linux-gnu/bits/gthr-default.h:864
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00007f4aca263fcd in Eigen::NonBlockingThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc630/cms/cmssw-patch/CMSSW_10_1_X_2018-02-11-1100/external/slc6_amd64_gcc630/lib/libtensorflow_cc.so
#4  0x00007f4aca264a29 in Eigen::NonBlockingThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc630/cms/cmssw-patch/CMSSW_10_1_X_2018-02-11-1100/external/slc6_amd64_gcc630/lib/libtensorflow_cc.so
#5  0x00007f4aca262847 in std::_Function_handler<void (), tensorflow::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc630/cms/cmssw-patch/CMSSW_10_1_X_2018-02-11-1100/external/slc6_amd64_gcc630/lib/libtensorflow_cc.so
#6  0x00007f4b23a58ecf in std::execute_native_thread_routine (__p=0x7f4ac4109830) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#7  0x000000318f207aa1 in start_thread () from /lib64/libpthread.so.0
#8  0x000000318eee8bcd in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f4afaaff700 (LWP 15312)):
#0  0x000000318eeacc4d in nanosleep () from /lib64/libc.so.6
#1  0x000000318eeacac0 in sleep () from /lib64/libc.so.6
#2  0x00007f4b16316c9e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02511/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-11-0000/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000000318f20e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#5  0x000000318f2095d8 in _L_lock_854 () from /lib64/libpthread.so.0
#6  0x000000318f2094a7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#7  0x00007f4aeea62e61 in DQMGlobalEDAnalyzer<(anonymous namespace)::Histograms>::globalBeginRun () from /cvmfs/cms-ib.cern.ch/nweek-02511/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-11-0000/lib/slc6_amd64_gcc630/pluginValidationDTRecHitsPlugins.so
#8  0x00007f4aeea6316a in edm::global::impl::RunCacheHolder<edm::global::EDAnalyzerBase, (anonymous namespace)::Histograms>::doBeginRun_ () from /cvmfs/cms-ib.cern.ch/nweek-02511/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-11-0000/lib/slc6_amd64_gcc630/pluginValidationDTRecHitsPlugins.so
#9  0x00007f4b25647f3b in edm::global::EDAnalyzerBase::doBeginRun(edm::RunPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02511/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-11-0000/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#10 0x00007f4b25584270 in edm::WorkerT<edm::global::EDAnalyzerBase>::implDoBegin(edm::RunPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02511/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-11-0000/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#11 0x00007f4b255564e6 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02511/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-11-0000/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#12 0x00007f4b2555675c in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02511/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-11-0000/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#13 0x00007f4b25556a2b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02511/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-11-0000/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#14 0x00007f4b25556c1e in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02511/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-11-0000/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#15 0x00007f4b2417142c in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f4b23067e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:509
#16 0x00007f4b24169c14 in tbb::internal::arena::process (this=0x7f4b23156d00, s=...) at ../../src/tbb/arena.cpp:160
#17 0x00007f4b2416874b in tbb::internal::market::process (this=0x7f4b23157580, j=...) at ../../src/tbb/market.cpp:693
#18 0x00007f4b241649f6 in tbb::internal::rml::private_worker::run (this=0x7f4b22f79080) at ../../src/tbb/private_server.cpp:270
#19 0x00007f4b24164c29 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#20 0x000000318f207aa1 in start_thread () from /lib64/libpthread.so.0
#21 0x000000318eee8bcd in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f4afb57e700 (LWP 15311)):
#0  0x000000318eeacc4d in nanosleep () from /lib64/libc.so.6
#1  0x000000318eeacac0 in sleep () from /lib64/libc.so.6
#2  0x00007f4b16316c9e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02511/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-11-0000/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000000318ee842fe in __memset_sse2 () from /lib64/libc.so.6
#5  0x00007f4b248af4f0 in TArrayF::Set(int) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc630/cms/cmssw-patch/CMSSW_10_1_X_2018-02-11-1100/external/slc6_amd64_gcc630/lib/libCore.so
#6  0x00007f4b1615fb2d in TH2F::TH2F(char const*, char const*, int, double, double, int, double, double) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc630/cms/cmssw-patch/CMSSW_10_1_X_2018-02-11-1100/external/slc6_amd64_gcc630/lib/libHist.so
#7  0x00007f4b15079035 in DQMStore::book2D(std::string const&, std::string const&, int, double, double, int, double, double) () from /cvmfs/cms-ib.cern.ch/nweek-02511/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-11-0000/lib/slc6_amd64_gcc630/libDQMServicesCore.so
#8  0x00007f4aeea5c5d9 in HRes1DHit::HRes1DHit(std::string const&, DQMStore::ConcurrentBooker&, bool, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02511/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-11-0000/lib/slc6_amd64_gcc630/pluginValidationDTRecHitsPlugins.so
#9  0x00007f4aeea577e4 in DTRecHitQuality::bookHistograms(DQMStore::ConcurrentBooker&, edm::Run const&, edm::EventSetup const&, (anonymous namespace)::Histograms&) const () from /cvmfs/cms-ib.cern.ch/nweek-02511/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-11-0000/lib/slc6_amd64_gcc630/pluginValidationDTRecHitsPlugins.so
#10 0x00007f4aeea59ecb in DQMGlobalEDAnalyzer<(anonymous namespace)::Histograms>::globalBeginRun(edm::Run const&, edm::EventSetup const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02511/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-11-0000/lib/slc6_amd64_gcc630/pluginValidationDTRecHitsPlugins.so
#11 0x00007f4aeea5a19a in edm::global::impl::RunCacheHolder<edm::global::EDAnalyzerBase, (anonymous namespace)::Histograms>::doBeginRun_(edm::Run const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02511/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-11-0000/lib/slc6_amd64_gcc630/pluginValidationDTRecHitsPlugins.so
#12 0x00007f4b25647f3b in edm::global::EDAnalyzerBase::doBeginRun(edm::RunPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02511/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-11-0000/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#13 0x00007f4b25584270 in edm::WorkerT<edm::global::EDAnalyzerBase>::implDoBegin(edm::RunPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02511/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-11-0000/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#14 0x00007f4b255564e6 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02511/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-11-0000/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#15 0x00007f4b2555675c in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02511/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-11-0000/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#16 0x00007f4b25556a2b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02511/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-11-0000/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#17 0x00007f4b25556c1e in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02511/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-11-0000/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#18 0x00007f4b2417142c in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f4b23077e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:509
#19 0x00007f4b24169c14 in tbb::internal::arena::process (this=0x7f4b23156d00, s=...) at ../../src/tbb/arena.cpp:160
#20 0x00007f4b2416874b in tbb::internal::market::process (this=0x7f4b23157580, j=...) at ../../src/tbb/market.cpp:693
#21 0x00007f4b241649f6 in tbb::internal::rml::private_worker::run (this=0x7f4b22f79180) at ../../src/tbb/private_server.cpp:270
#22 0x00007f4b24164c29 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#23 0x000000318f207aa1 in start_thread () from /lib64/libpthread.so.0
#24 0x000000318eee8bcd in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f4afbf7f700 (LWP 15310)):
#0  0x000000318eedf383 in poll () from /lib64/libc.so.6
#1  0x00007f4b16317234 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/nweek-02511/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-11-0000/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#2  0x00007f4b16317b8a in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/nweek-02511/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-11-0000/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  0x00007f4b16318db6 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/nweek-02511/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-11-0000/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007f4b257a76f0 in std::_Rb_tree<std::string, std::pair<std::string const, edm::Entry>, std::_Select1st<std::pair<std::string const, edm::Entry> >, std::less<std::string>, std::allocator<std::pair<std::string const, edm::Entry> > >::_M_erase(std::_Rb_tree_node<std::pair<std::string const, edm::Entry> >*) () from /cvmfs/cms-ib.cern.ch/nweek-02511/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-11-0000/lib/slc6_amd64_gcc630/libFWCoreParameterSet.so
#6  0x00007f4b257a76fc in std::_Rb_tree<std::string, std::pair<std::string const, edm::Entry>, std::_Select1st<std::pair<std::string const, edm::Entry> >, std::less<std::string>, std::allocator<std::pair<std::string const, edm::Entry> > >::_M_erase(std::_Rb_tree_node<std::pair<std::string const, edm::Entry> >*) () from /cvmfs/cms-ib.cern.ch/nweek-02511/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-11-0000/lib/slc6_amd64_gcc630/libFWCoreParameterSet.so
#7  0x00007f4b2579ecf2 in edm::ParameterSet::~ParameterSet() () from /cvmfs/cms-ib.cern.ch/nweek-02511/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-11-0000/lib/slc6_amd64_gcc630/libFWCoreParameterSet.so
#8  0x00007f4ae6f836ed in OuterTrackerMonitorTTStub::beginRun(edm::Run const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02511/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-11-0000/lib/slc6_amd64_gcc630/pluginDQMPhase2OuterTrackerAuto.so
#9  0x00007f4b255359d3 in edm::EDAnalyzer::doBeginRun(edm::RunPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02511/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-11-0000/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#10 0x00007f4b25583af0 in edm::WorkerT<edm::EDAnalyzer>::implDoBegin(edm::RunPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02511/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-11-0000/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#11 0x00007f4b255564e6 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02511/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-11-0000/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#12 0x00007f4b2555675c in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02511/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-11-0000/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#13 0x00007f4b25556a2b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02511/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-11-0000/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#14 0x00007f4b25556ef8 in void edm::SerialTaskQueueChain::actionToRun<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0> >::execute()::{lambda()#1}&>(edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0> >::execute()::{lambda()#1}&) () from /cvmfs/cms-ib.cern.ch/nweek-02511/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-11-0000/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#15 0x00007f4b25556f81 in edm::SerialTaskQueue::QueuedTask<void edm::SerialTaskQueueChain::push<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0> >::execute()::{lambda()#1}&>(edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0> >::execute()::{lambda()#1}&)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02511/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-11-0000/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#16 0x00007f4b2417142c in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f4b2306fe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:509
#17 0x00007f4b24169c14 in tbb::internal::arena::process (this=0x7f4b23156d00, s=...) at ../../src/tbb/arena.cpp:160
#18 0x00007f4b2416874b in tbb::internal::market::process (this=0x7f4b23157580, j=...) at ../../src/tbb/market.cpp:693
#19 0x00007f4b241649f6 in tbb::internal::rml::private_worker::run (this=0x7f4b22f79100) at ../../src/tbb/private_server.cpp:270
#20 0x00007f4b24164c29 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#21 0x000000318f207aa1 in start_thread () from /lib64/libpthread.so.0
#22 0x000000318eee8bcd in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f4b095bf700 (LWP 15302)):
#0  0x000000318f20f37d in waitpid () from /lib64/libpthread.so.0
#1  0x00007f4b16316ea7 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/nweek-02511/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-11-0000/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#2  0x00007f4b16317a95 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/nweek-02511/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-11-0000/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  0x00007f4b23a58ecf in std::execute_native_thread_routine (__p=0x7f4b09cb7f00) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#4  0x000000318f207aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x000000318eee8bcd in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f4b2370bc00 (LWP 15220)):
#0  0x000000318eeacc4d in nanosleep () from /lib64/libc.so.6
#1  0x000000318eeacac0 in sleep () from /lib64/libc.so.6
#2  0x00007f4b16316c9e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02511/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-11-0000/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f4af0a245f0 in edm::stream::ProducingModuleAdaptor<edm::FwdPtrCollectionFilter<reco::PFCandidate, reco::PdgIdSelectorHandler, reco::PFCandidateWithSrcPtrFactory>, edm::stream::EDFilterBase, edm::stream::EDFilterAdaptorBase>::wantsGlobalRuns() const () from /cvmfs/cms-ib.cern.ch/nweek-02511/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-11-0000/lib/slc6_amd64_gcc630/pluginCommonToolsParticleFlow_plugins.so
#5  0x00007f4b2554244b in void edm::GlobalSchedule::processOneGlobalAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0>::MyPrincipal&, edm::EventSetup const&, edm::ServiceToken const&, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02511/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-11-0000/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#6  0x00007f4b25542789 in void edm::beginGlobalTransitionAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)0>, edm::RunPrincipal, std::vector<edm::SubProcess, std::allocator<edm::SubProcess> > >(edm::WaitingTaskHolder, edm::Schedule&, edm::RunPrincipal&, edm::IOVSyncValue const&, edm::EventSetup const&, edm::ServiceToken const&, std::vector<edm::SubProcess, std::allocator<edm::SubProcess> >&) () from /cvmfs/cms-ib.cern.ch/nweek-02511/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-11-0000/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#7  0x00007f4b255e4203 in edm::EventProcessor::beginRun(edm::Hash<2> const&, unsigned int, bool&) () from /cvmfs/cms-ib.cern.ch/nweek-02511/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-11-0000/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#8  0x00007f4b255e4b82 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/nweek-02511/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-11-0000/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#9  0x000000000040da34 in main::{lambda()#1}::operator()() const ()
#10 0x000000000040c372 in main ()

Current Modules:

Module: OuterTrackerMonitorTTStub:OuterTrackerMonitorTTStub (crashed)
Module: DTRecHitQuality:rechivalidation
Module: DTSegment2DSLPhiQuality:seg2dsuperphivalidation
Module: none

A fatal system signal has occurred: segmentation violation
cmsbuild commented 6 years ago

A new Issue was created by @dan131riley Dan Riley.

@davidlange6, @Dr15Jones, @smuzaffar, @fabiocos can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

Dr15Jones commented 6 years ago

assign dqm

cmsbuild commented 6 years ago

New categories assigned: dqm

@kmaeshima,@vanbesien,@jfernan2,@vazzolini,@dmitrijus you have been requested to review this Pull request/Issue and eventually sign? Thanks

Dr15Jones commented 6 years ago

The copy of a ParameterSet is inefficient, but should not lead to a crash.

dan131riley commented 6 years ago

After digging into the implementation, I agree it shouldn't lead to a crash, even with the old ABI COW strings it ought to be safe. Will have to try memcheck on it.

dan131riley commented 6 years ago

And to clarify, I'd change

edm::ParameterSet psTTStub_Barrel_XY =  conf_.getParameter<edm::ParameterSet("TH2TTStub_Position");

to

const edm::ParameterSet& psTTStub_Barrel_XY =  conf_.getParameterSet("TH2TTStub_Position");

since there isn't a const& version of the templated getParameter<>

schneiml commented 5 years ago

@dan131riley , @Dr15Jones is there anything to do on our (DQM) side? It seems the problem was not fully understood from the core side.

dan131riley commented 5 years ago

We did conclude that OuterTrackerMonitorTTStub::beginRun was not responsible for the crash, so I'll close this ticket. The inefficient copying of ParameterSets is still there in OuterTrackerMonitorTTStub::bookHistograms, but that only runs once per run, so I guess it's not worth keeping the ticket open.