search

cms-sw / cmssw

CMS Offline Software
http://cms-sw.github.io/
Apache License 2.0
1.08k stars 4.3k forks source link

SEGV on unknown address 0x000000000000, RunManagerMT #21359

Closed davidlt closed 6 years ago

davidlt commented 6 years ago

CMSSW: CMSSW_10_0_ASAN_X_2017-11-12-1100 SCRAM_ARCH: slc6_amd64_gcc700 Workflow: 500199.0 Step: step1

Detailed report:

Internal FlatRandomEGun is initialzed
ASAN:DEADLYSIGNAL
=================================================================
==14731==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7fe8d15d85a3 bp 0x7fe8d039beb0 sp 0x7fe8d039bea0 T2)
==14731==The signal is caused by a READ memory access.
==14731==Hint: address points to the zero page.
    #0 0x7fe8d15d85a2 in RunManagerMT::terminateRun() /build/davidlt/CMSSW_10_0_ASAN_X_2017-11-12-1100/src/SimG4Core/Application/src/RunManagerMT.cc:254
    #1 0x7fe8d15d5866 in RunManagerMT::~RunManagerMT() /build/davidlt/CMSSW_10_0_ASAN_X_2017-11-12-1100/src/SimG4Core/Application/src/RunManagerMT.cc:92
    #2 0x7fe8d164f8d3 in void __gnu_cxx::new_allocator<RunManagerMT>::destroy<RunManagerMT>(RunManagerMT*) /cvmfs/cms-ib.cern.ch/nweek-02498/slc6_amd64_gcc700/external/gcc/7.0.0-mmelna/include/c++/7.2.1/ext/new_allocator.h:140
    #3 0x7fe8d164f8a6 in void std::allocator_traits<std::allocator<RunManagerMT> >::destroy<RunManagerMT>(std::allocator<RunManagerMT>&, RunManagerMT*) /cvmfs/cms-ib.cern.ch/nweek-02498/slc6_amd64_gcc700/external/gcc/7.0.0-mmelna/include/c++/7.2.1/bits/alloc_traits.h:487
    #4 0x7fe8d164f6a8 in std::_Sp_counted_ptr_inplace<RunManagerMT, std::allocator<RunManagerMT>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() /cvmfs/cms-ib.cern.ch/nweek-02498/slc6_amd64_gcc700/external/gcc/7.0.0-mmelna/include/c++/7.2.1/bits/shared_ptr_base.h:535
    #5 0x7fe903d9e2bb in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-11-12-1100/lib/slc6_amd64_gcc700/libDataFormatsCommon.so+0x512bb)
    #6 0x7fe8d164d60d in std::__shared_ptr<RunManagerMT, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() /cvmfs/cms-ib.cern.ch/nweek-02498/slc6_amd64_gcc700/external/gcc/7.0.0-mmelna/include/c++/7.2.1/bits/shared_ptr_base.h:1123
    #7 0x7fe8d164def9 in std::__shared_ptr<RunManagerMT, (__gnu_cxx::_Lock_policy)2>::reset() /cvmfs/cms-ib.cern.ch/nweek-02498/slc6_amd64_gcc700/external/gcc/7.0.0-mmelna/include/c++/7.2.1/bits/shared_ptr_base.h:1235
    #8 0x7fe8d15dfc78 in operator() /build/davidlt/CMSSW_10_0_ASAN_X_2017-11-12-1100/src/SimG4Core/Application/src/OscarMTMasterThread.cc:112
    #9 0x7fe8d15e30a7 in __invoke_impl<void, OscarMTMasterThread::OscarMTMasterThread(const edm::ParameterSet&)::<lambda()> > /cvmfs/cms-ib.cern.ch/nweek-02498/slc6_amd64_gcc700/external/gcc/7.0.0-mmelna/include/c++/7.2.1/bits/invoke.h:60
    #10 0x7fe8d15e2c62 in __invoke<OscarMTMasterThread::OscarMTMasterThread(const edm::ParameterSet&)::<lambda()> > /cvmfs/cms-ib.cern.ch/nweek-02498/slc6_amd64_gcc700/external/gcc/7.0.0-mmelna/include/c++/7.2.1/bits/invoke.h:95
    #11 0x7fe8d15e3483 in _M_invoke<0> /cvmfs/cms-ib.cern.ch/nweek-02498/slc6_amd64_gcc700/external/gcc/7.0.0-mmelna/include/c++/7.2.1/thread:234
    #12 0x7fe8d15e3454 in operator() /cvmfs/cms-ib.cern.ch/nweek-02498/slc6_amd64_gcc700/external/gcc/7.0.0-mmelna/include/c++/7.2.1/thread:243
    #13 0x7fe8d15e3433 in _M_run /cvmfs/cms-ib.cern.ch/nweek-02498/slc6_amd64_gcc700/external/gcc/7.0.0-mmelna/include/c++/7.2.1/thread:186
    #14 0x7fe901e45c7e in execute_native_thread_routine ../../../../../libstdc++-v3/src/c++11/thread.cc:83
    #15 0x326f807aa0 in start_thread (/lib64/libpthread.so.0+0x326f807aa0)
    #16 0x326f0e8bcc in __clone (/lib64/libc.so.6+0x326f0e8bcc)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /build/davidlt/CMSSW_10_0_ASAN_X_2017-11-12-1100/src/SimG4Core/Application/src/RunManagerMT.cc:254 in RunManagerMT::terminateRun()
Thread T2 created by T0 here:
    #0 0x7fe90463d210 in __interceptor_pthread_create ../../../../libsanitizer/asan/asan_interceptors.cc:243
    #1 0x7fe901e45f74 in __gthread_create /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc700/external/gcc/7.0.0-mmelna/gcc-branches_gcc-7-branch-254409/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/x86_64-unknown-linux-gnu/bits/gthr-default.h:662
    #2 0x7fe901e45f74 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) ../../../../../libstdc++-v3/src/c++11/thread.cc:163
    #3 0x7fe8d15e07c6 in OscarMTMasterThread::OscarMTMasterThread(edm::ParameterSet const&) /build/davidlt/CMSSW_10_0_ASAN_X_2017-11-12-1100/src/SimG4Core/Application/src/OscarMTMasterThread.cc:38
    #4 0x7fe8d1585e11 in OscarMTProducer::initializeGlobalCache(edm::ParameterSet const&) /build/davidlt/CMSSW_10_0_ASAN_X_2017-11-12-1100/src/SimG4Core/Application/plugins/OscarMTProducer.cc:139
    #5 0x7fe8d15b1194 in std::unique_ptr<OscarMTMasterThread, std::default_delete<OscarMTMasterThread> > edm::stream::impl::makeGlobal<OscarMTProducer, OscarMTMasterThread>(edm::ParameterSet const&, OscarMTMasterThread const*) (/build/davidlt/CMSSW_10_0_ASAN_X_2017-11-12-1100/biglib/slc6_amd64_gcc700/pluginSimulation.so+0xde4194)
    #6 0x7fe8d15b0b23 in edm::stream::ProducingModuleAdaptor<OscarMTProducer, edm::stream::EDProducerBase, edm::stream::EDProducerAdaptorBase>::ProducingModuleAdaptor(edm::ParameterSet const&) (/build/davidlt/CMSSW_10_0_ASAN_X_2017-11-12-1100/biglib/slc6_amd64_gcc700/pluginSimulation.so+0xde3b23)
    #7 0x7fe8d15b03a7 in std::_MakeUniq<edm::stream::ProducingModuleAdaptor<OscarMTProducer, edm::stream::EDProducerBase, edm::stream::EDProducerAdaptorBase> >::__single_object std::make_unique<edm::stream::ProducingModuleAdaptor<OscarMTProducer, edm::stream::EDProducerBase, edm::stream::EDProducerAdaptorBase>, edm::ParameterSet const&>(edm::ParameterSet const&) (/build/davidlt/CMSSW_10_0_ASAN_X_2017-11-12-1100/biglib/slc6_amd64_gcc700/pluginSimulation.so+0xde33a7)
    #8 0x7fe8d15afc35 in std::unique_ptr<edm::stream::EDProducerAdaptorBase, std::default_delete<edm::stream::EDProducerAdaptorBase> > edm::MakeModuleHelper<edm::stream::EDProducerAdaptorBase>::makeModule<OscarMTProducer>(edm::ParameterSet const&) (/build/davidlt/CMSSW_10_0_ASAN_X_2017-11-12-1100/biglib/slc6_amd64_gcc700/pluginSimulation.so+0xde2c35)
    #9 0x7fe8d15af73e in edm::WorkerMaker<OscarMTProducer>::makeModule(edm::ParameterSet const&) const (/build/davidlt/CMSSW_10_0_ASAN_X_2017-11-12-1100/biglib/slc6_amd64_gcc700/pluginSimulation.so+0xde273e)
    #10 0x7fe904172925 in edm::Maker::makeModule(edm::MakeModuleParams const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-11-12-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so+0x362925)
    #11 0x7fe90405d9da in edm::Factory::makeModule(edm::MakeModuleParams const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-11-12-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so+0x24d9da)
    #12 0x7fe90439ccfb in edm::ModuleRegistry::getModule(edm::MakeModuleParams const&, std::string const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-11-12-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so+0x58ccfb)
    #13 0x7fe903f9448f in edm::WorkerRegistry::getWorker(edm::WorkerParams const&, std::string const&) (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-11-12-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so+0x18448f)
    #14 0x7fe9041db189 in edm::WorkerManager::getWorker(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, std::string const&) (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-11-12-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so+0x3cb189)
    #15 0x7fe904123339 in edm::StreamSchedule::fillWorkers(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, std::string const&, bool, std::vector<edm::WorkerInPath, std::allocator<edm::WorkerInPath> >&, std::vector<std::string, std::allocator<std::string> > const&) (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-11-12-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so+0x313339)
    #16 0x7fe90412639f in edm::StreamSchedule::fillTrigPath(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, int, std::string const&, std::shared_ptr<edm::HLTGlobalStatus>, std::vector<std::string, std::allocator<std::string> > const&) (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-11-12-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so+0x31639f)
    #17 0x7fe904133d55 in edm::StreamSchedule::StreamSchedule(std::shared_ptr<edm::TriggerResultInserter>, std::vector<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> >, std::allocator<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> > > >&, std::vector<edm::propagate_const<std::shared_ptr<edm::EndPathStatusInserter> >, std::allocator<edm::propagate_const<std::shared_ptr<edm::EndPathStatusInserter> > > >&, std::shared_ptr<edm::ModuleRegistry>, edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::PreallocationConfiguration const&, edm::ProductRegistry&, edm::BranchIDListHelper&, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration>, bool, edm::StreamID, edm::ProcessContext const*) (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-11-12-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so+0x323d55)
    #18 0x7fe9044146a2 in edm::Schedule::Schedule(edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::ProductRegistry&, edm::BranchIDListHelper&, edm::ThinnedAssociationsHelper&, edm::SubProcessParentageHelper const*, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration>, bool, edm::PreallocationConfiguration const&, edm::ProcessContext const*) (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-11-12-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so+0x6046a2)
    #19 0x7fe9043be02f in edm::ScheduleItems::initSchedule(edm::ParameterSet&, bool, edm::PreallocationConfiguration const&, edm::ProcessContext const*) (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-11-12-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so+0x5ae02f)
    #20 0x7fe9042aca32 in edm::EventProcessor::init(std::shared_ptr<edm::ProcessDesc>&, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy) (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-11-12-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so+0x49ca32)
    #21 0x7fe9042b630f in edm::EventProcessor::EventProcessor(std::shared_ptr<edm::ProcessDesc>, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy) (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-11-12-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so+0x4a630f)
    #22 0x412c3d in main::{lambda()#1}::operator()() const (/cvmfs/cms-ib.cern.ch/nweek-02498/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-11-12-1100/bin/slc6_amd64_gcc700/cmsRun+0x412c3d)
    #23 0x40d322 in main (/cvmfs/cms-ib.cern.ch/nweek-02498/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-11-12-1100/bin/slc6_amd64_gcc700/cmsRun+0x40d322)
    #24 0x326f01ed1c in __libc_start_main (/lib64/libc.so.6+0x326f01ed1c)

==14731==ABORTING

The problem happens here: https://github.com/cms-sw/cmssw/blob/master/SimG4Core/Application/src/RunManagerMT.cc#L252

m_userRunAction pointer value is 0xbebebebebebebebe. Now ASan has 0xbe as pattern for malloc_fill_byte: Value used to fill the newly allocated memory.

cmsbuild commented 6 years ago

A new Issue was created by @davidlt .

@davidlange6, @Dr15Jones, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

Dr15Jones commented 6 years ago

assign simulation

cmsbuild commented 6 years ago

New categories assigned: simulation

@mdhildreth,@civanch you have been requested to review this Pull request/Issue and eventually sign? Thanks

Dr15Jones commented 6 years ago

m_userRunAction is only set here https://github.com/cms-sw/cmssw/blob/master/SimG4Core/Application/src/RunManagerMT.cc#L236

which is called from https://github.com/cms-sw/cmssw/blob/master/SimG4Core/Application/src/RunManagerMT.cc#L196

Looking at the logic of the code, it is possible for an exception to happen before reading the initialization of m_userRunAction which means the destructor would be called and would see an uninitialized value for the member variable.

I beleive the fix is just to set m_userRunAction to nullptr in the constructor. Or even better, changed m_userRunAction to be a std::shared_ptr<> with a customized delete method.

davidlt commented 6 years ago

Could you propose a pull request?

civanch commented 6 years ago

There is one #21363

davidlt commented 6 years ago

I can confirm that PR mentioned above resolves the issue.