search

cms-sw / cmssw

CMS Offline Software
http://cms-sw.github.io/
Apache License 2.0
1.08k stars 4.3k forks source link

[8.0.X] Crash in FormulaEvaluator when parsing expression #14789

Closed blinkseb closed 8 years ago

blinkseb commented 8 years ago

Dear experts,

The JEC software heavily use the reco::FormulaEvalutor for evaluating JEC. However, we encounter a crash when trying to use a new expression. The full stack-trace is given at the bottom of this issue.

The formula causing the issue is:

[2]*([3]*([4]+[5]*TMath::Log(max([0],min([1],x))))*1./([6]+[7]*0.5*(1.+TMath::Erf((TMath::Log(x)-TMath::Log(225.))/0.4))))

Stack:

Thread 2 (Thread 0x7f92ce47f700 (LWP 28206)):
#0  0x0000003e81e0f37d in waitpid () from /lib64/libpthread.so.0
#1  0x00007f92dc859ad7 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_6/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007f92dc85a382 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_6/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007f92dec77a50 in std::(anonymous namespace)::execute_native_thread_routine (__p=<optimized out>) at ../../../../../libstdc++-v3/src/c++11/thread.cc:84
#4  0x0000003e81e07aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003e812e893d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f92de911ba0 (LWP 28036)):
#0  0x0000003e812df113 in poll () from /lib64/libc.so.6
#1  0x00007f92dc85a1e4 in full_read.constprop () from /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_6/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007f92dc85a44a in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_6/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007f92dc85a5cb in sig_dostack_then_abort () from /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_6/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007f92db8238b6 in (anonymous namespace)::ExpressionFinder::createEvaluator(__gnu_cxx::__normal_iterator<char const*, std::string>, __gnu_cxx::__normal_iterator<char const*, std::string>, std::shared_ptr<reco::formula::BinaryOperatorEvaluatorBase>) const () from /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_6/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#6  0x00007f92db823234 in (anonymous namespace)::ExpressionFinder::createEvaluator(__gnu_cxx::__normal_iterator<char const*, std::string>, __gnu_cxx::__normal_iterator<char const*, std::string>, std::shared_ptr<reco::formula::BinaryOperatorEvaluatorBase>) const () from /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_6/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#7  0x00007f92db826d35 in (anonymous namespace)::createBinaryOperatorEvaluator((anonymous namespace)::ExpressionFinder const&, __gnu_cxx::__normal_iterator<char const*, std::string>, __gnu_cxx::__normal_iterator<char const*, std::string>) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_6/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#8  0x00007f92db82228b in (anonymous namespace)::ExpressionFinder::createEvaluator(__gnu_cxx::__normal_iterator<char const*, std::string>, __gnu_cxx::__normal_iterator<char const*, std::string>, std::shared_ptr<reco::formula::BinaryOperatorEvaluatorBase>) const () from /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_6/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#9  0x00007f92db826d35 in (anonymous namespace)::createBinaryOperatorEvaluator((anonymous namespace)::ExpressionFinder const&, __gnu_cxx::__normal_iterator<char const*, std::string>, __gnu_cxx::__normal_iterator<char const*, std::string>) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_6/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#10 0x00007f92db82228b in (anonymous namespace)::ExpressionFinder::createEvaluator(__gnu_cxx::__normal_iterator<char const*, std::string>, __gnu_cxx::__normal_iterator<char const*, std::string>, std::shared_ptr<reco::formula::BinaryOperatorEvaluatorBase>) const () from /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_6/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#11 0x00007f92db826745 in (anonymous namespace)::createBinaryOperatorEvaluator((anonymous namespace)::ExpressionFinder const&, __gnu_cxx::__normal_iterator<char const*, std::string>, __gnu_cxx::__normal_iterator<char const*, std::string>) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_6/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#12 0x00007f92db82228b in (anonymous namespace)::ExpressionFinder::createEvaluator(__gnu_cxx::__normal_iterator<char const*, std::string>, __gnu_cxx::__normal_iterator<char const*, std::string>, std::shared_ptr<reco::formula::BinaryOperatorEvaluatorBase>) const () from /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_6/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#13 0x00007f92db823234 in (anonymous namespace)::ExpressionFinder::createEvaluator(__gnu_cxx::__normal_iterator<char const*, std::string>, __gnu_cxx::__normal_iterator<char const*, std::string>, std::shared_ptr<reco::formula::BinaryOperatorEvaluatorBase>) const () from /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_6/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#14 0x00007f92db82667d in (anonymous namespace)::createBinaryOperatorEvaluator((anonymous namespace)::ExpressionFinder const&, __gnu_cxx::__normal_iterator<char const*, std::string>, __gnu_cxx::__normal_iterator<char const*, std::string>) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_6/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#15 0x00007f92db82228b in (anonymous namespace)::ExpressionFinder::createEvaluator(__gnu_cxx::__normal_iterator<char const*, std::string>, __gnu_cxx::__normal_iterator<char const*, std::string>, std::shared_ptr<reco::formula::BinaryOperatorEvaluatorBase>) const () from /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_6/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#16 0x00007f92db826d35 in (anonymous namespace)::createBinaryOperatorEvaluator((anonymous namespace)::ExpressionFinder const&, __gnu_cxx::__normal_iterator<char const*, std::string>, __gnu_cxx::__normal_iterator<char const*, std::string>) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_6/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#17 0x00007f92db82228b in (anonymous namespace)::ExpressionFinder::createEvaluator(__gnu_cxx::__normal_iterator<char const*, std::string>, __gnu_cxx::__normal_iterator<char const*, std::string>, std::shared_ptr<reco::formula::BinaryOperatorEvaluatorBase>) const () from /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_6/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#18 0x00007f92db826d35 in (anonymous namespace)::createBinaryOperatorEvaluator((anonymous namespace)::ExpressionFinder const&, __gnu_cxx::__normal_iterator<char const*, std::string>, __gnu_cxx::__normal_iterator<char const*, std::string>) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_6/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#19 0x00007f92db82228b in (anonymous namespace)::ExpressionFinder::createEvaluator(__gnu_cxx::__normal_iterator<char const*, std::string>, __gnu_cxx::__normal_iterator<char const*, std::string>, std::shared_ptr<reco::formula::BinaryOperatorEvaluatorBase>) const () from /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_6/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#20 0x00007f92db823234 in (anonymous namespace)::ExpressionFinder::createEvaluator(__gnu_cxx::__normal_iterator<char const*, std::string>, __gnu_cxx::__normal_iterator<char const*, std::string>, std::shared_ptr<reco::formula::BinaryOperatorEvaluatorBase>) const () from /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_6/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#21 0x00007f92db826d35 in (anonymous namespace)::createBinaryOperatorEvaluator((anonymous namespace)::ExpressionFinder const&, __gnu_cxx::__normal_iterator<char const*, std::string>, __gnu_cxx::__normal_iterator<char const*, std::string>) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_6/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#22 0x00007f92db82228b in (anonymous namespace)::ExpressionFinder::createEvaluator(__gnu_cxx::__normal_iterator<char const*, std::string>, __gnu_cxx::__normal_iterator<char const*, std::string>, std::shared_ptr<reco::formula::BinaryOperatorEvaluatorBase>) const () from /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_6/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#23 0x00007f92db827687 in reco::FormulaEvaluator::FormulaEvaluator(std::string const&) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_6/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#24 0x00007f92c95c9666 in SimpleJetCorrector::SimpleJetCorrector(JetCorrectorParameters const&) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_6/lib/slc6_amd64_gcc530/libCondFormatsJetMETObjects.so
#25 0x00007f92c95c1969 in FactorizedJetCorrectorCalculator::FactorizedJetCorrectorCalculator(std::vector<JetCorrectorParameters, std::allocator<JetCorrectorParameters> > const&) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_6/lib/slc6_amd64_gcc530/libCondFormatsJetMETObjects.so
#26 0x00007f92b2dbda8e in GammaJetFilter::GammaJetFilter(edm::ParameterSet const&) () from /cmshome/fpreiato/Test_NewGammaJet/CMSSW_8_0_6/lib/slc6_amd64_gcc530/pluginJetMETCorrectionsGammaJetFilter.so
#27 0x00007f92b2ddc1ef in edm::WorkerMaker<GammaJetFilter>::makeModule(edm::ParameterSet const&) const () from /cmshome/fpreiato/Test_NewGammaJet/CMSSW_8_0_6/lib/slc6_amd64_gcc530/pluginJetMETCorrectionsGammaJetFilter.so
#28 0x00007f92dfffedff in edm::Maker::makeModule(edm::MakeModuleParams const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const () from /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_6/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#29 0x00007f92e001ad57 in edm::Factory::makeModule(edm::MakeModuleParams const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const () from /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_6/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#30 0x00007f92dffaf760 in edm::ModuleRegistry::getModule(edm::MakeModuleParams const&, std::string const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_6/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#31 0x00007f92e000bc6a in edm::WorkerRegistry::getWorker(edm::WorkerParams const&, std::string const&) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_6/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#32 0x00007f92e00189ab in edm::WorkerManager::getWorker(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, std::string const&) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_6/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#33 0x00007f92dfff04dd in edm::StreamSchedule::fillWorkers(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, std::string const&, bool, std::vector<edm::WorkerInPath, std::allocator<edm::WorkerInPath> >&, std::vector<std::string, std::allocator<std::string> >*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_6/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#34 0x00007f92dfff1452 in edm::StreamSchedule::fillTrigPath(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, int, std::string const&, std::shared_ptr<edm::HLTGlobalStatus>, std::vector<std::string, std::allocator<std::string> >*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_6/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#35 0x00007f92dfff4340 in edm::StreamSchedule::StreamSchedule(std::shared_ptr<edm::TriggerResultInserter>, std::shared_ptr<edm::ModuleRegistry>, edm::ParameterSet&, edm::service::TriggerNamesService&, edm::PreallocationConfiguration const&, edm::ProductRegistry&, edm::BranchIDListHelper&, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration>, bool, edm::StreamID, edm::ProcessContext const*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_6/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#36 0x00007f92e006699c in edm::Schedule::Schedule(edm::ParameterSet&, edm::service::TriggerNamesService&, edm::ProductRegistry&, edm::BranchIDListHelper&, edm::ThinnedAssociationsHelper&, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration>, bool, edm::PreallocationConfiguration const&, edm::ProcessContext const*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_6/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#37 0x00007f92dffb0d02 in edm::ScheduleItems::initSchedule(edm::ParameterSet&, bool, edm::PreallocationConfiguration const&, edm::ProcessContext const*) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_6/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#38 0x00007f92e003b559 in edm::EventProcessor::init(std::shared_ptr<edm::ProcessDesc>&, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_6/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#39 0x00007f92e003d6aa in edm::EventProcessor::EventProcessor(std::shared_ptr<edm::ProcessDesc>&, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy) () from /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_6/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#40 0x000000000040c6a1 in main::{lambda()#1}::operator()() const ()
#41 0x000000000040ac36 in main ()

Can an expert look at it? It's rather urgent, since this issue delays release of the residuals corrections for 8.0.X.

Thanks

davidlt commented 8 years ago

assign reconstruction

blinkseb commented 8 years ago

You'll probably need a way to reproduce the issue:

davidlange6 commented 8 years ago

what release are you using?

On Jun 6, 2016, at 3:32 PM, Sébastien Brochet notifications@github.com wrote:

You'll probably need a way to reproduce the issue:

• The JEC txt files in question is here: https://github.com/cms-jet/JECDatabase/blob/master/textFiles/Spring16_25nsV3_DATA/Spring16_25nsV3_DATA_L2L3Residual_AK4PFchs.txt • Basic instructions on how to use it: https://twiki.cern.ch/twiki/bin/view/CMSPublic/WorkBookJetEnergyCorrections#JetEnCorFWLite — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

blinkseb commented 8 years ago

I'm just the messenger here, many analyzers face the same issue, but from the stack-trace, the release is CMSSW_8_0_6

karavdin commented 8 years ago

The issue is observed with CMSSW_8_0_5 and CMSSW_8_0_6 releases.

cmsbuild commented 8 years ago

New categories assigned: reconstruction

@cvuosalo,@slava77 you have been requested to review this Pull request/Issue and eventually sign? Thanks

davidlange6 commented 8 years ago

From the release notes, I would guess you need at least CMSSW_8_0_8_patch1, as this pull request is needed

https://github.com/cms-sw/cmssw/pull/14494

On Jun 6, 2016, at 3:39 PM, Sébastien Brochet notifications@github.com wrote:

I'm just the messenger here, many analyzers face the same issue, but from the stack-trace, the release is CMSSW_8_0_6

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

blinkseb commented 8 years ago

Cool, thanks a lot! Analyzers will test using CMSSW_8_0_8_patch1 and report if it's working or not.

cmsbuild commented 8 years ago

A new Issue was created by @blinkseb (Sbastien Brochet).

@davidlange6, @smuzaffar, @degano, @davidlt, @Dr15Jones can you please review it and eventually sign/assign? Thanks.

cms-bot commands are list here cms-sw/cmssw#13029

blinkseb commented 8 years ago

It seems to work using CMSSW_8_0_8_patch1, sorry for the noise here. I'm closing the issue.

Thanks a lot for the quick feedback!