Closed srimanob closed 3 years ago
cmsbuild
commented
3 years ago A new Issue was created by @srimanob Phat Srimanobhas.
@Dr15Jones, @dpiparo, @silviodonato, @smuzaffar, @makortel, @qliphy can you please review it and eventually sign/assign? Thanks.
cms-bot commands are listed here
makortel
commented
3 years ago assign core
cmsbuild
commented
3 years ago New categories assigned: core
@Dr15Jones,@smuzaffar,@makortel you have been requested to review this Pull request/Issue and eventually sign? Thanks
makortel
commented
3 years ago @srimanob Do I understand correctly that yout get the stack trace (**) with a single-thread run? Does the job have any relevant printouts before the crash?
On (***) I have no idea. The message comes from ROOT (that we convert to an exception by default). @pcanal, do you have idea on that? @srimanob Do you have HOME/.root.mimes file?
makortel
commented
3 years ago On (***) it might be interesting to see the stack trace of the exception. You can obtain it with
gdb cmsRun
(gdb) catch throw
(gdb) run <your_config>
# wait until the breakpoint hits, possibly 'continue' if some exceptions are thrown before
(gdb) where
srimanob
commented
3 years ago Hi @makortel
@srimanob Do I understand correctly that yout get the stack trace (**) with a single-thread run? Does the job have any relevant printouts before the crash?
No, I don't have it. This is the error I got back from condor. I can try run again with printout. The issue I face comes from both ntherads = 1 or 8. The strange is that I can't reproduce it when I try to run on lxplus using the same script.
On (***) I have no idea. The message comes from ROOT (that we convert to an exception by default). @pcanal, do you have idea on that? @srimanob Do you have
HOME/.root.mimesfile?
I don't have HOME/.root.mimes file.
Regarding gdb, is it possible to put in shell script if I would like to try to do it on condor? As I can't reproduce both issues on lxplus.
Thanks very much.
dan131riley
commented
3 years ago wrt (***) see https://twiki.cern.ch/twiki/bin/view/CMSPublic/SWGuideCrabFaq#Exit_code_6_with_Fatal_Root_Erro
Apparently condor doesn't set the HOME environment variable. You could either follow the advice on the wiki, or do something like export HOME=/tmp in your script.
dan131riley
commented
3 years ago also wrt (**) there should have been some kind of error message just before the stack trace. You'll probably find it is something similar, as from the stack trace it looks to be in the initialization of libgui. In general, you shouldn't be linking to libgui in a batch job.
srimanob
commented
3 years ago Thanks very much, @dan131riley
I am trying to solve ROOT, will let you know. I still have no idea on libgui as I don't call it (at least from myself). the script is basically run cmsDriver config.
I used the same submission script with 11_2 and everything work out-of-box, any idea what actually changes in 11_3 which may cause this?
srimanob
commented
3 years ago By the way for the ROOT issue, does it mean RecoTauCleaner/pfTausProducerSansRefs module tries to use ROOT which makes reference to $HOME somehow? If that is the case, it should be fixed I think.
srimanob
commented
3 years ago Confirm that with setting $HOME to the job, the RECO step works well.
pcanal
commented
3 years ago libgui is being loaded via the plugin mechanism:
21 0x00002b1e6eb62041 in dlopen@@GLIBC_2.2.5 () from /lib64/libdl.so.2
#22 0x00002b1e6cc66167 in edmplugin::SharedLibrary::SharedLibrary(std::filesystem::__cxx11::path const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre6/lib/slc7_amd64_gcc900
/libFWCorePluginManager.so
#23 0x00002b1e6cc60676 in edmplugin::PluginManager::load(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char
>, std::allocator<char> > const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre6/lib/slc7_amd64_gcc900/libFWCorePluginManager.so
#24 0x00002b1e6cc5a7f5 in edmplugin::PluginFactoryBase::findPMaker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const () from /cvmfs/cms.cern.ch/slc7_amd64_
gcc900/cms/cmssw/CMSSW_11_3_0_pre6/lib/slc7_amd64_gcc900/libFWCorePluginManager.so
#25 0x00002b1e6ced4b78 in edm::Factory::findMaker(edm::MakeModuleParams const&) const () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre6/lib/slc7_amd64_gcc900/libFWCoreFramework
.so
#26 0x00002b1e6ced4d72 in edm::Factory::makeModule(edm::MakeModuleParams const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription c
onst&)>&) const () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre6/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#27 0x00002b1e6cee66e8 in edm::ModuleRegistry::getModule(edm::MakeModuleParams const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, edm::signalslot::Signal<
void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre6/lib/slc7_amd64_gcc900/
libFWCoreFramework.soso some CMSSW module is needing it. (or so it seems).
makortel
commented
3 years ago How could we figure out which plugin would be pulling in the dependence on libGui? (I did not spot anything suspicious in BuildFiles with git grep rootgui etc)
pcanal
commented
3 years ago From the stack trace, I would say that examine the function parameter in those frames:
#25 0x00002b1e6ced4b78 in edm::Factory::findMaker(edm::MakeModuleParams const&) const () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre6/lib/slc7_amd64_gcc900/libFWCoreFramework
.so
#26 0x00002b1e6ced4d72 in edm::Factory::makeModule(edm::MakeModuleParams const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription c
onst&)>&) const () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre6/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#27 0x00002b1e6cee66e8 in edm::ModuleRegistry::getModule(edm::MakeModuleParams const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, edm::signalslot::Signal<
void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre6/lib/slc7_amd64_gcc900/
libFWCoreFramework.soshould give the answer.
Dr15Jones
commented
3 years ago If you add
process.add_(cms.Service("PrintLoadingPlugins"))it will print each time a plugin is loaded and will say what was requested.
makortel
commented
3 years ago If you add
process.add_(cms.Service("PrintLoadingPlugins"))it will print each time a plugin is loaded and will say what was requested.
@srimanob Could you repeat (**) with the PrintLoadingPlugins Service and post (a link to the) the full log of the job?
srimanob
commented
3 years ago Hi All,
Thanks for all information. I found that for the GEN step in my local test, it crashed due to ROOT without HOME env also. With this set, everything runs fine.
Is this an expected change? Why does running CMSSW job without any private modules need $HOME for root? My condor script worked well before with 10_6 and 11_2 without the need to set HOME. I just try to run 10_6 with unset HOME interactively, everything seems to work properly in all steps.
makortel
commented
3 years ago Is this an expected change?
Combined (or caused by?) the evidence for the job loading ROOT's libGUI I'd say "no, not expected". Let's first figure out what causes the loading of libGUI and then see if addressing that also removes the issue with setting or not $HOME.
srimanob
commented
3 years ago Here is the log file from PrintLoadingPlugins without HOME env,
/afs/cern.ch/user/s/srimanob/public/GitIssue33466/run-withoutHOME.log
To reproduce the issue, one can use cmsDriver above (ZEE one), with CMSSW_11_3_0_pre6. unsetenv HOME first.
makortel
commented
3 years ago Thanks @srimanob, quoting here the printouts before exception
Getting> 'CMS EDM Framework Module' PoolOutputModule
from Not found
Loading> /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre6/lib/slc7_amd64_gcc900/pluginIOPoolOutputPlugins.so
Getting> 'CMS EDM Framework Module' PPSSimTrackProducer
from Not found
Loading> /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre6/lib/slc7_amd64_gcc900/pluginSimPPSPPSSimTrackProducerAuto.so
terminate called after throwing an instance of 'edm::Exception'
what(): An exception of category 'FatalRootError' occurred.
Additional Info:
[a] Fatal Root Error: @SUB=TSystem::ExpandFileName
input: $HOME/.root.mimes, output: $HOME/.root.mimes(plus a similar stack trace as in the issue description).
makortel
commented
3 years ago The SimPPS/PPSSimTrackProducer/plugins depends on hector
https://github.com/cms-sw/cmssw/blob/dafb55d2ee66014caabcfc36104dba18240388e9/SimPPS/PPSSimTrackProducer/plugins/BuildFile.xml#L6
and
$ ldd $(scram tool tag hector LIBDIR)/libHector.so | fgrep Gui
libGui.so => /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre6/external/slc7_amd64_gcc900/lib/libGui.so (0x00007f46015fa000)so hector seems to depend on libGui.
Can the libGui dependence be removed from hector?
FYI @cms-sw/simulation-l2 @cms-sw/ctpps-dpg-l2
makortel
commented
3 years ago assign simulation
cmsbuild
commented
3 years ago New categories assigned: simulation
@civanch,@mdhildreth you have been requested to review this Pull request/Issue and eventually sign? Thanks
jan-kaspar
commented
3 years ago FYI @clemencia @mundim
dan131riley
commented
3 years ago Odd, nothing seems to have changed. The hector Makefile does
ROOTLIBS= $(shell root-config --libs --glibs)and root-config --libs --glibs always includes -lGui. I guess the thing to do is remove the --glibs and see if it's really needed.
mundim
commented
3 years ago Indeed, this is somehow strange. No change in HECTOR has been implemented since several years ago, this is an external package under the responsibility of @forthommel at that time, but I don't know who is in charge of this package now. Just to mention, this is a special version for CMS with no graphics.
dan131riley
commented
3 years ago Indeed, this is somehow strange. No change in HECTOR has been implemented since several years ago, this is an external package under the responsibility of @forthommel at that time, but I don't know who is in charge of this package now. Just to mention, this is a special version for CMS with no graphics.
The condor HOME issue may be intermittent? Anyway, linking to the Gui lib is undesirable, and in a test compile it looks like the graphics decency was indeed removed, so I think the right thing to do is just remove the --glibs in the Makefile. I can do a PR since I've already done the test.
forthommel
commented
3 years ago Can the
libGuidependence be removed fromhector?
I kind of "inherited" the project indeed. A bit of refactoring can be done to remove any libGui dependence. Let me check and return to this thread.
dan131riley
commented
3 years ago Can the
libGuidependence be removed fromhector?I kind of "inherited" the project indeed. A bit of refactoring can be done to remove any libGui dependence. Let me check and return to this thread.
I tried the compile without and it seemed fine, so I went ahead with a PR, which you can approve if it checks out.
forthommel
commented
3 years ago Indeed! Thanks for being faster! Also implemented in the mainline repo: https://github.com/hectorhep/hector/commit/94f44778dd66aa0699ca689e8048e5b2627e18ea
mundim
commented
3 years ago Can the
libGuidependence be removed fromhector?I kind of "inherited" the project indeed. A bit of refactoring can be done to remove any libGui dependence. Let me check and return to this thread.
Hi @forthommel, in case it is not a big burden, HECTOR is currently not suited for Run3 simulation (14 TeV), because it has 13TeV hardcoded. I do have a private version where the energy is a parameter to be passed in the constructor. Maybe now the action is too urgent for that, but it might be a good idea to change that in the future. Sorry to raise this point in this tread...
forthommel
commented
3 years ago Hi Luiz
A while ago I started a v2 of Hector. More thread-safe (i.e. with next to no global variables hanging around) and more parameterisable (CM. energy, relative/absolute coordinates system, ...).
You may check at : https://github.com/hectorhep/hector2
I did not release it so far as it would require a full validation by experts. If you feel like starting something with that in mind, do not hesitate!
Cheers, Laurent
On Tue, 20 Apr, 2021 at 07:50, mundim @.***> wrote:
Can the libGui dependence be removed from hector?
I kind of "inherited" the project indeed. A bit of refactoring can be done to remove any libGui dependence. Let me check and return to this thread.
Hi @forthommel https://github.com/forthommel, in case it is not a big burden, HECTOR is currently not suited for Run3 simulation (14 TeV), because it has 13TeV hardcoded. I do have a private version where the energy is a parameter to be passed in the constructor. Maybe now the action is too urgent for that, but it might be a good idea to change that in the future. Sorry to raise this point in this tread...
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cms-sw/cmssw/issues/33466#issuecomment-823337775, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACQZWIYIIIWVQUK7XIUGULTJWIDZANCNFSM43F2FHHA.
srimanob
commented
3 years ago Hi @dan131riley @makortel
Great that the SIM step can be fixed, thanks. However, I think for the standard MC/data processing sequence we should not need Gui or interactive.
Looking back on the original thread, we also have an issue with the RECO step. The log,
/afs/cern.ch/user/s/srimanob/public/GitIssue33466/reco-withoutHOME.log,
shows
Getting> 'CMS EDM Framework Module' BoostedJetONNXJetTagsProducer
from Not found
Loading> /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre5/lib/slc7_amd64_gcc900/pluginRecoBTagONNXRuntimePlugins.so
20-Apr-2021 17:11:58 CEST Closed file file:step3_DIGIL1HLT.root
----- Begin Fatal Exception 20-Apr-2021 17:11:58 CEST-----------------------
An exception of category 'FatalRootError' occurred while
[0] Constructing the EventProcessor
[1] Constructing module: class=RecoTauCleaner label='pfTausProducerSansRefs'
Additional Info:
[a] Fatal Root Error: @SUB=TSystem::ExpandFileName
input: $HOME/.root.mimes, output: $HOME/.root.mimes
----- End Fatal Exception -------------------------------------------------I am trying to go back in time at which release we can run without $HOME, and I find that all of the above cmsDrivers run fine without $HOME in 11_1_0. (one may need --runUnscheduled for 11_1 PAT).
Do you have an idea @cms-sw/reconstruction-l2 ? Should we expect interactive usage in the RECO sequence?
Or this is something in ROOT that we don't aware of (as we don't have change for Hector as mentioned above).
mundim
commented
3 years ago Hi Luiz A while ago I started a v2 of Hector. More thread-safe (i.e. with next to no global variables hanging around) and more parameterisable (CM. energy, relative/absolute coordinates system, ...). You may check at : https://github.com/hectorhep/hector2 I did not release it so far as it would require a full validation by experts. If you feel like starting something with that in mind, do not hesitate! Cheers, Laurent …
hi @forthommel, that is indeed a good news. I'll see if I can do it in parallel with the other issues in the simulation. thank you.
makortel
commented
3 years ago I went over all shared objects in /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre6/lib/slc7_amd64_gcc900 and the following depend on libGui
libCalibCalorimetryEcalCorrelatedNoiseAnalysisAlgos.so{lib,plugin}Fireworks*.so (ok)libRecoRomanPotRecoFP420.solibSimTransportPPSProtonTransport.solibUtilitiesPPS.sopluginCSCGeometryValidationPlugins.sopluginCalibCalorimetryEcalCorrelatedNoiseAnalysisModules.sopluginDTGeometryValidationPlugins.sopluginFastSimulationCTPPSFastTrackingProducerAuto.sopluginGEMGeometryValidationPlugins.sopluginME0GeometryValidationPlugins.sopluginRPCGeometryValidationPlugins.sopluginRecoRomanPotRecoFP420Plugins.sopluginSimPPSPPSSimTrackProducerAuto.sopluginSimTransportHectorProducer.soAll the PPS and RomanPot related packages are caused by the hector dependence (IIRC).
dan131riley
commented
3 years ago I ran the step4 RECO from above with HOME unset, and reproduced the exception. I reran with LD_DEBUG=files. None of the libraries listed above were loaded. Loading libGui actually seems to start with the plug-in handler loading libROOTEve.so:
27293: file=libGui.so [0]; needed by /cvmfs/cms.cern.ch/slc7_amd64_gcc900/lcg/root/6.22.06/lib/libROOTEve.so [0]
27293: file=libGui.so [0]; generating link map
27293: dynamic: 0x00007fd063a539e8 base: 0x00007fd0635f6000 size: 0x00000000004772f8
27293: entry: 0x00007fd0637bb520 phdr: 0x00007fd0635f6040 phnum: 9So far I haven't found any dependency to account for this.
dan131riley
commented
3 years ago It's coming from a call to TEnum::GetEnum() from edm::TypeWithDict::byName() here:
which somehow ends up loading libROOTEve.so which pulls in libGui.so. Some typical selectors for pfTausProducerSansRefs:
signalChargedHadrCands().size() = 3
abs(charge())-1
signalPiZeroCandidates().size() = 0 | signalPiZeroCandidates()[0].pt > 2.5
leadCand().isNonnull()
isolationPFChargedHadrCandsPtSum() + isolationPFGammaCandsEtSum()Here's the stack trace:
#0 0x00007ff9e1e2332e in __cxxabiv1::__cxa_throw (obj=0x7ff98de48b00, tinfo=0x7ff9e3dc5e60 <typeinfo for edm::Exception>, dest=0x7ff9e3d3efc2 <edm::Exception::~Exception()>) at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:78
#1 0x00007ff9d59274bf in (anonymous namespace)::RootErrorHandlerImpl(int, char const*, char const*) [clone .cold] () from /mnt/data1/dsr/tmp/CMSSW_11_3_0_pre1/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#2 0x00007ff9e2fcd9d6 in ErrorHandler () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/external/slc7_amd64_gcc900/lib/libCore.so
#3 0x00007ff9e2fcda78 in Error(char const*, char const*, ...) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/external/slc7_amd64_gcc900/lib/libCore.so
#4 0x00007ff9e30127cf in TSystem::ExpandFileName(char const*, char*, int) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/external/slc7_amd64_gcc900/lib/libCore.so
#5 0x00007ff9e30128bd in TSystem::ExpandFileName(TString&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/external/slc7_amd64_gcc900/lib/libCore.so
#6 0x00007ff97cf00625 in TGResourcePool::TGResourcePool(TGClient*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/external/slc7_amd64_gcc900/lib/libGui.so
#7 0x00007ff97ce54fea in TGClient::TGClient(char const*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/external/slc7_amd64_gcc900/lib/libGui.so
#8 0x00007ff97cde7574 in _GLOBAL__sub_I_TGClient.cxx () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/external/slc7_amd64_gcc900/lib/libGui.so
#9 0x00007ff9e43539c3 in _dl_init_internal () from /lib64/ld-linux-x86-64.so.2
#10 0x00007ff9e435859e in dl_open_worker () from /lib64/ld-linux-x86-64.so.2
#11 0x00007ff9e43537d4 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#12 0x00007ff9e4357b8b in _dl_open () from /lib64/ld-linux-x86-64.so.2
#13 0x00007ff9e2178fab in dlopen_doit () from /lib64/libdl.so.2
#14 0x00007ff9e43537d4 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#15 0x00007ff9e21795ad in _dlerror_run () from /lib64/libdl.so.2
#16 0x00007ff9e2179041 in dlopen@@GLIBC_2.2.5 () from /lib64/libdl.so.2
#17 0x00007ff9c182a8b5 in cling::utils::platform::DLOpen(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/external/slc7_amd64_gcc900/lib/libCling.so
#18 0x00007ff9c1740cd6 in cling::DynamicLibraryManager::loadLibrary(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bool) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/external/slc7_amd64_gcc900/lib/libCling.so
#19 0x00007ff9c16b2224 in TCling::Load(char const*, bool) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/external/slc7_amd64_gcc900/lib/libCling.so
#20 0x00007ff9e30132a0 in TSystem::Load(char const*, char const*, bool) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/external/slc7_amd64_gcc900/lib/libCore.so
#21 0x00007ff9e2fa3b61 in TROOT::LoadClass(char const*, char const*, bool) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/external/slc7_amd64_gcc900/lib/libCore.so
#22 0x00007ff9c16b0303 in TCling::ShallowAutoLoadImpl(char const*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/external/slc7_amd64_gcc900/lib/libCling.so
#23 0x00007ff9c16bf7f6 in TCling::DeepAutoLoadImpl(char const*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/external/slc7_amd64_gcc900/lib/libCling.so
#24 0x00007ff9c16bfe0c in TCling::AutoLoad(char const*, bool) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/external/slc7_amd64_gcc900/lib/libCling.so
#25 0x00007ff9e30aef0f in TEnum::GetEnum(char const*, TEnum::ESearchAction) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/external/slc7_amd64_gcc900/lib/libCore.so
#26 0x00007ff9e43a8ca8 in edm::TypeWithDict::byName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/lib/slc7_amd64_gcc900/libFWCoreReflection.so
#27 0x00007ff9e43a919d in edm::TypeWithDict::TypeWithDict(TMethodArg*, long) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/lib/slc7_amd64_gcc900/libFWCoreReflection.so
#28 0x00007ff9aca4d77d in reco::checkMethod(edm::FunctionWithDict const&, edm::TypeWithDict const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/lib/slc7_amd64_gcc900/libCommonToolsUtils.so
#29 0x00007ff9aca4de4b in reco::findMethod(edm::TypeWithDict const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, char const*, int&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/lib/slc7_amd64_gcc900/libCommonToolsUtils.so
#30 0x00007ff9aca30c69 in reco::parser::MethodSetter::push(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, char const*, bool) const () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/lib/slc7_amd64_gcc900/libCommonToolsUtils.so
#31 0x00007ff9aca31672 in reco::parser::MethodSetter::operator()(char const*, char const*) const () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/lib/slc7_amd64_gcc900/libCommonToolsUtils.so
#32 0x00007ff9aca496d9 in boost::spirit::classic::impl::concrete_parser<boost::spirit::classic::action<boost::spirit::classic::sequence<boost::spirit::classic::sequence<boost::spirit::classic::sequence<boost::spirit::classic::chlit<char>, boost::spirit::classic::rule<boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> >, boost::spirit::classic::nil_t, boost::spirit::classic::nil_t> >, boost::spirit::classic::kleene_star<boost::spirit::classic::sequence<boost::spirit::classic::chlit<char>, boost::spirit::classic::rule<boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> >, boost::spirit::classic::nil_t, boost::spirit::classic::nil_t> > > >, boost::spirit::classic::assertive_parser<reco::parser::SyntaxErrors, boost::spirit::classic::chlit<char> > >, reco::parser::MethodSetter>, boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> >, boost::spirit::classic::nil_t>::do_parse_virtual(boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> > const&) const () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/lib/slc7_amd64_gcc900/libCommonToolsUtils.so
[...]
#43 0x00007ff9aca404bb in boost::spirit::classic::impl::concrete_parser<boost::spirit::classic::action<boost::spirit::classic::rule<boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> >, boost::spirit::classic::nil_t, boost::spirit::classic::nil_t>, reco::parser::CutSetter>, boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> >, boost::spirit::classic::nil_t>::do_parse_virtual(boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> > const&) const () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/lib/slc7_amd64_gcc900/libCommonToolsUtils.so
#44 0x00007ff9aca3b6e7 in reco::parser::cutParser(edm::TypeWithDict const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<reco::parser::SelectorBase>&, bool) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/lib/slc7_amd64_gcc900/libCommonToolsUtils.so
#45 0x00007ff9892f3b25 in reco::tau::RecoTauStringCleanerPlugin::RecoTauStringCleanerPlugin(edm::ParameterSet const&, edm::ConsumesCollector&&) () from /mnt/data1/dsr/tmp/CMSSW_11_3_0_pre1/lib/slc7_amd64_gcc900/pluginRecoTauTagRecoTauPlugins.so
#46 0x00007ff9892f404f in edmplugin::PluginFactory<reco::tau::RecoTauCleanerPlugin* (edm::ParameterSet const&, edm::ConsumesCollector&&)>::PMaker<reco::tau::RecoTauStringCleanerPlugin>::create(edm::ParameterSet const&, edm::ConsumesCollector&&) const () from /mnt/data1/dsr/tmp/CMSSW_11_3_0_pre1/lib/slc7_amd64_gcc900/pluginRecoTauTagRecoTauPlugins.so
#47 0x00007ff9892a1280 in RecoTauCleanerImpl<std::vector<reco::PFTau, std::allocator<reco::PFTau> > >::RecoTauCleanerImpl(edm::ParameterSet const&) () from /mnt/data1/dsr/tmp/CMSSW_11_3_0_pre1/lib/slc7_amd64_gcc900/pluginRecoTauTagRecoTauPlugins.so
#48 0x00007ff9892a19e7 in edm::stream::ProducingModuleAdaptor<RecoTauCleanerImpl<std::vector<reco::PFTau, std::allocator<reco::PFTau> > >, edm::stream::EDProducerBase, edm::stream::EDProducerAdaptorBase>::setupStreamModules() () from /mnt/data1/dsr/tmp/CMSSW_11_3_0_pre1/lib/slc7_amd64_gcc900/pluginRecoTauTagRecoTauPlugins.soThanks @dan131riley. Would you be able to check the value of name?
dan131riley
commented
3 years ago @makortel vector<ROOT::Experimental::REveTableEntry>::size_type.
Also I tried reproducing in the ROOT6 master IB, which I believe has debug symbols, but it doesn't replicate there. That's consistent with it only occurring in some releases, or it could indicate a problem that has been fixed in later ROOT releases.
#27 0x00007fc5efb5eeae in edm::TypeWithDict::byName (name="vector<ROOT::Experimental::REveTableEntry>::size_type", property=32) at /mnt/data1/dsr/tmp/CMSSW_11_3_0_pre1/src/FWCore/Reflection/src/TypeWithDict.cc:168
vector<ROOT::Experimental::REveTableEntry>::size_type.
I wonder how such type gets queried.
Also I tried reproducing in the ROOT6 master IB, which I believe has debug symbols, but it doesn't replicate there. That's consistent with it only occurring in some releases, or it could indicate a problem that has been fixed in later ROOT releases.
@pcanal Would you have any idea what could be causing this issue, and if that is already been fixed in ROOT master or 6.22 branch?
dan131riley
commented
3 years ago Also doesn't replicate in CMSSW_12_0_ROOT624_X_2021-04-19-2300. I put in some print statements to see what gets looked up, in CMSSW_11_3_0_pre1 I get the sequence
Trying enum bool
Trying enum double
Trying enum float
Trying enum int
Trying enum void
Trying enum reco::HitPattern::HitCategory
Trying enum reco::HitPattern::HitCategory
Trying enum reco::RecoTauPiZero::PiZeroAlgorithm
Trying enum reco::RecoTauPiZero::PiZeroAlgorithm
Trying enum reco::RecoTauPiZero::PiZeroAlgorithm
Trying enum reco::PFRecoTauChargedHadron::PFRecoTauChargedHadronAlgorithm
Trying enum reco::PFRecoTauChargedHadron::PFRecoTauChargedHadronAlgorithm
Trying enum reco::PFRecoTauChargedHadron::PFRecoTauChargedHadronAlgorithm
Trying enum reco::PFRecoTauChargedHadron::PFRecoTauChargedHadronAlgorithm
Trying enum reco::PFRecoTauChargedHadron::PFRecoTauChargedHadronAlgorithm
Trying enum reco::PFRecoTauChargedHadron::PFRecoTauChargedHadronAlgorithm
Trying enum reco::PFRecoTauChargedHadron::PFRecoTauChargedHadronAlgorithm
Trying enum reco::PFRecoTauChargedHadron::PFRecoTauChargedHadronAlgorithm
Trying enum reco::PFRecoTauChargedHadron::PFRecoTauChargedHadronAlgorithm
Trying enum unsigned long
Trying enum vector<ROOT::Experimental::REveTableEntry>::size_typewhile in the 6.24 IB it goes
Trying enum reco::PFRecoTauChargedHadron::PFRecoTauChargedHadronAlgorithm
Trying enum reco::PFRecoTauChargedHadron::PFRecoTauChargedHadronAlgorithm
Trying enum reco::PFRecoTauChargedHadron::PFRecoTauChargedHadronAlgorithm
Trying enum reco::PFRecoTauChargedHadron::PFRecoTauChargedHadronAlgorithm
Trying enum reco::PFRecoTauChargedHadron::PFRecoTauChargedHadronAlgorithm
Trying enum reco::PFRecoTauChargedHadron::PFRecoTauChargedHadronAlgorithm
Trying enum reco::PFRecoTauChargedHadron::PFRecoTauChargedHadronAlgorithm
Trying enum reco::PFRecoTauChargedHadron::PFRecoTauChargedHadronAlgorithm
Trying enum reco::PFRecoTauChargedHadron::PFRecoTauChargedHadronAlgorithm
Trying enum vector<TProtoClass::TProtoRealData>::size_type
Trying enum vector<TProtoClass::TProtoRealData>::size_typeThere are some possibly relevant @pcanal commits that went into 6.23
Dr15Jones
commented
3 years ago Perhaps ROOT is trying to find std::vector<>::size_type and is ignoring the template parameter? If so, it might just pull in the first entry it finds in its read of .rootmap files and that might vary build to build.
pcanal
commented
3 years ago If I understood correctly, the failing case has:
#27 0x00007fc5efb5eeae in edm::TypeWithDict::byName (name="vector<ROOT::Experimental::REveTableEntry>::size_type", property=32) at /mnt/data1/dsr/tmp/CMSSW_11_3_0_pre1/src/FWCore/Reflection/src/TypeWithDict.cc:168Where the autoload is being ask to provide information about this class, and thus it (ought to) load information about the template parameter.
This edm::TypeWithDict::byName call comes from:
#27 0x00007ff9e43a919d in edm::TypeWithDict::TypeWithDict(TMethodArg*, long) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/lib/slc7_amd64_gcc900/libFWCoreReflection.so
#28 0x00007ff9aca4d77d in reco::checkMethod(edm::FunctionWithDict const&, edm::TypeWithDict const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/lib/slc7_amd64_gcc900/libCommonToolsUtils.so
#29 0x00007ff9aca4de4b in reco::findMethod(edm::TypeWithDict const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, char const*, int&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/lib/slc7_amd64_gcc900/libCommonToolsUtils.so
#30 0x00007ff9aca30c69 in reco::parser::MethodSetter::push(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, char const*, bool) const () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/lib/slc7_amd64_gcc900/libCommonToolsUtils.so
#31 0x00007ff9aca31672 in reco::parser::MethodSetter::operator()(char const*, char const*) const () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/lib/slc7_amd64_gcc900/libCommonToolsUtils.so
#32 0x00007ff9aca496d9 in boost::spirit::classic::impl::concrete_parser<boost::spirit::classic::action<boost::spirit::classic::sequence<boost::spirit::classic::sequence<boost::spirit::classic::sequence<boost::spirit::classic::chlit<char>, boost::spirit::classic::rule<boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> >, boost::spirit::classic::nil_t, boost::spirit::classic::nil_t> >, boost::spirit::classic::kleene_star<boost::spirit::classic::sequence<boost::spirit::classic::chlit<char>, boost::spirit::classic::rule<boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> >, boost::spirit::classic::nil_t, boost::spirit::classic::nil_t> > > >, boost::spirit::classic::assertive_parser<reco::parser::SyntaxErrors, boost::spirit::classic::chlit<char> > >, reco::parser::MethodSetter>, boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> >, boost::spirit::classic::nil_t>::do_parse_virtual(boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> > const&) const () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/lib/slc7_amd64_gcc900/libCommonToolsUtils.so
[...]
#43 0x00007ff9aca404bb in boost::spirit::classic::impl::concrete_parser<boost::spirit::classic::action<boost::spirit::classic::rule<boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> >, boost::spirit::classic::nil_t, boost::spirit::classic::nil_t>, reco::parser::CutSetter>, boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> >, boost::spirit::classic::nil_t>::do_parse_virtual(boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> > const&) const () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/lib/slc7_amd64_gcc900/libCommonToolsUtils.so
#44 0x00007ff9aca3b6e7 in reco::parser::cutParser(edm::TypeWithDict const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<reco::parser::SelectorBase>&, bool) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/lib/slc7_amd64_gcc900/libCommonToolsUtils.soSo I assume that the vector is the type of the argument of a function being looked at by the cut_parser.
It is plausible that this function (since it involves the new Eve implementation: REveTableEntry) has its parameter changed.
Dr15Jones
commented
3 years ago It is plausible that this function (since it involves the new Eve implementation: REveTableEntry) has its parameter changed.
No cmsRun jobs should be asking for anything from Eve.
pcanal
commented
3 years ago So we need more information from the input to the cutparser on why it made that request.
Dr15Jones
commented
3 years ago What we need to know is the arguments passed to reco::findMethod since that should say which function was called.
Dr15Jones
commented
3 years ago So looking at the two selectors
signalChargedHadrCands().size() = 3
signalPiZeroCandidates().size() = 0 | signalPiZeroCandidates()[0].pt > 2.5I see that signalChargedHadrCands() is defined here
https://github.com/cms-sw/cmssw/blob/6d2f66057131baacc2fcbdd203588c41c885b42c/DataFormats/TauReco/interface/PFTau.h#L90
and signalPiZeroCandidates() is here
https://github.com/cms-sw/cmssw/blob/6d2f66057131baacc2fcbdd203588c41c885b42c/DataFormats/TauReco/interface/PFTau.h#L145
Then the call to size() returns a std::vector<T>::size_type value
https://en.cppreference.com/w/cpp/container/vector/size
So this supports my theory that ROOT doesn't track the different std::vector<T>::size_type enums and instead just picks one convenient one (say std::vector<ROOT::Experimental::REveTableEntry>::size_type) as a stand in for all std::vector<T>::size_type uses by TClass.
dan131riley
commented
3 years ago The selector to reco::parser::cutParser() is
signalPiZeroCandidates().size() = 0 | signalPiZeroCandidates()[0].pt > 2.5and the args to reco::findMethod()
#31 0x00007fd5768e9673 in reco::findMethod (t=..., name="operator[]", args=std::vector of length 1, capacity 1 = {...}, fixuppedArgs=std::vector of length 0, capacity 0, iIterator=0x7fd557b6c7fe "[0].pt > 2.5", oError=@0x7fff62b7fb6c: 1) at /mnt/data1/dsr/tmp/CMSSW_11_3_0_pre1/src/CommonTools/Utils/src/findMethod.cc:170
(gdb) p t
$1 = (const edm::TypeWithDict &) @0x7fff62b7fb90: {
ti_ = 0x7fd577567088 <typeinfo for std::vector<reco::RecoTauPiZero, std::allocator<reco::RecoTauPiZero> >>,
class_ = 0x7fd557773600,
enum_ = 0x0,
dataType_ = 0x0,
arrayDimensions_ = {
myP = {
m_value = 0x0
}
},
property_ = 1114112
}
(gdb) p name
$2 = "operator[]"
(gdb) p iIterator
$5 = 0x7fd557b6c7fe "[0].pt > 2.5"Stack trace (-70 lines of boost::spirit):
#0 0x00007fd5abf8932e in __cxxabiv1::__cxa_throw (obj=0x7fd557b31e80, tinfo=0x7fd5adf43e60 <typeinfo for edm::Exception>, dest=0x7fd5adebcfc2 <edm::Exception::~Exception()>) at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:78
#1 0x00007fd590feddd2 in (anonymous namespace)::RootErrorHandlerImpl (level=3000, location=0x7fd5ad2da5c7 "TSystem::ExpandFileName", message=0x7fff62b7b930 "input: $HOME/.root.mimes, output: $HOME/.root.mimes") at /mnt/data1/dsr/tmp/CMSSW_11_3_0_pre1/src/FWCore/Services/plugins/InitRootHandlers.cc:284
#2 0x00007fd590fee2ea in (anonymous namespace)::RootErrorHandler (level=3000, location=0x7fd5ad2da5c7 "TSystem::ExpandFileName", message=0x7fff62b7b930 "input: $HOME/.root.mimes, output: $HOME/.root.mimes") at /mnt/data1/dsr/tmp/CMSSW_11_3_0_pre1/src/FWCore/Services/plugins/InitRootHandlers.cc:306
#3 0x00007fd5ad1339d6 in ErrorHandler () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/external/slc7_amd64_gcc900/lib/libCore.so
#4 0x00007fd5ad133a78 in Error(char const*, char const*, ...) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/external/slc7_amd64_gcc900/lib/libCore.so
#5 0x00007fd5ad1787cf in TSystem::ExpandFileName(char const*, char*, int) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/external/slc7_amd64_gcc900/lib/libCore.so
#6 0x00007fd5ad1788bd in TSystem::ExpandFileName(TString&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/external/slc7_amd64_gcc900/lib/libCore.so
#7 0x00007fd5462e4625 in TGResourcePool::TGResourcePool(TGClient*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/external/slc7_amd64_gcc900/lib/libGui.so
#8 0x00007fd546238fea in TGClient::TGClient(char const*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/external/slc7_amd64_gcc900/lib/libGui.so
#9 0x00007fd5461cb574 in _GLOBAL__sub_I_TGClient.cxx () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/external/slc7_amd64_gcc900/lib/libGui.so
#10 0x00007fd5ae4d19c3 in _dl_init_internal () from /lib64/ld-linux-x86-64.so.2
#11 0x00007fd5ae4d659e in dl_open_worker () from /lib64/ld-linux-x86-64.so.2
#12 0x00007fd5ae4d17d4 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#13 0x00007fd5ae4d5b8b in _dl_open () from /lib64/ld-linux-x86-64.so.2
#14 0x00007fd5ac2defab in dlopen_doit () from /lib64/libdl.so.2
#15 0x00007fd5ae4d17d4 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#16 0x00007fd5ac2df5ad in _dlerror_run () from /lib64/libdl.so.2
#17 0x00007fd5ac2df041 in dlopen@@GLIBC_2.2.5 () from /lib64/libdl.so.2
#18 0x00007fd58b42a8b5 in cling::utils::platform::DLOpen(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/external/slc7_amd64_gcc900/lib/libCling.so
#19 0x00007fd58b340cd6 in cling::DynamicLibraryManager::loadLibrary(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bool) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/external/slc7_amd64_gcc900/lib/libCling.so
#20 0x00007fd58b2b2224 in TCling::Load(char const*, bool) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/external/slc7_amd64_gcc900/lib/libCling.so
#21 0x00007fd5ad1792a0 in TSystem::Load(char const*, char const*, bool) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/external/slc7_amd64_gcc900/lib/libCore.so
#22 0x00007fd5ad109b61 in TROOT::LoadClass(char const*, char const*, bool) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/external/slc7_amd64_gcc900/lib/libCore.so
#23 0x00007fd58b2b0303 in TCling::ShallowAutoLoadImpl(char const*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/external/slc7_amd64_gcc900/lib/libCling.so
#24 0x00007fd58b2bf7f6 in TCling::DeepAutoLoadImpl(char const*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/external/slc7_amd64_gcc900/lib/libCling.so
#25 0x00007fd58b2bfe0c in TCling::AutoLoad(char const*, bool) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/external/slc7_amd64_gcc900/lib/libCling.so
#26 0x00007fd5ad214f0f in TEnum::GetEnum(char const*, TEnum::ESearchAction) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre1/external/slc7_amd64_gcc900/lib/libCore.so
#27 0x00007fd5ae516f0e in edm::TypeWithDict::byName (name="vector<ROOT::Experimental::REveTableEntry>::size_type", property=32) at /mnt/data1/dsr/tmp/CMSSW_11_3_0_pre1/src/FWCore/Reflection/src/TypeWithDict.cc:170
#28 0x00007fd5ae517bd9 in edm::TypeWithDict::TypeWithDict (this=0x7fff62b7efa0, arg=0x7fd557b3b7b0, property=0) at /mnt/data1/dsr/tmp/CMSSW_11_3_0_pre1/src/FWCore/Reflection/src/TypeWithDict.cc:345
#29 0x00007fd5ae517b5c in edm::TypeWithDict::TypeWithDict (this=0x7fff62b7efa0, arg=0x7fd557b3b7b0) at /mnt/data1/dsr/tmp/CMSSW_11_3_0_pre1/src/FWCore/Reflection/src/TypeWithDict.cc:342
#30 0x00007fd5768e8559 in reco::checkMethod (mem=..., type=..., args=std::vector of length 1, capacity 1 = {...}, fixuppedArgs=std::vector of length 0, capacity 0) at /mnt/data1/dsr/tmp/CMSSW_11_3_0_pre1/src/CommonTools/Utils/src/findMethod.cc:85
#31 0x00007fd5768e9673 in reco::findMethod (t=..., name="operator[]", args=std::vector of length 1, capacity 1 = {...}, fixuppedArgs=std::vector of length 0, capacity 0, iIterator=0x7fd557b6c7fe "[0].pt > 2.5", oError=@0x7fff62b7fb6c: 1) at /mnt/data1/dsr/tmp/CMSSW_11_3_0_pre1/src/CommonTools/Utils/src/findMethod.cc:170
#32 0x00007fd5768b08e4 in reco::parser::MethodSetter::push (this=0x7fd55764bc70, name="operator[]", args=std::vector of length 1, capacity 1 = {...}, begin=0x7fd557b6c7fe "[0].pt > 2.5", deep=true) at /mnt/data1/dsr/tmp/CMSSW_11_3_0_pre1/src/CommonTools/Utils/src/MethodSetter.cc:61
#33 0x00007fd5768b077d in reco::parser::MethodSetter::operator() (this=0x7fd55764bc70, begin=0x7fd557b6c7fe "[0].pt > 2.5", end=0x7fd557b6c801 ".pt > 2.5") at /mnt/data1/dsr/tmp/CMSSW_11_3_0_pre1/src/CommonTools/Utils/src/MethodSetter.cc:51
#34 0x00007fd5768d9cb4 in boost::spirit::classic::attributed_action_policy<boost::spirit::classic::nil_t>::call<reco::parser::MethodSetter, char const*> (actor=..., first=@0x7fff62b81590: 0x7fd557b6c7fe "[0].pt > 2.5", last=@0x7fff62b825a0: 0x7fd557b6c801 ".pt > 2.5") at /cvmfs/cms.cern.ch/slc7_amd64_gcc900/external/boost/1.72.0-ghbfee4/include/boost/spirit/home/classic/core/scanner/scanner.hpp:147
[...]
#108 0x00007fd5768bda62 in boost::spirit::classic::parse<char, boost::spirit::classic::sequence<boost::spirit::classic::impl::entry_grammar<reco::parser::Grammar, 0, boost::spirit::classic::parser_context<boost::spirit::classic::nil_t> >, boost::spirit::classic::end_parser>, boost::spirit::classic::space_parser> (str=0x7fd557b6c7c0 "signalPiZeroCandidates().size() = 0 | signalPiZeroCandidates()[0].pt > 2.5", p=..., skip=...) at /cvmfs/cms.cern.ch/slc7_amd64_gcc900/external/boost/1.72.0-ghbfee4/include/boost/spirit/home/classic/core/scanner/impl/skipper.ipp:173
#109 0x00007fd5768b9147 in reco::parser::cutParser (t=..., cut="signalPiZeroCandidates().size() = 0 | signalPiZeroCandidates()[0].pt > 2.5", sel=std::shared_ptr<reco::parser::SelectorBase> (empty) = {...}, lazy=false) at /mnt/data1/dsr/tmp/CMSSW_11_3_0_pre1/src/CommonTools/Utils/src/cutParser.cc:24
#110 0x00007fd552d7a8d3 in reco::parser::cutParser<reco::PFTau> (cut="signalPiZeroCandidates().size() = 0 | signalPiZeroCandidates()[0].pt > 2.5", sel=std::shared_ptr<reco::parser::SelectorBase> (empty) = {...}, lazy=false) at /mnt/data1/dsr/tmp/CMSSW_11_3_0_pre1/src/CommonTools/Utils/interface/cutParser.h:14
#111 0x00007fd552d79161 in StringCutObjectSelector<reco::PFTau, false>::StringCutObjectSelector (this=0x7fd557b5d238, cut="signalPiZeroCandidates().size() = 0 | signalPiZeroCandidates()[0].pt > 2.5", lazy=false) at /mnt/data1/dsr/tmp/CMSSW_11_3_0_pre1/src/CommonTools/Utils/interface/StringCutObjectSelector.h:18
#112 0x00007fd552ea1553 in reco::tau::RecoTauStringCleanerPlugin::RecoTauStringCleanerPlugin (this=0x7fd557b5d200, pset=..., iC=...) at /mnt/data1/dsr/tmp/CMSSW_11_3_0_pre1/src/RecoTauTag/RecoTau/plugins/RecoTauStringCleanerPlugin.cc:41And the argment type passed to std::vector<T>::operator[] is std::vector<T>::size_type.
https://en.cppreference.com/w/cpp/container/vector/operator_at
Dr15Jones
commented
3 years ago So looking at the TypeWithDict code we can see that the string "vector<ROOT::Experimental::REveTableEntry>::size_type" comes from TMethodArg::GetTypeName()
https://github.com/cms-sw/cmssw/blob/b2180834deb82044f9e8817f22d149b9f77ba7a9/FWCore/Reflection/src/TypeWithDict.cc#L342-L343
Here is the report from https://hypernews.cern.ch/HyperNews/CMS/get/edmFramework/3920.html
With the following cmsDrivers (*), I found the issue when trying to run on the condor. Note that, everything runs fine on lxplus.
(*)
cmsDriver.py ZEE_14TeV_TuneCP5_cfi --mc --conditions auto:phase1_2021_realistic -n 500 --era Run3 --eventcontent FEVTDEBUG -s GEN --datatier GEN --geometry DB:Extended --beamspot Run3RoundOptics25ns13TeVLowSigmaZ --python step1_ZEE_GEN_temp.py --no_exec --fileout file:step1_GEN.root --nThreads 1 --customise_commands "from IOMC.RandomEngine.RandomServiceHelper import RandomNumberServiceHelper ; randSvc = RandomNumberServiceHelper(process.RandomNumberGeneratorService) ; randSvc.populate()\n process.source.firstLuminosityBlock = cms.untracked.uint32(3)"cmsDriver.py step2 --mc --conditions auto:phase1_2021_realistic -n -1 --era Run3 --eventcontent FEVTDEBUG -s SIM --datatier GEN-SIM --beamspot Run3RoundOptics25ns13TeVLowSigmaZ --geometry DB:Extended --python step2_SIM_GFlashNo.py --no_exec --filein file:step1_GEN.root --fileout file:step2_SIM.root --nThreads 8 --customise_commands "from IOMC.RandomEngine.RandomServiceHelper import RandomNumberServiceHelper ; randSvc = RandomNumberServiceHelper(process.RandomNumberGeneratorService) ; randSvc.populate()" --customise Configuration/DataProcessing/Utils.addMonitoringcmsDriver.py step3 --mc --conditions auto:phase1_2021_realistic -s DIGI:pdigi_valid,L1,DIGI2RAW,HLT:@relval2021 --datatier GEN-SIM-DIGI-RAW -n -1 --geometry DB:Extended --era Run3 --eventcontent FEVTDEBUGHLT --python step3_DIGIL1HLT.py --no_exec --filein file:step2_SIM.root --fileout file:step3_DIGIL1HLT.root --nThreads 8 --customise_commands "from IOMC.RandomEngine.RandomServiceHelper import RandomNumberServiceHelper ; randSvc = RandomNumberServiceHelper(process.RandomNumberGeneratorService) ; randSvc.populate()"cmsDriver.py step4 --mc --conditions auto:phase1_2021_realistic -n -1 --era Run3 --eventcontent MINIAODSIM,DQM -s RAW2DIGI,L1Reco,RECO,RECOSIM,EI,PAT,VALIDATION:@standardValidation+@miniAODValidation,DQM:@standardDQM+@ExtraHLT+@miniAODDQM --datatier MINIAODSIM,DQMIO --geometry DB:Extended --python step4_RECO.py --no_exec --filein file:step3_DIGIL1HLT.root --fileout file:step4_RECO.root --nThreads 8(**)
(***)