katilp
commented
1 year ago Start with example datasets:
/ADDmonoPhoton_MD-1_d-3_TuneCP5_13TeV-pythia8/RunIISummer20UL16NanoAODv9-106X_mcRun2_asymptotic_v17-v2/NANOAODSIM /BBH_HToJPsiG_JPsiToMuMu_TuneCP5_13TeV-madgraph-pythia8/RunIISummer20UL16MiniAODv2-106X_mcRun2_asymptotic_v17-v1/MINIAODSIM
Expected changes in the scripts:
Called from interface.py:
- option
create-das-json-storedoes not need to query configs and parent, we get them all from mcm, in practice:- remove these lines for the parent loop
- for the record, the dasgoclient query for query "dataset" should also return
prep_idwhich can be used for the McM query
- option
create-mcm-storewill not need to proceed through input/output datasets, now we can work entirely withprep_ids, in practice:- looping through parents can be replaced by querying the chain ( = the list containing
pred_ids of the datasets in the provenance chain)
- looping through parents can be replaced by querying the chain ( = the list containing
get-conf-fileswill then get the config file list only from McM, some changes in the code may be neededcreat-recordswill have similar updates as were done in the collision record scripts
lhe_generators.py is called separately (see e.g. 2015 readme):
- NB, the gridpack location for pre-2017 production has changed
- the config files have e.g.
/cvmfs/cms.cern.ch/phys_generator/gridpacks/slc6_amd64_gcc481/13TeV/madgraph/V5_2.2.2/SingleBp/Bpb_M900GeV_W9GeV_Zb_LH_tarball.tar.xz(in an example LHE config file) but the /cvmfs area has them separately:-bash-4.2$ ls /cvmfs/cms.cern.ch/phys_generator/gridpacks/ 13p6TeV gridpacks pre2017 slc6_amd64_gcc472 slc6_amd64_gcc530 slc7_amd64_gcc630 UL 14TeV lhe_merger RunII slc6_amd64_gcc481 slc6_amd64_gcc630 slc7_amd64_gcc700 untar 2017 mg_amg_patch RunIII slc6_amd64_gcc491 slc6_amd64_gcc700 slc7_amd64_gcc820 2018 PdmV RunUL slc6_amd64_gcc493 slc7_amd64_gcc10 slc7_amd64_gcc900 -bash-4.2$ ls /cvmfs/cms.cern.ch/phys_generator/gridpacks/pre2017 13TeV 14TeV
- the config files have e.g.
The current script gets the provenance information as follows
As the processing scheme has changed from UL processing (no input datasets before AODSIM as they were transient) this won't work anymore.
The query flow should be changed to go directly to the chain:
For an example dataset /ADDmonoPhoton_MD-1_d-3_TuneCP5_13TeV-pythia8/RunIISummer20UL16NanoAODv9-106X_mcRun2_asymptotic_v17-v2/NANOAODSIM:
On the web GUI:
Query by the output file name:
https://cms-pdmv.cern.ch/mcm/requests?produce=%2FADDmonoPhoton_MD-1_d-3_TuneCP5_13TeV-pythia8%2FRunIISummer20UL16NanoAODv9-106X_mcRun2_asymptotic_v17-v2%2FNANOAODSIM&page=0&shown=140737488355327
https://cms-pdmv.cern.ch/mcm/chained_requests?contains=EXO-RunIISummer20UL16NanoAODv9-00205&page=0
then for each request of the query and get the dicts in the respective pages.
On the command line
Using
pred_idfrom dasget the chain from the dictionary:
curl -s -k https://cms-pdmv.cern.ch/mcm/public/restapi/requests/get/EXO-RunIISummer20UL16NanoAODv9-00205get the id of the chained request
auth-get-sso-cookie -u https://cms-pdmv.cern.ch/mcm -o cookies.txt(see also docs)then, for each step in the chain, get the full dict or what is needed, e.g.
An example with the LHE step: