Prod-v18.1.X

[bortigno]

Production of 2018. Continuing the work of #66 on branch prod-v18.1.X.

[bortigno]

Added JEC from DB #60. I merged already to branch prod-v18.1.X but did a PR to master_2018_102X.

[bortigno]

Updating samples #65 I wrote a simple script checkSamplesOnDAS.py (now pushed in my sandbox) that is using the DBS api to check the actual name of the samples and cross check their existence (typos, validity, etc)

[bortigno]

2a7b882bbfaf01f6e482307abbada3449e7e28e8 Merged branch #65 with latest and greatest 2018 samples.

[bortigno]

Ready to launch.

git tag -m "First attempt of prod-v18.1.X" prod-v18.1.0
>> crab_2019_05_21_17_35-prod-v18.1.1

I need to make a little hack for the DYJets (ZJets_hiM_MG) that is still in PRODUCTION. In the crab config of the sample I added the following line:

Data.allowNonValidInputDataset = True

And now testing directly on this one with --dryrun.

crab submit -c crab_2019_05_21_17_35-prod-v18.1.1/configs/ZJets_hiM_MG.py

There is an error related to the DB file:

...
==== CMSSW Stack Execution FAILED at Tue May 21 15:45:25 2019 ====
======== CMSSW OUTPUT STARTING ========
NOTICE: lines longer than 3000 characters will be truncated
== CMSSW: Setting up Frontier log level
== CMSSW: Beginning CMSSW wrapper script
== CMSSW:  slc6_amd64_gcc700 scramv1 CMSSW
== CMSSW: Performing SCRAM setup...
== CMSSW: Completed SCRAM setup
== CMSSW: Retrieving SCRAM project...
== CMSSW: Completed SCRAM project
== CMSSW: Executing CMSSW
== CMSSW: cmsRun  -j FrameworkJobReport.xml PSet.py
== CMSSW: ----- Begin Fatal Exception 21-May-2019 17:45:25 CEST-----------------------
== CMSSW: An exception of category 'StdException' occurred while
== CMSSW:    [0] Constructing the EventProcessor
== CMSSW:    [1] Constructing ESSource: class=PoolDBESSource label='jec'
== CMSSW: Exception Message:
== CMSSW: A std::exception was thrown.
== CMSSW: Connection on "sqlite:data/JEC/Autumn18_V8_MC.db" cannot be established ( CORAL : "ConnectionPool::getSessionFromNewConnection" from "CORAL/Services/ConnectionService" )
== CMSSW: ----- End Fatal Exception -------------------------------------------------
== CMSSW: Complete
== CMSSW: process id is 22877 status is 66
======== CMSSW OUTPUT FINSHING ========
ERROR: Caught WMExecutionFailure - code = 66 - name = CmsRunFailure - detail = Error running cmsRun
{'arguments': ['/bin/bash', '/tmp/bortigno/tmpZa_YqF/cmsRun-main.sh', '', 'slc6_amd64_gcc700', 'scramv1', 'CMSSW', 'CMSSW_10_2_11_patch1', 'FrameworkJobReport.xml', 'cmsRun', 'PSet.py', 'sandbox.tar.gz', '', '']}
Return code: 66
...

I think the path to the file is not correct for crab. Checking the other data/ files I use in the config to see if the have the same path.

It appears I need to give the path from Ntupliser/... I changed it in crab/template/EDAnlyser.py and I am testing it now after committing it.

git commit -m "bugfix on the db file path. For the crab config need to be from Ntupliser" crab/templates/EDAnalyzer.py
python crab/make_crab_script.py
# editing the Jets config to allow non valid status
vi crab_2019_05_21_17_57-prod-v18.1.1-1-g68875f8/configs/ZJets_hiM_MG.py 
crab submit -c crab_2019_05_21_17_57-prod-v18.1.1-1-g68875f8/configs/ZJets_hiM_MG.py --dryrun

I had problem with sqlite file until I find out that I need to use the sqlite_fip:<fileName> for crab, which is using FileInPath technology. More info at the twiki https://twiki.cern.ch/twiki/bin/view/CMSPublic/WorkBookJetEnergyCorrections#JecSqliteFile and https://twiki.cern.ch/twiki/bin/view/CMSPublic/SWGuideEdmFileInPath

Now I launched with v18.1.2. On a screen of "lxplus602"

[bortigno]

Snapshot of the production https://monit-grafana.cern.ch/dashboard/snapshot/Mywi6W1LLZ6a6lQn3hiz5l9htpJ3VvCy

[bortigno]

Monitoring link https://monit-grafana.cern.ch/d/cmsTMGlobal/cms-tasks-monitoring-globalview?orgId=11&var-user=bortigno&var-site=All&var-current_url=%2Fd%2FcmsTMDetail%2Fcms_task_monitoring&var-task=All&from=1557302148292&to=1558511748292

[bortigno]

Now the next step is to launch a 3 leptons skim. Let's call it v18.1.2.skim3lep

After changing the nMuon skim values to 3 in crab/templates/EDAnalyzer.py

git commit -m "Skim 3 leptons" crab/templates/EDAnalyzer.py
git tag -m "Same as v18.1.2 but skimming 3 leptons." prod-v18.1.2.skim3l
python crab/make_crab_script.py 

Then again I need to make a little hack for the DYJets (ZJets_hiM_MG) that is still in PRODUCTION. In the crab config of the sample I added the following line:

Data.allowNonValidInputDataset = True

now testing and launching the production:

crab submit -c crab_2019_05_22_10_18-prod-v18.1.2.skim3l/configs/ZJets_hiM_MG.py --dryrun

output of which is positive:

Will use CRAB configuration file crab_2019_05_22_10_18-prod-v18.1.2.skim3l/configs/ZJets_hiM_MG.py
Importing CMSSW configuration crab_2019_05_22_10_18-prod-v18.1.2.skim3l/analyzers/ZJets_hiM_MG.py
Finished importing CMSSW configuration crab_2019_05_22_10_18-prod-v18.1.2.skim3l/analyzers/ZJets_hiM_MG.py
Sending the request to the server
Success: Your task has been delivered to the CRAB3 server.
Waiting for task to be processed
Checking task status
Task status: NEW
Please wait...
Task status: NEW
Please wait...
Task status: QUEUED
Please wait...
Task status: UPLOADED

Creating temporary directory for dry run sandbox in /tmp/bortigno/tmpcpX9dP
Executing test, please wait...

Using FileBased splitting
Task consists of 164 jobs to process 770 files
The longest job will process 5 files, with an estimated processing time of 2855 minutes
The average job will process 4 files, with an estimated processing time of 1972 minutes
The shortest job will process 1 files, with an estimated processing time of 331 minutes
The estimated memory requirement is 839 MB

Timing quantities given below are ESTIMATES. Keep in mind that external factors
such as transient file-access delays can reduce estimate reliability.

For ~480 minute jobs, use:
Data.unitsPerJob = 1
You will need to submit a new task

Dry run requested: task paused
To continue processing, use 'crab proceed'

Log file is /afs/cern.ch/work/b/bortigno/x2mm18_10211p1/src/Ntupliser/DiMuons/logs/crab_ZJets_hiM_MG_2019_05_22_10_18_prod2018_prod-v18p1p2pskim3l/crab.log

so I can launch in screen :

bortigno@lxplus623:~/workspace/x2mm18_10211p1/src/Ntupliser/DiMuons$ screen
cmsenv
./crab_2019_05_22_10_18-prod-v18.1.2.skim3l/submit_all.sh

[bortigno]

Two releases prepared and a PR #71 open to master_2018

[bortigno]

Monitoring the samples I realised there was a problem with the dataset name of SingleMu_2018D. I fixed it with 5b226d8 and relaunched it.

[bortigno]

Some more technical info for documentation.

# here the last commit I had was the tag for the skim d2caf2f
git checkout prod-v18.1.X 
# here I fix the bug
vi python/Samples.py
git commit -m "bugfix. Fix dataset name for SingleMu2018D" python/Samples.py
  [prod-v18.1.X 5b226d8] bugfix. Fix dataset name for SingleMu2018D
   1 file changed, 2 insertions(+), 2 deletions(-)

git push
  Counting objects: 5, done.
  Delta compression using up to 10 threads.
  Compressing objects: 100% (5/5), done.
  Writing objects: 100% (5/5), 535 bytes | 535.00 KiB/s, done.
  Total 5 (delta 3), reused 0 (delta 0)
  remote: Resolving deltas: 100% (3/3), completed with 3 local objects.
  To github.com:UFLX2MuMu/Ntupliser.git
     d2caf2f..5b226d8  prod-v18.1.X -> prod-v18.1.X

python crab/make_crab_script.py 
  Getting annotated tag...
  Production using code version prod-v18.1.2.skim3l-1-g5b226d8 starting
  crab production directory = crab_2019_05_22_15_48-prod-v18.1.2.skim3l-1-g5b226d8
  Creating analyzer and crab config for SingleMu_2018D
    * Wrote crab_2019_05_22_15_48-prod-v18.1.2.skim3l-1-g5b226d8/analyzers/SingleMu_2018D.py
    * Wrote crab_2019_05_22_15_48-prod-v18.1.2.skim3l-1-g5b226d8/configs/SingleMu_2018D.py
  Creating submit_all.sh and check_all.sh scripts

./crab_2019_05_22_15_48-prod-v18.1.2.skim3l-1-g5b226d8/submit_all.sh 

Then here I wanted to move back to the tag without the 3lepton skim (prod-v18.1.2) and apply the fix on top of it and then launch.

git checkout tags/prod-v18.1.2
  M DiMuons/crab/make_crab_script.py
  Note: checking out 'tags/prod-v18.1.2'.
  You are in 'detached HEAD' state. You can look around, make experimental
  changes and commit them, and you can discard any commits you make in this
  state without impacting any branches by performing another checkout.
  HEAD is now at 527a112 bugfix on the db file path. For the crab ned to use sqlite_fip (file in path).

git checkout -b prod-v18.1.2-bugfix
  M DiMuons/crab/make_crab_script.py
  Switched to a new branch 'prod-v18.1.2-bugfix'

git cherry-pick 5b226d8
python crab/make_crab_script.py 
  Getting annotated tag...
  Production using code version prod-v18.1.2-1-g2666d57 starting
  crab production directory = crab_2019_05_22_16_01-prod-v18.1.2-1-g2666d57
  Creating analyzer and crab config for SingleMu_2018D
    * Wrote crab_2019_05_22_16_01-prod-v18.1.2-1-g2666d57/analyzers/SingleMu_2018D.py
    * Wrote crab_2019_05_22_16_01-prod-v18.1.2-1-g2666d57/configs/SingleMu_2018D.py

./crab_2019_05_22_16_01-prod-v18.1.2-1-g2666d57/submit_all.sh 

[bortigno]

Now the branch has been messed out, or at least has been split in two: prod-v18.1.2-bugfix --> for the 2 lepton skim prod-v18.1.2 --> for the 3lepton skim

[bortigno]

Some extra development. I added to the sample database the ggZZ samples.

git checkout issue-65-update-samples-2018
git merge prod-v18.1.2-bugfix
# here adding the ggZZ samples
vi python/Samples.py
git commit -m "Icnluding ggZZ 2018 samples in the sample database" python/Samples.py
  [issue-65-update-samples-2018 c71db59] Icnluding ggZZ 2018 samples in the sample database
   1 file changed, 30 insertions(+)
python checkSamplesOnDAS.py 
  /DYJetsToLL_M-105To160_TuneCP5_PSweights_13TeV-madgraphMLM-pythia8/RunIIAutumn18MiniAOD-102X_upgrade2018_realistic_v15-v1/MINIAODSIM does not correspond to any VALID DAS sample.
  Dataset [{'dataset': '/DYJetsToLL_M-105To160_TuneCP5_PSweights_13TeV-madgraphMLM-pythia8/RunIIAutumn18MiniAOD-102X_upgrade2018_realistic_v15-v1/MINIAODSIM'}] is in PRODUCTION state.

git checkout prod-v18.1.X
git merge --no-ff -m "Merging the ggZZ inclusion from issue65-update-samples-2018" issue-65-update-samples-2018
git push
# selection to run only on the missing files
vi crab/make_crab_script.py 
python crab/make_crab_script.py 
  Getting annotated tag...
  Production using code version prod-v18.1.2.skim3l-4-gf7ca38c starting
  crab production directory = crab_2019_05_22_17_04-prod-v18.1.2.skim3l-4-gf7ca38c

  Creating analyzer and crab config for ggZZ_4mu
  * Wrote crab_2019_05_22_17_04-prod-v18.1.2.skim3l-4-gf7ca38c/analyzers/ggZZ_4mu.py
  * Wrote crab_2019_05_22_17_04-prod-v18.1.2.skim3l-4-gf7ca38c/configs/ggZZ_4mu.py

  Creating analyzer and crab config for ggZZ_2e2mu
  * Wrote crab_2019_05_22_17_04-prod-v18.1.2.skim3l-4-gf7ca38c/analyzers/ggZZ_2e2mu.py
  * Wrote crab_2019_05_22_17_04-prod-v18.1.2.skim3l-4-gf7ca38c/configs/ggZZ_2e2mu.py

  Creating analyzer and crab config for ggZZ_2mu2tau
  * Wrote crab_2019_05_22_17_04-prod-v18.1.2.skim3l-4-gf7ca38c/analyzers/ggZZ_2mu2tau.py
  * Wrote crab_2019_05_22_17_04-prod-v18.1.2.skim3l-4-gf7ca38c/configs/ggZZ_2mu2tau.py

  Creating analyzer and crab config for ggZZ_2e2tau
  * Wrote crab_2019_05_22_17_04-prod-v18.1.2.skim3l-4-gf7ca38c/analyzers/ggZZ_2e2tau.py
  * Wrote crab_2019_05_22_17_04-prod-v18.1.2.skim3l-4-gf7ca38c/configs/ggZZ_2e2tau.py

  Creating analyzer and crab config for ggZZ_2mu2nu
  * Wrote crab_2019_05_22_17_04-prod-v18.1.2.skim3l-4-gf7ca38c/analyzers/ggZZ_2mu2nu.py
  * Wrote crab_2019_05_22_17_04-prod-v18.1.2.skim3l-4-gf7ca38c/configs/ggZZ_2mu2nu.py

  Creating analyzer and crab config for ggZZ_4tau
  * Wrote crab_2019_05_22_17_04-prod-v18.1.2.skim3l-4-gf7ca38c/analyzers/ggZZ_4tau.py
  * Wrote crab_2019_05_22_17_04-prod-v18.1.2.skim3l-4-gf7ca38c/configs/ggZZ_4tau.py

./crab_2019_05_22_17_04-prod-v18.1.2.skim3l-4-gf7ca38c/submit_all.sh 

I am not going to launch these samples for the 2 lepton skim.

[bortigno]

The MC of the 3lepton skim didn't work for some file system problem.

Need to resubmit. Before resubmitting with tag prod-v18.1.3.skim3l I added the tag name to the output dir to make it easier to connect the two 0d22752. I also reduced the number of events per job for the data to reduce the number of failure due to too much clock time and too much RAM.

I also, as usual, modified crab_2019_05_27_13_00-prod-v18.1.3.skim3l/configs/ZJets_hiM_MG.py to include the dataset in PRODUCTION.

Then submitted

./crab_2019_05_27_13_00-prod-v18.1.3.skim3l/submit_all.sh

[bortigno]

Andrew discovered another problem: the v18.1.3.skim3l production was a 3 muon skim instead of a 3 lepton skim. I need to re-submit another production: prod-v18.1.4.skim3l fixing this issue in 238a131.

At the same time I killed the prod-v18.1.3.skim3l remaining jobs. I should also removed them from EOS.

[bortigno]

Fixing a bug on the output dir name in 8d9c3ed. Creating a new tag and resubmitting prod-v18.1.6.skim3l