Prod-v17.2.X - Githubissues

bortigno commented 4 years ago

Issue tracking the production v17.2.X Working on branch prod-v17.2.X

Related to productions tracked in #114 and #115 .

bortigno commented 4 years ago

Here the script for the production is a bit outdated. I need to backport a few things from 2018.

git cherry-pick 72e9260eb1b45c03aaa6b9183f6475ce6da7a1e6
git cherry-pick 426e298e992f3251462d61ffcc5652beb968a783
git cherry-pick 0a205f04353ab210da9d1a417a5b8bc88be4fa90
git cherry-pick 0d22752ad7b7849ab560a502d9173d953d147b94
git cherry-pick ff6316f0ae183594a8dd386db64ba0b5bc581bd4
git cherry-pick 8d9c3ed85268f27896aa9e112ad3e691205d384e
git cherry-pick 01cc34c7a7b48e2c07f90552c95b79fa00dcdd4d
git cherry-pick 0750fe79a5acdf86f88bbe604ec9c1638234ad73
git cherry-pick 445c468a47b14ab2ac103261e874db2509f4e1f5

Making sure of what has been modified:

git diff --name-only master_2017_94X 
DiMuons/crab/make_crab_script.py
DiMuons/crab/templates/crab_config.py
DiMuons/python/Samples.py

All expected. Now I adjust the output dir name to 2017.

git commit -m "Updating crab script to 2017 production after backport of 2018 development" crab/make_crab_script.py

[prod-v17.2.X f8e70bf] Updating crab script to 2017 production after backport of 2018 development 1 file changed, 3 insertions(+), 3 deletions(-)

bortigno commented 4 years ago

I need to update the Sample.py from Samples_2017_94X_v2.py before running a production.

bortigno commented 4 years ago

Ok, updated the sample database using Samples_2017_94X_v2 and some updates from the PdmV twiki https://twiki.cern.ch/twiki/bin/viewauth/CMS/PdmV2017Analysis .

I also updated the pointers from the test analysers. All in 1009a5a and 13380ab (minor bug fix). Now tested with test_analyzer_mc.py and test_analyzer_data.py.

bortigno commented 4 years ago

Creating a tag now and testing a production

git tag -m "Test production v17.2.X" prod-v17.2.0
python crab/make_crab_script.py -s SingleMu_2017B H2Mu_gg_125_NLO ZJets_AMC

Getting annotated tag... Production using code version prod-v17.2.0 starting Production output dir /store/user/bortigno/h2mm/ntuples/2017/94X/prod-v17.2.0 Sample list: [<python.Samples.sample instance at 0x7faa481f3488>, <python.Samples.sample instance at 0x7faa481f3830>, <python.Samples.sample instance at 0x7faa481f3ea8>] crab production directory = crab_2019_12_04_17_56-prod-v17.2.0

Creating analyzer and crab config for SingleMu_2017B

Wrote crab_2019_12_04_17_56-prod-v17.2.0/analyzers/SingleMu_2017B.py

Wrote crab_2019_12_04_17_56-prod-v17.2.0/configs/SingleMu_2017B.py

Creating analyzer and crab config for H2Mu_gg_125_NLO

Wrote crab_2019_12_04_17_56-prod-v17.2.0/analyzers/H2Mu_gg_125_NLO.py

Wrote crab_2019_12_04_17_56-prod-v17.2.0/configs/H2Mu_gg_125_NLO.py

Creating analyzer and crab config for ZJets_AMC

Wrote crab_2019_12_04_17_56-prod-v17.2.0/analyzers/ZJets_AMC.py

Wrote crab_2019_12_04_17_56-prod-v17.2.0/configs/ZJets_AMC.py

Creating submit_all.sh and check_all.sh scripts

Submitting...

./crab_2019_12_04_17_56-prod-v17.2.0/submit_all.sh

bortigno commented 4 years ago

./crab_2019_12_04_17_56-prod-v17.2.0/check_all.sh

Submit fail for space problems (again...)

bortigno commented 4 years ago

eos quota /eos/cms/store/user/bortigno
# pre-configuring default route to /eos/user/b/bortigno/
# -use $EOSHOME variable to override

By user:
┏━> Quota Node: /eos/cms/store/user/
┌──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┐
│user      │used bytes│logi bytes│used files│aval bytes│aval logib│aval files│ filled[%]│vol-status│ino-status│
└──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┘
 bortigno    429.60 GB  214.80 GB    10.00 K    2.00 TB    1.00 TB    10.00 K    21.48 %         ok   exceeded 
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

By group:
┏━> Quota Node: /eos/cms/store/user/
┌──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┐
│group     │used bytes│logi bytes│used files│aval bytes│aval logib│aval files│ filled[%]│vol-status│ino-status│
└──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┘
 zh          396.80 TB  198.40 TB     2.77 M  800.00 TB  400.00 TB    15.00 M    49.60 %         ok         ok 
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

bortigno commented 4 years ago

Removing the 2018 production that is replicated in the group space

rm -rf /eos/cms/store/user/bortigno/h2mm/ntuples/2018/102X/prod-v18.1.6.skim3l*

bortigno commented 4 years ago

Resubmitting

./crab_2019_12_04_18_08-prod-v17.2.0/submit_all.sh

Checking

./crab_2019_12_04_18_08-prod-v17.2.0/check_all.sh

Now they are submitted. Grafana links: SingleMu2018 H2MuMu ZJets_AMC

bortigno commented 4 years ago

prod2017_prod-v17p2p0 test production had 80% success rate for data, 93.3% on signal and 98.5% on ZJets.

Recurring errors are 10034, 8021, 8002, 50660 the last two very common in data.

Data main issues ( memory issues!!! )

29 jobs failed with exit code 8002 14 jobs failed with exit code 50660 1 jobs failed with exit code 50664

8002 - std::exception (other than bad_alloc) 50660 - Application terminated by wrapper because using too much RAM (RSS)

MC main issues ( mainly site errors )

ALERT! 10034 - Required application version is not found at the site (see HERE.) ALERT! 8021 - FileReadError (May be a site error) The following link can help, please check: https://twiki.cern.ch/twiki/bin/view/CMSPublic/WorkBookWhichRelease#DifferentReleases

bortigno commented 4 years ago

Backporting some extra options from 2018 on the make_crab_script.py:

git pull
git cherry-pick 40c3a84

[prod-v17.2.X c221352] Trying automated job splitting Date: Thu Dec 5 16:18:49 2019 +0100 1 file changed, 6 insertions(+), 6 deletions(-)
git cherry-pick fa70119880148aa137cb318d92b88440db6a3c20
[prod-v17.2.X e0fe8e5] Setting longer jobs Date: Fri Dec 6 11:23:37 2019 +0100 1 file changed, 2 insertions(+), 2 deletions(-)
git cherry-pick 3b89b5f
after solving some minor conflicts [prod-v17.2.X 03e4d17] Adding testing options and username fetching for output dir. Date: Fri Dec 6 12:32:03 2019 +0100 1 file changed, 18 insertions(+), 15 deletions(-)

bortigno commented 4 years ago

@eyigitba will produce 2017

bortigno commented 4 years ago

The 2017 EDAnalyzer.py is a bit outdated. I updated it and now testing it

git commit -m "Update EDAnalyzer" crab/templates/EDAnalyzer.py

[prod-v17.2.X 2c23bac] Update EDAnalyzer 1 file changed, 43 insertions(+), 24 deletions(-)

python crab/make_crab_script.py -t
./crab_2019_12_09_18_38-prod-v17.2.1-1-g2c23bac/submit_all.sh

bortigno commented 4 years ago

Updated the EDAnalyzer and the test_ntupliser config.

git commit -m "Update EDAnalyzers and test_ntupliser config" test/test_ntupliser_data.py test/test_ntupliser_mc.py test/test_crab_config.py crab/templates/EDAnalyzer.py

[prod-v17.2.X 99fdd3d] Update EDAnalyzers and test_ntupliser config Date: Mon Dec 9 18:38:04 2019 +0100 4 files changed, 167 insertions(+), 97 deletions(-)

bortigno commented 4 years ago

@eyigitba 2017 is ready for production.

eyigitba commented 4 years ago

Started production with tag prod-v17.2.0.skim3l.

eyigitba commented 4 years ago

Production of 2017 3 lepton skim ntuples is completed. There were failing jobs on a couple of datasets that couldn't be resubmitted. The tasks for following datasets were killed and resubmitted with a new lumi mask to only produce the missing lumi sections:

SingleMu_2017B
SingleMu_2017C
SingleMu_2017D
SingleMu_2017E
SingleMu_2017F

This is done as described here:

https://twiki.cern.ch/twiki/bin/view/CMSPublic/CRAB3FAQ#Dealing_with_a_growing_input_dat

The only difference is we are just using the missinLumi.py file instead of taking differences between lumiMasks.

After the production of the new tasks is completed, I 'hadd'ed all the files 5 files to 1 and moved them to higgs group space without removing the non-hadded files. We can remove them in the future if need be. While doing this I copied all the SingleMu tuples in the most recent date directory and hadded them there.

One more thing to note is that earlier I hadded the DY MC samples 20 to 1, and removed the non-hadded ntuples to save some space. Everything should be fine but we should keep in mind.

The location of the files are below:

/eos/cms/store/group/phys_higgs/HiggsExo/H2Mu/UF/ntuples/2017/94X_v4/prod-v17.2.0.skim3l/

UFLX2MuMu / Ntupliser

Prod-v17.2.X #116