Luminosity calculation for CMS primary datasets

katilp commented 8 years ago

Need to add the instructions (somewhere) on how to compute the luminosity for primary datasets. These instructions are available in CMS-internal pages https://twiki.cern.ch/twiki/bin/view/CMS/TWikiLUM (general information about the luminosity group) and https://twiki.cern.ch/twiki/bin/viewauth/CMS/LumiCalc (quite good instructions on how to use the tool for Run 1 data) Achim will have a look and we can then extract the relevant part of the instructions. We will need to decide how to add this on the portal (a page like getting started or an item in a collection).

tiborsimko commented 8 years ago

Indeed, I get access denied. @RaoOfPhysics can you please have a look and advise on creating either a template page or a documentation record?

katilp commented 8 years ago

We can wait to see how the luminosity calculation goes with the 2011 examples. For the 2011-jets example, Matias is using a script pixelLumiCalc.py which is described in the twiki page linked above (pdf of that page attached here)

CMS_LumiCalc.pdf

katilp commented 7 years ago

Working on the luminosity issue, two options

use pixelLumiCalc.py (as above 1138#issuecomment-236139738) and find out what needs to be added for it to work on Open Data VM (used in Run1 analyses)
use bril and check if it works in Open Data VM (new for Run2 analyses, should work for Run1 as well.

Prefer the brilsw option to avoid two different luminosity tools for Run1 and Run2.

Further details from the luminosity group (Zhen Xie, Jonas Daugalas):

Bril toolkit (brilws) relies on brilconda python environment (no other CMS tools are needed).

Brilconda environment and bril toolkit are not just different options, both are needed.

External user should install bril toolkit locally. User could also install brilconda environment locally, but because brilconda is a big bundle, we suggest that users use centrally installed environment e.g. at /cvmfs/cms.cern.ch/brilconda . The purpose of step "a" is for users to set PATH to brilconda (installed somewhere) before installing/using bril toolkit.

Usage instructions can be found at http://cms-service-lumi.web.cern.ch/cms-service-lumi/brilwsdoc.html Quickstart example to get online luminosity of run 286302 for every lumisection: $ brilcalc lumi -r 286302 --byls

NB: The central installation to e.g. in /cvmfs/cms.cern.ch/brilconda needs to be done, it does not exist yet.

Instructions:

a. export PATH=$HOME/.local/bin:/cvmfs/cms.cern.ch/brilconda/bin:$PATH b. pip install --install-option="--prefix=$HOME/.local" brilws

a. is OK, but b. gives an error:

[cms-opendata@localhost ~]$ pip install --install-option="--prefix=$HOME/.local" brilws Traceback (most recent call last): File "/usr/bin/pip", line 5, in from pkg_resources import load_entry_point File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 16, in import sys, os, zipimport, time, re, imp, types ImportError: /usr/lib/python2.6/lib-dynload/timemodule.so: wrong ELF class: ELFCLASS32 [cms-opendata@localhost ~]$

because there is no central installation of brilconda on /cvmfs/cms.cern.ch/brilconda/ yet. Asked whether this could be provided by the lumi group (it is available on afs and nfs).

katilp commented 7 years ago

Note the information about trigger scales in http://cms-service-lumi.web.cern.ch/cms-service-lumi/brilwsdoc.html#triggerrules

Effective luminosity in a given HLT path is : total_luminosity/(HLT_prescale*L1_prescale).

HLT_prescales can be found for specific run number ranges in trigger configurations in http://opendata.cern.ch/record/1700 (open a file and search for prescales).
Cannot check if the L1 prescales can be found, the L1 records cannot be found ( #1204)

katilp commented 7 years ago

We have agreed with the CMS lumi group that they will provide a simple ascii file with run numbers, lumi sections and luminosity for each lumi section. This will be a record on the portal. We should an additional collection "CMS Luminosity information", and it will contain a record for each year.

It seems that we need to have a collection of collections including many of the existing collections (such as trigger information, condition data and this new luminosity information) @tiborsimko would that be OK?

@RaoOfPhysics we need to add lumi section definition to the glossary: "A lumi section is a fixed time period in data taking, appr 24 seconds" (feel free to correct)

In addition, we will add a single luminosity number together with json file (records 1000, 1001).

katilp commented 7 years ago

For 2011-2012 Run1 (but not for 2010: "Failed to find data table for the requested time range." as the current database only starts from 2011 ) we can use brilcalc to produce the luminosity table:

Instructions in https://cms-service-lumi.web.cern.ch/cms-service-lumi/brilwsdoc.html

on lxplus

cmsrel CMSSW_5_3_32 (although the exact version is not relevant)
cd CMSSW_5_3_32
cmsenv

setenv PATH $HOME/.local/bin:/afs/cern.ch/cms/lumi/brilconda-1.1.7/bin:$PATH
 pip install --install-option="--prefix=$HOME/.local" brilws
 rehash

wget http://opendata.cern.ch/record/1001/files/Cert_160404-180252_7TeV_ReRecoNov08_Collisions11_JSON.txt

brilcalc lumi -i Cert_160404-180252_7TeV_ReRecoNov08_Collisions11_JSON.txt > lumi.txt

produces a table starting with

#Data tag : v1 , Norm tag: onlineresult
+-------------+-------------------+------+------+----------------+---------------+
| run:fill    | time              | nls  | ncms | delivered(/ub) | recorded(/ub) |
+-------------+-------------------+------+------+----------------+---------------+
| 160431:1615 | 03/14/11 03:14:43 | 200  | 200  | 7366.884       | 7366.875      |
| 160577:1622 | 03/16/11 01:44:38 | 53   | 53   | 1719.888       | 1631.386      |
| 160578:1622 | 03/16/11 02:20:03 | 175  | 175  | 5201.105       | 4952.155      |
| 160871:1636 | 03/19/11 02:43:57 | 141  | 141  | 109739.586     | 106354.581    |
| 160872:1636 | 03/19/11 03:50:05 | 38   | 38   | 28346.738      | 23977.109     |
| 160873:1636 | 03/19/11 04:20:20 | 147  | 147  | 104914.534     | 103031.004    |
| 160874:1636 | 03/19/11 05:38:49 | 68   | 68   | 45556.342      | 42570.074     |
| 160939:1639 | 03/20/11 13:48:42 | 123  | 123  | 155001.746     | 151507.313    |
| 160940:1639 | 03/20/11 14:46:32 | 79   | 79   | 95735.589      | 92460.374     |
| 160942:1639 | 03/20/11 15:24:03 | 12   | 12   | 14258.472      | 12350.249     |
| 160943:1639 | 03/20/11 15:34:52 | 53   | 53   | 61416.254      | 59658.876     |
| 160955:1640 | 03/20/11 19:51:24 | 197  | 197  | 693341.494     | 422157.885    |
| 160956:1640 | 03/20/11 21:20:37 | 64   | 64   | 210493.470     | 202147.070    |
| 160957:1640 | 03/20/11 21:49:09 | 952  | 952  | 2641928.099    | 2541799.909   |
| 160998:1642 | 03/21/11 13:23:30 | 251  | 251  | 702215.413     | 688943.153    |
| 161008:1642 | 03/21/11 15:15:48 | 76   | 76   | 198363.430     | 195653.446    |
| 161016:1642 | 03/21/11 17:06:50 | 299  | 299  | 689971.115     | 678523.986    |
| 161103:1644 | 03/22/11 10:17:36 | 99   | 99   | 401846.742     | 389219.344    |
| 161106:1644 | 03/22/11 11:06:02 | 25   | 25   | 95987.700      | 94512.571     |
| 161107:1644 | 03/22/11 11:21:50 | 28   | 28   | 104651.638     | 103130.874    |
| 161113:1644 | 03/22/11 11:37:33 | 23   | 23   | 83228.887      | 81323.086     |

We have the first and last numbers for RunA available in primary datasets description (Run period from run number 160404 to 173692) and the exact numbers in the datasets themselves.

katilp commented 7 years ago

The luminosity summary file for 2011: lumi.txt

katilp commented 7 years ago

Asked Dan Marlow about the 2010 luminosity, we got a list extracted from WBM, but the values come from the online measurements and are not corrected in any way. Nor are they filtered on valid LS. The file is here for record, but not to be used for estimating the luminosity for validated runs and LS: RunSummary.txt

katilp commented 7 years ago

Possible references for 2010 certified luminosity: https://indico.cern.ch/event/123705/contributions/1337348/attachments/72293/103639/azzolini-PVT-March2011-ApprovedPlot.pdf (CMS internal)

Table 6.14 in https://inspirehep.net/record/897920/files/CERN-THESIS-2010-210.pdf summing the three numbers for Muon triggers: 31.79 pb-1

Naming to be clarified: "Golden" and "Muon physics" JSON file? Is the normal JSON (i.e the one on the portal) "Golden"?

-> "Golden" is the normal JSON file.

katilp commented 7 years ago

Luminosity by lumi section for 2010 obtained by lumicalc (recovered from Mikko Voutilainen): https://cernbox.cern.ch/index.php/s/Eu4li5jKn78YIEt

katilp commented 7 years ago

Reference document for the 2010 luminosity calculation https://cds.cern.ch/record/1279145

katilp commented 7 years ago

The 2010 luminosity values have now been added to the lumi DB, and we can now use the brilcalc tool also for the integrated luminosity for 2010 data.

The first RunB run in the JSON file of validated runs http://opendata.cern.ch/record/1000 is 146428. Removing others results in RunB2010_JSON.txt and the corresponding luminosity list obtained on lxplus with

brilcalc lumi -i RunB2010_JSON.txt > 2010RunBlumi.txt is 2010RunBlumi.txt

katilp commented 7 years ago

A possible conclusion of this issue:

Additional information to be added to http://opendata.cern.ch/record/1000 and http://opendata.cern.ch/record/1001

1) Add a new field "Issues & Limitations" with the following text:

record 1000: "This list covers all p-p data taking in 2010. The public data (RunB) is part of this list, between run numbers 146428 and 149294"
record 1001: "The list covers all p-p data taking in 2011. The public data (RunA) is part of this list, between run numbers 160431 to 173692"

2) Add a new field "How to obtain the integrated luminosity corresponding to this list?" with the following text:

record 1000: "The integrated luminosity for validated runs and luminosity sections of the 2010 public data (RunB) is available in this table 2010RunBlumi.txt. (The integrated luminosity for validated runs and luminosity sections of all 2010 p-p data taking is available in 2010lumi.txt.) In your estimate for the integrated luminosity, check for which runs the trigger you have selected is active and sum the values for those runs. The uncertainty in the luminosity measurement of 2010 data should be considered as 11%. (ref https://cds.cern.ch/record/1279145/files/EWK-10-004-pas.pdf)"
record 1001: "The integrated luminosity for validated runs and luminosity sections of the 2011 public data (RunA) is available in this table 2011RunAlumi.txt. (The integrated luminosity for validated runs and luminosity sections of all 2010 p-p data taking is available in 2011lumi.txt.) In your estimate for the integrated luminosity, check for which runs the trigger you have selected is active and sum the values for those runs. The uncertainty in the luminosity measurement of 2011 data should be considered as 2.2% (ref https://cds.cern.ch/record/1434360/files/SMP-12-008-pas.pdf) If you are using prescaled triggers, you can find the prescale factors for different run ranges in trigger configuration files listed in http://opendata.cern.ch/record/1700 (under "PrescaleService"). Note that these factors may change during a run , and the change of prescales (run, lumi section, index of prescales) is recorded in prescale2011.txt. In this case you may also need detailed list of luminosity by lumi section in (https://cernbox.cern.ch/index.php/s/ggjnUw99fAsuNZx TO BE COPIED from this link to the portal).

TBD:

[x] check the message and the check list in the end of the table above (runs is json but not in results)
- this is normal: it happens because Run and LS has been declared good through offline validation but online luminometers have not picked signal and therefore there is no value for them in the lumi DB
- [ ] add a brief explanation about this
[x] provide the table for 2011 RunA
[ ] check the full text with the lumi group, also the reference to the errors: for 2011, there is also an earlier PAS with 4.5%. (ref https://cds.cern.ch/record/1376102/files/EWK-11-001-pas.pdf)
[x] decide whether the record should also include the luminosity table for the full list (RunA + RunB for each year)
- include also the full list for the sake of consistency (json for the full list - > luminosity for the full list), and if so decided, to make it easier to make full data available in the future

katilp commented 7 years ago

Luminosity by lumisection in csv brilcalc lumi --byls -i RunB2010_JSON.txt -o checklumils.csv

Brilcalc also gives online changes for trigger prescales:

[lxplus059.cern.ch]~/scratch1/CMSSW_5_3_32 $brilcalc trg --prescale -r 173243 +--------+-------+----------+ | run | cmsls | prescidx | +--------+-------+----------+ | 173243 | 1 | 5 | | 173243 | 14 | 4 | +--------+-------+----------+

TBC: cmsls in the lumi section, prescidx is the index of the prescale sets, which can be found the in trigger config files (trigger menus) for different run ranges in http://opendata.cern.ch/record/1700 Example, for run 173243, go to see the corresponding trigger config i.e. http://opendata.cern.ch/record/3528 and http://opendata.cern.ch/record/3528/files/cdaq_physics_Run2011_3e33_v1.1_HLT_V3.py look for PrescaleService and find:

process.PrescaleService = cms.Service( "PrescaleService",
    lvl1DefaultLabel = cms.untracked.string( "1e33" ),
    lvl1Labels = cms.vstring( "5e33",
       "4e33",
       "3e33",
       "2.5e33",
       "2e33",
       "1.4e33",
       "1e33",
       "7e32",
       "5e32" ),
    prescaleTable = cms.VPSet(
      cms.PSet(
        pathName = cms.string( "HLT_Activity_Ecal_SC7_v8" ),
        prescales = cms.vuint32( 150, 100, 75, 65, 50, 35, 25, 15, 10 )
      ),
      cms.PSet(
        pathName = cms.string( "HLT_L1SingleJet16_v4" ),
        prescales = cms.vuint32( 50, 50, 50, 50, 50, 50, 50, 50, 50 )
      ),

In this case from ls 1, the set of prescales n=5 is used, from ls 14, n=4 is used. Numbering from n=0?

katilp commented 7 years ago

The instructions above (https://github.com/cernopendata/opendata.cern.ch/issues/1138#issuecomment-292106419) have been updated with the list of prescales (brilcalc trg -- prescale) for 2011 (run numbers 160403 - 183126). The corresponding list for 2012 (run numbers 190389 - 211812) is in prescale2012.txt

The detailed luminosity by lumi section is obtained with brilcalc lumi --byls -i Cert_160404-180252_7TeV_ReRecoNov08_Collisions11_JSON.txt -o 2011lumibyls.csv

https://cernbox.cern.ch/index.php/s/ggjnUw99fAsuNZx

@tiborsimko This is now ready to go

tiborsimko commented 7 years ago

Note: add "preliminary information" warning in the list of validated runs.

tiborsimko commented 7 years ago

Note: just discussed with @katilp that the luminosity information records could be separated out and put into new collection "CMS Luminosity Information".... with attached summary tables and perhaps also the prescale file.

tiborsimko commented 7 years ago

@katilp Can you please check http://opendata.cern.ch/collection/CMS-Luminosity-Information and let me know any quick-fixes to make for the lumi information, in case it comes up this afternoon

tiborsimko commented 7 years ago

CMS Validated Runs records are also updated regarding the run range limitations:

but we don't particularly link to the luminosity information:

Please let me know in this issue if we should interlink them in some nice way, or whether this is OK and the issue can be closed?

tiborsimko commented 7 years ago

BTW @ArtemisLav you can assign DOIs for the two new luminosity records and add any standard fields I may have omitted, e.g. I noticed I should have probably put 2010 in the field 264 in the record 1050...

katilp commented 7 years ago

@tiborsimko Looks OK. For readability, it would be good to have separated in paragraphs. In 1051, Please add "HLT trigger" so that it becomes "find the HLT trigger prescale factors for different run ranges"...

and add a mention that the detailed list is for the validated runs i.e. "detailed list of luminosity by lumi section in 2011lumibyls.csv for the list of validated runs and lumi sections (http://opendata.cern.ch/record/1001)"

ArtemisLav commented 7 years ago

@tiborsimko done; let me know when it is live, so that I can mint the DOIs.

tiborsimko commented 7 years ago

@ArtemisLav Deployed in PROD, you can register those two DOIs.

tiborsimko commented 7 years ago

@katilp All done.

cernopendata / opendata.cern.ch

Luminosity calculation for CMS primary datasets #1138