search

cms-l1t-offline / cmssw

CMS Offline Software
cms-sw.github.io/cmssw
Apache License 2.0
17 stars 27 forks source link

L1TriggerObjectMap and Correlation Triggers #362

Open rekovic opened 8 years ago

rekovic commented 8 years ago

In the HN [1] there was a mention of similar (potentially related) problems with DoubleMu triggers with correlation, which seem to have caused the corresponding HLT paths not to fire [2].

I tracked this down and identified that HLTL1TSeed filter does not pass because of the FAIL decisions in the L1TriggerObjectMap.

Looking in MuOnia dataset, in run 273291, events that PASS in the uGT for DoubleMu0er1p6_dEta_Max1p8 do FAIL in L1TriggerObjectMap [3]. This is not the case for run 274240, where L1TriggerObjectMap and unpacked uGT decisions agree fully and both PASS.

I re-run the L1TriggerObjectMap (i.e. re-emulate uGT) with the latest greatest version of the emulator, and the same problem is observed, that is to say the decision in the ObjectMap is still FAIL for those events. To reproduce this follow the recipe:

cmsrel CMSSW_8_0_9
cd CMSSW_8_0_9/src
cmsenv
git cms-init
git remote add cms-l1t-offline git@github.com:cms-l1t-offline/cmssw.git
git fetch cms-l1t-offline
git checkout -b mydevel
git cms-merge-topic -u cms-l1t-offline:l1t-integration-v67.3
git cms-addpkg L1Trigger/L1TCommon L1Trigger/Configuration
scram b -j 8

#######################################################
#
# re-make L1TriggerObjectMap, and dump L1TriggerObjectMap decisions
#
#######################################################
cmsDriver.py --python_filename=l1t_test_RAW2DIGI_L1THLTDemo.py -s RAW2DIGI --era=Run2_2016 --customise=L1Trigger/L1TCommon/customiseDemo.L1THLTDemo --customise=L1Trigger/Configuration/customiseUtils.L1TTurnOffUnpackStage2GtGmtAndCalo --conditions=auto:run2_data -n 10 --data --no_output --filein=root://cms-xrd-global.cern.ch///store/data/Run2016B/MuOnia/RAW/v2/000/273/291/00000/3A8F68C5-CB18-E611-B040-02163E011C13.root --customise=L1Trigger/Configuration/customiseUtils.L1TGlobalDigisSummary --customise=L1Trigger/Configuration/customiseUtils.L1TAddInfoOutput --customise_commands 'process.l1tGlobalSummary.AlgInputTag = cms.InputTag("hltGtStage2ObjectMap") \n' |& tee run273291_dump_command_hltGtStage2ObjectMap.log

#######################################################
#
# re-make L1TriggerObjectMap, but dump unpacked GT decisions
#
#######################################################
cmsDriver.py --python_filename=l1t_test_RAW2DIGI_L1THLTDemo.py -s RAW2DIGI --era=Run2_2016 --customise=L1Trigger/L1TCommon/customiseDemo.L1THLTDemo --customise=L1Trigger/Configuration/customiseUtils.L1TTurnOffUnpackStage2GtGmtAndCalo --conditions=auto:run2_data -n 10 --data --no_output --filein=root://cms-xrd-global.cern.ch///store/data/Run2016B/MuOnia/RAW/v2/000/273/291/00000/3A8F68C5-CB18-E611-B040-02163E011C13.root --customise=L1Trigger/Configuration/customiseUtils.L1TGlobalDigisSummary --customise=L1Trigger/Configuration/customiseUtils.L1TAddInfoOutput --customise_commands 'process.l1tGlobalSummary.AlgInputTag = cms.InputTag("hltGtStage2Digis") \n' |& tee run273291_dump_command_hltGtStage2Digi.log

Grep in log files for the L1TriggerObjectMap decisions decisions for L1_DoubleMu0er1p6_dEta_Max1p8: 

 L1T menu Name   : L1Menu_Collisions2016_v1c
 L1T menu Version: 0.1
 L1T menu Comment: 2016 pp physics menu. Algorithms with correlation conditions except for b32 and b35 as place holders

    Bit                  Algorithm Name                  Init    PScd  Final   PS Factor     Masked    Veto
============================================================================================================
 >  grep "32              L1_DoubleMu0er1p6_dEta_Max1p8" run273291_dump_command_hltGtStage2ObjectMap.log
      32              L1_DoubleMu0er1p6_dEta_Max1p8         0      0      0         1          1        0
      32              L1_DoubleMu0er1p6_dEta_Max1p8         0      0      0         1          1        0
      32              L1_DoubleMu0er1p6_dEta_Max1p8         0      0      0         1          1        0
      32              L1_DoubleMu0er1p6_dEta_Max1p8         0      0      0         1          1        0
      32              L1_DoubleMu0er1p6_dEta_Max1p8         0      0      0         1          1        0
      32              L1_DoubleMu0er1p6_dEta_Max1p8         0      0      0         1          1        0
      32              L1_DoubleMu0er1p6_dEta_Max1p8         0      0      0         1          1        0
      32              L1_DoubleMu0er1p6_dEta_Max1p8         0      0      0         1          1        0
      32              L1_DoubleMu0er1p6_dEta_Max1p8         0      0      0         1          1        0
      32              L1_DoubleMu0er1p6_dEta_Max1p8         0      0      0         1          1        0
      32              L1_DoubleMu0er1p6_dEta_Max1p8         0      0      0         1          1        0

Grep in log files for the unpacked uGT decisions decisions for L1_DoubleMu0er1p6_dEta_Max1p8: 

 L1T menu Name   : L1Menu_Collisions2016_v1c
 L1T menu Version: 0.1
 L1T menu Comment: 2016 pp physics menu. Algorithms with correlation conditions except for b32 and b35 as place holders

    Bit                  Algorithm Name                  Init    PScd  Final   PS Factor     Masked    Veto
============================================================================================================
 >  grep "32              L1_DoubleMu0er1p6_dEta_Max1p8" run273291_dump_command_hltGtStage2Digi.log
      32              L1_DoubleMu0er1p6_dEta_Max1p8         0      0      0         1          1        0
      32              L1_DoubleMu0er1p6_dEta_Max1p8         1      1      1         1          1        0
      32              L1_DoubleMu0er1p6_dEta_Max1p8         0      0      0         1          1        0
      32              L1_DoubleMu0er1p6_dEta_Max1p8         1      1      1         1          1        0
      32              L1_DoubleMu0er1p6_dEta_Max1p8         0      0      0         1          1        0
      32              L1_DoubleMu0er1p6_dEta_Max1p8         1      1      1         1          1        0
      32              L1_DoubleMu0er1p6_dEta_Max1p8         1      1      1         1          1        0
      32              L1_DoubleMu0er1p6_dEta_Max1p8         1      1      1         1          1        0
      32              L1_DoubleMu0er1p6_dEta_Max1p8         0      0      0         1          1        0
      32              L1_DoubleMu0er1p6_dEta_Max1p8         0      0      0         1          1        0
      32              L1_DoubleMu0er1p6_dEta_Max1p8         5      5      5         1          1        0

[1] https://hypernews.cern.ch/HyperNews/CMS/get/L1Trigger/1375.html [2] https://indico.cern.ch/event/532752/contributions/2170252/subcontributions/196797/attachments/1292112/1924972/BPH_TSG_15062016.pdf [3] /afs/cern.ch/user/r/rekovic/public/forL1TStage2/HLTL1T/273291/run_273291.detailedInfo.log.L1_DoubleMu0er1p6_dEta_Max1p8_OR_L1_DoubleMu0er1p6_dEta_Max1p8_OS [4] /afs/cern.ch/user/r/rekovic/public/forL1TStage2/HLTL1T/274240/run_274240.detailedInfo.log.L1_DoubleMu0er1p6_dEta_Max1p8_OR_L1_DoubleMu0er1p6_dEta_Max1p8_OS

rekovic commented 8 years ago

Using the GT --conditions=80X_dataRun2_Prompt_v8 does change the menu used, from L1Menu_Collisions2016_v1c to L1Menu_Collisions2016_v2c.
However, it does not change the results of the re-emulation of uGT. Still FAIL events for L1_DoubleMu0er1p6_dEta_Max1p8.

cmsDriver.py --python_filename=l1t_test_RAW2DIGI_L1THLTDemo.py -s RAW2DIGI --era=Run2_2016 --customise=L1Trigger/L1TCommon/customiseDemo.L1THLTDemo --customise=L1Trigger/Configuration/customiseUtils.L1TTurnOffUnpackStage2GtGmtAndCalo --conditions=80X_dataRun2_Prompt_v8 -n 10 --data --no_output --filein=root://cms-xrd-global.cern.ch///store/data/Run2016B/MuOnia/RAW/v2/000/273/291/00000/3A8F68C5-CB18-E611-B040-02163E011C13.root --customise=L1Trigger/Configuration/customiseUtils.L1TGlobalDigisSummary --customise=L1Trigger/Configuration/customiseUtils.L1TAddInfoOutput --customise_commands 'process.l1tGlobalSummary.AlgInputTag = cms.InputTag("hltGtStage2ObjectMap") \n' |& tee run273291_dump_command_hltGtStage2ObjectMap_80X_dataRun2_Prompt_v8.log

Grep in log file to see re-emulated decisions:

 L1T menu Name   : L1Menu_Collisions2016_v2c
 L1T menu Version: 0.1
 L1T menu Comment: 2016 pp physics menu. Algorithms with correlation conditions except for b32 and b35 as place holders

    Bit                  Algorithm Name                  Init    PScd  Final   PS Factor     Masked    Veto
============================================================================================================
 > grep "32              L1_DoubleMu0er1p6_dEta_Max1p8" run273291_dump_command_hltGtStage2ObjectMap_80X_dataRun2_Prompt_v8.log       32              L1_DoubleMu0er1p6_dEta_Max1p8         0      0      0         1          1        0      32              L1_DoubleMu0er1p6_dEta_Max1p8         0      0      0         1          1        0      32              L1_DoubleMu0er1p6_dEta_Max1p8         0      0      0         1          1        0
      32              L1_DoubleMu0er1p6_dEta_Max1p8         0      0      0         1          1        0
      32              L1_DoubleMu0er1p6_dEta_Max1p8         0      0      0         1          1        0
      32              L1_DoubleMu0er1p6_dEta_Max1p8         0      0      0         1          1        0
      32              L1_DoubleMu0er1p6_dEta_Max1p8         0      0      0         1          1        0
      32              L1_DoubleMu0er1p6_dEta_Max1p8         0      0      0         1          1        0
      32              L1_DoubleMu0er1p6_dEta_Max1p8         0      0      0         1          1        0
      32              L1_DoubleMu0er1p6_dEta_Max1p8         0      0      0         1          1        0
      32              L1_DoubleMu0er1p6_dEta_Max1p8         0      0      0         1          1        0
tmatsush commented 8 years ago

@rekovic

I followed the instruction and had the same results as you reported. However, if I perform the following, I got the same results between re-emulated and unpacked.

cp /afs/cern.ch/user/t/tmatsush/public/L1MenuDev/L1Menu_Collisions2016_v2c_ugt_1board.xml L1Trigger/L1TGlobal/data/Luminosity/startup/

then apply the following patch to pick up the menu;

--- L1Trigger/L1TGlobal/python/hackConditions_cff.py
+++ L1Trigger/L1TGlobal/python/hackConditions_cff.py
@@ -32,5 +32,5 @@ if eras.stage2L1Trigger.isChosen():
     from L1Trigger.L1TGlobal.StableParameters_cff import *
 #    from L1Trigger.L1TGlobal.GlobalParameters_cff import *
 #   from L1Trigger.L1TGlobal.PrescalesVetos_cff import *
-#   from L1Trigger.L1TGlobal.TriggerMenu_cff import *
-#   TriggerMenu.L1TriggerMenuFile = cms.string('L1Menu_Collisions2015_25nsStage1_v7_uGT.xml')
+    from L1Trigger.L1TGlobal.TriggerMenu_cff import *
+    TriggerMenu.L1TriggerMenuFile = cms.string('L1Menu_Collisions2016_v2c_ugt_1board.xml')

@kkotov https://hypernews.cern.ch/HyperNews/CMS/get/calibrations/2332.html https://cms-conddb.cern.ch/cmsDbBrowser/list/Prod/tags/L1Menu_Collisions2016_v2c_ugt_1board_xml It seems that the menu in GT is not quite the same to the one we used for the firmware. Do you have any idea how this can happen ?

gennai commented 8 years ago

Do I understand correctly that our worse nightmare, i.e. the L1 menu used at HLT to get the candidates passing the L1 paths is not the same as the one used at L1 to select the events, has happened ? I thought there were checks in place to avoid this to happen …

Best, S.

Il giorno 23 giu 2016, alle ore 20:24, tmatsush notifications@github.com<mailto:notifications@github.com> ha scritto:

@rekovichttps://github.com/rekovic

I followed the instruction and had the same result. However, if I perform the following, I got the same result as in data;

cp /afs/cern.ch/user/t/tmatsush/public/L1MenuDev/L1Menu_Collisions2016_v2c_ugt_1board.xmlhttp://cern.ch/user/t/tmatsush/public/L1MenuDev/L1Menu_Collisions2016_v2c_ugt_1board.xml L1Trigger/L1TGlobal/data/Luminosity/startup/

then apply the following patch to pick up the menu;

--- L1Trigger/L1TGlobal/python/hackConditions_cff.py +++ L1Trigger/L1TGlobal/python/hackConditions_cff.py @@ -32,5 +32,5 @@ if eras.stage2L1Trigger.isChosen(): from L1Trigger.L1TGlobal.StableParameters_cff import *

from L1Trigger.L1TGlobal.GlobalParameters_cff import *

from L1Trigger.L1TGlobal.PrescalesVetos_cff import *

-# from L1Trigger.L1TGlobal.TriggerMenu_cff import * -# TriggerMenu.L1TriggerMenuFile = cms.string('L1Menu_Collisions2015_25nsStage1_v7_uGT.xml')

@kkotovhttps://github.com/kkotov https://hypernews.cern.ch/HyperNews/CMS/get/calibrations/2332.html https://cms-conddb.cern.ch/cmsDbBrowser/list/Prod/tags/L1Menu_Collisions2016_v2c_ugt_1board_xml It seems that the menu in GT is not quite the same to the one we used for the firmware. Do you have any idea how this can happen ?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/cms-l1t-offline/cmssw/issues/362#issuecomment-228139902, or mute the threadhttps://github.com/notifications/unsubscribe/AFZviSTVVB-eKLgZ6qP4a2g31U7i0EBMks5qOs9zgaJpZM4I82vk.

tmatsush commented 8 years ago

The menu dump for L1_DoubleMu0er1p6_dEta_Max1p8 by specifying "L1Menu_Collisions2016_v2c_ugt_1board.xml" in hackConditions_cff.py

 Condition name:     MuonMuonCorrelation_16040223250608453060
  Condition category: CondCorrelation
  Condition type:     l1t::Type2cor
  Object types:       Mu  Mu 
  " >= " flag:        1
  Condition chip:     0
  Relative BX:     0

  First sub-condition category:  1
  Second sub-condition category: 1

  First sub-condition index:  0
  Second sub-condition index: 1

  Correlation parameters [ hex ]
    Cut Type:  1
    minEtaCutValue        = 0
    maxEtaCutValue        = 1800
    precEtaCut            = 3
    minPhiCutValue        = 140453374823912
    maxPhiCutValue        = 0
    precPhiCut            = 4013884653
    minDRCutValue         = 25
    maxDRCutValue         = 140453721203472
    precDRCut             = 3984175640
    minMassCutValue       = 140452933114096
    maxMassCutValue       = 140453391453440
    precMassCut           = 0
    chargeCorrelation  = 1

However, the menu dump without specifying the menu has wrong 'precEtaCut'

  Condition name:     MuonMuonCorrelation_16040223250608453060
  Condition category: CondCorrelation
  Condition type:     l1t::Type2cor
  Object types:       Mu  Mu 
  " >= " flag:        1
  Condition chip:     0
  Relative BX:     0

  First sub-condition category:  1
  Second sub-condition category: 1

  First sub-condition index:  0
  Second sub-condition index: 1

  Correlation parameters [ hex ]
    Cut Type:  1
    minEtaCutValue        = -9223372036854775808
    maxEtaCutValue        = -9223372036854775808
    precEtaCut            = 4294967295
    minPhiCutValue        = 140692796091880
    maxPhiCutValue        = 0
    precPhiCut            = 2929587437
    minDRCutValue         = 25
    maxDRCutValue         = 140693155074832
    precDRCut             = 2899878424
    minMassCutValue       = 140692353385424
    maxMassCutValue       = 140692805681408
    precMassCut           = 0
    chargeCorrelation  = 1

This can be reproduced by replacing utm library comes with older CMSSW release (e.g. CMSSW_8_0_7_patch2) by following the instruction at https://gitlab.cern.ch/cms-l1t-utm/utm, but checking out with git checkout 6b05e0f. Actually this hash tag is before we achieved perfect agreement between firmware/emulator for correlation conditions.

So I conclude that the O2O was performed in non-standard CMSSW configuration as was the case when we introduced L1Menu_Collisions2016_v3 at P5 (at that time it was CMSSW_8_0_1 with custom utm library). This problem was identified already and we will use the same CMSSW environment for O2O and firmware/emulator processes in future.

tmatsush commented 8 years ago

@blwiner FYI

fwyzard commented 8 years ago

Hi @tmatsush , thanks for tracking down the source of the problem.

I'm glad to hear that it should be in the past now... do you think it would be possible to track down the range of runs that were affected by it ?

Looking at GlobalAlgBlk.h I see that it now supports reading the UUIDs of the menu and firmware loaded in the uGT; does L1TUtmTriggerMenu.h support reading the same information for the menu loaded in the database ?

In this specific case, if we had compared those information between the uGT and the database, would it have caught the problem online ?

kkotov commented 8 years ago

@tmatsush Hmm, I recall using the CMSSW_8_0_1 with the release-default UTM library for this menu. If I understood correctly, this UTM version was inadequate for O2Oing the menu, rhight? Do you want me to try overwriting this MC tag with a proper payload and IOV starting at 2?

fwyzard commented 8 years ago

Please do not change the MC tag, a more correct approach would be to try to create a new one.

kkotov commented 8 years ago

Ok, I've made a new tag for this payload: https://cms-conddb.cern.ch/cmsDbBrowser/list/Prod/tags/L1Menu_Collisions2016_v2c_ugt_1board_xml_UTM_r45210-xsd310-patch @fwyzard should I queue it for the same list of GTs as for the previous menu?

tmatsush commented 8 years ago

The algorithms affected by the problem are correlation conditions enabled in L1Menu_Collisions2016_v1c and L1Menu_Collisions2016_v2c, i.e. bit 32 and bit 35.

uGT can/will include 32-bit hash values of MENU NAME and FW UUID. These hash values can be compared offline using the same hash generator available in UTM library with the MENU NAME and FW UUID included in the menu.

The problem we are discussing can only be prevented by running uGT emulator both with xml file and with offline database, then comparing the results. In order to ease identify the problem in future, UTM revision ID can be encoded in the comment strings of the menu.

tmatsush commented 8 years ago

@blwiner From which CMSSW release, correlation conditions in the emulator use LUT ?

fwyzard commented 8 years ago

@kkotov: I assume that by now all MC global tags use the v4 L1 menu, so there is no need to queue the new version of the v2c menu. However, you could double check with the AlCa conveners.

fwyzard commented 8 years ago

@tmatsush thanks for the summary. So the affected runs are those that used the v1c and v2c menus - according to the .json file that would be all collision runs up to 273302 - and the affected triggers are only

.A

blwiner commented 8 years ago

It has been a while since the LUTs went into the l1t integration release. @mulhearn or @rekovic might need to comment about when that got merged with a CMSSW release.

Does anyone know how to look back at old PRs? I'm not sure how to do that on GitHub. I might be able to find the PR that had the LUT information. It has been at least 2-3 months.

-b

On Fri, Jun 24, 2016 at 5:54 PM, tmatsush notifications@github.com wrote:

@blwiner https://github.com/blwiner From which CMSSW release, correlation conditions in the emulator use LUT ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cms-l1t-offline/cmssw/issues/362#issuecomment-228471393, or mute the thread https://github.com/notifications/unsubscribe/AElaBor8CranKbiT9uUjOFLRfgRDPfPqks5qPFINgaJpZM4I82vk .

B. Winer

fwyzard commented 8 years ago

On 25 June 2016 at 17:30, blwiner notifications@github.com wrote:

Does anyone know how to look back at old PRs? I'm not sure how to do that on GitHub. I might be able to find the PR that had the LUT information. It has been at least 2-3 months.

https://github.com/cms-l1t-offline/cmssw/pulls

By default it shows the open PRs, but if you click on "✔ 212 Closed" it will show the clodes ones instead.

.A

Strategy is a system of expedients.

Field Marshall Helmuth Carl Bernard Graf von Moltke

tmatsush commented 8 years ago

CMSSW_8_0_5 was at P5 when L1Menu_Collisions2016_v1c was first used in physics run. CMSSW_8_0_6 was at P5 when L1Menu_Collisions2016_v2c was first used in physics run. The correlation conditions implemented in uGT emulator uses floating point calculation in both releases. http://cmslxr.fnal.gov/source/L1Trigger/L1TGlobal/src/CorrCondition.cc?v=CMSSW_8_0_5 http://cmslxr.fnal.gov/source/L1Trigger/L1TGlobal/src/CorrCondition.cc?v=CMSSW_8_0_6

Even if we used the correct versions of the CMSSW/UTM as deployed at P5 for O2O, the implementation of correlation conditions was not the same as the firmware uses LUT based implementation. Therefore we would not have got the same results between firmware and emulator even without O2O issues.