fwyzard
commented
3 years ago assign hlt, heterogeneous
Open fwyzard opened 3 years ago
fwyzard
commented
3 years ago assign hlt, heterogeneous
cmsbuild
commented
3 years ago New categories assigned: heterogeneous,hlt
@fwyzard,@Martin-Grunewald,@makortel,@missirol you have been requested to review this Pull request/Issue and eventually sign? Thanks
cmsbuild
commented
3 years ago A new Issue was created by @fwyzard Andrea Bocci.
@Dr15Jones, @perrotta, @dpiparo, @makortel, @smuzaffar, @qliphy can you please review it and eventually sign/assign? Thanks.
cms-bot commands are listed here
fwyzard
commented
3 years ago @silviodonato now there is a GitHuib issue...
fwyzard
commented
3 years ago The HCAL discrepancies are partially addressed by @mariadalfonso in #35355 / #35357
mmusich
commented
3 years ago assign trk-dpg
mmusich
commented
3 years ago assign hcal-dpg
cmsbuild
commented
3 years ago New categories assigned: hcal-dpg,trk-dpg
@mmusich,@georgia14,@tsusa,@mseidel42 you have been requested to review this Pull request/Issue and eventually sign? Thanks
fwyzard
commented
3 years ago thanks
VinInn
commented
3 years ago what does Other mean?
fwyzard
commented
3 years ago Other means that the EDFilter responsible for accepting/rejecting the event (i.e. the last module that runs on a path) is different in the two cases; it's usually safe to ignore.
VinInn
commented
3 years ago Than it would be better to run with 1K or even 10K events just to understand which are seriously systematic affected. IMHO the only worth a deeper analysis from the list above are the two HLT_DoubleMu4 events and what is going on with all those HLT_QuadPFJet and their various combinatorial thresholds... (and the bizarre behavior of HLT_PFHT350_v19 wrt all other HLT_PFHTxyz_v17 (are there other v19????)
silviodonato
commented
3 years ago Than it would be better to run with 1K or even 10K events just to understand which are seriously systematic affected. IMHO the only worth a deeper analysis from the list above are the two HLT_DoubleMu4 events and what is going on with all those HLT_QuadPFJet and their various combinatorial thresholds... (and the bizarre behavior of HLT_PFHT350_v19 wrt all other HLT_PFHTxyz_v17 (are there other v19????)
The difference between HLT_PFHT350_v19 and HLT_PFHT370_v17 is only in the L1seeds, caloHT cut (320 vs 300), and PFHT cut (370 vs 350). All the filters cut on the same objects (hltHtMhtJet30 and hltPFHTJet30).
tsusa
commented
3 years ago @fwyzard is there a similar comparison between gpu and cpu (non-legacy wf) ?
fwyzard
commented
3 years ago Hi @tsusa,
the instructions above under Pixel local reconstruction changes should compare
while those under Pixel track reconstruction changes should compare
Are you looking for something else ?
tsusa
commented
3 years ago Hi @fwyzard, yes, the comparison
fwyzard
commented
3 years ago You can make this comparison by creating the configuration for the Pixel local reconstruction changes, running it on the CPU (e.g. setting CUDA_VISIBLE_DEVICES= or running on a machine without GPUs) and comparing the results with those of the legacy reconstruction.
VinInn
commented
3 years ago legacy pixel local reconstruction is NOT supposed to be identical to "new pixel local reconstruction", (and not required to)
fwyzard
commented
3 years ago Well, this is news to me. Why not ? What is different ?
mmusich
commented
3 years ago e.g. treatment of the generic CPE errors? https://indico.cern.ch/event/968978/contributions/4078289/attachments/2134046/3594257/Pixel%20GPU%20code%20Validation.pdf#page=11
fwyzard
commented
3 years ago ah, yes - still, how much different do we expect it to be ?
(and anyway this is outside the scope of the discrepancy discussed in this issue, which should use the new reconstruction for both CPU and GPU workflows)
mmusich
commented
3 years ago ah, yes - still, how much different do we expect it to be ?
There are plots in the talk linked above. I let you judged by yourself. Mind that the situation has been improved by https://github.com/cms-sw/cmssw/pull/34286
VinInn
commented
3 years ago run twice on gpu. Got:
Found 100 matching events, out of which 0 have different HLT resultsthis is for running only pixel tracking patatrack on GPU vs CPU (so Calo si always Legacy) file:
/store/relval/CMSSW_12_1_0_pre3/RelValTTbar_14TeV/GEN-SIM-DIGI-RAW/PU_121X_mcRun3_2021_realistic_v2-v1/10000/0eb14c4a-e363-424a-9c0c-2688c7d32c74.root
release: CMSSW_12_1_X_2021-09-28-2300
i.e.
#User-defined customization functions
#from HLTrigger.Configuration.customizeHLTforPatatrack import customizeHLTforPatatrack
#process = customizeHLTforPatatrack(process)
from HLTrigger.Configuration.customizeHLTforPatatrack import *
process = customiseCommon(process)
process = customisePixelLocalReconstruction(process)
process = customisePixelTrackReconstruction(process)hltDiff -o output-cpuTK.root -n output-gpuTK.root | grep -v "\- \- \~"
...
Found 100 matching events, out of which 100 have different HLT results
Events Accepted Gained Lost Other Trigger
100 36 +1 - - HLT_Mu3_PFJet40_v16
100 1 +1 - ~1 HLT_Mu7p5_Track2_Jpsi_v11
100 36 +1 - - HLT_Mu3_L1SingleMu5orSingleMu7_v1
100 19 +1 - - HLT_Mu12_v3
100 42 +1 - - HLT_DiPFJetAve80_v13
100 14 - -1 - HLT_DiPFJetAve60_HFJEC_v15
100 91 +1 - - HLT_AK8PFJet60_v15
100 64 +1 - - HLT_AK8PFJet80_v15
100 19 - -1 - HLT_AK8PFJet140_v15
100 50 +1 - - HLT_PFJet80_v20
100 78 - -1 - HLT_PFJetFwd25_v3
100 29 - -1 - HLT_PFJetFwd40_v19
100 88 - -1 - HLT_AK8PFJetFwd40_v15
100 18 +1 - - HLT_AK8PFJetFwd60_v14
100 7 +1 - ~1 HLT_PFMETNoMu110_PFMHTNoMu110_IDTight_v20
100 7 +1 - ~1 HLT_MonoCentralPFJet80_PFMETNoMu110_PFMHTNoMu110_IDTight_v20
100 1 +1 - - HLT_SingleJet30_Mu12_SinglePFJet40_v11
100 1 +1 - - HLT_Mu12_DoublePFJets40_CaloBTagDeepCSV_p71_v2
100 18 +1 - - HLT_Mu8_TrkIsoVVL_v12
100 2 +1 - - HLT_Mu8_TrkIsoVVL_Ele23_CaloIdL_TrackIdL_IsoVL_DZ_v13
100 2 +1 - - HLT_Mu8_TrkIsoVVL_Ele23_CaloIdL_TrackIdL_IsoVL_DZ_PFDiJet30_v1
100 2 +1 - - HLT_Mu8_TrkIsoVVL_Ele23_CaloIdL_TrackIdL_IsoVL_DZ_CaloDiJet30_v1
100 3 +1 - - HLT_Mu8_TrkIsoVVL_Ele23_CaloIdL_TrackIdL_IsoVL_v11
100 2 +1 - - HLT_Mu12_TrkIsoVVL_Ele23_CaloIdL_TrackIdL_IsoVL_v7
100 1 +1 - - HLT_Mu12_TrkIsoVVL_Ele23_CaloIdL_TrackIdL_IsoVL_DZ_v15
100 0 +1 - - HLT_DoubleMu3_DZ_PFMET90_PFMHT90_v10
100 2 - -1 - HLT_Mu4_TrkIsoVVL_DiPFJet90_40_DEta3p5_MJJ750_HTT300_PFMETNoMu60_v15
100 2 - -1 - HLT_Mu8_TrkIsoVVL_DiPFJet40_DEta3p5_MJJ750_HTT300_PFMETNoMu60_v16
100 5 - -1 - HLT_Mu3er1p5_PFJet100er2p5_PFMET70_PFMHT70_IDTight_v2
100 1 +1 - - HLT_Mu3er1p5_PFJet100er2p5_PFMET90_PFMHT90_IDTight_v2
100 5 - -1 - HLT_Mu3er1p5_PFJet100er2p5_PFMETNoMu70_PFMHTNoMu70_IDTight_v2
100 23 +1 - - HLT_Mu8_v12
100 20 - -1 - HLT_Ele8_CaloIdL_TrackIdL_IsoVL_PFJet30_v16
100 19 - -1 - HLT_Ele12_CaloIdL_TrackIdL_IsoVL_PFJet30_v18
100 18 - -1 - HLT_Ele15_CaloIdL_TrackIdL_IsoVL_PFJet30_v3
100 13 - -1 - HLT_Ele23_CaloIdL_TrackIdL_IsoVL_PFJet30_v18
100 22 +1 -1 - HLT_Ele8_CaloIdM_TrackIdM_PFJet30_v18
100 16 - -1 - HLT_Ele17_CaloIdM_TrackIdM_PFJet30_v16
100 14 - -1 - HLT_Ele23_CaloIdM_TrackIdM_PFJet30_v18
100 6 - -1 - HLT_PFHT400_SixPFJet32_v8
100 5 - -1 - HLT_PFHT400_FivePFJet_100_100_60_30_30_v8
100 20 +1 - - HLT_PFHT350MinPFJet15_v9
100 50 +1 - - HLT_AK4PFJet80_v19
100 86 - -1 - MC_PFBTagDeepCSV_v10
100 6 - -1 ~1 HLT_PFMETTypeOne100_PFMHT100_IDTight_PFHT60_v9
100 6 +1 - - HLT_QuadPFJet105_88_76_15_v5
100 4 +1 - - HLT_QuadPFJet111_90_80_15_v5
100 22 +1 - - HLT_Mu9_IP0_part0_v2
100 23 +1 - - HLT_TrkMu6NoFiltersNoVtx_v1
100 17 +1 - - HLT_TrkMu16NoFiltersNoVtx_v1
100 100 - -100 - Status_OnCPU
100 0 +100 - - Status_OnGPUand with larger statistics
Found 1000 matching events, out of which 1000 have different HLT results
Events Accepted Gained Lost Other Trigger
1000 16 +1 - - HLT_AK8PFJet360_TrimMass30_v18
1000 7 +1 - - HLT_AK8PFJet420_TrimMass30_v11
1000 398 +2 - - HLT_Mu3_PFJet40_v16
1000 10 +1 - ~2 HLT_Mu7p5_Track2_Jpsi_v11
1000 407 +2 - - HLT_Mu3_L1SingleMu5orSingleMu7_v1
1000 140 +2 -1 - HLT_Ele20_WPTight_Gsf_v6
1000 155 +1 -1 ~1 HLT_Ele15_WPLoose_Gsf_v3
1000 148 +1 -1 ~1 HLT_Ele17_WPLoose_Gsf_v3
1000 142 +1 -1 ~1 HLT_Ele20_WPLoose_Gsf_v6
1000 137 +1 - - HLT_Ele20_eta2p1_WPLoose_Gsf_v6
1000 121 +1 -1 - HLT_Ele27_WPTight_Gsf_v16
1000 120 +1 -1 - HLT_Ele28_WPTight_Gsf_v1
1000 113 +1 -1 - HLT_Ele30_WPTight_Gsf_v1
1000 108 +1 -1 - HLT_Ele32_WPTight_Gsf_v15
1000 104 +1 -1 - HLT_Ele35_WPTight_Gsf_v9
1000 96 +1 - - HLT_Ele35_WPTight_Gsf_L1EGMT_v5
1000 90 +1 -1 - HLT_Ele38_WPTight_Gsf_v9
1000 87 +1 - - HLT_Ele40_WPTight_Gsf_v9
1000 109 +1 -1 - HLT_Ele32_WPTight_Gsf_L1DoubleEG_v9
1000 21 - -1 ~1 HLT_Ele24_eta2p1_WPTight_Gsf_LooseChargedIsoPFTau30_eta2p1_CrossL1_v13
1000 31 +2 - ~3 HLT_IsoMu20_eta2p1_LooseChargedIsoPFTau27_eta2p1_CrossL1_v12
1000 26 +1 - ~3 HLT_IsoMu20_eta2p1_LooseChargedIsoPFTauHPS27_eta2p1_CrossL1_v4
1000 24 +1 - ~3 HLT_IsoMu20_eta2p1_MediumChargedIsoPFTauHPS27_eta2p1_CrossL1_v1
1000 23 +1 - ~4 HLT_IsoMu20_eta2p1_TightChargedIsoPFTauHPS27_eta2p1_CrossL1_v1
1000 22 +1 - ~3 HLT_IsoMu20_eta2p1_LooseChargedIsoPFTauHPS27_eta2p1_TightID_CrossL1_v1
1000 20 +1 - ~3 HLT_IsoMu20_eta2p1_MediumChargedIsoPFTauHPS27_eta2p1_TightID_CrossL1_v1
1000 19 +1 - ~4 HLT_IsoMu20_eta2p1_TightChargedIsoPFTauHPS27_eta2p1_TightID_CrossL1_v1
1000 37 +3 - ~1 HLT_IsoMu27_LooseChargedIsoPFTau20_Trk1_eta2p1_SingleL1_v5
1000 32 +1 - ~1 HLT_IsoMu27_LooseChargedIsoPFTauHPS20_Trk1_eta2p1_SingleL1_v1
1000 30 +1 - ~1 HLT_IsoMu27_MediumChargedIsoPFTauHPS20_Trk1_eta2p1_SingleL1_v1
1000 27 +1 - ~2 HLT_IsoMu27_TightChargedIsoPFTauHPS20_Trk1_eta2p1_SingleL1_v1
1000 144 +1 - - HLT_IsoMu20_v15
1000 127 +1 - - HLT_IsoMu24_v13
1000 119 +1 - - HLT_IsoMu24_eta2p1_v15
1000 121 +1 - - HLT_IsoMu27_v16
1000 275 +2 - - HLT_Mu12_v3
1000 258 +1 - - HLT_Mu15_v3
1000 221 +1 - - HLT_Mu20_v12
1000 167 +1 - - HLT_Mu27_v13
1000 966 - -1 - HLT_DiPFJet15_FBEta3_NoCaloMatched_v17
1000 254 +5 -3 - HLT_DiPFJet25_FBEta3_NoCaloMatched_v17
1000 969 +1 - - HLT_DiPFJetAve40_v14
1000 742 +1 - - HLT_DiPFJetAve60_v14
1000 466 +2 -3 - HLT_DiPFJetAve80_v13
1000 107 - -1 - HLT_DiPFJetAve140_v13
1000 28 - -1 - HLT_DiPFJetAve200_v13
1000 955 +2 -1 - HLT_DiPFJetAve25_HFJEC_v17
1000 825 +2 -2 - HLT_DiPFJetAve35_HFJEC_v17
1000 127 +2 -1 - HLT_DiPFJetAve60_HFJEC_v15
1000 45 - -1 - HLT_DiPFJetAve80_HFJEC_v16
1000 891 +3 -2 - HLT_AK8PFJet60_v15
1000 656 +6 -6 - HLT_AK8PFJet80_v15
1000 217 +3 -4 - HLT_AK8PFJet140_v15
1000 85 - -1 - HLT_AK8PFJet200_v15
1000 27 +1 -2 - HLT_AK8PFJet320_v16
1000 978 +3 - - HLT_PFJet40_v21
1000 568 +6 -5 - HLT_PFJet80_v20
1000 177 - -1 - HLT_PFJet140_v19
1000 56 - -1 - HLT_PFJet200_v19
1000 839 +3 -4 - HLT_PFJetFwd25_v3
1000 281 - -4 - HLT_PFJetFwd40_v19
1000 75 - -1 - HLT_PFJetFwd60_v19
1000 898 +8 -5 - HLT_AK8PFJetFwd40_v15
1000 186 +2 -2 - HLT_AK8PFJetFwd60_v14
1000 55 +1 -1 - HLT_AK8PFJetFwd80_v14
1000 630 +7 -3 - HLT_PFHT180_v17
1000 394 +2 -3 - HLT_PFHT250_v17
1000 163 +6 -2 - HLT_PFHT370_v17
1000 103 +2 - - HLT_PFHT430_v17
1000 41 - -1 - HLT_PFHT590_v17
1000 18 +1 - - HLT_PFHT780_v17
1000 48 +2 -3 ~2 HLT_PFMET110_PFMHT110_IDTight_v20
1000 38 +1 -4 ~2 HLT_PFMET120_PFMHT120_IDTight_v20
1000 23 - -1 ~1 HLT_PFMET130_PFMHT130_IDTight_v20
1000 19 +1 - - HLT_PFMET140_PFMHT140_IDTight_v20
1000 50 - -1 ~2 HLT_PFMET100_PFMHT100_IDTight_CaloBTagDeepCSV_3p1_v8
1000 37 +1 -3 ~2 HLT_PFMET110_PFMHT110_IDTight_CaloBTagDeepCSV_3p1_v8
1000 31 - -4 ~2 HLT_PFMET120_PFMHT120_IDTight_CaloBTagDeepCSV_3p1_v8
1000 16 - -1 ~1 HLT_PFMET130_PFMHT130_IDTight_CaloBTagDeepCSV_3p1_v8
1000 13 +1 - - HLT_PFMET140_PFMHT140_IDTight_CaloBTagDeepCSV_3p1_v8
1000 40 +2 -4 ~5 HLT_PFMET120_PFMHT120_IDTight_PFHT60_v9
1000 66 +1 -4 ~4 HLT_PFMETNoMu120_PFMHTNoMu120_IDTight_PFHT60_v9
1000 46 +3 -3 ~5 HLT_PFMETTypeOne120_PFMHT120_IDTight_PFHT60_v9
1000 53 +1 -1 ~2 HLT_PFMETTypeOne110_PFMHT110_IDTight_v12
1000 44 +2 -3 ~2 HLT_PFMETTypeOne120_PFMHT120_IDTight_v12
1000 29 - -1 ~1 HLT_PFMETTypeOne130_PFMHT130_IDTight_v12
1000 21 +1 - - HLT_PFMETTypeOne140_PFMHT140_IDTight_v11
1000 75 +2 -3 ~2 HLT_PFMETNoMu110_PFMHTNoMu110_IDTight_v20
1000 64 +1 -4 ~1 HLT_PFMETNoMu120_PFMHTNoMu120_IDTight_v20
1000 49 - -1 ~1 HLT_PFMETNoMu130_PFMHTNoMu130_IDTight_v19
1000 34 +2 - ~1 HLT_PFMETNoMu140_PFMHTNoMu140_IDTight_v19
1000 59 +1 -1 ~4 HLT_MonoCentralPFJet80_PFMETNoMu110_PFMHTNoMu110_IDTight_v20
1000 50 - -2 ~3 HLT_MonoCentralPFJet80_PFMETNoMu120_PFMHTNoMu120_IDTight_v20
1000 28 +2 - ~1 HLT_MonoCentralPFJet80_PFMETNoMu140_PFMHTNoMu140_IDTight_v19
1000 30 +4 -1 - HLT_SingleJet30_Mu12_SinglePFJet40_v11
1000 23 +4 -1 - HLT_Mu12_DoublePFJets40_CaloBTagDeepCSV_p71_v2
1000 7 +1 -1 - HLT_Mu12_DoublePFJets100_CaloBTagDeepCSV_p71_v2
1000 7 - -1 - HLT_Mu12_DoublePFJets40MaxDeta1p6_DoubleCaloBTagDeepCSV_p71_v2
1000 6 - -1 - HLT_Mu12_DoublePFJets54MaxDeta1p6_DoubleCaloBTagDeepCSV_p71_v2
1000 5 - -1 ~1 HLT_Mu12_DoublePFJets62MaxDeta1p6_DoubleCaloBTagDeepCSV_p71_v2
1000 641 +1 - - HLT_DoublePFJets40_CaloBTagDeepCSV_p71_v2
1000 99 +2 - - HLT_DoublePFJets100_CaloBTagDeepCSV_p71_v2
1000 20 - -1 - HLT_DoublePFJets116MaxDeta1p6_DoubleCaloBTagDeepCSV_p71_v2
1000 253 +2 -2 - HLT_Mu8_TrkIsoVVL_v12
1000 23 +1 -2 ~2 HLT_Mu8_TrkIsoVVL_Ele23_CaloIdL_TrackIdL_IsoVL_DZ_v13
1000 23 +1 -2 ~2 HLT_Mu8_TrkIsoVVL_Ele23_CaloIdL_TrackIdL_IsoVL_DZ_PFDiJet30_v1
1000 23 +1 -2 ~2 HLT_Mu8_TrkIsoVVL_Ele23_CaloIdL_TrackIdL_IsoVL_DZ_CaloDiJet30_v1
1000 13 +1 -1 ~4 HLT_Mu8_TrkIsoVVL_Ele23_CaloIdL_TrackIdL_IsoVL_DZ_PFDiJet30_PFBtagDeepCSV_1p5_v1
1000 17 - -2 ~3 HLT_Mu8_TrkIsoVVL_Ele23_CaloIdL_TrackIdL_IsoVL_DZ_CaloDiJet30_CaloBtagDeepCSV_1p5_v1
1000 25 +1 -2 ~2 HLT_Mu8_TrkIsoVVL_Ele23_CaloIdL_TrackIdL_IsoVL_v11
1000 198 +1 -2 - HLT_Mu17_TrkIsoVVL_v13
1000 185 +1 -2 - HLT_Mu19_TrkIsoVVL_v4
1000 77 +2 -1 - HLT_BTagMu_AK4DiJet20_Mu5_v13
1000 71 +2 -1 - HLT_BTagMu_AK4DiJet40_Mu5_v13
1000 32 +1 - - HLT_BTagMu_AK4DiJet70_Mu5_v13
1000 15 - -2 ~1 HLT_Mu23_TrkIsoVVL_Ele12_CaloIdL_TrackIdL_IsoVL_DZ_v15
1000 16 - -2 ~1 HLT_Mu23_TrkIsoVVL_Ele12_CaloIdL_TrackIdL_IsoVL_v7
1000 20 +1 -2 ~2 HLT_Mu12_TrkIsoVVL_Ele23_CaloIdL_TrackIdL_IsoVL_v7
1000 18 +1 -2 ~2 HLT_Mu12_TrkIsoVVL_Ele23_CaloIdL_TrackIdL_IsoVL_DZ_v15
1000 9 +1 - - HLT_Photon75_R9Id90_HE10_IsoM_EBOnly_CaloMJJ400_PFJetsMJJ600DEta3_v5
1000 12 +1 -1 - HLT_Photon35_TwoProngs35_v1
1000 103 - -1 ~1 HLT_IsoMu24_TwoProngs35_v1
1000 24 +2 -3 - HLT_DoubleMu3_DZ_PFMET50_PFMHT60_v10
1000 14 +2 - - HLT_DoubleMu3_DZ_PFMET70_PFMHT70_v10
1000 7 +1 - - HLT_DoubleMu3_DZ_PFMET90_PFMHT90_v10
1000 14 +1 -1 ~3 HLT_DiJet110_35_Mjj650_PFMET110_v9
1000 10 +1 - ~1 HLT_DiJet110_35_Mjj650_PFMET120_v9
1000 8 +1 - ~2 HLT_DiJet110_35_Mjj650_PFMET130_v9
1000 106 +1 - - HLT_Ele30_eta2p1_WPTight_Gsf_CentralPFJet35_EleCleaned_v13
1000 92 +2 - - HLT_Ele28_eta2p1_WPTight_Gsf_HT150_v13
1000 2 - -1 - HLT_Ele28_HighEta_SC20_Mass55_v13
1000 18 +2 - - HLT_Ele15_IsoVVVL_PFHT450_CaloBTagDeepCSV_4p5_v8
1000 12 +2 - ~1 HLT_Ele15_IsoVVVL_PFHT450_PFMET50_v16
1000 19 +2 - - HLT_Ele15_IsoVVVL_PFHT450_v16
1000 7 - -1 ~2 HLT_Mu4_TrkIsoVVL_DiPFJet90_40_DEta3p5_MJJ750_HTT300_PFMETNoMu60_v15
1000 8 - -1 ~3 HLT_Mu8_TrkIsoVVL_DiPFJet40_DEta3p5_MJJ750_HTT300_PFMETNoMu60_v16
1000 16 +1 -2 - HLT_Mu15_IsoVVVL_PFHT450_PFMET50_v15
1000 20 +1 -1 - HLT_Mu15_IsoVVVL_PFHT450_v15
1000 16 +1 -1 - HLT_Mu50_IsoVVVL_PFHT450_v15
1000 38 +1 -1 ~3 HLT_Mu3er1p5_PFJet100er2p5_PFMET70_PFMHT70_IDTight_v2
1000 25 +2 - - HLT_Mu3er1p5_PFJet100er2p5_PFMET90_PFMHT90_IDTight_v2
1000 45 - -1 ~2 HLT_Mu3er1p5_PFJet100er2p5_PFMETNoMu70_PFMHTNoMu70_IDTight_v2
1000 40 +1 -1 ~2 HLT_Mu3er1p5_PFJet100er2p5_PFMETNoMu80_PFMHTNoMu80_IDTight_v2
1000 35 +1 - - HLT_Mu3er1p5_PFJet100er2p5_PFMETNoMu90_PFMHTNoMu90_IDTight_v2
1000 327 +2 - - HLT_Mu8_v12
1000 248 +1 - - HLT_Mu17_v13
1000 227 +1 - - HLT_Mu19_v4
1000 194 +2 -1 ~2 HLT_Ele8_CaloIdL_TrackIdL_IsoVL_PFJet30_v16
1000 184 +1 -1 ~1 HLT_Ele12_CaloIdL_TrackIdL_IsoVL_PFJet30_v18
1000 179 +1 -1 ~1 HLT_Ele15_CaloIdL_TrackIdL_IsoVL_PFJet30_v3
1000 151 - -1 ~1 HLT_Ele23_CaloIdL_TrackIdL_IsoVL_PFJet30_v18
1000 219 +3 -2 ~1 HLT_Ele8_CaloIdM_TrackIdM_PFJet30_v18
1000 171 +1 -2 - HLT_Ele17_CaloIdM_TrackIdM_PFJet30_v16
1000 157 - -2 - HLT_Ele23_CaloIdM_TrackIdM_PFJet30_v18
1000 29 +5 - ~5 HLT_PFHT330PT30_QuadPFJet_75_60_45_40_TriplePFBTagDeepCSV_4p5_v3
1000 158 +2 -1 ~4 HLT_PFHT330PT30_QuadPFJet_75_60_45_40_v9
1000 33 +1 - ~6 HLT_PFHT400_SixPFJet32_DoublePFBTagDeepCSV_2p94_v8
1000 48 +1 -1 ~5 HLT_PFHT400_SixPFJet32_v8
1000 28 +2 - ~6 HLT_PFHT450_SixPFJet36_PFBTagDeepCSV_1p59_v7
1000 32 +2 - ~6 HLT_PFHT450_SixPFJet36_v7
1000 88 +2 -3 ~3 HLT_PFHT400_FivePFJet_100_100_60_30_30_v8
1000 58 +1 -2 ~7 HLT_PFHT400_FivePFJet_100_100_60_30_30_DoublePFBTagDeepCSV_4p5_v8
1000 43 +1 -2 ~1 HLT_PFHT400_FivePFJet_120_120_60_30_30_DoublePFBTagDeepCSV_4p5_v8
1000 190 +3 -2 - HLT_PFHT350_v19
1000 247 +2 - - HLT_PFHT350MinPFJet15_v9
1000 932 +4 -2 - HLT_AK4PFJet50_v19
1000 568 +6 -5 - HLT_AK4PFJet80_v19
1000 385 +3 - - HLT_AK4PFJet100_v19
1000 262 +1 -1 - HLT_AK4PFJet120_v18
1000 43 +1 -1 ~3 HLT_IsoTrackHB_v4
1000 825 +10 -7 - MC_PFBTagDeepCSV_v10
1000 25 +2 - ~4 MC_DoubleEle5_CaloIdL_MW_v15
1000 156 +2 -1 - MC_Ele5_WPTight_Gsf_v8
1000 175 +2 - ~1 MC_IsoMu_v15
1000 41 - -1 - MC_DoubleMu_TrkIsoVVL_DZ_v11
1000 17 +1 - ~1 HLT_MediumChargedIsoPFTau50_Trk30_eta2p1_1pr_MET90_v12
1000 11 +1 - ~1 HLT_MediumChargedIsoPFTau50_Trk30_eta2p1_1pr_MET100_v12
1000 10 +1 - ~1 HLT_MediumChargedIsoPFTau50_Trk30_eta2p1_1pr_MET110_v8
1000 7 +1 - - HLT_MediumChargedIsoPFTau50_Trk30_eta2p1_1pr_MET120_v8
1000 56 +1 - - HLT_IsoMu27_MET90_v3
1000 12 +1 - ~4 HLT_DoubleTightChargedIsoPFTau35_Trk1_eta2p1_Reg_v12
1000 11 +1 - ~4 HLT_DoubleTightChargedIsoPFTau35_Trk1_TightID_eta2p1_Reg_v12
1000 7 +1 - - HLT_Photon75_R9Id90_HE10_IsoM_EBOnly_PFJetsMJJ600DEta3_v5
1000 67 - -1 ~4 HLT_PFMET100_PFMHT100_IDTight_PFHT60_v9
1000 92 - -1 ~7 HLT_PFMETNoMu100_PFMHTNoMu100_IDTight_PFHT60_v9
1000 75 - -2 ~4 HLT_PFMETTypeOne100_PFMHT100_IDTight_PFHT60_v9
1000 19 +2 -2 - HLT_DoubleMu3_DCA_PFMET50_PFMHT60_v10
1000 21 +1 - ~6 HLT_QuadPFJet98_83_71_15_DoublePFBTagDeepCSV_1p3_7p7_VBF1_v8
1000 15 - -1 ~6 HLT_QuadPFJet103_88_75_15_DoublePFBTagDeepCSV_1p3_7p7_VBF1_v8
1000 163 +2 -1 ~2 HLT_QuadPFJet98_83_71_15_v5
1000 115 +1 -3 ~2 HLT_QuadPFJet103_88_75_15_v5
1000 110 +2 -2 ~2 HLT_QuadPFJet105_88_76_15_v5
1000 88 +2 -1 ~1 HLT_QuadPFJet111_90_80_15_v5
1000 9 - -1 - HLT_AK8PFJet330_TrimMass30_PFAK8BTagDeepCSV_p1_v2
1000 11 +1 - - HLT_Diphoton30_18_R9IdL_AND_HE_AND_IsoCaloId_NoPixelVeto_v2
1000 93 +1 - - HLT_Mu9_IP5_ToCSCS_v1
1000 93 +1 - - HLT_Mu9_IP5_part0_v2
1000 93 +1 - - HLT_Mu9_IP5_part1_v2
1000 93 +1 - - HLT_Mu9_IP5_part2_v2
1000 93 +1 - - HLT_Mu9_IP5_part3_v2
1000 93 +1 - - HLT_Mu9_IP5_part4_v2
1000 103 +1 - - HLT_Mu8_IP5_ToCSCS_v1
1000 103 +1 - - HLT_Mu8_IP5_part0_v2
1000 103 +1 - - HLT_Mu8_IP5_part1_v2
1000 103 +1 - - HLT_Mu8_IP5_part2_v2
1000 103 +1 - - HLT_Mu8_IP5_part3_v2
1000 103 +1 - - HLT_Mu8_IP5_part4_v2
1000 123 +1 - - HLT_Mu8_IP3_ToCSCS_v1
1000 123 +1 - - HLT_Mu8_IP3_part0_v3
1000 123 +1 - - HLT_Mu8_IP3_part1_v3
1000 123 +1 - - HLT_Mu8_IP3_part2_v3
1000 123 +1 - - HLT_Mu8_IP3_part3_v3
1000 123 +1 - - HLT_Mu8_IP3_part4_v3
1000 303 +2 - - HLT_Mu9_IP0_part0_v2
1000 113 +1 - - HLT_Mu9_IP3_part0_v2
1000 328 +2 - - HLT_TrkMu6NoFiltersNoVtx_v1
1000 252 +2 - - HLT_TrkMu16NoFiltersNoVtx_v1
1000 1000 - -1000 - Status_OnCPU
1000 0 +1000 - - Status_OnGPUPlease note that for running with a recent IB, you will need https://github.com/cms-sw/cmssw/pull/35497 . Thanks to Vincenzo for reporting the problem and testing the fix.
fwyzard
commented
3 years ago @tsusa
Hi @fwyzard, yes, the comparison
* the legacy pixel tracks (on CPU) running on top of the cpu pixel local reconstruction (on CPU) * the legacy pixel tracks (on CPU) running on top of the new pixel local reconstruction (on GPU)
I've finally tried it, and this comparison does not show any differences. This is in line with the fact that the customisation uses the legacy pixel local reconstruction when running on the CPU, instead of the new SoA-based one.
VinInn
commented
3 years ago I have compared CMSSW_12_1_X_2021-09-23-2300 with CMSSW_12_1_SKYLAKEAVX512_X_2021-09-23-2300 on CPU (Patatrack wf as above)
result:
hltDiff -o output-cpusseTK_1K.root -n output-cpuavxTK_1K.root | grep -v "\- \- \~"
...
Found 1000 matching events, out of which 535 have different HLT results
Events Accepted Gained Lost Other Trigger
1000 15 - -1 - HLT_DoubleMu4_Mass3p8_DZ_PFHT350_v8
1000 398 +1 -1 - HLT_Mu3_PFJet40_v16
1000 140 - -1 - HLT_Ele20_WPTight_Gsf_v6
1000 155 - -2 ~1 HLT_Ele15_WPLoose_Gsf_v3
1000 148 - -2 ~1 HLT_Ele17_WPLoose_Gsf_v3
1000 142 - -2 ~1 HLT_Ele20_WPLoose_Gsf_v6
1000 121 - -1 - HLT_Ele27_WPTight_Gsf_v16
1000 120 - -1 - HLT_Ele28_WPTight_Gsf_v1
1000 113 - -1 - HLT_Ele30_WPTight_Gsf_v1
1000 108 - -1 - HLT_Ele32_WPTight_Gsf_v15
1000 104 - -1 - HLT_Ele35_WPTight_Gsf_v9
1000 90 - -1 - HLT_Ele38_WPTight_Gsf_v9
1000 109 - -1 - HLT_Ele32_WPTight_Gsf_L1DoubleEG_v9
1000 21 +1 - - HLT_Ele24_eta2p1_WPTight_Gsf_LooseChargedIsoPFTau30_eta2p1_CrossL1_v13
1000 31 +1 - ~3 HLT_IsoMu20_eta2p1_LooseChargedIsoPFTau27_eta2p1_CrossL1_v12
1000 26 +2 - ~1 HLT_IsoMu20_eta2p1_LooseChargedIsoPFTauHPS27_eta2p1_CrossL1_v4
1000 24 +2 - ~1 HLT_IsoMu20_eta2p1_MediumChargedIsoPFTauHPS27_eta2p1_CrossL1_v1
1000 23 +2 - ~1 HLT_IsoMu20_eta2p1_TightChargedIsoPFTauHPS27_eta2p1_CrossL1_v1
1000 22 +2 - ~1 HLT_IsoMu20_eta2p1_LooseChargedIsoPFTauHPS27_eta2p1_TightID_CrossL1_v1
1000 20 +2 - ~1 HLT_IsoMu20_eta2p1_MediumChargedIsoPFTauHPS27_eta2p1_TightID_CrossL1_v1
1000 19 +2 - ~1 HLT_IsoMu20_eta2p1_TightChargedIsoPFTauHPS27_eta2p1_TightID_CrossL1_v1
1000 13 +1 - ~2 HLT_IsoMu24_eta2p1_MediumChargedIsoPFTau35_Trk1_eta2p1_Reg_CrossL1_v12
1000 13 +1 - ~2 HLT_IsoMu24_eta2p1_MediumChargedIsoPFTau35_Trk1_TightID_eta2p1_Reg_CrossL1_v12
1000 11 +1 - ~2 HLT_IsoMu24_eta2p1_TightChargedIsoPFTau35_Trk1_eta2p1_Reg_CrossL1_v12
1000 11 +1 - ~2 HLT_IsoMu24_eta2p1_TightChargedIsoPFTau35_Trk1_TightID_eta2p1_Reg_CrossL1_v12
1000 11 +1 - ~2 HLT_IsoMu24_eta2p1_TightChargedIsoPFTauHPS35_Trk1_eta2p1_Reg_CrossL1_v1
1000 10 +1 - ~2 HLT_IsoMu24_eta2p1_MediumChargedIsoPFTauHPS35_Trk1_TightID_eta2p1_Reg_CrossL1_v1
1000 9 +1 - ~2 HLT_IsoMu24_eta2p1_TightChargedIsoPFTauHPS35_Trk1_TightID_eta2p1_Reg_CrossL1_v1
1000 12 +1 - ~2 HLT_IsoMu24_eta2p1_MediumChargedIsoPFTauHPS35_Trk1_eta2p1_Reg_CrossL1_v4
1000 15 +1 - ~2 HLT_IsoMu24_eta2p1_MediumChargedIsoPFTauHPS30_Trk1_eta2p1_Reg_CrossL1_v1
1000 37 +1 -1 ~1 HLT_IsoMu27_LooseChargedIsoPFTau20_Trk1_eta2p1_SingleL1_v5
1000 32 +1 - ~1 HLT_IsoMu27_LooseChargedIsoPFTauHPS20_Trk1_eta2p1_SingleL1_v1
1000 30 +1 - ~1 HLT_IsoMu27_MediumChargedIsoPFTauHPS20_Trk1_eta2p1_SingleL1_v1
1000 27 +1 - ~1 HLT_IsoMu27_TightChargedIsoPFTauHPS20_Trk1_eta2p1_SingleL1_v1
1000 144 +2 - ~1 HLT_IsoMu20_v15
1000 127 +2 - ~1 HLT_IsoMu24_v13
1000 119 +2 - ~1 HLT_IsoMu24_eta2p1_v15
1000 121 +1 - ~1 HLT_IsoMu27_v16
1000 999 +1 - - HLT_DiPFJet25_NoCaloMatched_v16
1000 966 +6 -7 - HLT_DiPFJet15_FBEta3_NoCaloMatched_v17
1000 254 +31 -29 - HLT_DiPFJet25_FBEta3_NoCaloMatched_v17
1000 969 +5 -1 - HLT_DiPFJetAve40_v14
1000 742 +6 -5 - HLT_DiPFJetAve60_v14
1000 466 +5 -6 - HLT_DiPFJetAve80_v13
1000 107 +2 - - HLT_DiPFJetAve140_v13
1000 990 +1 -1 - HLT_DiPFJetAve15_HFJEC_v17
1000 955 +3 -5 - HLT_DiPFJetAve25_HFJEC_v17
1000 825 +12 -4 - HLT_DiPFJetAve35_HFJEC_v17
1000 127 +13 -14 - HLT_DiPFJetAve60_HFJEC_v15
1000 45 +3 -4 - HLT_DiPFJetAve80_HFJEC_v16
1000 16 - -1 - HLT_DiPFJetAve100_HFJEC_v16
1000 891 +15 -16 - HLT_AK8PFJet60_v15
1000 656 +10 -14 - HLT_AK8PFJet80_v15
1000 217 +7 -5 - HLT_AK8PFJet140_v15
1000 85 +2 -1 - HLT_AK8PFJet200_v15
1000 40 +1 - - HLT_AK8PFJet260_v16
1000 27 - -1 - HLT_AK8PFJet320_v16
1000 6 - -1 - HLT_AK8PFJet450_v16
1000 978 +5 - - HLT_PFJet40_v21
1000 826 +7 -6 - HLT_PFJet60_v21
1000 568 +12 -8 - HLT_PFJet80_v20
1000 177 +4 -4 - HLT_PFJet140_v19
1000 56 +1 -2 - HLT_PFJet200_v19
1000 999 +1 - - HLT_PFJetFwd15_v3
1000 839 +27 -24 - HLT_PFJetFwd25_v3
1000 281 +15 -20 - HLT_PFJetFwd40_v19
1000 75 +1 -1 - HLT_PFJetFwd60_v19
1000 1000 - -1 - HLT_AK8PFJetFwd25_v3
1000 898 +35 -35 - HLT_AK8PFJetFwd40_v15
1000 186 +12 -9 - HLT_AK8PFJetFwd60_v14
1000 55 +3 -2 - HLT_AK8PFJetFwd80_v14
1000 630 +4 -9 - HLT_PFHT180_v17
1000 394 +6 -6 - HLT_PFHT250_v17
1000 163 +4 -2 - HLT_PFHT370_v17
1000 103 +2 - - HLT_PFHT430_v17
1000 64 +1 - - HLT_PFHT510_v17
1000 41 +1 -2 - HLT_PFHT590_v17
1000 18 +2 -1 - HLT_PFHT780_v17
1000 9 - -1 - HLT_PFHT890_v17
1000 15 - -3 ~8 HLT_PFHT500_PFMET100_PFMHT100_IDTight_v12
1000 12 - -1 ~9 HLT_PFHT500_PFMET110_PFMHT110_IDTight_v12
1000 8 - -1 ~2 HLT_PFHT700_PFMET95_PFMHT95_IDTight_v12
1000 48 +4 -4 ~7 HLT_PFMET110_PFMHT110_IDTight_v20
1000 38 +1 -4 ~10 HLT_PFMET120_PFMHT120_IDTight_v20
1000 23 - -1 ~8 HLT_PFMET130_PFMHT130_IDTight_v20
1000 19 +1 -1 ~5 HLT_PFMET140_PFMHT140_IDTight_v20
1000 50 +3 -6 ~6 HLT_PFMET100_PFMHT100_IDTight_CaloBTagDeepCSV_3p1_v8
1000 37 +3 -3 ~7 HLT_PFMET110_PFMHT110_IDTight_CaloBTagDeepCSV_3p1_v8
1000 31 - -4 ~9 HLT_PFMET120_PFMHT120_IDTight_CaloBTagDeepCSV_3p1_v8
1000 16 - -1 ~5 HLT_PFMET130_PFMHT130_IDTight_CaloBTagDeepCSV_3p1_v8
1000 13 +1 -1 ~4 HLT_PFMET140_PFMHT140_IDTight_CaloBTagDeepCSV_3p1_v8
1000 40 +1 -4 ~11 HLT_PFMET120_PFMHT120_IDTight_PFHT60_v9
1000 66 +2 -5 ~10 HLT_PFMETNoMu120_PFMHTNoMu120_IDTight_PFHT60_v9
1000 46 +2 -6 ~9 HLT_PFMETTypeOne120_PFMHT120_IDTight_PFHT60_v9
1000 53 +6 -4 ~7 HLT_PFMETTypeOne110_PFMHT110_IDTight_v12
1000 44 +2 -6 ~8 HLT_PFMETTypeOne120_PFMHT120_IDTight_v12
1000 29 - -5 ~8 HLT_PFMETTypeOne130_PFMHT130_IDTight_v12
1000 21 +1 -1 ~5 HLT_PFMETTypeOne140_PFMHT140_IDTight_v11
1000 75 +4 -7 ~8 HLT_PFMETNoMu110_PFMHTNoMu110_IDTight_v20
1000 64 +2 -5 ~9 HLT_PFMETNoMu120_PFMHTNoMu120_IDTight_v20
1000 49 +3 -5 ~5 HLT_PFMETNoMu130_PFMHTNoMu130_IDTight_v19
1000 34 +2 -1 ~4 HLT_PFMETNoMu140_PFMHTNoMu140_IDTight_v19
1000 59 +2 -4 ~8 HLT_MonoCentralPFJet80_PFMETNoMu110_PFMHTNoMu110_IDTight_v20
1000 50 +1 -2 ~11 HLT_MonoCentralPFJet80_PFMETNoMu120_PFMHTNoMu120_IDTight_v20
1000 39 +1 -2 ~6 HLT_MonoCentralPFJet80_PFMETNoMu130_PFMHTNoMu130_IDTight_v19
1000 28 +2 - ~4 HLT_MonoCentralPFJet80_PFMETNoMu140_PFMHTNoMu140_IDTight_v19
1000 8 +1 - - HLT_PFMETTypeOne200_BeamHaloCleaned_v9
1000 5 - -1 ~1 HLT_Mu12_DoublePFJets62MaxDeta1p6_DoubleCaloBTagDeepCSV_p71_v2
1000 641 +1 -3 - HLT_DoublePFJets40_CaloBTagDeepCSV_p71_v2
1000 99 +2 -2 - HLT_DoublePFJets100_CaloBTagDeepCSV_p71_v2
1000 20 - -1 ~1 HLT_DoublePFJets116MaxDeta1p6_DoubleCaloBTagDeepCSV_p71_v2
1000 103 +1 - ~2 HLT_IsoMu24_TwoProngs35_v1
1000 24 +1 -4 - HLT_DoubleMu3_DZ_PFMET50_PFMHT60_v10
1000 14 +2 -3 ~2 HLT_DoubleMu3_DZ_PFMET70_PFMHT70_v10
1000 7 - -2 - HLT_DoubleMu3_DZ_PFMET90_PFMHT90_v10
1000 106 +1 - - HLT_Ele30_eta2p1_WPTight_Gsf_CentralPFJet35_EleCleaned_v13
1000 92 +1 -1 - HLT_Ele28_eta2p1_WPTight_Gsf_HT150_v13
1000 18 - -1 - HLT_Ele15_IsoVVVL_PFHT450_CaloBTagDeepCSV_4p5_v8
1000 12 +1 - ~1 HLT_Ele15_IsoVVVL_PFHT450_PFMET50_v16
1000 19 - -1 - HLT_Ele15_IsoVVVL_PFHT450_v16
1000 14 - -1 - HLT_Ele50_IsoVVVL_PFHT450_v16
1000 7 +1 -1 ~1 HLT_Mu4_TrkIsoVVL_DiPFJet90_40_DEta3p5_MJJ750_HTT300_PFMETNoMu60_v15
1000 8 +1 -2 ~3 HLT_Mu8_TrkIsoVVL_DiPFJet40_DEta3p5_MJJ750_HTT300_PFMETNoMu60_v16
1000 3 +2 - ~3 HLT_Mu10_TrkIsoVVL_DiPFJet40_DEta3p5_MJJ750_HTT350_PFMETNoMu60_v15
1000 16 +1 -2 - HLT_Mu15_IsoVVVL_PFHT450_PFMET50_v15
1000 20 +1 - - HLT_Mu15_IsoVVVL_PFHT450_v15
1000 16 +1 - - HLT_Mu50_IsoVVVL_PFHT450_v15
1000 10 - -1 - HLT_Mu15_IsoVVVL_PFHT600_v19
1000 38 +3 -3 ~5 HLT_Mu3er1p5_PFJet100er2p5_PFMET70_PFMHT70_IDTight_v2
1000 32 +3 -2 ~5 HLT_Mu3er1p5_PFJet100er2p5_PFMET80_PFMHT80_IDTight_v2
1000 25 +4 -1 ~4 HLT_Mu3er1p5_PFJet100er2p5_PFMET90_PFMHT90_IDTight_v2
1000 21 +2 -4 ~1 HLT_Mu3er1p5_PFJet100er2p5_PFMET100_PFMHT100_IDTight_v2
1000 45 +3 -4 ~3 HLT_Mu3er1p5_PFJet100er2p5_PFMETNoMu70_PFMHTNoMu70_IDTight_v2
1000 40 +4 -2 ~1 HLT_Mu3er1p5_PFJet100er2p5_PFMETNoMu80_PFMHTNoMu80_IDTight_v2
1000 35 +3 -2 ~3 HLT_Mu3er1p5_PFJet100er2p5_PFMETNoMu90_PFMHTNoMu90_IDTight_v2
1000 31 +2 -2 ~4 HLT_Mu3er1p5_PFJet100er2p5_PFMETNoMu100_PFMHTNoMu100_IDTight_v2
1000 194 +1 -1 - HLT_Ele8_CaloIdL_TrackIdL_IsoVL_PFJet30_v16
1000 184 +1 -1 - HLT_Ele12_CaloIdL_TrackIdL_IsoVL_PFJet30_v18
1000 179 +1 -1 - HLT_Ele15_CaloIdL_TrackIdL_IsoVL_PFJet30_v3
1000 151 - -1 - HLT_Ele23_CaloIdL_TrackIdL_IsoVL_PFJet30_v18
1000 219 +1 -2 - HLT_Ele8_CaloIdM_TrackIdM_PFJet30_v18
1000 171 +1 -2 - HLT_Ele17_CaloIdM_TrackIdM_PFJet30_v16
1000 157 - -2 - HLT_Ele23_CaloIdM_TrackIdM_PFJet30_v18
1000 29 +3 - ~8 HLT_PFHT330PT30_QuadPFJet_75_60_45_40_TriplePFBTagDeepCSV_4p5_v3
1000 158 +4 -2 ~8 HLT_PFHT330PT30_QuadPFJet_75_60_45_40_v9
1000 33 +1 -2 ~4 HLT_PFHT400_SixPFJet32_DoublePFBTagDeepCSV_2p94_v8
1000 48 +1 -2 ~3 HLT_PFHT400_SixPFJet32_v8
1000 28 - -1 ~6 HLT_PFHT450_SixPFJet36_PFBTagDeepCSV_1p59_v7
1000 32 +1 -2 ~4 HLT_PFHT450_SixPFJet36_v7
1000 88 +1 -5 ~7 HLT_PFHT400_FivePFJet_100_100_60_30_30_v8
1000 58 +1 -7 ~10 HLT_PFHT400_FivePFJet_100_100_60_30_30_DoublePFBTagDeepCSV_4p5_v8
1000 43 +1 -5 ~5 HLT_PFHT400_FivePFJet_120_120_60_30_30_DoublePFBTagDeepCSV_4p5_v8
1000 190 +6 -2 - HLT_PFHT350_v19
1000 247 +6 -10 - HLT_PFHT350MinPFJet15_v9
1000 5 +1 -1 - HLT_Photon60_R9Id90_CaloIdL_IsoL_DisplacedIdL_PFHT350MinPFJet15_v11
1000 361 +1 - - HLT_AK4CaloJet120_v9
1000 996 - -1 - HLT_AK4PFJet30_v19
1000 932 +9 -6 - HLT_AK4PFJet50_v19
1000 568 +12 -8 - HLT_AK4PFJet80_v19
1000 385 +7 -9 - HLT_AK4PFJet100_v19
1000 262 +5 -3 - HLT_AK4PFJet120_v18
1000 825 +7 -5 - MC_PFBTagDeepCSV_v10
1000 1000 - -2 - MC_AK8PFJets_v17
1000 156 - -1 - MC_Ele5_WPTight_Gsf_v8
1000 175 +2 - ~1 MC_IsoMu_v15
1000 2 - -1 - HLT_MediumChargedIsoPFTau200HighPtRelaxedIso_Trk50_eta2p1_v12
1000 16 - -1 ~5 HLT_RsqMR300_Rsq0p09_MR200_v15
1000 13 - -1 ~11 HLT_RsqMR300_Rsq0p09_MR200_4jet_v15
1000 8 +1 - ~12 HLT_RsqMR320_Rsq0p09_MR200_4jet_v15
1000 56 +1 - ~1 HLT_IsoMu27_MET90_v3
1000 2 +1 - ~2 HLT_VBF_DoubleLooseChargedIsoPFTau20_Trk1_eta2p1_v3
1000 67 +5 -7 ~15 HLT_PFMET100_PFMHT100_IDTight_PFHT60_v9
1000 92 +8 -6 ~18 HLT_PFMETNoMu100_PFMHTNoMu100_IDTight_PFHT60_v9
1000 75 +6 -4 ~14 HLT_PFMETTypeOne100_PFMHT100_IDTight_PFHT60_v9
1000 19 +1 -2 - HLT_DoubleMu3_DCA_PFMET50_PFMHT60_v10
1000 21 +5 -3 ~10 HLT_QuadPFJet98_83_71_15_DoublePFBTagDeepCSV_1p3_7p7_VBF1_v8
1000 15 +4 -1 ~5 HLT_QuadPFJet103_88_75_15_DoublePFBTagDeepCSV_1p3_7p7_VBF1_v8
1000 12 +4 -1 ~6 HLT_QuadPFJet111_90_80_15_DoublePFBTagDeepCSV_1p3_7p7_VBF1_v8
1000 6 - - ~12 HLT_QuadPFJet98_83_71_15_PFBTagDeepCSV_1p3_VBF2_v8
1000 163 +3 -5 ~6 HLT_QuadPFJet98_83_71_15_v5
1000 115 +2 -3 ~2 HLT_QuadPFJet103_88_75_15_v5
1000 110 +3 -5 ~2 HLT_QuadPFJet105_88_76_15_v5
1000 88 +3 -1 ~4 HLT_QuadPFJet111_90_80_15_v5
1000 10 +1 -1 - HLT_AK8PFJet330_TrimMass30_PFAK8BTagDeepCSV_p17_v2
1000 9 - -1 - HLT_AK8PFJet330_TrimMass30_PFAK8BTagDeepCSV_p1_v2
1000 13 - -1 - HLT_AK8PFJet330_TrimMass30_PFAK8BoostedDoubleB_p02_v3
1000 20 - -1 - HLT_AK8PFJet330_TrimMass30_PFAK8BoostedDoubleB_np2_v2
1000 20 - -1 - HLT_AK8PFJet330_TrimMass30_PFAK8BoostedDoubleB_np4_v2
1000 14 +5 -1 ~4 HLT_QuadPFJet105_88_76_15_DoublePFBTagDeepCSV_1p3_7p7_VBF1_v8and here comparing cpu vs gpu in CMSSW_12_1_SKYLAKEAVX512_X_2021-09-23-2300 (for TK patatrack only)
hltDiff -o output-cpuavxTK_1K.root -n output-gpuavxTK_1K.root | grep -v "\- \- \~"
Found 1000 matching events, out of which 1000 have different HLT results
Events Accepted Gained Lost Other Trigger
1000 7 +1 - - HLT_AK8PFJet420_TrimMass30_v11
1000 4 +1 - - HLT_AK8PFHT750_TrimMass50_v12
1000 398 +2 - - HLT_Mu3_PFJet40_v16
1000 10 +1 - ~2 HLT_Mu7p5_Track2_Jpsi_v11
1000 407 +2 - - HLT_Mu3_L1SingleMu5orSingleMu7_v1
1000 139 +2 - - HLT_Ele20_WPTight_Gsf_v6
1000 153 +1 - - HLT_Ele15_WPLoose_Gsf_v3
1000 146 +1 - - HLT_Ele17_WPLoose_Gsf_v3
1000 140 +1 - - HLT_Ele20_WPLoose_Gsf_v6
1000 137 +1 - - HLT_Ele20_eta2p1_WPLoose_Gsf_v6
1000 120 +1 - - HLT_Ele27_WPTight_Gsf_v16
1000 119 +1 - - HLT_Ele28_WPTight_Gsf_v1
1000 112 +1 - - HLT_Ele30_WPTight_Gsf_v1
1000 107 +1 - - HLT_Ele32_WPTight_Gsf_v15
1000 103 +1 - - HLT_Ele35_WPTight_Gsf_v9
1000 96 +1 - - HLT_Ele35_WPTight_Gsf_L1EGMT_v5
1000 89 +1 - - HLT_Ele38_WPTight_Gsf_v9
1000 87 +1 - - HLT_Ele40_WPTight_Gsf_v9
1000 108 +1 - - HLT_Ele32_WPTight_Gsf_L1DoubleEG_v9
1000 22 - -1 ~1 HLT_Ele24_eta2p1_WPTight_Gsf_LooseChargedIsoPFTau30_eta2p1_CrossL1_v13
1000 32 +2 - ~3 HLT_IsoMu20_eta2p1_LooseChargedIsoPFTau27_eta2p1_CrossL1_v12
1000 28 +1 - ~3 HLT_IsoMu20_eta2p1_LooseChargedIsoPFTauHPS27_eta2p1_CrossL1_v4
1000 26 +1 - ~3 HLT_IsoMu20_eta2p1_MediumChargedIsoPFTauHPS27_eta2p1_CrossL1_v1
1000 25 +1 - ~4 HLT_IsoMu20_eta2p1_TightChargedIsoPFTauHPS27_eta2p1_CrossL1_v1
1000 24 +1 -1 ~3 HLT_IsoMu20_eta2p1_LooseChargedIsoPFTauHPS27_eta2p1_TightID_CrossL1_v1
1000 22 +1 -1 ~3 HLT_IsoMu20_eta2p1_MediumChargedIsoPFTauHPS27_eta2p1_TightID_CrossL1_v1
1000 21 +1 -1 ~4 HLT_IsoMu20_eta2p1_TightChargedIsoPFTauHPS27_eta2p1_TightID_CrossL1_v1
1000 37 +3 - ~1 HLT_IsoMu27_LooseChargedIsoPFTau20_Trk1_eta2p1_SingleL1_v5
1000 33 +1 - ~1 HLT_IsoMu27_LooseChargedIsoPFTauHPS20_Trk1_eta2p1_SingleL1_v1
1000 31 +1 - ~1 HLT_IsoMu27_MediumChargedIsoPFTauHPS20_Trk1_eta2p1_SingleL1_v1
1000 28 +1 - ~2 HLT_IsoMu27_TightChargedIsoPFTauHPS20_Trk1_eta2p1_SingleL1_v1
1000 146 +1 - - HLT_IsoMu20_v15
1000 129 +1 - - HLT_IsoMu24_v13
1000 121 +1 - - HLT_IsoMu24_eta2p1_v15
1000 122 +1 - - HLT_IsoMu27_v16
1000 275 +2 - - HLT_Mu12_v3
1000 258 +1 - - HLT_Mu15_v3
1000 221 +1 - - HLT_Mu20_v12
1000 167 +1 - - HLT_Mu27_v13
1000 965 +1 -1 - HLT_DiPFJet15_FBEta3_NoCaloMatched_v17
1000 256 +2 -4 - HLT_DiPFJet25_FBEta3_NoCaloMatched_v17
1000 743 +3 -1 - HLT_DiPFJetAve60_v14
1000 465 +4 -4 - HLT_DiPFJetAve80_v13
1000 109 - -1 - HLT_DiPFJetAve140_v13
1000 953 +2 -3 - HLT_DiPFJetAve25_HFJEC_v17
1000 833 +1 -2 - HLT_DiPFJetAve35_HFJEC_v17
1000 126 +1 -1 - HLT_DiPFJetAve60_HFJEC_v15
1000 890 +2 -3 - HLT_AK8PFJet60_v15
1000 652 +6 -2 - HLT_AK8PFJet80_v15
1000 219 +2 -5 - HLT_AK8PFJet140_v15
1000 86 - -1 - HLT_AK8PFJet200_v15
1000 983 +1 - - HLT_PFJet40_v21
1000 827 +1 -1 - HLT_PFJet60_v21
1000 572 +6 -2 - HLT_PFJet80_v20
1000 177 - -1 - HLT_PFJet140_v19
1000 842 +2 -4 - HLT_PFJetFwd25_v3
1000 276 +1 -3 - HLT_PFJetFwd40_v19
1000 999 - -1 - HLT_AK8PFJetFwd25_v3
1000 898 +6 -4 - HLT_AK8PFJetFwd40_v15
1000 189 +1 - - HLT_AK8PFJetFwd60_v14
1000 56 +1 - - HLT_AK8PFJetFwd80_v14
1000 625 +6 -3 - HLT_PFHT180_v17
1000 394 +4 -4 - HLT_PFHT250_v17
1000 165 +4 -3 - HLT_PFHT370_v17
1000 105 +3 - - HLT_PFHT430_v17
1000 19 - -1 - HLT_PFHT780_v17
1000 12 +1 - ~2 HLT_PFHT500_PFMET100_PFMHT100_IDTight_v12
1000 11 +1 - ~3 HLT_PFHT500_PFMET110_PFMHT110_IDTight_v12
1000 7 +1 - - HLT_PFHT700_PFMET95_PFMHT95_IDTight_v12
1000 48 +2 -4 ~2 HLT_PFMET110_PFMHT110_IDTight_v20
1000 35 +1 -1 ~1 HLT_PFMET120_PFMHT120_IDTight_v20
1000 19 +1 - - HLT_PFMET140_PFMHT140_IDTight_v20
1000 47 - -1 ~1 HLT_PFMET100_PFMHT100_IDTight_CaloBTagDeepCSV_3p1_v8
1000 37 +1 -3 ~1 HLT_PFMET110_PFMHT110_IDTight_CaloBTagDeepCSV_3p1_v8
1000 27 +1 -1 - HLT_PFMET120_PFMHT120_IDTight_CaloBTagDeepCSV_3p1_v8
1000 37 +1 -1 ~1 HLT_PFMET120_PFMHT120_IDTight_PFHT60_v9
1000 63 - -2 ~1 HLT_PFMETNoMu120_PFMHTNoMu120_IDTight_PFHT60_v9
1000 42 +1 -2 ~1 HLT_PFMETTypeOne120_PFMHT120_IDTight_PFHT60_v9
1000 55 +1 -2 ~2 HLT_PFMETTypeOne110_PFMHT110_IDTight_v12
1000 40 +1 -2 ~1 HLT_PFMETTypeOne120_PFMHT120_IDTight_v12
1000 24 +2 - - HLT_PFMETTypeOne130_PFMHT130_IDTight_v12
1000 21 - -1 - HLT_PFMETTypeOne140_PFMHT140_IDTight_v11
1000 72 +2 -3 ~4 HLT_PFMETNoMu110_PFMHTNoMu110_IDTight_v20
1000 61 - -2 ~1 HLT_PFMETNoMu120_PFMHTNoMu120_IDTight_v20
1000 35 +1 -1 ~1 HLT_PFMETNoMu140_PFMHTNoMu140_IDTight_v19
1000 57 +2 -2 ~5 HLT_MonoCentralPFJet80_PFMETNoMu110_PFMHTNoMu110_IDTight_v20
1000 49 - -2 ~3 HLT_MonoCentralPFJet80_PFMETNoMu120_PFMHTNoMu120_IDTight_v20
1000 30 +1 -1 ~2 HLT_MonoCentralPFJet80_PFMETNoMu140_PFMHTNoMu140_IDTight_v19
1000 9 - -1 - HLT_PFMETTypeOne200_BeamHaloCleaned_v9
1000 30 +4 -1 - HLT_SingleJet30_Mu12_SinglePFJet40_v11
1000 23 +4 -1 - HLT_Mu12_DoublePFJets40_CaloBTagDeepCSV_p71_v2
1000 7 +1 -1 - HLT_Mu12_DoublePFJets100_CaloBTagDeepCSV_p71_v2
1000 7 - -1 - HLT_Mu12_DoublePFJets40MaxDeta1p6_DoubleCaloBTagDeepCSV_p71_v2
1000 6 - -1 - HLT_Mu12_DoublePFJets54MaxDeta1p6_DoubleCaloBTagDeepCSV_p71_v2
1000 4 - -1 ~1 HLT_Mu12_DoublePFJets62MaxDeta1p6_DoubleCaloBTagDeepCSV_p71_v2
1000 639 +2 - - HLT_DoublePFJets40_CaloBTagDeepCSV_p71_v2
1000 99 +2 - - HLT_DoublePFJets100_CaloBTagDeepCSV_p71_v2
1000 253 +2 -2 - HLT_Mu8_TrkIsoVVL_v12
1000 23 +1 -2 ~2 HLT_Mu8_TrkIsoVVL_Ele23_CaloIdL_TrackIdL_IsoVL_DZ_v13
1000 23 +1 -2 ~2 HLT_Mu8_TrkIsoVVL_Ele23_CaloIdL_TrackIdL_IsoVL_DZ_PFDiJet30_v1
1000 23 +1 -2 ~2 HLT_Mu8_TrkIsoVVL_Ele23_CaloIdL_TrackIdL_IsoVL_DZ_CaloDiJet30_v1
1000 13 +1 -1 ~4 HLT_Mu8_TrkIsoVVL_Ele23_CaloIdL_TrackIdL_IsoVL_DZ_PFDiJet30_PFBtagDeepCSV_1p5_v1
1000 17 - -2 ~3 HLT_Mu8_TrkIsoVVL_Ele23_CaloIdL_TrackIdL_IsoVL_DZ_CaloDiJet30_CaloBtagDeepCSV_1p5_v1
1000 25 +1 -2 ~2 HLT_Mu8_TrkIsoVVL_Ele23_CaloIdL_TrackIdL_IsoVL_v11
1000 198 +1 -2 - HLT_Mu17_TrkIsoVVL_v13
1000 185 +1 -2 - HLT_Mu19_TrkIsoVVL_v4
1000 77 +2 -1 - HLT_BTagMu_AK4DiJet20_Mu5_v13
1000 71 +2 -1 - HLT_BTagMu_AK4DiJet40_Mu5_v13
1000 32 +1 - - HLT_BTagMu_AK4DiJet70_Mu5_v13
1000 15 - -2 ~1 HLT_Mu23_TrkIsoVVL_Ele12_CaloIdL_TrackIdL_IsoVL_DZ_v15
1000 16 - -2 ~1 HLT_Mu23_TrkIsoVVL_Ele12_CaloIdL_TrackIdL_IsoVL_v7
1000 20 +1 -2 ~2 HLT_Mu12_TrkIsoVVL_Ele23_CaloIdL_TrackIdL_IsoVL_v7
1000 18 +1 -2 ~2 HLT_Mu12_TrkIsoVVL_Ele23_CaloIdL_TrackIdL_IsoVL_DZ_v15
1000 12 +1 -1 - HLT_Photon35_TwoProngs35_v1
1000 104 - -1 ~1 HLT_IsoMu24_TwoProngs35_v1
1000 21 +1 -2 - HLT_DoubleMu3_DZ_PFMET50_PFMHT60_v10
1000 13 - -2 - HLT_DoubleMu3_DZ_PFMET70_PFMHT70_v10
1000 5 +2 - - HLT_DoubleMu3_DZ_PFMET90_PFMHT90_v10
1000 14 - -1 ~4 HLT_DiJet110_35_Mjj650_PFMET110_v9
1000 107 +1 - - HLT_Ele30_eta2p1_WPTight_Gsf_CentralPFJet35_EleCleaned_v13
1000 92 +2 - - HLT_Ele28_eta2p1_WPTight_Gsf_HT150_v13
1000 17 +1 - - HLT_Ele15_IsoVVVL_PFHT450_CaloBTagDeepCSV_4p5_v8
1000 18 +1 - - HLT_Ele15_IsoVVVL_PFHT450_v16
1000 15 +1 -1 - HLT_Mu15_IsoVVVL_PFHT450_PFMET50_v15
1000 21 - -1 - HLT_Mu15_IsoVVVL_PFHT450_v15
1000 17 - -1 - HLT_Mu50_IsoVVVL_PFHT450_v15
1000 38 - -1 - HLT_Mu3er1p5_PFJet100er2p5_PFMET70_PFMHT70_IDTight_v2
1000 33 - -1 - HLT_Mu3er1p5_PFJet100er2p5_PFMET80_PFMHT80_IDTight_v2
1000 28 +1 -1 - HLT_Mu3er1p5_PFJet100er2p5_PFMET90_PFMHT90_IDTight_v2
1000 19 +1 - ~1 HLT_Mu3er1p5_PFJet100er2p5_PFMET100_PFMHT100_IDTight_v2
1000 36 +1 -1 - HLT_Mu3er1p5_PFJet100er2p5_PFMETNoMu90_PFMHTNoMu90_IDTight_v2
1000 9 - -1 - HLT_DiMu4_Ele9_CaloIdL_TrackIdL_DZ_Mass3p8_v17
1000 4 - -1 - HLT_DiMu9_Ele9_CaloIdL_TrackIdL_DZ_v17
1000 5 - -1 - HLT_DiMu9_Ele9_CaloIdL_TrackIdL_v17
1000 327 +2 - - HLT_Mu8_v12
1000 248 +1 - - HLT_Mu17_v13
1000 227 +1 - - HLT_Mu19_v4
1000 194 +1 -1 ~2 HLT_Ele8_CaloIdL_TrackIdL_IsoVL_PFJet30_v16
1000 218 +1 - ~1 HLT_Ele8_CaloIdM_TrackIdM_PFJet30_v18
1000 32 +5 - ~5 HLT_PFHT330PT30_QuadPFJet_75_60_45_40_TriplePFBTagDeepCSV_4p5_v3
1000 160 +2 -1 ~5 HLT_PFHT330PT30_QuadPFJet_75_60_45_40_v9
1000 32 +3 -1 ~5 HLT_PFHT400_SixPFJet32_DoublePFBTagDeepCSV_2p94_v8
1000 47 +1 -2 ~4 HLT_PFHT400_SixPFJet32_v8
1000 27 +3 - ~5 HLT_PFHT450_SixPFJet36_PFBTagDeepCSV_1p59_v7
1000 31 +2 - ~5 HLT_PFHT450_SixPFJet36_v7
1000 84 +3 -2 ~3 HLT_PFHT400_FivePFJet_100_100_60_30_30_v8
1000 52 +3 -2 ~6 HLT_PFHT400_FivePFJet_100_100_60_30_30_DoublePFBTagDeepCSV_4p5_v8
1000 39 +2 -2 ~1 HLT_PFHT400_FivePFJet_120_120_60_30_30_DoublePFBTagDeepCSV_4p5_v8
1000 194 +5 -4 - HLT_PFHT350_v19
1000 243 +1 - - HLT_PFHT350MinPFJet15_v9
1000 935 +1 - - HLT_AK4PFJet50_v19
1000 572 +6 -2 - HLT_AK4PFJet80_v19
1000 383 +2 - - HLT_AK4PFJet100_v19
1000 264 +2 -1 - HLT_AK4PFJet120_v18
1000 43 +1 -1 ~3 HLT_IsoTrackHB_v4
1000 827 +11 -6 - MC_PFBTagDeepCSV_v10
1000 25 +2 - ~4 MC_DoubleEle5_CaloIdL_MW_v15
1000 155 +2 - - MC_Ele5_WPTight_Gsf_v8
1000 177 +2 - ~1 MC_IsoMu_v15
1000 41 - -1 - MC_DoubleMu_TrkIsoVVL_DZ_v11
1000 17 +1 - ~1 HLT_MediumChargedIsoPFTau50_Trk30_eta2p1_1pr_MET90_v12
1000 11 +1 - ~1 HLT_MediumChargedIsoPFTau50_Trk30_eta2p1_1pr_MET100_v12
1000 10 +1 - ~1 HLT_MediumChargedIsoPFTau50_Trk30_eta2p1_1pr_MET110_v8
1000 7 +1 - - HLT_MediumChargedIsoPFTau50_Trk30_eta2p1_1pr_MET120_v8
1000 4 - -1 ~1 HLT_Rsq0p35_v15
1000 3 - -1 ~1 HLT_Rsq0p40_v15
1000 15 - -1 ~2 HLT_RsqMR300_Rsq0p09_MR200_v15
1000 13 - -1 ~2 HLT_RsqMR320_Rsq0p09_MR200_v15
1000 12 - -1 ~4 HLT_RsqMR300_Rsq0p09_MR200_4jet_v15
1000 9 - -1 ~4 HLT_RsqMR320_Rsq0p09_MR200_4jet_v15
1000 57 +1 - - HLT_IsoMu27_MET90_v3
1000 12 +1 - ~3 HLT_DoubleTightChargedIsoPFTau35_Trk1_eta2p1_Reg_v12
1000 11 +1 - ~3 HLT_DoubleTightChargedIsoPFTau35_Trk1_TightID_eta2p1_Reg_v12
1000 65 +1 -1 ~3 HLT_PFMET100_PFMHT100_IDTight_PFHT60_v9
1000 94 +1 -2 ~4 HLT_PFMETNoMu100_PFMHTNoMu100_IDTight_PFHT60_v9
1000 77 +1 -1 ~3 HLT_PFMETTypeOne100_PFMHT100_IDTight_PFHT60_v9
1000 18 +1 - - HLT_DoubleMu3_DCA_PFMET50_PFMHT60_v10
1000 23 +1 - ~5 HLT_QuadPFJet98_83_71_15_DoublePFBTagDeepCSV_1p3_7p7_VBF1_v8
1000 18 +2 - ~3 HLT_QuadPFJet103_88_75_15_DoublePFBTagDeepCSV_1p3_7p7_VBF1_v8
1000 15 +1 - ~2 HLT_QuadPFJet111_90_80_15_DoublePFBTagDeepCSV_1p3_7p7_VBF1_v8
1000 6 +1 - ~5 HLT_QuadPFJet98_83_71_15_PFBTagDeepCSV_1p3_VBF2_v8
1000 161 +2 -1 ~2 HLT_QuadPFJet98_83_71_15_v5
1000 114 +1 - ~2 HLT_QuadPFJet103_88_75_15_v5
1000 108 +2 -1 ~1 HLT_QuadPFJet105_88_76_15_v5
1000 90 +1 - ~1 HLT_QuadPFJet111_90_80_15_v5
1000 10 +1 - - HLT_AK8PFJet330_TrimMass30_PFAK8BTagDeepCSV_p17_v2
1000 8 +1 - - HLT_AK8PFJet330_TrimMass30_PFAK8BTagDeepCSV_p1_v2
1000 12 +1 - - HLT_AK8PFJet330_TrimMass30_PFAK8BoostedDoubleB_p02_v3
1000 11 +1 - - HLT_Diphoton30_18_R9IdL_AND_HE_AND_IsoCaloId_NoPixelVeto_v2
1000 93 +1 - - HLT_Mu9_IP5_ToCSCS_v1
1000 93 +1 - - HLT_Mu9_IP5_part0_v2
1000 93 +1 - - HLT_Mu9_IP5_part1_v2
1000 93 +1 - - HLT_Mu9_IP5_part2_v2
1000 93 +1 - - HLT_Mu9_IP5_part3_v2
1000 93 +1 - - HLT_Mu9_IP5_part4_v2
1000 103 +1 - - HLT_Mu8_IP5_ToCSCS_v1
1000 103 +1 - - HLT_Mu8_IP5_part0_v2
1000 103 +1 - - HLT_Mu8_IP5_part1_v2
1000 103 +1 - - HLT_Mu8_IP5_part2_v2
1000 103 +1 - - HLT_Mu8_IP5_part3_v2
1000 103 +1 - - HLT_Mu8_IP5_part4_v2
1000 123 +1 - - HLT_Mu8_IP3_ToCSCS_v1
1000 123 +1 - - HLT_Mu8_IP3_part0_v3
1000 123 +1 - - HLT_Mu8_IP3_part1_v3
1000 123 +1 - - HLT_Mu8_IP3_part2_v3
1000 123 +1 - - HLT_Mu8_IP3_part3_v3
1000 123 +1 - - HLT_Mu8_IP3_part4_v3
1000 303 +2 - - HLT_Mu9_IP0_part0_v2
1000 113 +1 - - HLT_Mu9_IP3_part0_v2
1000 18 +2 - ~2 HLT_QuadPFJet105_88_76_15_DoublePFBTagDeepCSV_1p3_7p7_VBF1_v8
1000 328 +2 - - HLT_TrkMu6NoFiltersNoVtx_v1
1000 252 +2 - - HLT_TrkMu16NoFiltersNoVtx_v1
1000 1000 - -1000 - Status_OnCPU
1000 0 +1000 - - Status_OnGPUMaybe we should change the issue in "HLT Regression due to differences in floating point operation reassociation(and math-lib precision)"
VinInn
commented
3 years ago and here in CMSSW_12_1_SKYLAKEAVX512_X_2021-09-23-2300 comparing cmsRun with cmsRunVDT
Found 1000 matching events, out of which 0 have different HLT results
AdrianoDee
commented
3 years ago Just dropping a comparison here using the configuration above (adding this just to check the effect on hlt full tracks). To be sure the comparison is done on the same machine forcing CPU or GPU offload (with forceGpuOffload)
/RelValTTbar_14TeV/CMSSW_12_0_0_pre6-PU_120X_mcRun3_2021_realistic_v4_JIRA_129-v1/GEN-SIM-DIGI-RAWCMSSW_12_1_0_pre3Results: plots. The comparison is done between CPU and two runs on GPU (on the same events). There are just really minor fluctuations.
VinInn
commented
3 years ago about regression in Pixel local reco: see https://hypernews.cern.ch/HyperNews/CMS/get/pixelOfflineSW/1587/1/1.html
VinInn
commented
3 years ago The fixes to make the GPU as close as possible to CPU for what concern Pixel Clusterizer produce these changes
hltDiff -o output-gpuTKfix_1K.root -n output-gpuTK_1K.root | grep -v "\- \- \~"
.....
Found 1000 matching events, out of which 36 have different HLT results
Events Accepted Gained Lost Other Trigger
1000 140 +1 - - HLT_Ele20_WPTight_Gsf_v6
1000 32 +1 - - HLT_IsoMu20_eta2p1_LooseChargedIsoPFTau27_eta2p1_CrossL1_v12
1000 26 +1 - - HLT_IsoMu20_eta2p1_LooseChargedIsoPFTauHPS27_eta2p1_CrossL1_v4
1000 24 +1 - - HLT_IsoMu20_eta2p1_MediumChargedIsoPFTauHPS27_eta2p1_CrossL1_v1
1000 23 +1 - - HLT_IsoMu20_eta2p1_TightChargedIsoPFTauHPS27_eta2p1_CrossL1_v1
1000 21 +2 - - HLT_IsoMu20_eta2p1_LooseChargedIsoPFTauHPS27_eta2p1_TightID_CrossL1_v1
1000 19 +2 - - HLT_IsoMu20_eta2p1_MediumChargedIsoPFTauHPS27_eta2p1_TightID_CrossL1_v1
1000 18 +2 - - HLT_IsoMu20_eta2p1_TightChargedIsoPFTauHPS27_eta2p1_TightID_CrossL1_v1
1000 39 +1 - - HLT_IsoMu27_LooseChargedIsoPFTau20_Trk1_eta2p1_SingleL1_v5
1000 32 +1 - - HLT_IsoMu27_LooseChargedIsoPFTauHPS20_Trk1_eta2p1_SingleL1_v1
1000 30 +1 - - HLT_IsoMu27_MediumChargedIsoPFTauHPS20_Trk1_eta2p1_SingleL1_v1
1000 27 +1 - - HLT_IsoMu27_TightChargedIsoPFTauHPS20_Trk1_eta2p1_SingleL1_v1
1000 255 +1 - - HLT_DiPFJet25_FBEta3_NoCaloMatched_v17
1000 956 +1 -1 - HLT_DiPFJetAve25_HFJEC_v17
1000 892 +1 -1 - HLT_AK8PFJet60_v15
1000 827 - -1 - HLT_PFJet60_v21
1000 570 - -1 - HLT_PFJet80_v20
1000 839 - -1 - HLT_PFJetFwd25_v3
1000 900 +1 - - HLT_AK8PFJetFwd40_v15
1000 185 +1 - - HLT_AK8PFJetFwd60_v14
1000 633 +2 -1 - HLT_PFHT180_v17
1000 48 - -1 - HLT_PFMET110_PFMHT110_IDTight_v20
1000 36 - -1 - HLT_PFMET110_PFMHT110_IDTight_CaloBTagDeepCSV_3p1_v8
1000 12 +1 - - HLT_Mu8_TrkIsoVVL_Ele23_CaloIdL_TrackIdL_IsoVL_DZ_PFDiJet30_PFBtagDeepCSV_1p5_v1
1000 25 - -2 - HLT_DoubleMu3_DZ_PFMET50_PFMHT60_v10
1000 19 - -3 - HLT_DoubleMu3_DZ_PFMET70_PFMHT70_v10
1000 13 +1 - - HLT_DiJet110_35_Mjj650_PFMET110_v9
1000 10 +1 - - HLT_DiJet110_35_Mjj650_PFMET120_v9
1000 8 +1 - - HLT_DiJet110_35_Mjj650_PFMET130_v9
1000 13 +1 - - HLT_Ele15_IsoVVVL_PFHT450_PFMET50_v16
1000 33 +1 - - HLT_PFHT330PT30_QuadPFJet_75_60_45_40_TriplePFBTagDeepCSV_4p5_v3
1000 58 - -1 - HLT_PFHT400_FivePFJet_100_100_60_30_30_DoublePFBTagDeepCSV_4p5_v8
1000 43 - -1 - HLT_PFHT400_FivePFJet_120_120_60_30_30_DoublePFBTagDeepCSV_4p5_v8
1000 250 - -1 - HLT_PFHT350MinPFJet15_v9
1000 570 - -1 - HLT_AK4PFJet80_v19
1000 263 - -1 - HLT_AK4PFJet120_v18
1000 43 +1 -1 ~1 HLT_IsoTrackHB_v4
1000 828 +2 -2 - MC_PFBTagDeepCSV_v10
1000 156 +1 - - MC_Ele5_WPTight_Gsf_v8
1000 176 +1 - - MC_IsoMu_v15
1000 17 - -1 ~1 HLT_DoubleMediumChargedIsoPFTau35_Trk1_eta2p1_Reg_v12
1000 12 +1 - ~1 HLT_DoubleTightChargedIsoPFTau35_Trk1_eta2p1_Reg_v12
1000 11 +1 - ~1 HLT_DoubleTightChargedIsoPFTau35_Trk1_TightID_eta2p1_Reg_v12
1000 21 - -2 - HLT_DoubleMu3_DCA_PFMET50_PFMHT60_v10
1000 21 +1 - ~1 HLT_QuadPFJet98_83_71_15_DoublePFBTagDeepCSV_1p3_7p7_VBF1_v8
1000 13 +1 - ~1 HLT_QuadPFJet103_88_75_15_DoublePFBTagDeepCSV_1p3_7p7_VBF1_v8
1000 11 +1 - ~1 HLT_QuadPFJet111_90_80_15_DoublePFBTagDeepCSV_1p3_7p7_VBF1_v8
1000 13 +1 - ~1 HLT_QuadPFJet105_88_76_15_DoublePFBTagDeepCSV_1p3_7p7_VBF1_v8The modifications, (channelThreshold to 1000 (actually is "wrong" on CPU i.e., if you wish, in the original menu) and clusterThreshold lower by ONE electron, are well below the systematics due to the pixel detector behavior evolution even in a single run. The fact that the rate of some triggers changes by more than 10% is an issue in itself outside the scope of this technical one.
silviodonato
commented
3 years ago Hi all,
I just want to report that I see very large differences when using 2018 data.
In particular the filter hltVerticesPFFilter is rejecting away all the events when using GPU (0% efficiency). On CPU we get an efficiency of 99% (as expected).
You can reproduce using CMSSW_12_1_0_pre4 and the configuration made by
hltGetConfiguration /dev/CMSSW_12_1_0/GRun/V4 --globaltag auto:run3_hlt --data --customise HLTrigger/Configuration/customizeHLTforPatatrack.customizeHLTforPatatrack,HLTrigger/Configuration/customizeHLTforCMSSW.customiseFor2018Input --input file:/eos/cms/store/data/Run2018D/EphemeralHLTPhysics7/RAW/v1/000/323/790/00000/B543D251-40F1-CB46-A6A1-046CF3D78D6D.root --era Run3 --path HLTriggerFirstPath,MC_PFBTagDeepCSV_v10,HLTriggerFinalPath --unprescale --output minimal --max-events 10(you need git cms-merge-topic 35584 to get hltGetConfiguration working with the new Run-3 database).
You can find all configurations and results in
/afs/cern.ch/work/s/sdonato/public/GPUdiff/CMSSW_12_1_0_pre4/src
process.hltVerticesPFSelector = cms.EDFilter( "PrimaryVertexObjectFilter",
filterParams = cms.PSet(
maxZ = cms.double( 24.0 ),
minNdof = cms.double( 4.0 ),
maxRho = cms.double( 2.0 ),
pvSrc = cms.InputTag( "hltVerticesPF" )
),
src = cms.InputTag( "hltVerticesPF" )
)
process.hltVerticesPFFilter = cms.EDFilter( "VertexSelector",
src = cms.InputTag( "hltVerticesPFSelector" ),
cut = cms.string( "!isFake" ),
filter = cms.bool( True )
)Do you get the same behaviour with Run3 MC ?
VinInn
commented
3 years ago auto:run3_hlt --era Run3 on 2018 data?
silviodonato
commented
3 years ago Do you get the same behaviour with Run3 MC ?
Run3 MC looks ok 100% efficiency in both GPU and CPU
silviodonato
commented
3 years ago auto:run3_hlt --era Run3 on 2018 data?
thanks, indeed using --era Run2 solved the problem! :-)
silviodonato
commented
3 years ago After the fix (--era Run2), running on data (/eos/cms/store/data/Run2018D/EphemeralHLTPhysics7/RAW/v1/000/323/790/00000/)
I still see larger difference especially in MC_PFBTagDeepCSV_v10 (10%-20%) wrt https://github.com/cms-sw/cmssw/issues/35376#issuecomment-932945377.
I used CMSSW_12_1_0_pre4 + #35566 #35638 #35570 #35584
Events Accepted Gained Lost Other Trigger
20673 11 +1 - - HLT_AK8PFJet380_TrimMass30_v11
20673 5 - -1 - HLT_AK8PFJet420_TrimMass30_v11
20673 5 - - ~1 HLT_AK8PFHT750_TrimMass50_v12
20673 1 - - ~1 HLT_AK8PFHT800_TrimMass50_v12
20673 1 - - ~1 HLT_AK8PFHT850_TrimMass50_v11
20673 1 - - ~1 HLT_AK8PFHT900_TrimMass50_v11
20673 1 - - ~1 HLT_Trimuon5_3p5_2_Upsilon_Muon_v5
20673 0 - - ~1 HLT_TrimuonOpen_5_3p5_2_Upsilon_Muon_v3
20673 3 - - ~16 HLT_DoubleEle25_CaloIdL_MW_v4
20673 3 - - ~16 HLT_DoubleEle27_CaloIdL_MW_v4
20673 2 - - ~14 HLT_DoubleEle33_CaloIdL_MW_v17
20673 0 - - ~4 HLT_DoubleEle8_CaloIdM_TrackIdM_Mass8_DZ_PFHT350_v20
20673 1 - - ~4 HLT_DoubleEle8_CaloIdM_TrackIdM_Mass8_PFHT350_v20
20673 3 - - ~12 HLT_Ele27_Ele37_CaloIdL_MW_v4
20673 0 - - ~1 HLT_Mu27_Ele37_CaloIdL_MW_v5
20673 0 - - ~1 HLT_DoubleMu3_TkMu_DsTau3Mu_v4
20673 13 - - ~1 HLT_DoublePhoton33_CaloIdL_v6
20673 2 - - ~7 HLT_DiEle27_WPTightCaloOnly_L1DoubleEG_v4
20673 60 +2 -1 ~18 HLT_Ele27_WPTight_Gsf_v16
20673 59 +2 -1 ~17 HLT_Ele28_WPTight_Gsf_v1
20673 50 +2 -1 ~16 HLT_Ele30_WPTight_Gsf_v1
20673 46 +1 -1 ~14 HLT_Ele32_WPTight_Gsf_v15
20673 37 +1 -1 ~10 HLT_Ele35_WPTight_Gsf_v9
20673 32 +1 -1 ~9 HLT_Ele35_WPTight_Gsf_L1EGMT_v5
20673 31 +1 -1 ~9 HLT_Ele38_WPTight_Gsf_v9
20673 21 - -1 ~5 HLT_Ele40_WPTight_Gsf_v9
20673 46 +1 -1 ~14 HLT_Ele32_WPTight_Gsf_L1DoubleEG_v9
20673 1 - - ~3 HLT_Ele24_eta2p1_WPTight_Gsf_LooseChargedIsoPFTau30_eta2p1_CrossL1_v13
20673 0 - - ~3 HLT_Ele24_eta2p1_WPTight_Gsf_LooseChargedIsoPFTauHPS30_eta2p1_CrossL1_v1
20673 0 - - ~3 HLT_Ele24_eta2p1_WPTight_Gsf_MediumChargedIsoPFTauHPS30_eta2p1_CrossL1_v1
20673 0 - - ~3 HLT_Ele24_eta2p1_WPTight_Gsf_TightChargedIsoPFTauHPS30_eta2p1_CrossL1_v1
20673 0 - - ~3 HLT_Ele24_eta2p1_WPTight_Gsf_LooseChargedIsoPFTauHPS30_eta2p1_TightID_CrossL1_v1
20673 0 - - ~3 HLT_Ele24_eta2p1_WPTight_Gsf_MediumChargedIsoPFTauHPS30_eta2p1_TightID_CrossL1_v1
20673 0 - - ~3 HLT_Ele24_eta2p1_WPTight_Gsf_TightChargedIsoPFTauHPS30_eta2p1_TightID_CrossL1_v1
20673 463 +1 - - HLT_ZeroBias_Beamspot_v4
20673 1 - - ~3 HLT_IsoMu20_eta2p1_LooseChargedIsoPFTau27_eta2p1_CrossL1_v12
20673 0 - - ~3 HLT_IsoMu20_eta2p1_LooseChargedIsoPFTauHPS27_eta2p1_CrossL1_v4
20673 0 - - ~3 HLT_IsoMu20_eta2p1_MediumChargedIsoPFTauHPS27_eta2p1_CrossL1_v1
20673 0 - - ~3 HLT_IsoMu20_eta2p1_TightChargedIsoPFTauHPS27_eta2p1_CrossL1_v1
20673 0 - - ~3 HLT_IsoMu20_eta2p1_LooseChargedIsoPFTauHPS27_eta2p1_TightID_CrossL1_v1
20673 0 - - ~3 HLT_IsoMu20_eta2p1_MediumChargedIsoPFTauHPS27_eta2p1_TightID_CrossL1_v1
20673 0 - - ~3 HLT_IsoMu20_eta2p1_TightChargedIsoPFTauHPS27_eta2p1_TightID_CrossL1_v1
20673 0 - - ~3 HLT_IsoMu27_LooseChargedIsoPFTau20_Trk1_eta2p1_SingleL1_v5
20673 0 - - ~3 HLT_IsoMu27_LooseChargedIsoPFTauHPS20_Trk1_eta2p1_SingleL1_v1
20673 0 - - ~3 HLT_IsoMu27_MediumChargedIsoPFTauHPS20_Trk1_eta2p1_SingleL1_v1
20673 0 - - ~3 HLT_IsoMu27_TightChargedIsoPFTauHPS20_Trk1_eta2p1_SingleL1_v1
20673 74 +2 - ~1 HLT_IsoMu24_v13
20673 65 +2 - - HLT_IsoMu24_eta2p1_v15
20673 64 +2 - ~1 HLT_IsoMu27_v16
20673 48 +1 - - HLT_IsoMu30_v4
20673 3 - - ~1 HLT_Mu25_TkMu0_Onia_v8
20673 157 +1 - - HLT_Mu27_v13
20673 2 - - ~2 HLT_TkMu100_v2
20673 359 +7 -10 - HLT_DiPFJet15_NoCaloMatched_v16
20673 33 +5 -5 - HLT_DiPFJet25_NoCaloMatched_v16
20673 189 +16 -15 - HLT_DiPFJet15_FBEta3_NoCaloMatched_v17
20673 7 +2 -2 - HLT_DiPFJet25_FBEta3_NoCaloMatched_v17
20673 43 - -1 - HLT_DiPFJetAve260_v14
20673 21 +1 - - HLT_DiPFJetAve320_v14
20673 270 +14 -17 - HLT_DiPFJetAve15_HFJEC_v17
20673 20 +3 -1 - HLT_DiPFJetAve25_HFJEC_v17
20673 149 +14 -9 - HLT_AK8PFJet15_v3
20673 417 +1 -4 - HLT_AK8PFJet25_v3
20673 149 +14 -9 - HLT_AK8PFJet40_v16
20673 4 +2 - - HLT_AK8PFJet60_v15
20673 96 +3 -2 - HLT_AK8PFJet260_v16
20673 6 +1 - - HLT_AK8PFJet450_v16
20673 405 +4 -5 - HLT_PFJet15_v3
20673 122 +3 -10 - HLT_PFJet25_v3
20673 11 - -1 - HLT_PFJet40_v21
20673 77 +3 -2 - HLT_PFJet260_v20
20673 4 +1 - - HLT_PFJet450_v21
20673 385 +2 -6 - HLT_PFJetFwd15_v3
20673 80 +4 -8 - HLT_PFJetFwd25_v3
20673 4 - -1 - HLT_PFJetFwd40_v19
20673 409 +4 -7 - HLT_AK8PFJetFwd25_v3
20673 134 +10 -8 - HLT_AK8PFJetFwd40_v15
20673 84 +3 -4 - HLT_PFHT510_v17
20673 37 +3 - - HLT_PFHT590_v17
20673 16 +1 - - HLT_PFHT780_v17
20673 3 +1 - ~2 HLT_PFHT500_PFMET100_PFMHT100_IDTight_v12
20673 12 +1 - ~3 HLT_PFMET110_PFMHT110_IDTight_v20
20673 8 - -3 - HLT_PFMET120_PFMHT120_IDTight_v20
20673 5 - -1 ~1 HLT_PFMET130_PFMHT130_IDTight_v20
20673 4 - -1 ~2 HLT_PFMET140_PFMHT140_IDTight_v20
20673 4 - - ~1 HLT_PFMET100_PFMHT100_IDTight_CaloBTagDeepCSV_3p1_v8
20673 5 - -1 - HLT_PFMET120_PFMHT120_IDTight_PFHT60_v9
20673 5 - -1 - HLT_PFMETNoMu120_PFMHTNoMu120_IDTight_PFHT60_v9
20673 7 - -1 - HLT_PFMETTypeOne120_PFMHT120_IDTight_PFHT60_v9
20673 15 +1 - ~3 HLT_PFMETTypeOne110_PFMHT110_IDTight_v12
20673 10 - -1 - HLT_PFMETTypeOne120_PFMHT120_IDTight_v12
20673 7 - -1 ~1 HLT_PFMETTypeOne130_PFMHT130_IDTight_v12
20673 5 - -1 ~2 HLT_PFMETTypeOne140_PFMHT140_IDTight_v11
20673 12 +1 - ~3 HLT_PFMETNoMu110_PFMHTNoMu110_IDTight_v20
20673 8 - -3 - HLT_PFMETNoMu120_PFMHTNoMu120_IDTight_v20
20673 5 - -1 ~1 HLT_PFMETNoMu130_PFMHTNoMu130_IDTight_v19
20673 4 - -1 ~2 HLT_PFMETNoMu140_PFMHTNoMu140_IDTight_v19
20673 4 - -1 - HLT_MonoCentralPFJet80_PFMETNoMu120_PFMHTNoMu120_IDTight_v20
20673 3 - -1 ~1 HLT_MonoCentralPFJet80_PFMETNoMu130_PFMHTNoMu130_IDTight_v19
20673 2 - -1 ~1 HLT_MonoCentralPFJet80_PFMETNoMu140_PFMHTNoMu140_IDTight_v19
20673 3 - - ~1 HLT_Mu8_TrkIsoVVL_Ele23_CaloIdL_TrackIdL_IsoVL_DZ_v13
20673 2 - - ~1 HLT_Mu8_TrkIsoVVL_Ele23_CaloIdL_TrackIdL_IsoVL_DZ_PFDiJet30_v1
20673 2 - - ~1 HLT_Mu8_TrkIsoVVL_Ele23_CaloIdL_TrackIdL_IsoVL_DZ_CaloDiJet30_v1
20673 1 - - ~1 HLT_Mu8_TrkIsoVVL_Ele23_CaloIdL_TrackIdL_IsoVL_DZ_PFDiJet30_PFBtagDeepCSV_1p5_v1
20673 1 - - ~1 HLT_Mu8_TrkIsoVVL_Ele23_CaloIdL_TrackIdL_IsoVL_DZ_CaloDiJet30_CaloBtagDeepCSV_1p5_v1
20673 4 +1 - - HLT_Mu8_TrkIsoVVL_Ele23_CaloIdL_TrackIdL_IsoVL_v11
20673 1 - - ~1 HLT_Ele23_Ele12_CaloIdL_TrackIdL_IsoVL_DZ_v19
20673 2 - - ~1 HLT_Ele23_Ele12_CaloIdL_TrackIdL_IsoVL_v19
20673 0 - - ~1 HLT_Mu23_TrkIsoVVL_Ele12_CaloIdL_TrackIdL_IsoVL_DZ_v15
20673 3 - - ~1 HLT_Mu23_TrkIsoVVL_Ele12_CaloIdL_TrackIdL_IsoVL_v7
20673 2 - - ~2 HLT_Mu12_DoublePhoton20_v5
20673 3 - - ~1 HLT_TriplePhoton_20_20_20_CaloIdLV2_v3
20673 2 - - ~1 HLT_TriplePhoton_20_20_20_CaloIdLV2_R9IdVL_v3
20673 5 +1 - ~2 HLT_Photon35_TwoProngs35_v1
20673 28 +1 - ~3 HLT_IsoMu24_TwoProngs35_v1
20673 30 +1 -1 - HLT_Dimuon0_Upsilon_L1_4p5er2p0M_v7
20673 43 +1 -1 - HLT_Dimuon0_Upsilon_NoVertexing_v7
20673 30 +1 -1 - HLT_Dimuon0_Upsilon_L1_5M_v8
20673 133 - -1 - HLT_Dimuon0_LowMass_L1_0er1p5R_v7
20673 206 - -1 - HLT_Dimuon0_LowMass_L1_4R_v7
20673 8 - - ~5 HLT_Tau3Mu_Mu7_Mu1_TkMu1_Tau15_v4
20673 8 - - ~5 HLT_Tau3Mu_Mu7_Mu1_TkMu1_Tau15_Charge1_v4
20673 3 - - ~5 HLT_Tau3Mu_Mu7_Mu1_TkMu1_IsoTau15_v4
20673 3 - - ~5 HLT_Tau3Mu_Mu7_Mu1_TkMu1_IsoTau15_Charge1_v4
20673 9 - -1 ~1 HLT_DiJet110_35_Mjj650_PFMET110_v9
20673 5 +2 -1 ~2 HLT_DiJet110_35_Mjj650_PFMET120_v9
20673 3 - - ~1 HLT_DiJet110_35_Mjj650_PFMET130_v9
20673 1 - - ~2 HLT_TripleJet110_35_35_Mjj650_PFMET110_v9
20673 1 - - ~5 HLT_TripleJet110_35_35_Mjj650_PFMET120_v9
20673 1 - - ~1 HLT_TripleJet110_35_35_Mjj650_PFMET130_v9
20673 10 +1 -1 ~11 HLT_Ele30_eta2p1_WPTight_Gsf_CentralPFJet35_EleCleaned_v13
20673 2 - - ~13 HLT_Ele28_eta2p1_WPTight_Gsf_HT150_v13
20673 3 - -1 ~17 HLT_Ele28_HighEta_SC20_Mass55_v13
20673 0 - - ~1 HLT_Ele15_IsoVVVL_PFHT450_CaloBTagDeepCSV_4p5_v8
20673 2 - - ~1 HLT_Mu15_IsoVVVL_PFHT450_CaloBTagDeepCSV_4p5_v8
20673 6 +1 - - HLT_Mu3er1p5_PFJet100er2p5_PFMET70_PFMHT70_IDTight_v2
20673 8 +2 -1 ~2 HLT_Mu3er1p5_PFJet100er2p5_PFMETNoMu70_PFMHTNoMu70_IDTight_v2
20673 1 - - ~4 HLT_TrkMu12_DoubleTrkMu5NoFiltersNoVtx_v6
20673 1 - - ~4 HLT_TrkMu16_DoubleTrkMu6NoFiltersNoVtx_v12
20673 1 - - ~1 HLT_TrkMu17_DoubleTrkMu8NoFiltersNoVtx_v13
20673 0 +1 - ~1 HLT_Ele8_CaloIdL_TrackIdL_IsoVL_PFJet30_v16
20673 0 - - ~1 HLT_Ele12_CaloIdL_TrackIdL_IsoVL_PFJet30_v18
20673 0 - - ~1 HLT_Ele15_CaloIdL_TrackIdL_IsoVL_PFJet30_v3
20673 1 - - ~1 HLT_Ele8_CaloIdM_TrackIdM_PFJet30_v18
20673 5 - -1 ~2 HLT_Ele50_CaloIdVT_GsfTrkIdT_PFJet165_v18
20673 0 +1 - ~3 HLT_PFHT330PT30_QuadPFJet_75_60_45_40_TriplePFBTagDeepCSV_4p5_v3
20673 95 +6 -2 ~8 HLT_PFHT330PT30_QuadPFJet_75_60_45_40_v9
20673 2 +1 - ~3 HLT_PFHT400_SixPFJet32_DoublePFBTagDeepCSV_2p94_v8
20673 9 - - ~3 HLT_PFHT400_SixPFJet32_v8
20673 5 - - ~4 HLT_PFHT450_SixPFJet36_PFBTagDeepCSV_1p59_v7
20673 6 - - ~4 HLT_PFHT450_SixPFJet36_v7
20673 21 +2 -2 ~5 HLT_PFHT400_FivePFJet_100_100_60_30_30_v8
20673 4 - -1 ~6 HLT_PFHT400_FivePFJet_100_100_60_30_30_DoublePFBTagDeepCSV_4p5_v8
20673 3 - -1 ~3 HLT_PFHT400_FivePFJet_120_120_60_30_30_DoublePFBTagDeepCSV_4p5_v8
20673 360 +13 -3 - HLT_PFHT350_v19
20673 462 +12 -8 - HLT_PFHT350MinPFJet15_v9
20673 54 +1 - - HLT_Photon60_R9Id90_CaloIdL_IsoL_v5
20673 41 +1 -1 - HLT_Photon60_R9Id90_CaloIdL_IsoL_DisplacedIdL_v5
20673 6 - -1 ~2 HLT_Photon60_R9Id90_CaloIdL_IsoL_DisplacedIdL_PFHT350MinPFJet15_v11
20673 44 - - ~2 HLT_DiSC30_18_EIso_AND_HE_Mass70_v13
20673 47 +5 -5 - HLT_AK4PFJet30_v19
20673 16850 +241 -152 - MC_AK4PFJets_v17
20673 2461 +315 -245 ~173 MC_PFBTagDeepCSV_v10
20673 18477 +152 -125 - MC_PFHT_v16
20673 13596 - - ~1 MC_AK4CaloJetsFromPV_v8
20673 13848 +1 - - MC_CaloBTagDeepCSV_v8
20673 19217 +179 -105 - MC_AK8PFJets_v17
20673 20370 +7 -3 - MC_AK8TrimPFJets_v17
20673 20415 +1 - - MC_AK8PFHT_v16
20673 1 - - ~2 MC_DoubleEle5_CaloIdL_MW_v15
20673 49 - -3 ~1 MC_IsoMu_v15
20673 1890 +1 - - AlCa_EcalPi0EBonly_v13
20673 0 - - ~5 HLT_MediumChargedIsoPFTau180HighPtRelaxedIso_Trk50_eta2p1_1pr_v11
20673 2 - -1 ~5 HLT_MediumChargedIsoPFTau180HighPtRelaxedIso_Trk50_eta2p1_v12
20673 2 - -1 ~4 HLT_MediumChargedIsoPFTau200HighPtRelaxedIso_Trk50_eta2p1_v12
20673 0 - - ~5 HLT_MediumChargedIsoPFTau220HighPtRelaxedIso_Trk50_eta2p1_v12
20673 0 - - ~1 HLT_Ele16_Ele12_Ele8_CaloIdL_TrackIdL_v9
20673 0 - - ~2 HLT_Rsq0p35_v15
20673 0 - - ~2 HLT_Rsq0p40_v15
20673 4 +1 - ~4 HLT_RsqMR300_Rsq0p09_MR200_v15
20673 2 - - ~4 HLT_RsqMR320_Rsq0p09_MR200_v15
20673 1 - - ~1 HLT_RsqMR300_Rsq0p09_MR200_4jet_v15
20673 1 - - ~1 HLT_RsqMR320_Rsq0p09_MR200_4jet_v15
20673 0 - - ~3 HLT_IsoMu27_MET90_v3
20673 16 +2 -2 ~18 HLT_DoubleMediumChargedIsoPFTau35_Trk1_eta2p1_Reg_v12
20673 11 +2 -1 ~18 HLT_DoubleMediumChargedIsoPFTau35_Trk1_TightID_eta2p1_Reg_v12
20673 12 +2 -1 ~18 HLT_DoubleTightChargedIsoPFTau35_Trk1_eta2p1_Reg_v12
20673 9 +2 -1 ~18 HLT_DoubleTightChargedIsoPFTau35_Trk1_TightID_eta2p1_Reg_v12
20673 13 - -1 ~20 HLT_DoubleTightChargedIsoPFTauHPS35_Trk1_eta2p1_Reg_v1
20673 11 +1 -1 ~20 HLT_DoubleMediumChargedIsoPFTauHPS35_Trk1_TightID_eta2p1_Reg_v1
20673 16 - -1 ~20 HLT_DoubleMediumChargedIsoPFTauHPS35_Trk1_eta2p1_Reg_v4
20673 10 - -1 ~20 HLT_DoubleTightChargedIsoPFTauHPS35_Trk1_TightID_eta2p1_Reg_v1
20673 8 - - ~13 HLT_DoubleMediumChargedIsoPFTauHPS40_Trk1_eta2p1_Reg_v1
20673 7 - - ~13 HLT_DoubleTightChargedIsoPFTauHPS40_Trk1_eta2p1_Reg_v1
20673 5 - - ~13 HLT_DoubleMediumChargedIsoPFTauHPS40_Trk1_TightID_eta2p1_Reg_v1
20673 5 - - ~13 HLT_DoubleTightChargedIsoPFTauHPS40_Trk1_TightID_eta2p1_Reg_v1
20673 1 - - ~3 HLT_VBF_DoubleLooseChargedIsoPFTau20_Trk1_eta2p1_v3
20673 0 - - ~4 HLT_VBF_DoubleLooseChargedIsoPFTauHPS20_Trk1_eta2p1_v1
20673 0 - - ~4 HLT_VBF_DoubleMediumChargedIsoPFTauHPS20_Trk1_eta2p1_v1
20673 0 - - ~4 HLT_VBF_DoubleTightChargedIsoPFTauHPS20_Trk1_eta2p1_v1
20673 15 - -1 ~4 HLT_PFMET100_PFMHT100_IDTight_PFHT60_v9
20673 15 - -1 ~4 HLT_PFMETNoMu100_PFMHTNoMu100_IDTight_PFHT60_v9
20673 18 - -2 ~3 HLT_PFMETTypeOne100_PFMHT100_IDTight_PFHT60_v9
20673 6 +1 - - HLT_DoubleMu2_Jpsi_DoubleTrk1_Phi1p05_v6
20673 2 - - ~1 HLT_DoubleMu2_Jpsi_DoubleTkMu0_Phi_v5
20673 2 +2 - - HLT_QuadPFJet98_83_71_15_DoublePFBTagDeepCSV_1p3_7p7_VBF1_v8
20673 4 +3 - ~3 HLT_QuadPFJet103_88_75_15_DoublePFBTagDeepCSV_1p3_7p7_VBF1_v8
20673 3 +2 - ~2 HLT_QuadPFJet111_90_80_15_DoublePFBTagDeepCSV_1p3_7p7_VBF1_v8
20673 0 +1 - ~3 HLT_QuadPFJet98_83_71_15_PFBTagDeepCSV_1p3_VBF2_v8
20673 0 - - ~7 HLT_QuadPFJet103_88_75_15_PFBTagDeepCSV_1p3_VBF2_v8
20673 0 - - ~6 HLT_QuadPFJet105_88_76_15_PFBTagDeepCSV_1p3_VBF2_v8
20673 0 - - ~4 HLT_QuadPFJet111_90_80_15_PFBTagDeepCSV_1p3_VBF2_v8
20673 124 +4 -3 - HLT_QuadPFJet98_83_71_15_v5
20673 158 +7 -5 ~8 HLT_QuadPFJet103_88_75_15_v5
20673 148 +8 -4 ~4 HLT_QuadPFJet105_88_76_15_v5
20673 120 +6 -3 ~5 HLT_QuadPFJet111_90_80_15_v5
20673 2 +1 - ~2 HLT_AK8PFJet330_TrimMass30_PFAK8BTagDeepCSV_p17_v2
20673 1 - - ~2 HLT_AK8PFJet330_TrimMass30_PFAK8BTagDeepCSV_p1_v2
20673 9 +1 - ~2 HLT_AK8PFJet330_TrimMass30_PFAK8BoostedDoubleB_p02_v3
20673 16 - - ~2 HLT_AK8PFJet330_TrimMass30_PFAK8BoostedDoubleB_np2_v2
20673 16 - - ~2 HLT_AK8PFJet330_TrimMass30_PFAK8BoostedDoubleB_np4_v2
20673 207 +1 - - HLT_Mu9_IP5_ToCSCS_v1
20673 207 +1 - - HLT_Mu9_IP5_part0_v2
20673 207 +1 - - HLT_Mu9_IP5_part1_v2
20673 207 +1 - - HLT_Mu9_IP5_part2_v2
20673 207 +1 - - HLT_Mu9_IP5_part3_v2
20673 207 +1 - - HLT_Mu9_IP5_part4_v2
20673 213 +1 - - HLT_Mu8_IP5_ToCSCS_v1
20673 213 +1 - - HLT_Mu8_IP5_part0_v2
20673 213 +1 - - HLT_Mu8_IP5_part1_v2
20673 213 +1 - - HLT_Mu8_IP5_part2_v2
20673 213 +1 - - HLT_Mu8_IP5_part3_v2
20673 213 +1 - - HLT_Mu8_IP5_part4_v2
20673 315 +2 - - HLT_Mu8_IP3_ToCSCS_v1
20673 315 +2 - - HLT_Mu8_IP3_part0_v3
20673 315 +2 - - HLT_Mu8_IP3_part1_v3
20673 315 +2 - - HLT_Mu8_IP3_part2_v3
20673 315 +2 - - HLT_Mu8_IP3_part3_v3
20673 315 +2 - - HLT_Mu8_IP3_part4_v3
20673 811 +1 - - HLT_Mu9_IP0_part0_v2
20673 306 +2 - - HLT_Mu9_IP3_part0_v2
20673 4 +2 - ~3 HLT_QuadPFJet105_88_76_15_DoublePFBTagDeepCSV_1p3_7p7_VBF1_v8
20673 20673 - -20673 - Status_OnCPU
20673 0 +20673 - - Status_OnGPU
We expect that #35915 #35902 #36088 reduce these differences. I will post an update with the results obtained by Ganesh
silviodonato
commented
2 years ago From the latest Ganesh's results https://docs.google.com/spreadsheets/d/1MOAzxYSRKRjqDvkQKUd6a9SAT0mNO--_3Zc3w9JCtQ0/edit#gid=2096617240 obtained in CMSSW_12_2_0_pre2 + #36088 we can say:
I confirm to see such GPU/CPU differences in CMSSW_12_2_0_pre3, you can use
hltGetConfiguration /dev/CMSSW_12_2_0/GRun --globaltag auto:run2_data --data --customise HLTrigger/Configuration/customizeHLTforCMSSW.customiseFor2018Input,HLTrigger/Configuration/customizeHLTforPatatrack.customiseCommon,HLTrigger/Configuration/customizeHLTforPatatrack.customisePixelLocalReconstruction --input file:/eos/cms/store/data/Run2018D/EphemeralHLTPhysics7/RAW/v1/000/323/790/00000/B543D251-40F1-CB46-A6A1-046CF3D78D6D.root --era Run2_2018 --output minimal --max-events -1The interesting part is that I tried to use some homemade paths for debugging (/users/sdonato/GPUtest/PixelBtag/V23) and I got:
100000 6153 - -1 - HLT_PixelGlobalBTagDeepCSV0p17_v1
100000 978 +1 - - HLT_PixelGlobalBTagDeepCSV0p30_v1
100000 10243 +1 - - HLT_PixelGlobalBTagDeepCSV0p1_v1
100000 12228 +574 -876 ~497 MC_PFBTagDeepCSV_v10
100000 0 +100000 - - Status_OnCPU
100000 100000 - -100000 - Status_OnGPUwhere HLT_PixelGlobalBTagDeepCSV0pXX_v1 cuts on b-tagging evaluated using directly global pixel tracks.
MC_PFBTagDeepCSV_v10 uses Particle Flow and "full" tracking (pixel+strips)
This suggests that the pixel tracks are ok, but there is something weird happens in Particle Flow or in the full tracking (eg. something like #35923).
For the record, running two times with GPU, I got (GPU vs GPU):
Events Accepted Gained Lost Other Trigger
100000 12228 +281 -283 ~183 MC_PFBTagDeepCSV_v10a bug like #35923 might explain why we are getting such a large differences even comparing GPU vs GPU.
fwyzard
commented
2 years ago hi @silviodonato
The interesting part is that I tried to use some homemade paths for debugging (
/users/sdonato/GPUtest/PixelBtag/V23) and I got:100000 6153 - -1 - HLT_PixelGlobalBTagDeepCSV0p17_v1 100000 978 +1 - - HLT_PixelGlobalBTagDeepCSV0p30_v1 100000 10243 +1 - - HLT_PixelGlobalBTagDeepCSV0p1_v1 100000 12228 +574 -876 ~497 MC_PFBTagDeepCSV_v10 100000 0 +100000 - - Status_OnCPU 100000 100000 - -100000 - Status_OnGPUwhere
HLT_PixelGlobalBTagDeepCSV0pXX_v1cuts on b-tagging evaluated using directly global pixel tracks.
in this test, did you customise only the pixel local reco or also the pixel tracks ?
silviodonato
commented
2 years ago in this test, did you customise only the pixel local reco or also the pixel tracks ?
I used only pixel local reco
#User-defined customization functions
from HLTrigger.Configuration.customizeHLTforCMSSW import customiseFor2018Input
process = customiseFor2018Input(process)
from HLTrigger.Configuration.customizeHLTforPatatrack import customiseCommon
process = customiseCommon(process)
from HLTrigger.Configuration.customizeHLTforPatatrack import customisePixelLocalReconstruction
process = customisePixelLocalReconstruction(process)I copied the python file in /eos/home-s/sdonato/public/CMSSW_12_2_0_pre3/src/hlt_GPU.py
silviodonato
commented
2 years ago I investigated a bit more about the origin of the differences. I considered the case CPU vs GPU when only the pixel local reco is customized (see message above). First of all, I noticed that the differences are 'fluctuating', ie. if you select an event with a large difference and you ran again on the same event you might get a small difference. So I ran with "keep *" and selected a posteriori an event with a large difference in pT in the leading PF jet (116 GeV vs 107 GeV) and I looked at the input objects. The pt of the pixel tracks - sorted by pT - look identical:
difference, CPU, GPU
0.0 30.18839617398052 30.18839617398052
0.0 21.56895506988423 21.56895506988423
-0.0004511632673462884 19.690113529090368 19.690564692357714
0.0 10.324574928061926 10.324574928061926
0.0 7.387472328452399 7.387472328452399
0.0 5.349614808496968 5.349614808496968
0.0 4.5758975428997335 4.5758975428997335
0.0 4.462715082227141 4.462715082227141
0.0 4.322723376184752 4.322723376184752
0.0 3.958396060412112 3.958396060412112The differences appear the the Kalman filter.
In the first step, pT of hltIter0PFlowCtfWithMaterialTracks
difference, CPU, GPU
-0.0001917870921701592 23.91527577096683 23.915467558059
4.570622465993772 17.32298355299593 12.752361087002157
5.953023987817948 12.719457549278475 6.766433561460527
2.3039737408562226 6.766433087351903 4.46245934649568
0.07099019766110803 4.46245934649568 4.391469148834572
1.1454919242038355 4.391469148834572 3.2459772246307366
0.06380459588703191 3.2459772246307366 3.1821726287437047
0.19308527028399158 3.1821726287437047 2.989087358459713
0.12274472954427695 2.989087358459713 2.866342628915436
0.3567856013825934 2.866342628915436 2.5095570275328427pT of hltIter1Merged
difference, CPU, GPU
-32.84278624068943 23.91527577096683 56.75806201165626
-6.505120658972366 17.410346899086633 23.915467558059
-0.005744998247548594 12.746616088754608 12.752361087002157
0.0 11.083732508826449 11.083732508826449
-4.741086243598147e-07 6.766433087351903 6.766433561460527
0.0 4.46245934649568 4.46245934649568
0.0 4.391469148834572 4.391469148834572
0.0 3.2459772246307366 3.2459772246307366
0.0 3.1821726287437047 3.1821726287437047
0.0 2.989087358459713 2.989087358459713pT of hltMergedTracks
difference, CPU, GPU
0.11290763047638563 56.87096964213264 56.75806201165626
-0.0001917870921701592 23.91527577096683 23.915467558059
4.657985812084476 17.410346899086633 12.752361087002157
1.6628835799281596 12.746616088754608 11.083732508826449
0.25730624949656367 11.083732508826449 10.826426259329885
1.2104215518827193 10.826411053077246 9.615989501194527
2.6735175973749046 9.615989501194527 6.942471903819622
0.17603786182021075 6.942471423280738 6.766433561460527
2.3039737408562226 6.766433087351903 4.46245934649568
0.07099019766110803 4.46245934649568 4.391469148834572On 21 Dec, 2021, at 7:14 PM, Silvio Donato @.***> wrote:
I investigated a bit more about the origin of the differences. I considered the case CPU vs GPU when only the pixel local reco is customized (see message above). First of all, I noticed that the differences are 'fluctuating', ie. if you select an event with a large difference and you ran again on the same event you might get a small difference. So I ran with "keep *" and selected a posteriori an event with a large difference in pT in the leading PF jet (116 GeV vs 107 GeV) and I looked at the input objects. The pt of the pixel tracks - sorted by pT - look identical:
difference, CPU, GPU 0.0 30.18839617398052 30.18839617398052 0.0 21.56895506988423 21.56895506988423 -0.0004511632673462884 19.690113529090368 19.690564692357714 0.0 10.324574928061926 10.324574928061926 0.0 7.387472328452399 7.387472328452399 0.0 5.349614808496968 5.349614808496968 0.0 4.5758975428997335 4.5758975428997335 0.0 4.462715082227141 4.462715082227141 0.0 4.322723376184752 4.322723376184752 0.0 3.958396060412112 3.958396060412112
The differences appear the the Kalman filter. (not differences: mostly "missing" tracks)
In the first step, pT of hltIter0PFlowCtfWithMaterialTracks
difference, CPU, GPU -0.0001917870921701592 23.91527577096683 23.915467558059 4.570622465993772 17.32298355299593 12.752361087002157 here 17.3 is simply missing 5.953023987817948 12.719457549278475 6.766433561460527 2.3039737408562226 6.766433087351903 4.46245934649568 0.07099019766110803 4.46245934649568 4.391469148834572 1.1454919242038355 4.391469148834572 3.2459772246307366 0.06380459588703191 3.2459772246307366 3.1821726287437047 0.19308527028399158 3.1821726287437047 2.989087358459713 0.12274472954427695 2.989087358459713 2.866342628915436 0.3567856013825934 2.866342628915436 2.5095570275328427
pT of hltIter1Merged
difference, CPU, GPU -32.84278624068943 23.91527577096683 56.75806201165626 -6.505120658972366 17.410346899086633 23.915467558059 56 is "new" and 17 is missing (agaain) -0.005744998247548594 12.746616088754608 12.752361087002157 0.0 11.083732508826449 11.083732508826449 -4.741086243598147e-07 6.766433087351903 6.766433561460527 0.0 4.46245934649568 4.46245934649568 0.0 4.391469148834572 4.391469148834572 0.0 3.2459772246307366 3.2459772246307366 0.0 3.1821726287437047 3.1821726287437047 0.0 2.989087358459713 2.989087358459713
pT of hltMergedTracks
difference, CPU, GPU 0.11290763047638563 56.87096964213264 56.75806201165626 where 56.87 (the left one) come from? -0.0001917870921701592 23.91527577096683 23.915467558059 4.657985812084476 17.410346899086633 12.752361087002157 17 missing again (where?) 1.6628835799281596 12.746616088754608 11.083732508826449 0.25730624949656367 11.083732508826449 10.826426259329885 1.2104215518827193 10.826411053077246 9.615989501194527 2.6735175973749046 9.615989501194527 6.942471903819622 0.17603786182021075 6.942471423280738 6.766433561460527 2.3039737408562226 6.766433087351903 4.46245934649568 0.07099019766110803 4.46245934649568 4.391469148834572
I suggest: sort seeds by pt (or whatever) to stabilize CKF:
see https://cmssdt.cern.ch/lxr/source/RecoTracker/CkfPattern/src/CkfTrackCandidateMakerBase.cc#0231
enough to recompile with -DVI_SORTSEED
silviodonato
commented
2 years ago From the latest Ganesh's results https://docs.google.com/spreadsheets/d/1MOAzxYSRKRjqDvkQKUd6a9SAT0mNO--_3Zc3w9JCtQ0/edit#gid=2096617240 obtained in CMSSW_12_2_0_pre2 + #36088 we can say:
- everything is ok up to ECAL+HCAL (as expected)
- there are differences in the pixel local reco of data 2018 (~10x larger than the GPU-to-GPU differences)
- there are no additional differences when offloading Pixel Tracking on GPU (as expected)
- the GPU-to-GPU differences come only from the pixel local reconstruction (as expected)
I confirm to see such GPU/CPU differences in
CMSSW_12_2_0_pre3, you can usehltGetConfiguration /dev/CMSSW_12_2_0/GRun --globaltag auto:run2_data --data --customise HLTrigger/Configuration/customizeHLTforCMSSW.customiseFor2018Input,HLTrigger/Configuration/customizeHLTforPatatrack.customiseCommon,HLTrigger/Configuration/customizeHLTforPatatrack.customisePixelLocalReconstruction --input file:/eos/cms/store/data/Run2018D/EphemeralHLTPhysics7/RAW/v1/000/323/790/00000/B543D251-40F1-CB46-A6A1-046CF3D78D6D.root --era Run2_2018 --output minimal --max-events -1The interesting part is that I tried to use some homemade paths for debugging (
/users/sdonato/GPUtest/PixelBtag/V23) and I got:100000 6153 - -1 - HLT_PixelGlobalBTagDeepCSV0p17_v1 100000 978 +1 - - HLT_PixelGlobalBTagDeepCSV0p30_v1 100000 10243 +1 - - HLT_PixelGlobalBTagDeepCSV0p1_v1 100000 12228 +574 -876 ~497 MC_PFBTagDeepCSV_v10 100000 0 +100000 - - Status_OnCPU 100000 100000 - -100000 - Status_OnGPUwhere
HLT_PixelGlobalBTagDeepCSV0pXX_v1cuts on b-tagging evaluated using directly global pixel tracks.MC_PFBTagDeepCSV_v10uses Particle Flow and "full" tracking (pixel+strips) This suggests that the pixel tracks are ok, but there is something weird happens in Particle Flow or in the full tracking (eg. something like #35923). For the record, running two times with GPU, I got (GPU vs GPU):Events Accepted Gained Lost Other Trigger 100000 12228 +281 -283 ~183 MC_PFBTagDeepCSV_v10a bug like #35923 might explain why we are getting such a large differences even comparing GPU vs GPU.
Trying to reproduce this in more recent releases, I realized that there are no differences in HLT_PixelGlobalBTagDeepCSV*_v1 using legacy pixel tracking.
In CMSSW_12_2_0_pre3
using
hltGetConfiguration /users/sdonato/GPUtest/PixelBtag/V23 --globaltag auto:run2_data --data --input file:/eos/cms/store/data/Run2018D/EphemeralHLTPhysics7/RAW/v1/000/323/790/00000/B543D251-40F1-CB46-A6A1-046CF3D78D6D.root --era Run2_2018 --output minimal --max-events -1 --customise HLTrigger/Configuration/customizeHLTforCMSSW.customiseFor2018Input,HLTrigger/Configuration/customizeHLTforPatatrack.customiseCommon,HLTrigger/Configuration/customizeHLTforPatatrack.customisePixelLocalReconstruction(ie. no customisePixelTrackReconstruction)
Events Accepted Gained Lost Other Trigger
10000 1171 +39 -19 ~9 MC_PFBTagDeepCSV_v10
(no diff HLT_PixelGlobalBTagDeepCSV*_v1)Using
hltGetConfiguration /users/sdonato/GPUtest/PixelBtag/V23 --globaltag auto:run2_data --data --input file:/eos/cms/store/data/Run2018D/EphemeralHLTPhysics7/RAW/v1/000/323/790/00000/B543D251-40F1-CB46-A6A1-046CF3D78D6D.root --era Run2_2018 --output minimal --max-events -1 --customise HLTrigger/Configuration/customizeHLTforCMSSW.customiseFor2018Input,HLTrigger/Configuration/customizeHLTforPatatrack.customiseCommon,HLTrigger/Configuration/customizeHLTforPatatrack.customisePixelLocalReconstruction,HLTrigger/Configuration/customizeHLTforPatatrack.customisePixelTrackReconstruction(ie. with customisePixelTrackReconstruction)
I got
Events Accepted Gained Lost Other Trigger
10000 1506 +10 -12 - HLT_PixelGlobalBTagDeepCSV0p17_v1
10000 512 +8 -15 - HLT_PixelGlobalBTagDeepCSV0p30_v1
10000 83 +2 -3 - HLT_PixelGlobalBTagDeepCSV0p60_v1
10000 2831 +9 -17 - HLT_PixelGlobalBTagDeepCSV0p05_v1
10000 2397 +9 -18 - HLT_PixelGlobalBTagDeepCSV0p1_v1
10000 1192 +41 -27 ~10 MC_PFBTagDeepCSV_v10Using a more recent release (CMSSW_12_3_0_pre4 + #36721) and HLT menu (/users/sdonato/GPUtest/PixelBtag/V34, including CPU-only patatrack), and adding the regional full-tracking b-tagging:
Events Accepted Gained Lost Other Trigger
10000 2539 +21 -19 - HLT_PixelGlobalBTagDeepCSV0p17_v1
10000 1084 +22 -17 - HLT_PixelGlobalBTagDeepCSV0p30_v1
10000 249 +6 -5 - HLT_PixelGlobalBTagDeepCSV0p60_v1
10000 3917 +18 -16 - HLT_PixelGlobalBTagDeepCSV0p05_v1
10000 3463 +19 -18 - HLT_PixelGlobalBTagDeepCSV0p1_v1
10000 1347 +12 -20 - HLT_RegionalFullBTagDeepCSV0p17_v1
10000 3683 +14 -13 - HLT_RegionalFullBTagDeepCSV0p05_v1
10000 692 +16 -12 - HLT_RegionalFullBTagDeepCSV0p30_v1
10000 291 +2 -2 - HLT_RegionalFullBTagDeepCSV0p60_v1
10000 2335 +18 -12 - HLT_RegionalFullBTagDeepCSV0p1_v1
10000 1179 +20 -18 ~4 MC_PFBTagDeepCSV_v10
10000 1789 +19 -21 ~3 HLT_PFBTagDeepCSV0p17_v10
10000 2841 +24 -32 ~3 HLT_PFBTagDeepCSV0p1_v10
10000 4041 +16 -23 ~3 HLT_PFBTagDeepCSV0p05_v10
10000 971 +15 -14 ~4 HLT_PFBTagDeepCSV0p3_v10
10000 409 +6 -5 ~6 HLT_PFBTagDeepCSV0p6_v10or using quadruplets:
Events Accepted Gained Lost Other Trigger
10000 1449 +12 -16 - HLT_PixelGlobalBTagDeepCSV0p17_v1
10000 449 +7 -5 - HLT_PixelGlobalBTagDeepCSV0p30_v1
10000 75 +2 - - HLT_PixelGlobalBTagDeepCSV0p60_v1
10000 2772 +10 -20 - HLT_PixelGlobalBTagDeepCSV0p05_v1
10000 2307 +13 -20 - HLT_PixelGlobalBTagDeepCSV0p1_v1
10000 1110 +12 -4 - HLT_RegionalFullBTagDeepCSV0p17_v1
10000 3216 +9 -13 - HLT_RegionalFullBTagDeepCSV0p05_v1
10000 514 +9 -5 - HLT_RegionalFullBTagDeepCSV0p30_v1
10000 185 +1 -2 - HLT_RegionalFullBTagDeepCSV0p60_v1
10000 2043 +10 -7 - HLT_RegionalFullBTagDeepCSV0p1_v1
10000 850 +9 -9 ~2 MC_PFBTagDeepCSV_v10
10000 1350 +12 -14 ~1 HLT_PFBTagDeepCSV0p17_v10
10000 2217 +19 -13 ~1 HLT_PFBTagDeepCSV0p1_v10
10000 3239 +17 -15 ~1 HLT_PFBTagDeepCSV0p05_v10
10000 675 +8 -7 ~2 HLT_PFBTagDeepCSV0p3_v10
10000 254 +3 -5 ~2 HLT_PFBTagDeepCSV0p6_v10I cannot check if the differences comes from the pixel local reco or from the pixel tracking because of https://github.com/cms-sw/cmssw/issues/35668 .
The good news is that the size of the differences seems to be of a few percents (instead of ~10%)
PS. Before the holidays I tried to enable VI_SORTSEED flag, but it seemed to not improve the situation, but I wanted to study it better.
silviodonato
commented
2 years ago Ganesh updated the study of the GPU vs GPU differences with the latest HLT menu and release: https://docs.google.com/spreadsheets/d/1AsLsABXTWSCuMr3K7LRYZTeHXGgUzZ3IdgIma53XiVU/edit#gid=0
The most significant differences are in the tau and b-tagging paths (but also MET).
For some reason HLT_DoubleMediumDeepTauIsoPFTauHPS35_L2NN_eta2p1_v1 is the only path with a very large difference
Here some numbers:
CPU vs GPU
| trigger | Change (%) | Accepted OLD | Accepted NEW | Change | Change (sigma) | Gained | Lost | Total |
|---|---|---|---|---|---|---|---|---|
| HLT_DoubleMediumDeepTauIsoPFTauHPS35_L2NN_eta2p1_v1 | 23.54% | 792 | 754 | 182 | 6.47 | 72 | -110 | 1461195 |
| HLT_QuadPFJet105_88_76_15_DoublePFBTagDeepCSV_1p3_7p7_VBF1_v8 | 10.63% | 208 | 206 | 22 | 1.53 | 10 | -12 | 1461195 |
| HLT_PFHT400_FivePFJet_120_120_60_30_30_DoublePFBTagDeepCSV_4p5_v8 | 11.22% | 197 | 195 | 22 | 0.69 | 10 | -12 | 1461195 |
| HLT_PFMETNoMu120_PFMHTNoMu120_IDTight_v20 | 2.65% | 1017 | 1022 | 27 | 0.85 | 16 | -11 | 1461195 |
| HLT_IsoTrackHE_v4 | 4.36% | 279 | 271 | 12 | 0.20 | 2 | -10 | 1461195 |
| HLT_Ele28_WPTight_Gsf_v1 | 0.72% | 3760 | 3765 | 27 | 0.43 | 16 | -11 | 1461195 |
| HLT_IsoMu24_eta2p1_v15 | 0.12% | 4018 | 4013 | 5 | 0.08 | 0 | -5 | 1461195 |
This are the numbers obtained GPU vs GPU
| trigger | Change (%) | Accepted OLD | Accepted NEW | Change | Change (sigma) | Gained | Lost | Total |
|---|---|---|---|---|---|---|---|---|
| HLT_DoubleMediumDeepTauIsoPFTauHPS35_L2NN_eta2p1_v1 | 24.89% | 754 | 781 | 191 | 6.96 | 109 | -82 | 1461195 |
| HLT_QuadPFJet105_88_76_15_DoublePFBTagDeepCSV_1p3_7p7_VBF1_v8 | 4.85% | 206 | 206 | 10 | 0.70 | 5 | -5 | 1461195 |
| HLT_PFHT400_FivePFJet_120_120_60_30_30_DoublePFBTagDeepCSV_4p5_v8 | 4.63% | 195 | 194 | 9 | 0.28 | 4 | -5 | 1461195 |
| HLT_PFMETNoMu120_PFMHTNoMu120_IDTight_v20 | 0.88% | 1022 | 1023 | 9 | 0.28 | 5 | -4 | 1461195 |
| HLT_IsoTrackHE_v4 | 1.83% | 271 | 274 | 5 | 0.08 | 4 | -1 | 1461195 |
| HLT_Ele28_WPTight_Gsf_v1 | 0.08% | 3765 | 3766 | 3 | 0.05 | 2 | -1 | 1461195 |
| HLT_IsoMu24_eta2p1_v15 | 0.05% | 4013 | 4015 | 2 | 0.03 | 2 | 0 | 1461195 |
fwyzard
commented
2 years ago Are we sure that a CPU vs CPU reproducibility test would not show any discrepancy ?
silviodonato
commented
2 years ago Are we sure that a CPU vs CPU reproducibility test would not show any discrepancy ?
no, but I can ask about this
Tests done in multiple recent releases have shown that the HLT results are not consistent when running on GPU vs on CPU.
Here are the instruction to reproduce the issue using
CMSSW_12_1_0_pre3/dev/CMSSW_12_1_0/GRun/V1/RelValTTbar_14TeV/CMSSW_12_0_0_pre6-PU_120X_mcRun3_2021_realistic_v4_JIRA_129-v1/GEN-SIM-DIGI-RAW/store/relval/CMSSW_12_1_0_pre3/RelValTTbar_14TeV/GEN-SIM-DIGI-RAW/PU_121X_mcRun3_2021_realistic_v2-v1/10000/0eb14c4a-e363-424a-9c0c-2688c7d32c74.rootauto:phase1_2021_realistic; running on a previous release using the same global tag as the sample itself (120X_mcRun3_2021_realistic_v4) shows a similar behaviour.setup a CMSSW working area
extract the HLT configuration for running on GPU using the Run3 era
run the HLT menu on a GPU-equipped machine
compare the results
To disentangle the various effects, one can use different customisations on top of the HLT menu, running each resulting configuration with a GPU and without a GPU (that is, fully on the CPU). Replace the customisation at the bottom of the
hlt.pyfilewith a more fine-grained one, described below.
legacy configuration
Run the HLT menu unchanged, adding only the
Status_OnGPUandStatus_OnCPUpaths, without actually offloading any reconstruction to GPU:ECAL-only changes
To check the impact of running the ECAL reconstruction on GPU vs CPU, apply only the ECAL changes:
HCAL-only changes
To check the impact of running the HCAL reconstruction on GPU vs CPU, apply only the HCAL changes:
Pixel local reconstruction changes
To check the impact of running the Pixel local reconstruction on GPU vs CPU, apply only the Pixel changes:
Pixel track reconstruction changes
To check the impact of running the Pixel local reconstruction on GPU vs CPU, apply only the Pixel and Tracking changes. Clearly, for this comparison to be meaningful, the previous one needs to be understood first.
The ECAL-only comparison did not reveal significant differences.
The HCAL-only comparison showed significant differences in a few % of the events (order of 10% of the accepted events).
The Pixel local reconstruction comparison showed significant differences in a few % of the events (order of 10% of the accepted events), while affecting less paths than the HCAL one.
I think that looking at the Pixel track comparison makes sense only after fixing the local reconstruction one.
Updates
for running with recent IBs, please use https://github.com/cms-sw/cmssw/pull/35497 .
the 12.1.0-pre3 relvals can also be used, for example
/store/relval/CMSSW_12_1_0_pre3/RelValTTbar_14TeV/GEN-SIM-DIGI-RAW/PU_121X_mcRun3_2021_realistic_v2-v1/10000/0eb14c4a-e363-424a-9c0c-2688c7d32c74.root.