Add Validation and DQM to Phase2 HLT WF

AdrianoDee commented 2 years ago

This is to remind us that the wf running the Phase2 HLT (39434.75) is not running any validation or DQM for hlt objects.

This will be necessary in the mid-long term and would be useful in the short term for developments to spot eventual changes in the menu behavior (see e.g. https://github.com/cms-sw/cmssw/pull/39323).

cmsbuild commented 2 years ago

A new Issue was created by @AdrianoDee .

@Dr15Jones, @perrotta, @dpiparo, @rappoccio, @makortel, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

AdrianoDee commented 2 years ago

assign upgrade

AdrianoDee commented 2 years ago

assign hlt

cmsbuild commented 2 years ago

New categories assigned: upgrade,hlt

@AdrianoDee,@missirol,@srimanob,@Martin-Grunewald you have been requested to review this Pull request/Issue and eventually sign? Thanks

srimanob commented 2 years ago

I think the way is to run HLT at the ususal step (DIGI+L1+HLT). Then the validation will run as usual. No need to push for extra development.

srimanob commented 1 year ago

I make this https://github.com/cms-sw/cmssw/pull/39450 for the first attempt to converge on running HLT together with DIGI step.

slava77 commented 1 year ago

What is the current status? is this topic being tracked in another issue or outside of github?

@beaucero @SohamBhattacharya

srimanob commented 1 year ago

Hi @slava77 I have not finished it yet, sorry. I can make it after May 22.

slava77 commented 1 year ago

Hi @slava77 I have not finished it yet, sorry. I can make it after May 22.

I wanted to check if there was some progress Thanks.

srimanob commented 1 year ago

Hi @slava77 I've made a new PR to enable one-by-one HLT validation module in https://github.com/cms-sw/cmssw/pull/41898

srimanob commented 1 year ago

I think this PR can be closed as we have Phase-2 DQM in the release.

srimanob commented 1 year ago

+Upgrade

mmusich commented 1 year ago

https://github.com/cms-sw/cmssw/pull/42783 introduces some modification to run the Tracking DQM monitoring in Phase-2

SohamBhattacharya commented 1 year ago

Hi, I have naive question about this. I see two different approaches to adding HLT validation and DQM.

The one started by @srimanob here
One developed by @Sam-Harper and @finnlabe here (in cmssw)

If they're both trying to do the same (?) thing (i.e. validate HLT paths), is there a way to consolidate the two approaches to make it simpler?

mmusich commented 1 year ago

If they're both trying to do the same (?) thing (i.e. validate HLT paths), is there a way to consolidate the two approaches to make it simpler?

as far as I can tell , that's not the same thing. The issue here is more targeted to the offline validation of the objects used by the HLT (electrons, muons, tracks, jets, etc. etc.). Basically everything in DQMOffline/Trigger that is currently available for Run 3 (but not for phase-2).

SohamBhattacharya commented 1 year ago

If they're both trying to do the same (?) thing (i.e. validate HLT paths), is there a way to consolidate the two approaches to make it simpler?

as far as I can tell , that's not the same thing. The issue here is more targeted to the offline validation of the objects used by the HLT (electrons, muons, tracks, jets, etc. etc.). Basically everything in DQMOffline/Trigger that is currently available for Run 3 (but not for phase-2).

It seems to me that the updates to HLTriggerOffline in https://github.com/cms-sw/cmssw/pull/41898 are to do with validating the TriggerResults (validation for HLT_AK4PFPuppiJet520), which is similar to what's being done in 2 above. (please correct me if I'm wrong) While we absolutely need the validation of Phase-2 HLT objects in DQMOffline/Trigger, imo (from the pov of HLT-upgrade) we should first have the Phase-2 HLT path and filter validation. That would show us changes in not only the HLT objects (thus also effectively doing the job of DQMOffline/Trigger as far as the impact on the Phase-2 menu is considered), but also in the upstream quantities that the HLT filters depend on, like the L1 objects. For example, the egamma issue (fixed in https://github.com/cms-sw/cmssw/pull/42791) was missed because the filters were not monitored and changes at L1 were not propagated down to the filters used by the Phase-2 menu.

Hence I was wondering if there's a way to have a validation like this as part of the Phase-2 HLT WF.

[tagging @rovere as I'm not sure if he's in the thread]

mmusich commented 1 year ago

we should first have the Phase-2 HLT path and filter validation.

@SohamBhattacharya I guess you are free to twist the initial aim of this ticket, but in the initial message I read:

is not running any validation or DQM for hlt objects.

I think your proposal should go into a different issue!

SohamBhattacharya commented 1 year ago

we should first have the Phase-2 HLT path and filter validation.

@SohamBhattacharya I guess you are free to twist the initial aim of this ticket, but in the initial message I read:

is not running any validation or DQM for hlt objects.

I think your proposal should go into a different issue!

No problem at all with having this in a different issue. Only initiating the discussion here as the question of validation was rightfully raised in https://github.com/cms-sw/cmssw/pull/42791 -- so wanted to make it clear what kind of validation hlt-upgrade would like to focus on first. :)

srimanob commented 1 year ago

Hi @SohamBhattacharya

My PR (https://github.com/cms-sw/cmssw/pull/41898) is for Validation step (i.e. compare with MC) while @mmusich PR (https://github.com/cms-sw/cmssw/pull/42783) takes care for DQM (i.e. no need of GEN, MC. Something that you run on Tier-0 during data taking, and also regular relvals, both Data and MC).

The way that both PR work is the same, we create a sequence based on Run-3 and enable only modules which work for Phase-2. See:

Validation: https://github.com/cms-sw/cmssw/blob/master/HLTriggerOffline/Common/python/HLTValidation_cff.py#L30-L91 (I chose to not create new sequence, but use the original one and exclude what do not work.)
DQM: https://github.com/cms-sw/cmssw/blob/d9adfeaf74d34ee3c350e42f343f74d98d766d9e/DQMOffline/Trigger/python/DQMOffline_Trigger_cff.py#L193-L215 (Marco creates the specific Phase-2 sequence, and include only working module. Then use replace to replace the sequence.

If I understand @mmusich correctly, the suggestion is to try to enable this full sequence [1]. While on my part, POG, PAG should try to enable their trigger paths and re-enable [2]

[1]

offlineHLTSource4HLTMonitorPD = cms.Sequence(
    dqmInfoHLTMon *
    lumiMonitorHLTsequence *          # lumi
    sistripMonitorHLTsequence *       # strip
    sipixelMonitorHLTsequence *       # pixel
    BTVHLTOfflineSource *             # BTV
    bTagHLTTrackMonitoringSequence *  # BTV relative track efficiencies
    trackingMonitorHLT *              # tracking
    BTagAndProbeHLT *                 # BTag and Probe
    trackingMonitorHLTDisplacedJet*   # EXO : DisplacedJet Tracking 
    egmTrackingMonitorHLT *           # EGM tracking
    hltToOfflineTrackValidatorSequence *  # Relative Online to Offline performace
    vertexingMonitorHLT *             # vertexing
    particleNetMonitoringHLT          # HIG: monitoring of HLT PNET taggers (incl. comparisons to Offline PNET)
)

[2]

hltassociation = cms.Sequence(
    hltMultiTrackValidation
    +hltMultiPVValidation
    +egammaSelectors
    +ExoticaValidationProdSeq
    +hltMultiTrackValidationGsfTracks
    +hltMultiTrackValidationMuonTracks
    )

hltvalidationWithMC = cms.Sequence(
    HLTMuonVal
    +HLTTauVal
    +egammaValidationSequence
    +heavyFlavorValidationSequence
    +HLTJetMETValSeq
    +HLTSusyExoValSeq
    +HiggsValidationSequence
    +ExoticaValidationSequence
    +b2gHLTriggerValidation
    +SMPValidationSequence
    +hltbtagValidationSequence #too noisy for now
    +hltHCALdigisAnalyzer+hltHCALRecoAnalyzer+hltHCALNoiseRates # HCAL
)

mmusich commented 10 months ago

for the record https://github.com/cms-sw/cmssw/pull/43094 introduced tracking and vertexing HLT validation (w.r.t tracking particles) for the phase 2 setup.