search

cms-sw / cmssw

CMS Offline Software
http://cms-sw.github.io/
Apache License 2.0
1.06k stars 4.24k forks source link

Multiple RelVals: `Unknown trigger name` exception #44895

Closed iarspider closed 2 months ago

iarspider commented 2 months ago

RelVal 1042.0 is failing in multiple IBs with the following exception:

----- Begin Fatal Exception 02-May-2024 13:41:35 CEST-----------------------
An exception of category 'Configuration' occurred while
   [0] Processing  Event run: 305064 lumi: 38 event: 58033402 stream: 0
   [1] Running path 'ALCARECOStreamPPSCalTrackBasedSelOutPath'
   [2] Calling OutputModule prePrefetchSelection()
Exception Message:
EventSelector::init, An OutputModule is using SelectEvents
to request a trigger name that does not exist
The unknown trigger name is: pathALCARECOPPSCalTrackBasedSel
----- End Fatal Exception -------------------------------------------------

full log

iarspider commented 2 months ago

assign alca

cmsbuild commented 2 months ago

New categories assigned: alca

@saumyaphor4252,@perrotta,@consuegs you have been requested to review this Pull request/Issue and eventually sign? Thanks

cmsbuild commented 2 months ago

cms-bot internal usage

cmsbuild commented 2 months ago

A new Issue was created by @iarspider.

@smuzaffar, @rappoccio, @makortel, @antoniovilela, @Dr15Jones, @sextonkennedy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

perrotta commented 2 months ago

Also RelVal 1041 is failing in CMSSW_14_1_X_2024-05-02-1100 with the same exception. And RelVal 1040.1 is failing with the exception

----- Begin Fatal Exception 02-May-2024 09:43:06 CEST-----------------------
An exception of category 'Configuration' occurred while
   [0] Processing  Event run: 305064 lumi: 38 event: 58036426 stream: 2
   [1] Running path 'ALCARECOStreamSiPixelCalZeroBiasOutPath'
   [2] Calling OutputModule prePrefetchSelection()
Exception Message:
EventSelector::init, An OutputModule is using SelectEvents
to request a trigger name that does not exist
The unknown trigger name is: pathALCARECOSiPixelCalZeroBias
----- End Fatal Exception -------------------------------------------------

I suppose the issue is surfaced by the merging in that IB of #44767 , which throws a clear exception if OutputModule SelectEvents refers to a non-existent Path or Process.

The three failing workflows refer to 'RunExpressPhy2017F' (ExpressPhysics dataset for 2017).

As far as I can read from https://twiki.cern.ch/twiki/bin/viewauth/CMS/AlCaRecoMatrix, in Run2 there was not such a PPSCalTrackBasedSel path in ALCARECO, and in 2017 not even a SiPixelCalZeroBias. (To be honest, if I read the content of Configuration/AlCa/doc/pastAlCaRecoMatrix.py I get a different content: this is something that must be investigated further).

wf 1040.1 was introduced in #22455: my impression is that a configuration for 2018 was (wrongly) used for a 2017 AlCaReco: unfortunately, the logs of the tests cannot be recovered to verify whether any message was issued in them for the missing path.

The same for wf 1041.0 that was introduced (as previous 1040.2) with #33215, and wf 1042.0 that was introduced with #35631, but in both cases still with a configuration that does not seem to me to reflect the 2017 ALCARECO content.

This is mostly to tell that I am investigating. Let me add here @francescobrivio (as author of some of the PRs that considered the Express content for 2017), my fellow @cms-sw/alca-l2 , and @cms-sw/ctpps-dpg-l2 for a clarification about the PPS content in Express for the 2017.

And let me also add @makortel as author of #44767: did you notice Matti that after the CMSSW_14_1_X_2024-05-02-1100 the exception is not 'clear' any more (see the very same workflows failing in CMSSW_14_1 2024-05-02-2300)?. Do you understand why?

Dr15Jones commented 2 months ago

And let me also add @makortel as author of https://github.com/cms-sw/cmssw/pull/44767: did you notice Matti that after the CMSSW_14_1_X_2024-05-02-1100 the exception is not 'clear' any more (see the very same workflows failing in CMSSW_14_1 2024-05-02-2300)?. Do you understand why?

We discovered several bugs in #44767 which are now fixed in #44891. These bugs resulted in the assert sometimes going off rather than the exception message being printed. We did extensive testing with the new fix and now see the exception message being issued consistently.

makortel commented 2 months ago

Let's use this issue to report all the "unknown trigger names" from https://github.com/cms-sw/cmssw/pull/44891#issuecomment-2091954682

Workflow 1001.3

----- Begin Fatal Exception 03-May-2024 02:16:12 CEST-----------------------
An exception of category 'Configuration' occurred while
   [0] Processing  Event run: 355769 lumi: 35 event: 22779366 stream: 0
   [1] Running path 'ALCARECOStreamEcalESAlignOutPath'
   [2] Calling OutputModule prePrefetchSelection()
Exception Message:
EventSelector::init, An OutputModule is using SelectEvents
to request a trigger name that does not exist
The unknown trigger name is: pathALCARECOEcalESAlign
----- End Fatal Exception -------------------------------------------------
----- Begin Fatal Exception 03-May-2024 02:16:12 CEST-----------------------
An exception of category 'Configuration' occurred while
   [0] Processing  Event run: 355769 lumi: 35 event: 22311221 stream: 3
   [1] Running path 'ALCARECOStreamEcalUncalZElectronOutPath'
   [2] Calling OutputModule prePrefetchSelection()
Exception Message:
EventSelector::init, An OutputModule is using SelectEvents
to request a trigger name that does not exist
The unknown trigger name is: pathALCARECOEcalUncalZElectron
----- End Fatal Exception -------------------------------------------------

1040.1

----- Begin Fatal Exception 03-May-2024 01:53:50 CEST-----------------------
An exception of category 'Configuration' occurred while
   [0] Processing  Event run: 305064 lumi: 38 event: 58036426 stream: 1
   [1] Running path 'ALCARECOStreamSiPixelCalZeroBiasOutPath'
   [2] Calling OutputModule prePrefetchSelection()
Exception Message:
EventSelector::init, An OutputModule is using SelectEvents
to request a trigger name that does not exist
The unknown trigger name is: pathALCARECOSiPixelCalZeroBias
----- End Fatal Exception -------------------------------------------------

1041.0, 1042.0

----- Begin Fatal Exception 03-May-2024 01:53:35 CEST-----------------------
An exception of category 'Configuration' occurred while
   [0] Processing  Event run: 305064 lumi: 38 event: 58067786 stream: 2
   [1] Running path 'ALCARECOStreamPPSCalTrackBasedSelOutPath'
   [2] Calling OutputModule prePrefetchSelection()
Exception Message:
EventSelector::init, An OutputModule is using SelectEvents
to request a trigger name that does not exist
The unknown trigger name is: pathALCARECOPPSCalTrackBasedSel
----- End Fatal Exception -------------------------------------------------
makortel commented 2 months ago

And for reference, an earlier such issue https://github.com/cms-sw/cmssw/issues/44780 was fixed in https://github.com/cms-sw/cmssw/pull/44787

makortel commented 2 months ago

My private test of 1001.3 on a single thread gives also

----- Begin Fatal Exception 02-May-2024 21:13:44 CEST-----------------------
An exception of category 'Configuration' occurred while
   [0] Processing  Event run: 355769 lumi: 35 event: 22311221 stream: 0
   [1] Running path 'ALCARECOStreamTkAlV0sOutPath'
   [2] Calling OutputModule prePrefetchSelection()
Exception Message:
EventSelector::init, An OutputModule is using SelectEvents
to request a trigger name that does not exist
The unknown trigger name is: pathALCARECOTkAlK0s
----- End Fatal Exception -------------------------------------------------

so all these workflows might have more issues than reported so far.

mmusich commented 2 months ago

https://github.com/cms-sw/cmssw/pull/44914 should fix.