Online-DQM occasionally not picking up the correct IOVs ?

[missirol]

There have been cases in recent weeks/months where strange discrepancies were observed in the online-DQM outputs at P5.

• L1T prescales. On May-27 (2024), strange data-emulator mismatches in the L1T decisions were seen in online-DQM during run-381286 (link). The checks made at the time suggested that, while 381286 was a collisions run, the emulator plots in the online-DQM were using the L1T prescales of the trigger menu used in the previous run (which was a run with "circulating" beams, not collisions). The relevant tag is L1TGlobalPrescalesVetosFract_Stage2v1_hlt. No issues were seen in the O2O logs at the time, nor warnings or crashes anywhere. Below is the L1T report from the day after.

  During collisions run 381286, DQM shfter reported mismatches in EGamma and uGT data vs emulator. While EGamma issue has been reported before (expert checking) the uGT mismatch is new.

  • Emulator picked prescale conditions from circulating key for some reason that is not yet understood
  • Can be a problem related to DQM software

  This slide from L1T Techical Coordination suggests that a similar issue also occurred on Apr-25 (2024), see CMSLITDPG-1257.

• Data-emulator mismatches related to ECAL Barrel trigger primitives. On Aug-15 (2024), ECAL uploaded new conditions via O2O with IOV starting from run-384485, but during that run data-emulator mismatches showed up in the ECAL online-DQM outputs. This too may be consistent with the online-DQM jobs consuming conditions from an older IOV.

  • Links: elog, online DQM of run-384485, relevant CMSTalk post, tag EcalTPGLinearizationConst_v2_hlt.

  • Copying here text from an ECAL expert.

    There was a glitch in run 384485 which showed up as a data/emulator mismatch in EB. It looks like for that run the new pedestals were not propagated to either the electronics or the emulator. Our checks strongly indicate that it was the latter.

In both examples, the discrepancies disappeared after a new run was started.

At face value, both examples seem compatible with the cmsRun jobs in the online-DQM nodes not picking up the latest (and correct) IOVs, using instead older ones and thus leading to mismatches between real and emulated data in DQM outputs.

I think it would be helpful if DQM and AlCa-DB could investigate what happened in these cases (O2O logs, etc), with help from framework experts if needed.

If the issue is not specific to online-DQM, but generally related to the access to the conditions database, it could potentially affect the HLT jobs running online as well.

Maybe unrelated, a recent HLT crash possibly caused by a failure in accessing correct conditions (in that case, for the beamspot) is being discussed in #45555.

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @missirol.

@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[makortel]

assign dqm, alca, db

[cmsbuild]

New categories assigned: dqm,alca,db

@rvenditti,@syuvivida,@tjavaid,@nothingface0,@antoniovagnerini,@francescobrivio,@saumyaphor4252,@saumyaphor4252,@perrotta,@perrotta,@consuegs,@consuegs you have been requested to review this Pull request/Issue and eventually sign? Thanks

[missirol]

@cms-sw/dqm-l2 @cms-sw/db-l2 @cms-sw/alca-l2

Will you try to address this issue ?

[missirol]

@cms-sw/dqm-l2 @cms-sw/db-l2 @cms-sw/alca-l2

Still wondering if there will be some follow-up. Or the issue is not worth investigating ? Or should more info be provided ?

[perrotta]

@cms-sw/dqm-l2 @cms-sw/db-l2 @cms-sw/alca-l2

Still wondering if there will be some follow-up. Or the issue is not worth investigating ? Or should more info be provided ?

@missirol if the online-DQM jobs consumes conditions from an older IOV I think is an issue rooted in the online-DQM jobs. If needed (and if I can) I can help debugging, but @cms-sw/dqm-l2 should pinpoint first which are those jobs, where the issue could come from, etc.

[missirol]

if the online-DQM jobs consumes conditions from an older IOV I think is an issue rooted in the online-DQM jobs.

How can we be sure that this only affects the online-DQM [*] ? Could it be that the online-DQM is just the first (and only ?) place where such an issue would be spotted ?

In the cases given in the description, was anything strange noticed on the DB side and/or in the O2O logs ? (I understood in https://github.com/cms-sw/cmssw/issues/45555#issuecomment-2293642146 that O2O logs get eventually deleted, so maybe now it's too late to check). @cms-sw/db-l2

[*] From the description

If the issue is not specific to online-DQM, but generally related to the access to the conditions database, it could potentially affect the HLT jobs running online as well.