Developments for DQM Core must be tested also on Online DQM clients before deploying/merging

cms-sw / cmssw

CMS Offline Software

http://cms-sw.github.io/

Apache License 2.0

1.07k stars 4.28k forks source link

Developments for DQM Core must be tested also on Online DQM clients before deploying/merging #41713

Open sroychow opened 1 year ago

sroychow commented 1 year ago

In the recent collision runs, several MEs were not displayed on Online GUI( see link). Upon iterating with Central DQM experts, it was understood that the problem is caused by #41400 -

So when an Online ME is listed as a per-LS Monitoring Element as well, plots in the Online live mode are snapshots of every LS and in the end they are not properly merged. DQM experts are looking into the cause for this. They have deployed a temporary fix since run 367657 where all online MEs are removed from per LS ME config.

With the above motivation, the proposal is the following:-

Whenever a change to DQMStore or any DQM Core is made, the Online clients should also be tested on top of the proposed changes before the relevant PRs are merged.

cmsbuild commented 1 year ago

A new Issue was created by @sroychow Suvankar Roy Chowdhury.

@Dr15Jones, @perrotta, @dpiparo, @rappoccio, @makortel, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

makortel commented 1 year ago

assign dqm

cmsbuild commented 1 year ago

New categories assigned: dqm

@tjavaid,@micsucmed,@nothingface0,@rvenditti,@emanueleusai,@syuvivida,@pmandrik you have been requested to review this Pull request/Issue and eventually sign? Thanks

makortel commented 1 year ago

(I'd like to understand a bit better the testing suggestion)

the Online clients should also be tested

The clients defined in DQM/Integration/python/clients are tested (as far as I know/recall) in the IBs (could also be tested in PRs), but none of those have shown any failure since the master PR https://github.com/cms-sw/cmssw/pull/41196 was merged. Does that mean these tests are not sufficient (to reveal this kind of problems), they would need to be tested in the online environment, or did you mean something completely different?

mmusich commented 1 year ago

Does that mean these tests are not sufficient (to reveal this kind of problems), they would need to be tested in the online environment, or did you mean something completely different?

I guess a DQM bin-by-bin comparison of the online client output might have caught the issue. The clients don't fail but (some of) the histograms are not there.

syuvivida commented 1 year ago

+1 Agreed from the DQM side.

cmsbuild commented 1 year ago

This issue is fully signed and ready to be closed.

mmusich commented 1 year ago

+1 Agreed from the DQM side.

I natively would have thought that DQM would sign once a solution for the issue is proposed.