warn on too many filter units on cloud mode when they should not be

cmsdaq / DAQExpert

New expert system processing data model produced by DAQAggregator

1 stars 2 forks source link

Closed andreh12 closed 6 years ago

andreh12 commented 6 years ago

warn about fraction filter units in cloud mode too high in LHC states where the cloud VMs should not be running

warn if more than 3% of filter units in cloud mode AND
if LHC status is between FlatTop and BeamDump (both states included), see http://aliweb.cern.ch/system/files/LHC-OP-ES-0022-V1_5.pdf page 8 for state diagram

This should be a warning, not gated with 'no rate when expected', similar to how rcms.utilities.daqexpert.reasoning.logic.basic.Deadtime is used.

andreh12 commented 6 years ago

an example test case file is /daqexpert/snapshots/pro/cdaq/2017/9/7/12/1504785601261.json.gz

hsakulin commented 6 years ago

I wonder whether we should include stable beams and beam dump. Eventually we will be starting the cloud during the fill as the lumi is decreasing.

We should at least keep this in mind.

andreh12 commented 6 years ago

@hsakulin good point -- for the moment I would keep things simple.

When we start running cloud VMs as function of the luminosity we should

either have a formula saying how many should be there
or complain that the HLT load is above a given value and mention cloud VMs if there are many

Or we could just take the algorithm determining how many cloud VMs are started and check whether this is actually fulfilled.

andreh12 commented 6 years ago

we can close this when we have >= 2.9.11 in production