What needs to be done to replace the DAQ Doctor

[hsakulin]

1) Additional checks

• [x] High individual FED Deadtime
• [x] High Partition Deadtime
• [x] High global deadtime (with analysis of breakdown (individual issue #46)) (*)
• [x] (individual issue #17) Check that physics stream rates are within limits (configurable, different sets for different machine modes, may vary over time, exceptions for special runs, maybe tie to HLT Key or L1/HLT Key). ( finer check in the future: limits may depend on HLT menu, inst. luminosity, prescale-column )
• [x] (individual issue #28) Temperature checks
• [x] (individual issue #29) (check all other checks that used to be done by the DAQ doctor)

(*)

• [x] Use DeadTimeBeamActive for runs with beam in the machine (INJECTION PROBE to STABLE BEAMS)
• [x] Use DeadTime for cosmic runs (BEAM DUMP to SETUP)

2) New data sources

2a) New data source: Filter Farm Monitoring

• [x] (individual issue #47) meet with Srecko to find out if it is always there

• [x] (individual issue #17) HLT output rates per stream, stream sizes

• [x] (individual issue #44) CPU usage per machine

• [x] Fill level of RAM disks (individual issue #32), output disks (individual issue #45)

2b) New data source: errors from XMAS (spotlight)

2c) Where do we get the temperature monitoring from? (individual issue #28)

II=> 3) Audio alerts

• [x] LHC machine/beam mode change
• [x] DAQ state change
• [x] Subsystem going to RunningDegraded / RunningSoftErrordetected / Error
• [x] (individual issue #43) L1 or HLT rate out of limits
• [x] High dead time
• [x] No rate when expected etc.

4) Add support for audio

• [x] NM sending alerts to CMS-WOW in control room (test on office PC)

• [x] (individual issue #35)Receive audio message from DCS, WBM, DQM (see: https://svnweb.cern.ch/trac/cms-daqdemo/browser/DAQTools/Monitoring/trunk/DaqDottoressa/RequestHandler.pm?desc=1) External message also create a notification.

• [x] (individual issue #40) Tool for expert at home to listen to audio messages (in browser ? )

• [x] (individual issue #39) Add global dead time to plots. Use DeadTimeBeamActive all the time.

• [x] (individual issue #41) 6) Searchable archive

[gladky]

test case: fed 106 in warning >40% 2016-12-02T15:15:00

[gladky]

New Logic module has been introduced "Active Beam" for global deadtime check. Beam is considered active in following LHC Beam modes:

• INJECTION_PROBE_BEAM
• INJECTION_SETUP_BEAM
• INJECTION_PHYSICS_BEAM
• PREPARE_RAMP
• RAMP
• FLAT_TOP
• SQUEEZE
• ADJUST
• STABLE_BEAMS