Special instructions: HLT high throughput

[gladky]

From shifter bulletin

• High throughput running (2016-11-29, R. Mommsen) When running with a high throughput (>5.5 GB/s) out of FFF/HLT, please pay particular attention to any signs of problems on the BU appliances. Please ask the shift leader to switch to the “physics only” prescale column whenever any of these problems occur:
  • any yellow/red fields in the last 3 columns of the FFF table on DAQView which persist for more than a minute
  • any alert message on the F3 monitoring page which complain about Input/output errors
  • >20 continuous LS on the F3 monitoring page are not mini-merged (2nd row of green/red boxes) and further delays are building up
  • >60 LS on the F3 monitoring page are not macro-merged (3rd row of green/red boxes)

[gladky]

Currently we identify the high HLT bandwidth in LM "Too high HLT output bandwidth" and suggest following action if it surpasses the 1st threshold:

1. Talk to the trigger shifter and shift leader.
2. Have them check the pre-scale column. You may need to call the HLT DOC.

and following when surpasses 2nd threshold:

1. Talk to the trigger shifter and shift leader. Have them check the pre-scale column.
2. Check the per-stream bandwidths in F3Mon. You may need to call the HLT DOC.

• 1st threshold is currently set to 4.5 GB/s
• 2nd to 6 GB/s

[gladky]

• is this up to date?
• do you want to introduce it to expert? if yes:
  • do you want to synchronise the thresholds?
  • could we get all information we need to automate reasoning to system snapshot?

[mommsen]

The problem descriptions are still accurate to detect brewing problems. I have no idea how difficult it would be to get this information into the DAQExpert, and if it is worthwhile to do.

The thresholds are best guesses to avoid that the problem actually occurs. The thresholds should be reviewed once we have the B-physics parking stream, which should happen over the coming week(s).