cmsdaq / DAQExpert

New expert system processing data model produced by DAQAggregator
1 stars 2 forks source link

Special instructions: TOTDET deadtime #199

Closed gladky closed 5 years ago

gladky commented 6 years ago
  • If the run gets deadtime due to totem (TOTDET), this is likely a trigger rate issue. Check the rates coming from sub detecteors to understand what is the problem (Matthieu Marionneau 12-06-2017)
gladky commented 6 years ago

Currently we detect deadtime of each partition (including TOTDET) and we report:

Deadtime of partition(s) {{PROBLEM-PARTITION}} in subsystem(s) {{PROBLEM-SUBSYSTEM}} is {{DEADTIME}} the threshold is X%

gladky commented 6 years ago
  1. Is this up to date?
  2. shall we include this in expert?
jjhollar commented 6 years ago

Hi again,

Please see the other ticket (https://github.com/cmsdaq/DAQExpert/issues/184) - it's the same issue I mentioned where these are 2 of the slowest FEDs, so any time the trigger rate goes crazy they will be the ones appearing to cause large deadtime by throttling to a little more than 100kHz L1A rate.

This is still up to date. I think it's up to you if it should be included in the DAQ expert separately, or if you think it's already covered by the high trigger rate alarms. Personally my preference would be for the DAQ expert to always show the denominator when reporting deadtime as a percentage - there's a big difference between 95% of 100kHz and 95% of 40MHz :)

Jonathan

gladky commented 5 years ago

Covered by 2 LM monitoring TCDS rate:

HighTcdsInputRate

The TCDS trigger input rate is high: {{TCDS_TRIGGER_INPUT_RATE}}

  1. Ask the trigger shifter to check the prescale column
  2. Make an e-log entry

threshold: > 100 kHz

VeryHighTcdsInputRate

The TCDS trigger input rate is very high: {{TCDS_TRIGGER_INPUT_RATE}}

  1. Ask the trigger shifter to check the inputs to the L1 trigger (noisy towers, failed links)
  2. Make an e-log entry

threshold: > 200 kHz + dominates HighTcdsInputRate