cmsdaq / DAQExpert

New expert system processing data model produced by DAQAggregator
1 stars 2 forks source link

Special instructions: ECAL FED stuck #202

Closed gladky closed 5 years ago

gladky commented 6 years ago
  • ECAL If an ECAL FED gets stuck in 100% BUSY,
    • first try to just Stop and Start run
    • only if that doesn't work, red-recycle ECAL
    • Post an elog with the FED number (updated on October 17th 2016, M. Schoenenberger)
gladky commented 6 years ago

Currently we have special instructions for ECAL when FED stuck:

  1. StopAndStartTheRun (try up to 2 times)
  2. Problem fixed: Make an e-log entry.
  3. Problem not fixed: Red recycle ECAL
  4. Call the DOC for the ECAL

Note that

gladky commented 6 years ago
gladky commented 6 years ago

@giacomoCucciati could you please comment on this one?

giacomoCucciati commented 6 years ago

I don't remember the reason of a second stop&start if the first doesn't work. Probably also the second one will fail. So I would keep the instructions restoring a single StopAndStartTheRun.

Thanks

hsakulin commented 6 years ago

So let's just change the instructions to only do 1 start/stop and keep them both for 100% BUSY and 100% WARNING.

gladky commented 5 years ago

Final instructions for ECAL on FED stuck:

  1. \<\>,
  2. Problem fixed: Make an e-log entry including problematic FED number.
  3. Problem not fixed: \<\<StopAndStartTheRun>> with \<\<RedRecycle::ECAL>>
  4. Problem still not fixed: Call the DOC for the ECAL

Note that steps with angle brackets are automated, i.e. operator can issue them with a single click in DAQExpert dashboard, and soon they will be issued without operator intervention.

gladky commented 5 years ago

Moved to "covered by expert" section. Released in 2.12.3. Closing.