Special instructions: ES soft error every 10 seconds

[gladky]

• ES (Updated 20160804 following request of ECAL/ES DOC.) ES may ask for consecutive SoftErrorRecovery every ~10 seconds. What to do: 1 Stop the run and re-start it 1 if 1) doesn't work and DAQ is in the same condition as before, stop the run and red-recycle ES.

[gladky]

Currently we have ES specific instructions for this case:

1. Stop the run and re-start it
2. If 1) doesn't work and DAQ is in the same condition as before, stop the run and red-recycle ES

Note that:

• the threshold is 3 times in 10 minutes

[gladky]

• [ ] please confirm it's up to date, I will then move this to "covered by expert" section in shifter bulletin board.

[andreh12]

I noticed that the stop the run and re-start it and red-recycle are not actions -- would it make sense to make them actions ?

If 1. does not fix it, the shifter would have to skip the action at 1. on the second turn -- is this possible ?

Is see that we already have the following in the code: https://github.com/cmsdaq/DAQExpert/blob/667bd1e4bd9b8178c4b6920afdd20cb750145085/src/main/java/rcms/utilities/daqexpert/reasoning/logic/failures/FlowchartCase5.java#L35

so my guess is that (if confirmed) we can also change these to (semi-)automatic actions.

[gladky]

That's right @andreh12 , this is indeed good candidate for update.

[gladky]

@giacomoCucciati could you please comment on this one?

[giacomoCucciati]

About this issue, it is a while we don't see it. But I would keep this instruction in any case. Small correction, the stop&start at 1) will not probably work so it could be just downtime. If the DAQ shows 3 or more consecutive SoftErrorRecovery we should go directly for a red recycle.

Is it possible to do this modification?

[gladky]

@giacomoCucciati thank you! I will update it to this form:

1. stop the run and red-recycle ES

[gladky]

Updated recovery procedure in 2.12.2, moved to section "covered by expert"