Special instructions: TRACKER FED Stuck

[gladky]

From the Shifter Bulletin

TRACKER (2016-08-14 Erik Butz)

1. if tracker FED is stuck in 100% WARNING or BUSY: try TTS resync
2. if this doesn't solve the problem: stop the run, red recycle tracker and start new run
3. if also this does not resolve the problem: call the tracker DOC 165503 (backup=77704)
4. If a FED goes into disconnected, Tracker DOC recommends no to do a red-recycle but to call them directly. They will probably need to be put out to reboot the FED. Jose Ruiz 20-10-2016

In expert we have currently default instruction for FED Stuck:

1. StopAndStartTheRun with <<RedRecycle::{{SUBSYSTEM}}>> and <<GreenRecycle::{{SUBSYSTEM}}>> (try up to 2 times)",
2. "Problem fixed: Make an e-log entry. Call the DOC of the subsystem {{SUBSYSTEM}} to inform",
3. "Problem not fixed: Call the DOC for the subsystem {{SUBSYSTEM}}");

Current tracker specific instruction are as follows (only tracker in warn):

1. "Issue a TTCHardReset once",
2. "Problem fixed: Make an e-log entry.",
3. "Problem not fixed: Stop the run, start a new run"
4. "Problem still not fixed: Stop the run, red recycle TRACKER, start a new run",
5. "Call the DOC for the TRACKER")

With @andreh12 we've just discussed to remove TRACKER-WARNING special case and replace it with TRACKER special case that will cover both WARN and BUSY states.

Waiting for confirmation from TRACKER that this is what they want.

[andreh12]

Thanks for summarizing this !

I just had the tracker expert on the phone and he confirms that they want the message for both WARNING and BUSY, i.e. we can remove the condition for WARNING only.

[andreh12]

added to release 2.10.3

[gladky]

Closing the issue and removing this from the Shifter Bulletin

[gladky]

Note: The last part of the instructions:

If a FED goes into disconnected, Tracker DOC recommends no to do a red-recycle but to call them directly. They will probably need to be put out to reboot the FED. Jose Ruiz 20-10-2016

is compatible with or LM "FEDDisconnected" where we suggest to:

Check with the {{PROBLEM-SUBSYSTEM}} subsystem DOC

[andreh12]

on removing this from the shifter instructions: I'd be more in favour of leaving instructions which have been integrated into the DAQExpert there but change their color (e.g. a pale grey) and write somewhere that this color means 'implemented in DAQExpert'.

Otherwise subsystem experts may wonder why these have disappeared and put them there again etc.

[gladky]

The whole point was to make this list easier to read. Just yesterday we've realised that shifters do NOT read it carefully - few of the were not aware of new automatic recoveries. Maybe it would be enough to inform the subsystems about the fact that we've removed it and they are available in DAQExpert. Later we are planning to publish current DAQExpert logic capabilities in some more readable format than code for subsystems to verify what's currently in the system. @hsakulin @mommsen comments on that?

[andreh12]

I have been suspecting since long time that shifters will not read it carefully when it becomes too complex :-)

What about moving the implemented instructions to a separate (new) Twiki page (at least for the moment until we publish the logic content) ?

[hsakulin]

I like the idea of having two categories of instructions (on two pages or in two separate sections on the same page):

1) Not (yet) implemented in the DAQExpert or superseding DAQExpert

2) Implemented in the DAQExpert

The latter should always be in sync with what is deployed. Best would be if it could be auto-generated.

On 04 May 2018, at 12:22, andreh12 notifications@github.com wrote:

I have been suspecting since long time that shifters will not read it carefully when it becomes too complex :-)

What about moving the implemented instructions to a separate (new) Twiki page ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[gladky]

Ok. Do you want me to create this page or you do it yourself? I will then update it with instruction I removed before.

[hsakulin]

I would tend to keep it in a separate section on the same page. Like this it is easier to search for instructions.

This section should be visually different and should start with a statement saying that this section should NOT be edited by shifters.

However I would only make the change on the page visible to the shifters once you went through all instructions.

On 04 May 2018, at 14:48, Maciej Gladki notifications@github.com wrote:

Ok. Do you want me to create this page or you do it yourself? I will then update it with instruction I removed before.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.