search

cmsdaq / DAQExpert

New expert system processing data model produced by DAQAggregator
1 stars 2 forks source link

Special instructions: general - tracker APV OOS #215

Open gladky opened 6 years ago

gladky commented 6 years ago
  • If Tracker APVes are Out-Of-Sync at the start of a run, send a Resync
gladky commented 6 years ago

Contacted Erik Butz

erikbutz commented 6 years ago

The APVe's are part of TCDS now, so I don't know if they can still be in this particular state. On the other hand, if they are OOS, sending a resync is the way to go, so I think we can include this in the DAQ expert.

jhegeman commented 6 years ago

This must be out of date since a very long time. I don't think this actually happens any more with the TCDS-internal APVE implementation. Unless we observe this behaviour again, I would not even add this to the DAQExpert.

erikbutz commented 6 years ago

I don't object. If we think that the feature is sufficiently seldom to warrant a closer look rather than a standard action, we can leave it out.

hsakulin commented 6 years ago

Tracker APVe should be checked by the DAQ Expert in the same way as iCIs are checked.

For tracker partitions both the APVe and the iCI contribute to the over-all partition state. (Both are shown in DAQView). Whether the iCI or the APVe is stuck in Out-Of-Sync, a resync should be the first thing tried.

This might be implemented as sub-case of FC5.

jhegeman commented 6 years ago

Hi Hannes,

The DAQExpert should indeed check the APVEs, but please don't treat the APVEs in the same way as ICIs. They do behave differently.

Jeroen

hsakulin commented 6 years ago

Hi Jeroen, If I am not mistaken a resync works also for the APVe (And I think we sometimes had to use this even after the APVe was absorbed into TCDS). Can you give us any other instructions specific to the APVe? Hannes.

jhegeman commented 6 years ago

The TTS behaviour of the APVE lies closer to the original intention of the different TTS states than the tracker FED. So the original rules apply: a Resync for OOS, a HardReset (or, more likely, a red-recycle) for ERROR, etc. (So no Resync for WARNING, for example.)

hsakulin commented 6 years ago

The APVes are controlled by the TRACKER subsystem. So the red-recycle would be needed on TRACKER if the APVe was stuck in ERROR (and was not recovered by a hard reset).

The tracker instruction in FC5 would still fit the APVe quite well. (No matter in what state the partition is stuck, first try a hard reset, then a red recycle). We may do an unnecessary hard reset in case the APVe was stuck in W/B but the case probably never appears anyways).

jhegeman commented 6 years ago

Sure, let's use a HardReset then. We have come a long way from the original feature/instruction, but it does allow the DAQExpert to simply treat the APVEs just like the other tracker partitions.

gladky commented 6 years ago

Related instructions:

  • Strip Tracker
    • if APVEs go out-of-sync at the start of the run: Sending a TTC resync command should solve the issue (it might be necessary to retry a few times). If not, stop the run and redrecycle tracker.
      After the out-of-sync has gone away, it happens from time to time that one the tracker iCl's flicker between ready and busy states. If that's the case, stop the run and redrecycle tracker.
jhegeman commented 6 years ago

Hi Maciej,

Some small corrections:

Jeroen