search

HCALRunControl / levelOneHCALFM

HCAL Function Manager
https://twiki.cern.ch/twiki/bin/view/CMS/HCALFunctionManager
0 stars 6 forks source link

Start TA during starting from supervisor #416

Closed kakwok closed 6 years ago

kakwok commented 6 years ago

We used to have IgnoreTriggerForEnable=True in supervisor and FM starts the TA during running action. This should no longer be needed with EvmTrigFM scheduled to start last. Supervisor can start TA last during starting to officially start the run, FM then transit to running state when supervisor updated its global state to running.

Tested at 904 for (click-stop + auto-stop) x (multi + single partition run) Tested at P5 for sequencer runs with a series of click-stop (Run 315283). I have not encountered any stopping issue with click-stop and auto-stop, but the logs need to be studied in details to make sure it's working correctly.

kakwok commented 6 years ago

Supervisor seems to go through a A->R->D->R transition in this case. A->R because TA went to ready from click-stop R->D because FM issued disable D->R again after finishing disable. However, FM completed the stopping->configured transition based on the notification of A->R, so FM declared configured while supervisor are still disabling the apps. While this does not cause error, this is not ideal. 

2018-04-27 13:55:33 and 530 ms : hcal.cms.hcalutca01.p:28122.DummyTriggerAdapter.instance(2)
      INFO New state is:Ready
 2018-04-27 13:55:33 and 822 ms : hcal.cms.hcalutca01.p:28001.hcalSupervisor.instance(2)
      INFO Global state transition from A to R
 2018-04-27 13:55:34 and 192 ms : cms.hcalpro.rcms.fm.app.level1.HCALlevelTwoEventHandler
      INFO [HCAL LVL2 HCAL_HF]   Sending AsyncDisable to supervisor
 2018-04-27 13:55:34 and 194 ms : hcal.cms.hcalutca01.p:28001.hcalSupervisor.instance(2)
      INFO New state is:Disable
 2018-04-27 13:55:34 and 476 ms : cms.hcalpro.rcms.fm.app.level1.HCALEventHandler
      INFO [SethLog HCAL HCAL_HF] 2 received id: http://hcalutca01.cms:28001/urn:xdaq-application:lid=50, ToState: Ready
 2018-04-27 13:55:35 and 424 ms : hcal.cms.hcalutca01.p:28001.hcalSupervisor.instance(2)
      INFO New state is:Ready
kakwok commented 6 years ago

Auto-stop seems to work smoothly without above problem. Supervisor went through A->R only.

sethcooper commented 6 years ago

For click stop, we need to not stop the TA separately, since we are no longer starting it separately in the start sequence. This would make the auto-stop and click stop more symmetric as well. The auto-stop should have this same issue in general—I think it's just racing now so it doesn't always happen. This also means I think that the TA should not go to Ready by itself, but needs to signal in some other way that the run is done and the FM should stop its supervisor. (Probably the FM can just check the number of events requested/taken and stop on that condition.)

kakwok commented 6 years ago

To be worked with hcos MR https://gitlab.cern.ch/cmshcos/hcal/merge_requests/317 Release 14.9.2+