test suspender for PLC protection

[prjemian]

test the suspender for the PLC protection implemented in #69

[prjemian]

Suspends on any of these:

suspend_plc_protect_saxs_y = SuspendWhenChanged(plc_protect.SAXS_Y, expected_value=1)
suspend_plc_protect_waxs_x = SuspendWhenChanged(plc_protect.WAXS_X, expected_value=1)
suspend_plc_protect_ax = SuspendWhenChanged(plc_protect.AX, expected_value=1) 

# these PVs:
In [11]: plc_protect.SAXS_Y.pvname
Out[11]: '9idcLAX:plc:X11'

In [12]: plc_protect.WAXS_X.pvname
Out[12]: '9idcLAX:plc:X12'

In [13]: plc_protect.AX.pvname
Out[13]: '9idcLAX:plc:X13'

@kmpeters: Any idea how I can make these change to test how Bluesky will respond? Can we temporarily switch the PLC off or some other type of test?

[prjemian]

@jilavsky responded by offline email

I am not sure this is correct - depends how you are trying to implement these suspenders - if they will work on smaller subset of the code. These PVs routinely change as you move to specific state. So ax changes to 0 when USAXS is in beam etc. Basically, except for “Empty beam” state, there is always one in 0 state.

At this moment we are in WAXS mode and X12 = 0, others =1.

The sensors (X11, X12, X13) work as hard limits preventing other motors to move from “home” position unless the other two are in home position. Most of the time one of these switches is “off” as one of the motors is not in home position.
Since the X11, X12, X13 look to epics as hard limits of motors, BlueSky should get report from epics that motors have hit their hard limits and cannot move.

Note that in spec we have built in “wait for limit switch” function as these limit switches have been found to be kind of slow to report and it is common to see 0.1-0,2 seconds delay needed…

Basically, now waxs_x is out of home position. You cannot move ax negative and pin_y negative to protect instrument from move from home position. This is done by hardware which breaks limit switches on the cables.

---

Important: You need to watch for PLC “failed state” which is 9idcLAX:plc:Y0 (OK=1, Failed=0). This is when user breaks the lights and all motors loose power. We need to suspend operations here (and ideally send staff e-mail that we have a failure). This is real emergency case and needs to be dealt with - typically with human at the instrument… Basically, this case must be dealt with and BlueSky should stop everything since we already took away power from motors anyway and BS (cool acronym for BlueSky ;-) would get out of sync from real positions.

[prjemian]

I believe you are right. The Bluesky Suspender expects a Signal and here we are watching three (or possibly four). This implementation needs change before it works the way we expect.

[prjemian]

observation of how these PVs change (using camonitor and ignoring other changes in 9idcLAX:USAXS:state):

9idcLAX:plc:Y0                 2018-11-21 13:48:11.360218 High STATE MAJOR
9idcLAX:plc:X11                2018-12-04 11:11:31.709282 High STATE MAJOR
9idcLAX:plc:X12                2018-12-04 11:28:37.188613 High STATE MAJOR
9idcLAX:plc:X13                2018-12-04 11:28:39.247557 Low  
9idcLAX:USAXS:state            2018-12-04 12:34:06.067926 Moving USAXS to OpenBeamPath mode  
9idcLAX:plc:X13                2018-12-04 12:34:17.819914 High STATE MAJOR
9idcLAX:USAXS:state            2018-12-04 12:34:19.601259 USAXS moved to OpenBeamPath mode  
9idcLAX:USAXS:state            2018-12-04 12:37:00.554057 Moving USAXS to SAXS mode  
9idcLAX:plc:X11                2018-12-04 12:37:01.124131 Low  
9idcLAX:USAXS:state            2018-12-04 12:37:08.370723 SAXS Mode  
9idcLAX:USAXS:state            2018-12-04 12:42:10.875288 Moving USAXS to USAXS mode  
9idcLAX:plc:X11                2018-12-04 12:42:19.711745 High STATE MAJOR
9idcLAX:plc:X13                2018-12-04 12:42:20.997050 Low  
9idcLAX:USAXS:state            2018-12-04 12:42:33.475287 USAXS Mode  
9idcLAX:USAXS:state            2018-12-04 12:43:20.942940 Moving USAXS to OpenBeamPath mode  
9idcLAX:plc:X13                2018-12-04 12:43:32.777656 High STATE MAJOR
9idcLAX:USAXS:state            2018-12-04 12:43:34.476272 USAXS moved to OpenBeamPath mode  
9idcLAX:USAXS:state            2018-12-04 12:48:31.164178 Moving USAXS to Radiography mode  
9idcLAX:plc:X13                2018-12-04 12:48:32.526258 Low  
9idcLAX:USAXS:state            2018-12-04 12:48:51.365065 Radiography Mode  
9idcLAX:USAXS:state            2018-12-04 12:57:47.489962 Moving USAXS to SAXS mode  
9idcLAX:plc:X13                2018-12-04 12:57:59.432389 High STATE MAJOR
9idcLAX:plc:X11                2018-12-04 12:58:01.496235 Low  
9idcLAX:USAXS:state            2018-12-04 12:58:09.889962 starting SAXS  
9idcLAX:USAXS:state            2018-12-04 12:58:31.039961 Moving USAXS to WAXS mode  
9idcLAX:plc:X11                2018-12-04 12:58:39.842234 High STATE MAJOR
9idcLAX:plc:X12                2018-12-04 12:58:40.613231 Low  
9idcLAX:USAXS:state            2018-12-04 12:58:53.624252 Preparing for waxs  

[kmpeters]

@kmpeters: Any idea how I can make these change to test how Bluesky will respond? Can we temporarily switch the PLC off or some other type of test?

@prjemian, the easiest way to test is probably to use the 30-second override of the PLC protection to move one of the motors into an unsafe state. This would allow changing the state of the limit switch by hand.

[prjemian]

I'll work this up in a bluesky simulator (EPICS softIoc with a bunch of bo records and a readable object as a detector) and start counting the detector signal. Meanwhile, the EPICS bo records can be individually manipulated from a GUI such as caQtDM.

[prjemian]

simulator will be on a different subnet than USAXS so as to not disturb operations

[prjemian]

it's obvious we'll need this suspender for operations then:

suspend_plc_protect_operations_status = SuspendWhenChanged(
    plc_protect.operations_status, 
    expected_value=1)
RE.install_suspender(suspend_plc_protect_operations_status)

Also, need to modify stop_if_tripped() to send an email to staff ASAP, per comment above.

The individual suspenders on X11, X12, and X13 will need more work.

[prjemian]

Right. No suspenders on X11, X12, or X13. Otherwise, we could not change modes during a bluesky plan -- the mode change would suspend as soon as they all become High (an interim condition during any mode change).

The wait_for_interlock() method (a bluesky plan) has the desired logic already. That method will also need a yield from bps.null() either before or after the while block to ensure the plan emits at least one bluesky Msg.

[prjemian]

need to ensure that yield from plc_protect.wait_for_interlock() is called during mode changes

[prjemian]

verified:

usaxs@usaxscontrol ~/.../profile_bluesky/startup $ grep wait_for_interlock *.py
28-protection_plc.py:    def wait_for_interlock(self, verbose=True):
41-commands.py:    yield from plc_protect.wait_for_interlock()
41-commands.py:    # - should not be needed, we have plc_protect.wait_for_interlock() now. 
41-commands.py:    yield from plc_protect.wait_for_interlock()
41-commands.py:    yield from plc_protect.wait_for_interlock()

Such as:

def move_SAXSIn():
    # yield from plc_protect.stop_if_tripped()
    yield from bps.mv(
        ccd_shutter,        "close",
        ti_filter_shutter,  "close",
    )

    print("Moving to Pinhole SAXS mode")

    confirmUsaxsSaxsOutOfBeam()
    yield from plc_protect.wait_for_interlock()

[prjemian]

This code will be tested during #59

[prjemian]

there's more work to do on this, as shown from the simulator