ScalerMotorFlyer problems after initial user testing

[prjemian]

Tested the new ScalerMotorFlyer today with a servo motor (mr) on the USAXS instrument and encountered some problems.

• [x] 1. How to change the start, finish, and other parameters after the flyer device is created?
• [x] 2. Can the motor and/or scaler be changed after the device is created?
• [x] 3. Resolve this TODO item using self._motor.precision
• [x] 4. Make certain the scaler is NOT in autocount mode
• [x] 5. User asks what are the units of period? (Ambiguity regarding specifying time parameters or number of points.)
• [ ] 6. During the test scans, the scaler was not counting when the motor was moving from start to finish.
• [ ] 7. When the fly scan became stuck, one or more ^C did not interrupt the RunEngine. Only a ^\ could terminate the IPython console session.
• [x] 8. With fly_time_pad=2, got a Status object timeout. Increased to 10 and no timeout.
• [ ] 9. Motor velocity not reset when fly scan got stuck (such as when not exactly in taxi position, item 3)

[prjemian]

Addressing items 1 & 2 above, a wrapper plan was created locally that reconfigures a common flyer object, where scaler, motor, and other necessary parameters are configured. Then, yield from bp.fly([flyer], md=md) is called.

def my_fly_plan(scaler, motor, start, finish, fly_time, period, fly_time_pad=2, scaler_time_pad=2, md={}):
    flyer._motor = motor
    flyer._pos_start = start
    flyer._pos_finish = finish
    flyer._fly_time = fly_time
    flyer._fly_time_pad = fly_time_pad
    flyer._scaler = scaler
    flyer._scaler_time_pad = scaler_time_pad
    flyer._period = period

    yield from fly_with_stats([flyer], md=md)

[prjemian]

Item 3 was resolved locally as suggested by converting:

if self._motor.position != self._pos_start:  # TODO: within tolerance?

into

if round(self._motor.position - self._pos_start, self._motor.precision) != 0:

and adjusting the motor's .PREC field.

NOTE: Since ophyd only collects the motor.PREC field from the channel metadata, this value is only set when the object is first connected and will not update even if motor.PREC field is updated. A restart of the session solves this.

[prjemian]

Item 4 will be implemented in method _action_fly().

[prjemian]

@jilavsky:

Item 6 was caused by an incorrect local override of actions_thread() to fix item 3 (right fix, wrong method to override):

class MyScalerMotorFlyer(ScalerMotorFlyer):
    @run_in_thread
    def actions_thread(self):
        ...

which replaced: https://github.com/BCDA-APS/apstools/blob/8cd4236115d6dc913766f0a9679d476eb74656d0/apstools/devices/flyer_motor_scaler.py#L258

This removed the necessary parts of the ScalerMotorFlyer(). It should have replaced this identical looking line: https://github.com/BCDA-APS/apstools/blob/8cd4236115d6dc913766f0a9679d476eb74656d0/apstools/devices/flyer_motor_scaler.py#L341

which is part of the _action_taxi() method.

[prjemian]

Correct override should be:

class MyScalerMotorFlyer(ScalerMotorFlyer):

    def _action_taxi(self):
        """Move motor to start position."""
        self.mode.put("taxi")
        self.status_taxi = self._motor.move(self._pos_start, wait=False)
        self.status_taxi.wait()

        # arrived to within motor's precision?
        if round(self._motor.position - self._pos_start, self._motor.precision) != 0:
            raise RuntimeError(
                "Not in requested taxi position within precision:"
                f" requested={self._pos_start}"
                f" position={self._motor.position}"
                f" precision={self._motor.precision} digits"
            )

[prjemian]

Item 4 is resolved by an override of the _action_fly() method:

    def _action_fly(self):
        """
        Start the fly scan and wait for it to complete.

        Data will be accumulated in response to CA monitor events from the
        scaler.
        """
        import time

        BRIEF_PAUSE = 0.2

        self.mode.put("fly")

        # set the fly scan velocity
        velocity = abs(self._pos_finish - self._pos_start) / self._fly_time
        if velocity != self._motor.velocity.get():
            self._original_values.remember(self._motor.velocity)
            self._motor.velocity.put(velocity)

        # make the scaler idle
        self._original_values.remember(self._scaler.count)
        self._original_values.remember(self._scaler.count_mode)
        self._scaler.count_mode.put("OneShot")  # turn off auto count mode
        self._scaler.count.put("Done")  # stop the scaler from counting

        # set the scaler count time (allowance)
        self._original_values.remember(self._scaler.preset_time)
        count_time_allowance = self._fly_time + self._scaler_time_pad
        self._scaler.preset_time.put(count_time_allowance)

        # start acquiring, scaler update rate was set in _action_setup()
        self._scaler.time.subscribe(self._action_acquire_event)  # CA monitor

        # start scaler counting, THEN motor moving
        self._scaler.count.put("Count")
        self.status_fly = self._motor.move(self._pos_finish, wait=False)

        # wait for motor to be done moving
        motion_time_allowance = self._fly_time + self._fly_time_pad
        self.status_fly.wait(timeout=motion_time_allowance)
        self._action_acquire_event()  # last event

        self._scaler.count.put("Done")  # stop scaler counting
        self._scaler.time.unsubscribe_all()  # stop acquiring

[prjemian]

We did not test with a2rp which is a PZT, not a servo.

[prjemian]

Note that the my_fly_plan() could create its own flyer object (internal use only) with a custom device name. Instead of modifying a common object.

[prjemian]

This was a successful run in that it completed with a peak.

In [28]: run
Out[28]: 
BlueskyRun
  uid='729265e6-213f-4b2f-a52f-1bd5a755cc98'
  exit_status='success'
  2022-12-01 16:13:06.195 -- 2022-12-01 16:13:24.090
  Streams:
    * flyer_stats
    * primary

[prjemian]

[prjemian]

In [24]: listruns(num=40)
/home/beams11/USAXS/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/databroker/queries.py:89: PytzUsageWarning: The zone attribute is specific to pytz's interface; please migrate to a new time zone provider. For more details on how to do so, see https://pytz-deprecation-shim.readthedocs.io/en/latest/migration.html
  timezone = lz.zone
catalog: 20idb_usaxs
    scan_id                 time    plan_name   detectors
0         3  2022-12-01 16:41:54  my_fly_plan            
1         2  2022-12-01 16:40:34  my_fly_plan            
2         1  2022-12-01 16:39:56  my_fly_plan            
3         1  2022-12-01 16:35:09  my_fly_plan            
4         8  2022-12-01 16:31:58  my_fly_plan            
5         7  2022-12-01 16:31:21  my_fly_plan            
6         6  2022-12-01 16:23:29  my_fly_plan            
7         5  2022-12-01 16:21:28  my_fly_plan            
8         4  2022-12-01 16:13:06  my_fly_plan            
9         3  2022-12-01 16:11:30  my_fly_plan            
10        2  2022-12-01 16:09:51  my_fly_plan            
11        1  2022-12-01 16:07:05  my_fly_plan            
12        6  2022-12-01 15:56:32  my_fly_plan            
13        5  2022-12-01 15:55:53  my_fly_plan            
14        4  2022-12-01 15:53:11  my_fly_plan            
15        3  2022-12-01 15:50:46  my_fly_plan            
16        2  2022-12-01 15:49:52  my_fly_plan            
17        1  2022-12-01 15:47:57  my_fly_plan            
18        3  2022-12-01 15:46:49  my_fly_plan            
19        2  2022-12-01 15:46:14  my_fly_plan            
20        1  2022-12-01 15:44:13  my_fly_plan            
21      453  2022-12-01 14:45:10    tune_a2rp  [PD_USAXS]
22      452  2022-12-01 14:44:58      tune_ar  [PD_USAXS]
23      451  2022-12-01 14:44:45      tune_mr  [I0_USAXS]
24      450  2022-12-01 14:42:19    tune_a2rp  [PD_USAXS]
25      449  2022-12-01 14:42:07      tune_ar  [PD_USAXS]
26      448  2022-12-01 14:41:54      tune_mr  [I0_USAXS]
27      447  2022-12-01 14:41:30    tune_a2rp  [PD_USAXS]
28      446  2022-12-01 14:41:19      tune_ar  [PD_USAXS]
29      445  2022-12-01 14:41:04      tune_mr  [I0_USAXS]
30      444  2022-12-01 14:35:03    tune_a2rp  [PD_USAXS]
31      443  2022-12-01 14:34:51      tune_ar  [PD_USAXS]
32      442  2022-12-01 14:34:38      tune_mr  [I0_USAXS]
33      441  2022-12-01 14:31:52      tune_dy  [PD_USAXS]
34      440  2022-12-01 14:31:38      tune_dx  [PD_USAXS]
35      439  2022-12-01 14:31:05      tune_ar  [PD_USAXS]
36      438  2022-12-01 14:30:46    tune_a2rp  [PD_USAXS]
37      437  2022-12-01 14:24:08      tune_mr  [I0_USAXS]
38      436  2022-12-01 14:23:49      tune_mr  [I0_USAXS]
39      435  2022-12-01 14:20:19    tune_a2rp  [PD_USAXS]

[prjemian]

With @jilavsky, more testing today, noticed that we have observed some pathologies:

• first point is sometimes very negative
• usually get multiple scaler updates per motor update
• sometimes motor position does not update for much longer (see chart above)

EPICS Channel Access updates could be part of the problem (such as first & third bullets above), the update rate of the EPICS motor record is likely at the heart of the second bullet point.

@kmpeters: Can the update rate of motor .RBV field be changed by a CA user? How is this controlled?

[MarkRivers]

@kmpeters: Can the update rate of motor .RBV field be changed by a CA user? How is this controlled?

I think it is controlled by a value you give in the st.cmd file when creating the motor controller object.

[prjemian]

We're observing a motor .RBV update rate of ~5 Hz for both the servo motor and the PZT we tested. And then, the occasional period when it does not update at all.

[prjemian]

Test code: https://github.com/APS-USAXS/usaxs-bluesky/blob/master/user/test_flyer.py

[kmpeters]

We're observing a motor .RBV update rate of ~5 Hz for both the servo motor and the PZT we tested. And then, the occasional period when it does not update at all.

Both the Automation1 and PIGCS2 support are configured for moving poll frequencies of 10 Hz.

[prjemian]

Documentation for items 1 & 2 could be either in the source code or in the demo notebook. Probably more useful in the source code since there is no standalone .rst file for the module or class. (That might be another re-org of the documentation.)

[prjemian]

6. During the test scans, the scaler was not counting when the motor was moving from start to finish.

Is this fixed by ensuring count_mode="OneShot"?

[prjemian]

Per https://github.com/BCDA-APS/apstools/pull/777#issuecomment-1379151886, moving to 1.6.11 milestone and project to complete items 6, 7, & 9 (above).

[prjemian]

Work on this issue in preparation for the 1.6.10 milestone has been completed. Further work is scheduled for a future milestone now. The issue remains open.

[prjemian]

No access to the servo motor at this time.

[jilavsky]

No access to the servo motor at this time.

USAXS motors are still up with soft ioc and epics running. usaxscontrol is also up and while starting BlueSky may be challenge, motor hardware as well as LAX counting system is available. No amplifies, range changes etc. But if you need to scan motors, they are ready for @kmpeters. Kevin should be developing flyscanning for Automation1, so coordinate if you need to use the hardware. Do not expect much help from me - in Europe for June; I can remotely do something, but we do not have anyone on site with any understanding.