FailedStatus from ophyd while writing array to EpicsSignal

[prjemian]

We want to write to some array PVs (such as positioner arrays in the sscan record, array fields in the array calcs). We know we can do this with the EpicsSignal.put() method. It works. What fails is the QA process associated with the .set() method that assures the EPICS PV has reached the values we wrote.

When writing an array (list, np.ndarray, tuple) to an EpicsSignal connected to such a PV, the ophyd code stops with an FailedStatus exception that points to this chain:

• ophyd.utils._wait_for_value() line 295
• ophyd.utils._compare_maybe_enum() line 341
• np.core.numeric.allclose() line 2241
• np.core.numeric.isclose() line 2332, which fails due to abs(x-y)

One failure is that the two arrays passed must have the same shape (or this operation in numpy fails).

Another part of the failure is that EPICS Channel Access (used by EpicsSignal) always returns (since we have not told it to return less) the full array, not just the limited part we wanted to write.

Demonstrate the problem

Such an EPICS PV: pjgp:userArrayCalc1.AA

(bluesky_2024_2) jemian@otz ~ $ cainfo pjgp:userArrayCalc1.AA
pjgp:userArrayCalc1.AA
    State:            connected
    Host:             otz.xray.aps.anl.gov:5064
    Access:           read, write
    Native data type: DBF_DOUBLE
    Request type:     DBR_DOUBLE
    Element count:    8000

ophyd 1.9.0, numpy 1.26.4

In [1]: from ophyd import EpicsSignal

In [2]: pv = EpicsSignal("pjgp:userArrayCalc1.AA", name="pv")

In [3]: pv.connected
Out[3]: True

In [4]: len(pv.get())
Out[4]: 8000

In [5]: pv.get()[:10]
Out[5]: array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])

In [6]: pv.put([1,2,3,4])

In [7]: pv.get()[:10]
Out[7]: array([1., 2., 3., 4., 0., 0., 0., 0., 0., 0.])

Show the problem using EpicsSignal.set():

In [8]: pv.set([5,4,3,2,1])
Out[8]: Status(obj=EpicsSignal(read_pv='pjgp:userArrayCalc1.AA', name='pv', value=array([1., 2., 3., ..., 0., 0., 0.]), timestamp=631152000.0, tolerance=1e-05, auto_monitor=False, string=False, write_pv='pjgp:userArrayCalc1.AA', limits=False, put_complete=False), done=False, success=False)

pv: _set_and_wait(value=[5, 4, 3, 2, 1], timeout=None, atol=1e-05, rtol=None, kwargs={})
Traceback (most recent call last):
  File "/home/beams/JEMIAN/.conda/envs/bluesky_2024_2/lib/python3.11/site-packages/ophyd/signal.py", line 331, in set_thread
    self._set_and_wait(value, timeout, **kwargs)
  File "/home/beams/JEMIAN/.conda/envs/bluesky_2024_2/lib/python3.11/site-packages/ophyd/signal.py", line 302, in _set_and_wait
    return _set_and_wait(
           ^^^^^^^^^^^^^^
  File "/home/beams/JEMIAN/.conda/envs/bluesky_2024_2/lib/python3.11/site-packages/ophyd/utils/epics_pvs.py", line 240, in _set_and_wait
    _wait_for_value(
  File "/home/beams/JEMIAN/.conda/envs/bluesky_2024_2/lib/python3.11/site-packages/ophyd/utils/epics_pvs.py", line 295, in _wait_for_value
    while (val is not None and current_value is None) or not _compare_maybe_enum(
                                                             ^^^^^^^^^^^^^^^^^^^^
  File "/home/beams/JEMIAN/.conda/envs/bluesky_2024_2/lib/python3.11/site-packages/ophyd/utils/epics_pvs.py", line 341, in _compare_maybe_enum
    return np.allclose(
           ^^^^^^^^^^^^
  File "/home/beams/JEMIAN/.conda/envs/bluesky_2024_2/lib/python3.11/site-packages/numpy/core/numeric.py", line 2241, in allclose
    res = all(isclose(a, b, rtol=rtol, atol=atol, equal_nan=equal_nan))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/beams/JEMIAN/.conda/envs/bluesky_2024_2/lib/python3.11/site-packages/numpy/core/numeric.py", line 2351, in isclose
    return within_tol(x, y, atol, rtol)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/beams/JEMIAN/.conda/envs/bluesky_2024_2/lib/python3.11/site-packages/numpy/core/numeric.py", line 2332, in within_tol
    return less_equal(abs(x-y), atol + rtol * abs(y))
                          ~^~
ValueError: operands could not be broadcast together with shapes (5,) (8000,) 

numpy demo

In [9]: import numpy as np
In [11]: np.allclose(np.array([1,2,3,4]), np.array([4,3,2,1]))
Out[11]: False

In [12]: np.allclose(np.array([1,2,3,4]), np.array([4,3,2,1, 6, 6, 6]))
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[12], line 1
----> 1 np.allclose(np.array([1,2,3,4]), np.array([4,3,2,1, 6, 6, 6]))

File ~/.conda/envs/bluesky_2024_2/lib/python3.11/site-packages/numpy/core/numeric.py:2241, in allclose(a, b, rtol, atol, equal_nan)
   2170 @array_function_dispatch(_allclose_dispatcher)
   2171 def allclose(a, b, rtol=1.e-5, atol=1.e-8, equal_nan=False):
   2172     """
   2173     Returns True if two arrays are element-wise equal within a tolerance.
   2174 
   (...)
   2239 
   2240     """
-> 2241     res = all(isclose(a, b, rtol=rtol, atol=atol, equal_nan=equal_nan))
   2242     return bool(res)

File ~/.conda/envs/bluesky_2024_2/lib/python3.11/site-packages/numpy/core/numeric.py:2351, in isclose(a, b, rtol, atol, equal_nan)
   2349 yfin = isfinite(y)
   2350 if all(xfin) and all(yfin):
-> 2351     return within_tol(x, y, atol, rtol)
   2352 else:
   2353     finite = xfin & yfin

File ~/.conda/envs/bluesky_2024_2/lib/python3.11/site-packages/numpy/core/numeric.py:2332, in isclose.<locals>.within_tol(x, y, atol, rtol)
   2330 def within_tol(x, y, atol, rtol):
   2331     with errstate(invalid='ignore'), _no_nep50_warning():
-> 2332         return less_equal(abs(x-y), atol + rtol * abs(y))

ValueError: operands could not be broadcast together with shapes (4,) (7,) 

[prjemian]

While the documentation of numpy.allclose() that says the shapes can be different should be fixed, there is still the problem that EPICS returns the full array. Ophyd needs to trim that EPICS array down to the size of the target array.

Both fixes can be applied by inserting code just before this line: https://github.com/bluesky/ophyd/blob/9c77d86cd7c3b1c6e34d88b9b3eed27063c9b616/ophyd/utils/epics_pvs.py#L338

I propose inserting these lines:

    array_like = (list, np.ndarray, tuple)
    if isinstance(a, array_like) and isinstance(b, array_like):
        # 2024-07-30, prj: 
        # np.allclose(a, b) fails when both a & b are different shaped arrays
        # If only one is a numpy array, then np.allclose does not fail.
        # np.allclose() calls np.isclose() which has a comment that states
        # the two arrays "must be the same shape."
        a = np.array(a)  # target
        b = np.array(b)  # reported by EPICS
        if len(a.shape) == 1 and len(b.shape) == 1 and len(a) < len(b):
            # Some EPICS arrays always return full size, even if only less is written.
            # EPICS CA arrays are always 1-D.
            b = b[:len(a)]  # cut 1-D EPICS array down to requested size

        if a.shape != b.shape:
            return False

[prjemian]

A fix here is critical to our fly scans at the APS.

[prjemian]

With my fix added locally, the EpicsSignal.set() method does not raise the FailedStatus exception for arrays with different lengths.

[tacaswell]

If you put across CA to an array with less than the full array is the expected semantics:

• "here is your whole new array, please reduce your size to match"
• "here is a partial update, overwrite just the start of the array but leave the rest the same"

?

When we "put" to an array should we be resetting the size as well or is the issue that the monitor is providing up to max size rather than the current size?

[prjemian]

When we "put" to an array should we be resetting the size as well or is the issue that the monitor is providing up to max size rather than the current size?

Neither, actually. The current .put() works properly. The .get() always returns the full array (the max size). But the comparison should always truncate the array from .get() to the length of the provided array (the one sent to .put()) for the comparison of Did we get there yet?