Discussion: supporting structured-data control systems (pvAccess, Tango, etc.)

[klauer]

Supporting Structured-data Control Systems

We have gone back and forth about supporting control system protocols that are not EPICS V3 in ophyd: pvAccess, tango, etc.. Let's do what we can to come to an agreement (or at least an understanding) here in this issue.

Note that while the examples use Tango, the discussion has nothing to do with their API (I admittedly don't know much about it). The assumption is solely that these control systems supply structured data in some form, and it's up to us to shape it for our purposes.

Regardless of the choice that's made, the bluesky interface (which defines the interactions between ophyd and bluesky for the purposes of data collection) will not change. That is to say, whether a Tango device maps to an ophyd Device or Signal or Foobar, it does not matter from the perspective of bluesky. So, this discussion is solely about how to best handle the hardware abstraction in a way that makes sense in ophyd and for the user.

Stance 1: Blur the lines between Device/Signal

@tacaswell suggests:

I have previously argued that the distinction between Device and Signal should be made fuzzier, particularly as we move to v4 (or tango).

I think this could result in 2 paths (correct me if I'm wrong or there are other possibilities):

(1) TangoDevice and PvaDevice become their own thing, almost entirely unrelated to the currently existing Device. ophyd.Device is built around the assumption that it is comprised of various Components, which is no longer a valid assumption, unless TangoComponent and PvaComponent come along with it.

TangoDevice is not user-subclassed, but something that does the communication directly with the supplied API. read_attrs, or something similar, comes back as we no longer have Components:

dev = TangoDevice(..., read_attrs=['item0', 'item1'])

(2)

Signal can be structured.

1. Add TangoSignal, PvaSignal
2. Device does not change, and remains a "V3-only" construct
3. The hierarchy of Device is a container for Signals still applies
4. Signal may no longer be a leaf in the hierarchy, as it may contain some form of Signals logically underneath it.

sig = TangoSignal(...)
sig.sub_signal

Upsides:

1. Maximum amount of freedom in designing what these new Devices/Signals do
2. ...

Downsides:

1. May break some assumptions baked into ophyd
2. Might end up with a result that's completely unlike everything in ophyd currently
3. ...

Stance 2: Signal is a leaf of the tree - period.

@klauer believes that Signal and Device, as the basis for ophyd abstraction of the hardware underneath, remain useful in their current form and that the container relationship should strictly remain.

1. Device is a container of Signals
2. Device, in the case of EPICS V3, aggregates Signals into a logical - or useful - grouping.
3. Device, in the case of pvAccess or Tango, can piece together structured data or pick it apart a. This ability means that kind can be applied (or, of course, the deprecated read attributes lists) b. All of the Device methods and their underlying assumptions continue to work as expected
4. TangoSignal exposes the underlying structured Device, but picks off a single leaf of their hierachy, mapping onto an ophyd Signal. This happens either at the control layer or just in TangoSignal itself.

# Assuming some structured data along the lines of:
# device_a_identifier
#  |- item0
#  |- item1
#  |- item2
# device_z_identifier
#  |- item0
#  |- item1

class SingleMappedDevice(Device):
    'Simple mappings to a single Tango Device'
    a_item0 = Cpt(TangoSignal, '<device_a_identifier>', item='item0', kind='normal')
    a_item1 = Cpt(TangoSignal, '<device_a_identifier>', item='item1', kind='normal')
    a_item2 = Cpt(TangoSignal, '<device_a_identifier>', item='item2', kind='omitted')

class AggregateDevice(Device):
    'Aggregate some leaf signals of a structured Tango Device into our own'
    a_item0 = Cpt(TangoSignal, '<device_a_identifier>', item='item0', kind='normal')
    z_item0 = Cpt(TangoSignal, '<device_z_identifier>', item='item0', kind='normal')

Upsides:

1. Things continue to work as-is, without much difference to the end user
2. ...

Downsides:

1. Potentially more work for the user to set things up
2. ...

Related

659 - preliminary pvAccess support - only for normative types

652 - desire for control layer-related discussion

cc @tacaswell @ZLLentz @teddyrendahl @danielballan

[tacaswell]

I think I like the second path on stance 1 but see it as entirely consistent with stance 2.

Device is already a container of Signal or Device and we have currently arranged it so that the leaves are always Signal objects (either 1 epicsV3 pv, 2 epicsV3 pvs (for things with seperate setpoints and readbacks), 'derived', or fully synthetic). The distinction between the container-like-things and signal-like-things is if they recurse to their children to handle a user request (for get, put, read, trigger, ...) or 'leaves the system' to some other source of truth (it's own state or hitting the underlying control system) (in practice, we actually have a third case as the area detector filestore plugins actually do both, but....).

Given that Cpt already does not really care what types come through it I think something like

class MixedDevice(Device):
    a = Cpt(Signal)
    b = Cpt(EpicsSignal, 'pv_name')
    c = Cpt(EpicsMotor, 'motor_prefix')
    d = Cpt(EpicsV4, 'v4_pvname')
    e = Cpt(TangoDevice, 'tango_name')

makes sense and will (assuming we can put the requisite methods for the MixedDevice to recurse) just work. It can then be up to the V4/tango implementations if they want to present themselves as "leaves" where you either get the whole structure or not (and it all has the same kind) or to present themselves as "container like" and present the ability to tune what parts are read (and all of the kinds etc) by mirroring enough of the Device API (where "enough" should be documented!).

I'm sure we will find some places where we do make some isinstance checks which will need to be changed to some sort of duck-typing, but otherwise will get the best of both stances (maximum flexibility implementing the new structured nodes and things continue to 'just work' the way they do) and be consistent with ophyd as we know (and love?) it now.

[danielballan]

Signal and Device [.,.] remain useful in their current form and that the container relationship should strictly remain.

I agree with that. I think this describes the Signal/Device distinction as I understand it:

The distinction between the container-like-things and signal-like-things is if they recurse to their children to handle a user request [...]

I could be wrong, but I think the discussion around this statement:

I have previously argued that the distinction between Device and Signal should be made fuzzier, particularly as we move to v4 (or tango).

may be largely semantic. As I have previously said elsewhere, Signals and Devices are "just" things that implement The Interface from bluesky's point of view but within ophyd they have always been distinct. Devices already have certain methods that signals do not, such as stage(), because they manage coordinated configuration, and Signals do not. Adding extra methods to ophyd that further delineate the difference between a leaf and a container does not break anything from my point of view. As long as bluesky plans treat them alike, I don't see a problem.

Another interesting possibility to consider is that this class:

class SingleMappedDevice(Device):
    'Simple mappings to a single Tango Device'
    a_item0 = Cpt(TangoSignal, '<device_a_identifier>', item='item0', kind='normal')
    a_item1 = Cpt(TangoSignal, '<device_a_identifier>', item='item1', kind='normal')
    a_item2 = Cpt(TangoSignal, '<device_a_identifier>', item='item2', kind='omitted')

could be dynamically generated at connection time. In V3 world, we lose all structure over Channel Access, and so it has to be (re-)encoded on both sides. The ophyd side can get out of sync with the IOC side in the highly hypothetical situation that inter-group communication fails during an IOC upgrade. If I understand correctly, Tango ships an encoding of its structure at connection time, and one could imagine:

device = build_tango_device('<device_a_identifer>')

where build_tango_device both defines and instantiates an ophyd Device with (potentially) sub-Devices and ultimately TangoSignals.

---

As an aside, @klauer mentioned twice above the deprecated status of the *_attrs accessors. Our thinking on that at NSLS-II has evolved since the Kind feature first landed. Distributing the "source of truth" onto each individual object using .kind has been an improvement in our view, but read_attrs is still useful for setting or reviewing kind in bulk. The*_attrs accessors were initially included in the Kind feature only for backward-compatibility's sake, it's true, but in practice they seem worth keeping around forever -- whether or not they become essential for support structured-data control systems.

[tacaswell]

Devices already have certain methods that signals do not, such as stage(), because they manage coordinated configuration, and Signals do not.

This seemed like a choice of convenience (to keep the required API smaller) that Device calls stage on all of its children that have a stage method (we could have just as easily gone the other way and said everything must have stage method and the leaves' stage method is a no-op) rather than a deep design choice.

EpicsSignal out-of-the-box has two PVs attached to it so has some

but within ophyd they have always been distinct.

We have had almost from the beginning the filestore mixins which are used to MI with Device and inject their own 'leaf-like' ground truth:

https://github.com/NSLS-II/ophyd/blob/442271c06e8384c18f67176a9cef691edf154631/ophyd/areadetector/filestore_mixins.py#L368-L375

This is the difference between the public attributes on Device and Signal

--- /tmp/device.txt     2019-01-27 13:22:16.692655404 -0500
+++ /tmp/sig_txt        2019-01-27 13:22:32.963518788 -0500
@@ -1,11 +1,9 @@
-['OphydAttrList',
- 'SUB_ACQ_DONE',
+['SUB_META',
+ 'SUB_VALUE',
  'attr_name',
  'check_value',
+ 'cl',
  'clear_sub',
- 'component_names',
- 'configuration_attrs',
- 'configure',
  'connected',
  'describe',
  'describe_configuration',
@@ -13,37 +11,31 @@
  'dotted_name',
  'event_types',
  'get',
- 'get_device_tuple',
- 'get_instantiated_signals',
+ 'high_limit',
  'hints',
  'kind',
- 'lazy_wait_for_connection',
+ 'limits',
  'log',
+ 'low_limit',
+ 'metadata',
+ 'metadata_keys',
  'name',
  'parent',
- 'pause',
- 'prefix',
  'put',
  'read',
- 'read_attrs',
+ 'read_access',
  'read_configuration',
  'report',
- 'resume',
  'root',
- 'signal_names',
- 'stage',
- 'stage_sigs',
- 'stop',
+ 'rtolerance',
+ 'set',
  'subscribe',
  'subscriptions',
- 'summary',
+ 'timestamp',
+ 'tolerance',
  'trigger',
- 'trigger_signals',
- 'unstage',
  'unsubscribe',
  'unsubscribe_all',
+ 'value',
  'wait_for_connection',
- 'walk_components',
- 'walk_signals',
- 'walk_subdevice_classes',
- 'walk_subdevices']
+ 'write_access']

I think there is a good case that Signal should pick up some of the missing methods (stage, unstage, pause, resume, stop) as those are used by bluesky and so that when subclassing Signal you can safely call super() in those methods.

---

I keep coming back to :

• 'device-like' -> things who's bluesky interface's behavior you can affect from the ophyd layer at run time and init time
• 'signal-like' -> things who's bluesky interface's behavior you can affect from the ophyd layer (modulo fine-grained hinting? maybe all it's sub-parts must have the same hint?) only at init time

The*_attrs accessors were initially included in the Kind feature only for backward-compatibility's sake, it's true, but in practice they seem worth keeping around forever -- whether or not they become essential for support structured-data control systems.

I think this may be a good time to re-address the API for mucking with the behavior of 'device-like' objects.

[teddyrendahl]

I have previously argued that the distinction between Device and Signal should be made fuzzier, particularly as we move to v4 (or tango).

Based on the number of lines in this issue it is quite fuzzy already 😄

The distinction between the container-like-things and signal-like-things is if they recurse to their children to handle a user request [...]

I actually have a different definition that I haven't seen above. I believe that the distinction between the two is that Signal is the the smallest practical unit possible and contains an immutable group of access points to the underlying control layer. I added practical above because it is possible to connect only to the value of an EPICS PV or the connection state but we put both of those in EpicsSignal because:

1. We know every EpicsSignal has a connection state (immutable)
2. We know that whenever you are trying to get the value you need to ensure you are connected (practical)

Therefore I have no problem with our structured data leaf containing multiple control points underneath it as long as these are a non-negotiable part of the signal. In this case the Component, Device hierarchy would indicate a structure put together by Ophyd, not dictated by the control system.

In this case, I'm voting for a PVASignal that implements read to get the structure and convert it to a dictionary and an optional set. Then our leaf of PVASignal could still have smaller leaves for each attribute of the PVASignal

two_dim_array = PVASignal(...)
two_dim_array.read_attrs == ['array', 'width', 'height']
two_dim_array.width.get()

[tacaswell]

@teddyrendahl Interesting, That is a nice way of squaring that EpicsSignal has both a _read_pv and _write_pv but is not a Device circle.

Then our leaf of PVASignal could still have smaller leaves for each attribute of the PVASignal

However, this is making the 'leaves' mutable again.

[prjemian]

First time to contribute on this thread. I like and prefer the distinction between Device and Signal. Strongly. Check my knowledge here: the Signal is the fundamental component. A Device is composed from one or more Components, each describing a single Signal object. Hope that distinction remains. (Perhaps these remarks belong in #682 as well, but that discussion seems more about a duck-typing similarity between the two.)

Structured data presents a certain challenge. I envision to create automatically a new Device from the self-describing structure from the particular control system. These dynamic Devices could be created from a Device subclass specific to the chosen control system (TangoDevice, pvaDevice, areaDetectorDevice, IoTDevice, ...).

[tacaswell]

A Device is composed from one or more Components, each describing a single Signal object.

".. each describing a single Signal or Device object."

[klauer]

v2 is the future; closing