Extra diffractometer motors get set as `Kind.config`

[gfabbris]

The diffractometer device is setting every motor that is not _real or _pseudo as Kind.config. This is a problem because the position of these motors end up not being saved in the baseline. It looks like this is something that is done by ophyd.PseudoPositioner, but I couldn't understand how. Note that adding kind="normal" to the instance does not fix it. The only workaround that seems to work is to change the Kind after the diffractometer is initialized: https://github.com/APS-4ID-POLAR/ipython-polar/blob/43839e7a1d369b8155aa028cfa7f561e1314d75d/profile_bluesky/startup/instrument/devices/huber.py#L124

This seems to load fine, but I haven't tested running a scan.

@prjemian : Could you check this in a simulated diffractometer?

[prjemian]

I'll take a look. In hklpy

On Thu, Mar 25, 2021, 5:28 PM Gilberto Fabbris @.***> wrote:

The diffractometer device is setting every motor that is not _real or _pseudo as Kind.config. This is a problem because the position of these motors end up not being saved in the baseline. It looks like this is something that is done by ophyd.PseudoPositioner, but I couldn't understand how. Note that adding kind="normal" to the instance does not fix it. The only workaround that seems to work is to change the Kind after the diffractometer is initialized:

https://github.com/APS-4ID-POLAR/ipython-polar/blob/43839e7a1d369b8155aa028cfa7f561e1314d75d/profile_bluesky/startup/instrument/devices/huber.py#L124

This seems to load fine, but I haven't tested running a scan.

@prjemian https://github.com/prjemian : Could you check this in a simulated diffractometer?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/APS-4ID-POLAR/ipython-polar/issues/103, or unsubscribe https://github.com/notifications/unsubscribe-auth/AARMUMHDIVTI5IHHC7VAYMLTFO2JRANCNFSM4Z2HYVUQ .

[prjemian]

It's not actually setting every Component to kind=config but rather treating them as "config" in the .read() method. This is done deep in the ophyd.Device support. See the comments in the hklpy repo: https://github.com/bluesky/hklpy/issues/110#issuecomment-842653589.

[prjemian]

For your huber, add this (to the) constructor:

class FourCircleDiffractometer...
    def __init__(self, *args, **kwargs)
        super().__init__(*args, **kwargs)
        self.read_attrs += "x y z baseth basetth ath achi atth".split()

[gfabbris]

It looks like this solution doesn't work, the read_attrs gets messed up. I think that it's because it tries to populate the read_attrs in the super().__init__, but gets confused with the extra components.

In [2]: RE(bp.count([scalerd]))

Transient Scan ID: 423     Time: 2021-05-28 15:21:08
Persistent Unique Scan ID: 'd7840fa4-7b5c-493d-8c57-02cac7039203'
---------------------------------------------------------------------------                                     
KeyError                                  Traceback (most recent call last)                                     
<ipython-input-2-830ddbf07713> in <module>
----> 1 RE(bp.count([scalerd]))

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/run_engine.py in __call__(self, *args, **metadata_kw)
    805             self._task_fut.add_done_callback(set_blocking_event)
    806 
--> 807         self._resume_task(init_func=_build_task)
    808 
    809         if self._interrupted:

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/run_engine.py in _resume_task(self, init_func)
    929                     if (exc is not None
    930                             and not isinstance(exc, _RunEnginePanic)):
--> 931                         raise exc
    932 
    933     def install_suspender(self, suspender):

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/run_engine.py in _run(self)
   1498             exit_reason = str(err)
   1499             self.log.exception("Run aborted")
-> 1500             raise err
   1501         finally:
   1502             if not exit_reason:

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/run_engine.py in _run(self)
   1363                     else:
   1364                         try:
-> 1365                             msg = self._plan_stack[-1].send(resp)
   1366                         # We have exhausted the top generator
   1367                         except StopIteration:

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in __call__(self, plan)
   1305         plan = monitor_during_wrapper(plan, self.monitors)
   1306         plan = baseline_wrapper(plan, self.baseline)
-> 1307         return (yield from plan)
   1308 
   1309 

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in baseline_wrapper(plan, devices, name)
   1160         return (yield from plan)
   1161     else:
-> 1162         return (yield from plan_mutator(plan, insert_baseline))
   1163 
   1164 

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in monitor_during_wrapper(plan, signals)
    801     plan1 = plan_mutator(plan, insert_after_open)
    802     plan2 = plan_mutator(plan1, insert_before_close)
--> 803     return (yield from plan2)
    804 
    805 

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in fly_during_wrapper(plan, flyers)
    859     plan1 = plan_mutator(plan, insert_after_open)
    860     plan2 = plan_mutator(plan1, insert_before_close)
--> 861     return (yield from plan2)
    862 
    863 

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/plans.py in count(detectors, num, delay, per_shot, md)
     75                                       num=num, delay=delay))
     76 
---> 77     return (yield from inner_count())
     78 
     79 

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/utils.py in dec_inner(*inner_args, **inner_kwargs)
   1125                 plan = gen_func(*inner_args, **inner_kwargs)
   1126                 plan = wrapper(plan, *args, **kwargs)
-> 1127                 return (yield from plan)
   1128             return dec_inner
   1129         return dec

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in stage_wrapper(plan, devices)
    950         return (yield from plan)
    951 
--> 952     return (yield from finalize_wrapper(inner(), unstage_devices()))
    953 
    954 

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in finalize_wrapper(plan, final_plan, pause_for_debug)
    507     cleanup = True
    508     try:
--> 509         ret = yield from plan
    510     except GeneratorExit:
    511         cleanup = False

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in inner()
    948     def inner():
    949         yield from stage_devices()
--> 950         return (yield from plan)
    951 
    952     return (yield from finalize_wrapper(inner(), unstage_devices()))

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/utils.py in dec_inner(*inner_args, **inner_kwargs)
   1125                 plan = gen_func(*inner_args, **inner_kwargs)
   1126                 plan = wrapper(plan, *args, **kwargs)
-> 1127                 return (yield from plan)
   1128             return dec_inner
   1129         return dec

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in run_wrapper(plan, md)
    315         metadata to be passed into the 'open_run' message
    316     """
--> 317     rs_uid = yield from open_run(md)
    318 
    319     def except_plan(e):

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/plan_stubs.py in open_run(md)
    821     :func:`bluesky.plans_stubs.close_run`
    822     """
--> 823     return (yield Msg('open_run', **(md or {})))
    824 
    825 

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/utils.py in single_gen(msg)
    152         the input message
    153     '''
--> 154     return (yield msg)
    155 
    156 

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/utils.py in single_gen(msg)
    152         the input message
    153     '''
--> 154     return (yield msg)
    155 
    156 

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/plan_stubs.py in trigger_and_read(devices, name)
    909         return ret
    910     from .preprocessors import rewindable_wrapper
--> 911     return (yield from rewindable_wrapper(inner_trigger_and_read(),
    912                                           rewindable))
    913 

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in rewindable_wrapper(plan, rewindable)
    691                                             restore_rewindable()))
    692     else:
--> 693         return (yield from plan)
    694 
    695 

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/plan_stubs.py in inner_trigger_and_read()
    903         ret = {}  # collect and return readings to give plan access to them
    904         for obj in devices:
--> 905             reading = (yield from read(obj))
    906             if reading is not None:
    907                 ret.update(reading)

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/plan_stubs.py in read(obj)
     93         Msg('read', obj)
     94     """
---> 95     return (yield Msg('read', obj))
     96 
     97 

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/run_engine.py in _run(self)
   1423                         # exceptions (coming in via throw) can be
   1424                         # raised
-> 1425                         new_response = await coro(msg)
   1426 
   1427                     # special case `CancelledError` and let the outer

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/run_engine.py in _read(self, msg)
   1681             ...
   1682         else:
-> 1683             await current_run.read(msg, ret)
   1684 
   1685         return ret

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/bundlers.py in read(self, msg, reading)
    172             if obj not in self._describe_cache:
    173                 # Validate that there is no data key name collision.
--> 174                 data_keys = obj.describe()
    175                 self._describe_cache[obj] = data_keys
    176                 self._config_desc_cache[obj] = obj.describe_configuration()

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/ophyd/device.py in describe(self)
   1254         res = super().describe()
   1255         for _, component in self._get_components_of_kind(Kind.normal):
-> 1256             res.update(component.describe())
   1257         return res
   1258 

~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/ophyd/pseudopos.py in describe(self)
    169         low_limit, high_limit = self.limits
    170 
--> 171         for d in (desc[self.readback.name], desc[self.setpoint.name]):
    172             d['upper_ctrl_limit'] = high_limit
    173             d['lower_ctrl_limit'] = low_limit

KeyError: 'fourc_h'

In [3]: fourc.read_attrs
Out[3]: ['h', 'h.setpoint', 'k', 'k.setpoint', 'l', 'l.readback', 'theta', 'theta.user_readback', 'chi', 'chi.user_readback', 'phi',
'phi.user_readback', 'tth', 'tth.user_setpoint', 'x', 'x.user_readback', 'y', 'y.user_readback', 'y.user_setpoint', 'z', 'z.user_readback',
'baseth', 'baseth.user_setpoint', 'basetth', 'basetth.user_readback', 'ath', 'ath.user_readback', 'achi', 'achi.user_readback', 'atth',
'atth.user_readback']

In [4]: fourc.h.read_attrs
Out[4]: ['setpoint']

In [5]: fourc.h.kind
Out[5]: <Kind.config|normal: 3>

The correct settings should be:

In [1]: fourc.read_attrs
Out[1]: ['h', 'h.readback', 'h.setpoint', 'k', 'k.readback', 'k.setpoint', 'l', 'l.readback', 'l.setpoint', 'theta', 'theta.user_readback',
'theta.user_setpoint', 'chi', 'chi.user_readback', 'chi.user_setpoint', 'phi', 'phi.user_readback', 'phi.user_setpoint', 'tth',
'tth.user_readback', 'tth.user_setpoint', 'x', 'x.user_readback', 'x.user_setpoint', 'y', 'y.user_readback', 'y.user_setpoint', 'z',
'z.user_readback', 'z.user_setpoint', 'baseth', 'baseth.user_readback', 'baseth.user_setpoint', 'basetth', 'basetth.user_readback',
'basetth.user_setpoint', 'tablex', 'tablex.user_readback', 'tablex.user_setpoint', 'tabley', 'tabley.user_readback',
'tabley.user_setpoint', 'ath', 'ath.user_readback', 'ath.user_setpoint', 'achi', 'achi.user_readback', 'achi.user_setpoint', 'atth',
'atth.user_readback', 'atth.user_setpoint']

In [2]: fourc.h.kind
Out[2]: <Kind.config|normal: 3>

In [3]: fourc.h.read_attrs
Out[3]: ['readback', 'setpoint']

[prjemian]

Compare with this recent HOWTO. The ophyd.PseudoPositioner() class provides a _real attribute list to identify the attributes needed for the forward() and inverse() transformations. This optional attribute defaults to the list from read_attrs.

Rewrite as:

class FourCircleDiffractometer...

    _real = "x y z baseth basetth ath achi atth".split()

    def __init__(self, *args, **kwargs)
        super().__init__(*args, **kwargs)

[prjemian]

Aaaack! The _real list is not the additional axes. It should be the four axes in canonical order:

class FourCircleDiffractometer...

    _real = "theta chi phi tth".split()

    def __init__(self, *args, **kwargs)
        super().__init__(*args, **kwargs)

[gfabbris]

Yes, but this marks the other motors (x, y, z, ...) as config instead of normal (even if I explicitly initialize it as normal).

This is the only way that I found to make it work (after initializing fourc):

for attr in ("x", "y", "z", "baseth", "basetth", "ath", "achi", "atth",
             "tablex", "tabley"):
    getattr(fourc, attr).kind = "normal"

[prjemian]

Got it. I misunderstood the intent. You are describing the recent bug report. Ok. Make it work.