problem counting with ScalerCH

[prjemian]

per #7, switched EpicsScaler to ScalerCH but that produced a problem when using RE(bluesky.count([scaler]))

where

scaler = ScalerCH("xxx:scaler", name="scaler")

[prjemian]

@danielballan : Here's that error message:

In [9]: RE(bp.count([scaler]))
Transient Scan ID: 585     Time: 2018/02/10 17:39:28
Persistent Unique Scan ID: '038a851f-1d19-48fb-a4eb-6017bd571f0d'

exit status: fail
---------------------------------------------------------------------------
ValidationError                           Traceback (most recent call last)
<ipython-input-9-4922b2372926> in <module>()
----> 1 RE(bp.count([scaler]))

/home/mintadmin/Apps/BlueSky/lib/python3.6/site-packages/bluesky/run_engine.py in __call__(self, *args, **metadata_kw)
    666                     # it (unless it is a canceled error)
    667                     if exc is not None:
--> 668                         raise exc
    669 
    670             if self._interrupted:

/home/mintadmin/Apps/BlueSky/lib/python3.6/site-packages/bluesky/run_engine.py in _run(self)
   1114             self.log.error("Run aborted")
   1115             self.log.error("%r", err)
-> 1116             raise err
   1117         finally:
   1118             # Some done_callbacks may still be alive in other threads.

/home/mintadmin/Apps/BlueSky/lib/python3.6/site-packages/bluesky/run_engine.py in _run(self)
   1013                         resp = self._response_stack.pop()
   1014                         try:
-> 1015                             msg = self._plan_stack[-1].send(resp)
   1016                         # We have exhausted the top generator
   1017                         except StopIteration:

/home/mintadmin/Apps/BlueSky/lib/python3.6/site-packages/bluesky/preprocessors.py in __call__(self, plan)
   1226         plan = monitor_during_wrapper(plan, self.monitors)
   1227         plan = fly_during_wrapper(plan, self.flyers)
-> 1228         return (yield from plan)

/home/mintadmin/Apps/BlueSky/lib/python3.6/site-packages/bluesky/preprocessors.py in fly_during_wrapper(plan, flyers)
    780     plan1 = plan_mutator(plan, insert_after_open)
    781     plan2 = plan_mutator(plan1, insert_before_close)
--> 782     return (yield from plan2)
    783 
    784 

/home/mintadmin/Apps/BlueSky/lib/python3.6/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    136                     continue
    137                 else:
--> 138                     raise ex
    139         # if inserting / mutating, put new generator on the stack
    140         # and replace the current msg with the first element from the

/home/mintadmin/Apps/BlueSky/lib/python3.6/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     89             ret = result_stack.pop()
     90             try:
---> 91                 msg = plan_stack[-1].send(ret)
     92             except StopIteration as e:
     93                 # discard the exhausted generator

/home/mintadmin/Apps/BlueSky/lib/python3.6/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    136                     continue
    137                 else:
--> 138                     raise ex
    139         # if inserting / mutating, put new generator on the stack
    140         # and replace the current msg with the first element from the

/home/mintadmin/Apps/BlueSky/lib/python3.6/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     89             ret = result_stack.pop()
     90             try:
---> 91                 msg = plan_stack[-1].send(ret)
     92             except StopIteration as e:
     93                 # discard the exhausted generator

/home/mintadmin/Apps/BlueSky/lib/python3.6/site-packages/bluesky/preprocessors.py in monitor_during_wrapper(plan, signals)
    722     plan1 = plan_mutator(plan, insert_after_open)
    723     plan2 = plan_mutator(plan1, insert_before_close)
--> 724     return (yield from plan2)
    725 
    726 

/home/mintadmin/Apps/BlueSky/lib/python3.6/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    136                     continue
    137                 else:
--> 138                     raise ex
    139         # if inserting / mutating, put new generator on the stack
    140         # and replace the current msg with the first element from the

/home/mintadmin/Apps/BlueSky/lib/python3.6/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     89             ret = result_stack.pop()
     90             try:
---> 91                 msg = plan_stack[-1].send(ret)
     92             except StopIteration as e:
     93                 # discard the exhausted generator

/home/mintadmin/Apps/BlueSky/lib/python3.6/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    136                     continue
    137                 else:
--> 138                     raise ex
    139         # if inserting / mutating, put new generator on the stack
    140         # and replace the current msg with the first element from the

/home/mintadmin/Apps/BlueSky/lib/python3.6/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     89             ret = result_stack.pop()
     90             try:
---> 91                 msg = plan_stack[-1].send(ret)
     92             except StopIteration as e:
     93                 # discard the exhausted generator

/home/mintadmin/Apps/BlueSky/lib/python3.6/site-packages/bluesky/preprocessors.py in baseline_wrapper(plan, devices, name)
   1090     if not devices:
   1091         # no-op
-> 1092         return (yield from plan)
   1093     else:
   1094         return (yield from plan_mutator(plan, insert_baseline))

/home/mintadmin/Apps/BlueSky/lib/python3.6/site-packages/bluesky/plans.py in count(detectors, num, delay, md)
    110         return (yield from infinite_plan())
    111     else:
--> 112         return (yield from finite_plan())
    113 
    114 

/home/mintadmin/Apps/BlueSky/lib/python3.6/site-packages/bluesky/utils.py in dec_inner(*inner_args, **inner_kwargs)
    929                 plan = gen_func(*inner_args, **inner_kwargs)
    930                 plan = wrapper(plan, *args, **kwargs)
--> 931                 return (yield from plan)
    932             return dec_inner
    933         return dec

/home/mintadmin/Apps/BlueSky/lib/python3.6/site-packages/bluesky/preprocessors.py in stage_wrapper(plan, devices)
    871         return (yield from plan)
    872 
--> 873     return (yield from finalize_wrapper(inner(), unstage_devices()))
    874 
    875 

/home/mintadmin/Apps/BlueSky/lib/python3.6/site-packages/bluesky/preprocessors.py in finalize_wrapper(plan, final_plan, pause_for_debug)
    432     cleanup = True
    433     try:
--> 434         ret = yield from plan
    435     except GeneratorExit:
    436         cleanup = False

/home/mintadmin/Apps/BlueSky/lib/python3.6/site-packages/bluesky/preprocessors.py in inner()
    869     def inner():
    870         yield from stage_devices()
--> 871         return (yield from plan)
    872 
    873     return (yield from finalize_wrapper(inner(), unstage_devices()))

/home/mintadmin/Apps/BlueSky/lib/python3.6/site-packages/bluesky/utils.py in dec_inner(*inner_args, **inner_kwargs)
    929                 plan = gen_func(*inner_args, **inner_kwargs)
    930                 plan = wrapper(plan, *args, **kwargs)
--> 931                 return (yield from plan)
    932             return dec_inner
    933         return dec

/home/mintadmin/Apps/BlueSky/lib/python3.6/site-packages/bluesky/preprocessors.py in run_wrapper(plan, md)
    294     yield from contingency_wrapper(plan,
    295                                    except_plan=except_plan,
--> 296                                    else_plan=close_run)
    297     return rs_uid
    298 

/home/mintadmin/Apps/BlueSky/lib/python3.6/site-packages/bluesky/preprocessors.py in contingency_wrapper(plan, except_plan, else_plan, final_plan, pause_for_debug)
    494     cleanup = True
    495     try:
--> 496         ret = yield from plan
    497     except GeneratorExit:
    498         cleanup = False

/home/mintadmin/Apps/BlueSky/lib/python3.6/site-packages/bluesky/plans.py in finite_plan()
     79             now = time.time()  # Intercept the flow in its earliest moment.
     80             yield Msg('checkpoint')
---> 81             yield from bps.trigger_and_read(detectors)
     82             try:
     83                 d = next(delay)

/home/mintadmin/Apps/BlueSky/lib/python3.6/site-packages/bluesky/plan_stubs.py in trigger_and_read(devices, name)
    738     from .preprocessors import rewindable_wrapper
    739     return (yield from rewindable_wrapper(inner_trigger_and_read(),
--> 740                                           rewindable))
    741 
    742 

/home/mintadmin/Apps/BlueSky/lib/python3.6/site-packages/bluesky/preprocessors.py in rewindable_wrapper(plan, rewindable)
    612                                             restore_rewindable()))
    613     else:
--> 614         return (yield from plan)
    615 
    616 

/home/mintadmin/Apps/BlueSky/lib/python3.6/site-packages/bluesky/plan_stubs.py in inner_trigger_and_read()
    734             if reading is not None:
    735                 ret.update(reading)
--> 736         yield from save()
    737         return ret
    738     from .preprocessors import rewindable_wrapper

/home/mintadmin/Apps/BlueSky/lib/python3.6/site-packages/bluesky/plan_stubs.py in save()
     52     :func:`bluesky.plans.create`
     53     """
---> 54     return (yield Msg('save'))
     55 
     56 

/home/mintadmin/Apps/BlueSky/lib/python3.6/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    161         try:
    162             # yield out the 'current message' and collect the return
--> 163             inner_ret = yield msg
    164         except GeneratorExit:
    165             # special case GeneratorExit.  We must clean up all of our plans

/home/mintadmin/Apps/BlueSky/lib/python3.6/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    161         try:
    162             # yield out the 'current message' and collect the return
--> 163             inner_ret = yield msg
    164         except GeneratorExit:
    165             # special case GeneratorExit.  We must clean up all of our plans

/home/mintadmin/Apps/BlueSky/lib/python3.6/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    161         try:
    162             # yield out the 'current message' and collect the return
--> 163             inner_ret = yield msg
    164         except GeneratorExit:
    165             # special case GeneratorExit.  We must clean up all of our plans

/home/mintadmin/Apps/BlueSky/lib/python3.6/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    161         try:
    162             # yield out the 'current message' and collect the return
--> 163             inner_ret = yield msg
    164         except GeneratorExit:
    165             # special case GeneratorExit.  We must clean up all of our plans

/home/mintadmin/Apps/BlueSky/lib/python3.6/site-packages/bluesky/run_engine.py in _run(self)
   1064                         # exceptions (coming in via throw) can be
   1065                         # raised
-> 1066                         response = yield from coro(msg)
   1067                     # special case `CancelledError` and let the outer
   1068                     # exception block deal with it.

/home/mintadmin/Apps/BlueSky/lib/python3.6/site-packages/bluesky/run_engine.py in _save(self, msg)
   1553                        configuration=config, name=self._bundle_name,
   1554                        hints=hints, object_keys=object_keys)
-> 1555             yield from self.emit(DocumentNames.descriptor, doc)
   1556             self.log.debug("Emitted Event Descriptor with name %r containing "
   1557                            "data keys %r (uid=%r)", self._bundle_name,

/home/mintadmin/Apps/BlueSky/lib/python3.6/asyncio/coroutines.py in coro(*args, **kw)
    208         @functools.wraps(func)
    209         def coro(*args, **kw):
--> 210             res = func(*args, **kw)
    211             if (base_futures.isfuture(res) or inspect.isgenerator(res) or
    212                 isinstance(res, CoroWrapper)):

/home/mintadmin/Apps/BlueSky/lib/python3.6/site-packages/bluesky/run_engine.py in emit(self, name, doc)
   2143     def emit(self, name, doc):
   2144         "Process blocking callbacks and schedule non-blocking callbacks."
-> 2145         jsonschema.validate(doc, schemas[name])
   2146         self.dispatcher.process(name, doc)
   2147 

/home/mintadmin/Apps/BlueSky/lib/python3.6/site-packages/jsonschema/validators.py in validate(instance, schema, cls, *args, **kwargs)
    476         cls = validator_for(schema)
    477     cls.check_schema(schema)
--> 478     cls(schema, *args, **kwargs).validate(instance)

/home/mintadmin/Apps/BlueSky/lib/python3.6/site-packages/jsonschema/validators.py in validate(self, *args, **kwargs)
    121         def validate(self, *args, **kwargs):
    122             for error in self.iter_errors(*args, **kwargs):
--> 123                 raise error
    124 
    125         def is_type(self, instance, type):

ValidationError: Additional properties are not allowed ('' was unexpected)

Failed validating 'additionalProperties' in schema['patternProperties']['^([^./]+)$']:
    {'additionalProperties': False,
     'patternProperties': {'^([^./]+)$': {'$ref': '#/definitions/data_type'}},
     'title': 'data_type'}

On instance['data_keys']:
    {'': {'dtype': 'number',
          'lower_ctrl_limit': 0.0,
          'object_name': 'scaler',
          'precision': 0,
          'shape': [],
          'source': 'PV:prj:scaler1.S1',
          'units': '',
          'upper_ctrl_limit': 0.0},
     'scaler_time': {'dtype': 'number',
                     'lower_ctrl_limit': 0.0,
                     'object_name': 'scaler',
                     'precision': 3,
                     'shape': [],
                     'source': 'PV:prj:scaler1.T',
                     'units': '',
                     'upper_ctrl_limit': 0.0}}

[danielballan]

@tacaswell Any guidance here? Looks like this is dying in a jsonschema validation error when the documents are emitted, likely because ScalerCH is generating invalid (blank?) data key names.

[tacaswell]

The issue is that the name PV is an empty string so when (on init) the channels asks what it's name should be, it happily takes the empty string.

Either put names in and then re-call match_names or remove channels without names from the read attrs

[danielballan]

Hmm. It has a name -- name="scaler". Do the channels not name themselves by default? It seems like ScalerCH should blow up at init time if it has names/read_attrs that will generate invalid documents.

[tacaswell]

In the __init__ of the individual channels they call self.match_name which sets the channel's .name to what ever is in the name PV (which out the parent's prefix).

[danielballan]

OK, the key fact I was missing here is that scaler IOCs have a concept of names, which they expose through PVs. ScalerCH tries to account for that and sync to ophyd names to the names known to the IOC. This has not come up in any other ophyd Devices, because IOCs don't typically encoded user-readable names. In that context, I can interpret Tom's original comment:

Either put names in and then re-call match_names or remove channels without names from the read attrs

"Put names in" means put a string to the IOC. Then, sync the ophyd Device to that name using match_name. As a comment in ScalerChannel indicates, we could do that second step automatically by monitoring the IOC's name PV.

[danielballan]

See https://github.com/NSLS-II/ophyd/pull/509 which omits nameless channels from the read_attrs at init time.

[tacaswell]

@prjemian did you get this sorted?

[prjemian]

No, but not important now

On Feb 13, 2018 4:33 PM, "Thomas A Caswell" notifications@github.com wrote:

@prjemian https://github.com/prjemian did you get this sorted?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/APS-2BM-MIC/ipython-user2bmb/issues/10#issuecomment-365428222, or mute the thread https://github.com/notifications/unsubscribe-auth/ACLKMHBuV6m8VqzHWPHlzQoDnuMZzpklks5tUg22gaJpZM4SBIXn .

[prjemian]

All that needs to be done is to add labels to each desired scaler channel in EPICS?

On Feb 13, 2018 5:02 PM, "Pete Jemian" prjemian@gmail.com wrote:

No, but not important now

On Feb 13, 2018 4:33 PM, "Thomas A Caswell" notifications@github.com wrote:

@prjemian https://github.com/prjemian did you get this sorted?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/APS-2BM-MIC/ipython-user2bmb/issues/10#issuecomment-365428222, or mute the thread https://github.com/notifications/unsubscribe-auth/ACLKMHBuV6m8VqzHWPHlzQoDnuMZzpklks5tUg22gaJpZM4SBIXn .

[danielballan]

Yes, seems so. On Tue, Feb 13, 2018 at 6:13 PM Pete R Jemian notifications@github.com wrote:

All that needs to be done is to add labels to each desired scaler channel in EPICS?

On Feb 13, 2018 5:02 PM, "Pete Jemian" prjemian@gmail.com wrote:

No, but not important now

On Feb 13, 2018 4:33 PM, "Thomas A Caswell" notifications@github.com wrote:

@prjemian https://github.com/prjemian did you get this sorted?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub < https://github.com/APS-2BM-MIC/ipython-user2bmb/issues/10#issuecomment-365428222 , or mute the thread < https://github.com/notifications/unsubscribe-auth/ACLKMHBuV6m8VqzHWPHlzQoDnuMZzpklks5tUg22gaJpZM4SBIXn

.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/APS-2BM-MIC/ipython-user2bmb/issues/10#issuecomment-365438013, or mute the thread https://github.com/notifications/unsubscribe-auth/ACLIri1Ev4VPf9saRzgGS-kg4Aw9XVhxks5tUhcSgaJpZM4SBIXn .

[prjemian]

Tested this just now.

This works for me iff the channel names (.NM1, .NM2, ...) are defined before the call to ScalerCH(). If the names are changed after this call, the ophyd object does not learn about the new name.

Also, should read_attrs and/or configuration_attrs be populated initially with the channels that have names?

[tacaswell]

You can call sclr.match_names() to re-sync them. Maybe that should be done in the stage method? We could also install a monitor onto the name PVs, but then they could change mid-scan which would break lots of things.

NSLS-II/ophyd#509 filters to only channels with names.

[prjemian]

This has a workaround until ScalerCH is fixed up.