[fccd class] bug in fccd.exposure

ambarb commented 5 years ago

There was a typo in a script that

prevented any further RE exectution because a .set() was in progress
the fccd would change from 0.02s exposure to 0.3s exposure after a period of time (30s or more)
several quits of bluesky did not resolve
ioc restart resolved

So there is a recovery process, but this function should be more robust because it is 1) confusing for typical user if they make this mistake and 2) i was on a very bright bragg so it was lucky that I was here to block the beam instead of home sleeping.

@cmazzoli @wen-hu

def plan_end_200K():
    yield from  mv(inout, 'Out')
    #Rod scans
    print('\n\n****** bragg now *****\n\n')
    yield from mv(fccd.exposure,(0.02,0.3,18*2))  ###
    #yield from bps.sleep(60*5)
    yield from mv(fccd.exposure,(3,0,18*2))  ###

There should have been a plan after the first fccd.exposure instead of setting another exposure time.

Here is the traceback:

In [6]: RE(plan_end_200K())

****** bragg now *****

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
~/Beamline/ScienceComm/2018_11_MuR/20181120_dinner.py in <module>()
----> 1 RE(plan_end_200K())

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/run_engine.py in __call__(self, *args, **metadata_kw)
    668                     # it (unless it is a canceled error)
    669                     if exc is not None:
--> 670                         raise exc
    671 
    672             if self._interrupted:

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/run_engine.py in _run(self)
   1118             self.log.error("Run aborted")
   1119             self.log.error("%r", err)
-> 1120             raise err
   1121         finally:
   1122             # Some done_callbacks may still be alive in other threads.

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/run_engine.py in _run(self)
    996                         try:
    997                             msg = self._plan_stack[-1].throw(
--> 998                                 self._exception)
    999                         except Exception as e:
   1000                             # The current plan did not handle it,

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/preprocessors.py in __call__(self, plan)
   1236         plan = monitor_during_wrapper(plan, self.monitors)
   1237         plan = baseline_wrapper(plan, self.baseline)
-> 1238         return (yield from plan)

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/preprocessors.py in baseline_wrapper(plan, devices, name)
   1092         return (yield from plan)
   1093     else:
-> 1094         return (yield from plan_mutator(plan, insert_baseline))
   1095 
   1096 

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     73             # if we have a stashed exception, pass it along
     74             try:
---> 75                 msg = plan_stack[-1].throw(exception)
     76             except Exception as e:
     77                 # if we catch an exception,

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/preprocessors.py in monitor_during_wrapper(plan, signals)
    722     plan1 = plan_mutator(plan, insert_after_open)
    723     plan2 = plan_mutator(plan1, insert_before_close)
--> 724     return (yield from plan2)
    725 
    726 

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     73             # if we have a stashed exception, pass it along
     74             try:
---> 75                 msg = plan_stack[-1].throw(exception)
     76             except Exception as e:
     77                 # if we catch an exception,

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     73             # if we have a stashed exception, pass it along
     74             try:
---> 75                 msg = plan_stack[-1].throw(exception)
     76             except Exception as e:
     77                 # if we catch an exception,

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/preprocessors.py in fly_during_wrapper(plan, flyers)
    780     plan1 = plan_mutator(plan, insert_after_open)
    781     plan2 = plan_mutator(plan1, insert_before_close)
--> 782     return (yield from plan2)
    783 
    784 

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     73             # if we have a stashed exception, pass it along
     74             try:
---> 75                 msg = plan_stack[-1].throw(exception)
     76             except Exception as e:
     77                 # if we catch an exception,

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     73             # if we have a stashed exception, pass it along
     74             try:
---> 75                 msg = plan_stack[-1].throw(exception)
     76             except Exception as e:
     77                 # if we catch an exception,

~/Beamline/ScienceComm/2018_11_MuR/20181120_dinner.py in plan_end_200K()
     42     yield from mv(fccd.exposure,(0.02,0.3,18*2))  ###
     43     #yield from bps.sleep(60*5)
---> 44     yield from mv(fccd.exposure,(3,0,18*2))  ###
     45     yield from scan_L_aboveBragg()
     46     print('\n\n****** done above, now below *****\n\n')

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/plan_stubs.py in mv(*args)
    240     step, = utils.merge_cycler(cyl)
    241     for obj, val in step.items():
--> 242         ret = yield Msg('set', obj, val, group=group)
    243         status_objects.append(ret)
    244     yield Msg('wait', None, group=group)

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    161         try:
    162             # yield out the 'current message' and collect the return
--> 163             inner_ret = yield msg
    164         except GeneratorExit:
    165             # special case GeneratorExit.  We must clean up all of our plans

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    161         try:
    162             # yield out the 'current message' and collect the return
--> 163             inner_ret = yield msg
    164         except GeneratorExit:
    165             # special case GeneratorExit.  We must clean up all of our plans

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    161         try:
    162             # yield out the 'current message' and collect the return
--> 163             inner_ret = yield msg
    164         except GeneratorExit:
    165             # special case GeneratorExit.  We must clean up all of our plans

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    161         try:
    162             # yield out the 'current message' and collect the return
--> 163             inner_ret = yield msg
    164         except GeneratorExit:
    165             # special case GeneratorExit.  We must clean up all of our plans

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    161         try:
    162             # yield out the 'current message' and collect the return
--> 163             inner_ret = yield msg
    164         except GeneratorExit:
    165             # special case GeneratorExit.  We must clean up all of our plans

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/run_engine.py in _run(self)
   1068                         # exceptions (coming in via throw) can be
   1069                         # raised
-> 1070                         response = yield from coro(msg)
   1071                     # special case `CancelledError` and let the outer
   1072                     # exception block deal with it.

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/asyncio/coroutines.py in coro(*args, **kw)
    208         @functools.wraps(func)
    209         def coro(*args, **kw):
--> 210             res = func(*args, **kw)
    211             if (base_futures.isfuture(res) or inspect.isgenerator(res) or
    212                 isinstance(res, CoroWrapper)):

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/run_engine.py in _set(self, msg)
   1822         group = kwargs.pop('group', None)
   1823         self._movable_objs_touched.add(msg.obj)
-> 1824         ret = msg.obj.set(*msg.args, **kwargs)
   1825         p_event = asyncio.Event(loop=self.loop)
   1826         pardon_failures = self._pardon_failures

/epics/xf/23id/ipython/profile_collection/startup/csx1/devices/areadetector.py in set(self, exp)
    191             # Set delay generator
    192             self.parent.dg1.A.set(aa)
--> 193             self.parent.dg1.B.set(bb)
    194             self.parent.dg1.C.set(cc)
    195             self.parent.dg1.D.set(dd)

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/ophyd/signal.py in set(self, value, timeout, settle_time)
    928         '''
    929         if not self._put_complete:
--> 930             return super().set(value, timeout=timeout, settle_time=settle_time)
    931 
    932         # using put completion:

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/ophyd/signal.py in set(self, value, timeout, settle_time)
    179 
    180         if self._set_thread is not None:
--> 181             raise RuntimeError('Another set() call is still in progress')
    182 
    183         st = Status(self)

RuntimeError: Another set() call is still in progress

mrakitin commented 5 years ago

Is it still observed?

ambarb commented 5 years ago

probably. we can test on Monday before I warm up the camera and get back to you.

ambarb commented 5 years ago

also probably related to https://github.com/NSLS-II-CSX/xf23id1_profiles/issues/17

unsure if this is a ophyd device thing or a fccd ioc thing. @stuwilkins do you have an idea?

danielballan commented 5 years ago

See proposed fix in #19.

ambarb commented 3 years ago

maybe this is fixed. CSX should test to verify if this is still a problem

NSLS-II-CSX / profile_collection

[fccd class] bug in fccd.exposure #9