search

NSLS-II-CSX / profile_collection

BSD 3-Clause "New" or "Revised" License
0 stars 5 forks source link

[fccd] periodic issue that requires restart of bsui #25

Closed ambarb closed 1 year ago

ambarb commented 5 years ago

The following worked for a long time and then the plan got "hung up" . Had to restart bsui in order to regain control. This is an intermittent issue.


In [231]: RE(bpp.pchain(ct_dark_all(20), ct([fccd],num=1,md={'purpose':'test und offset -24 & fast shutter & 2x1 bflslt'})))                                                                         

Starting procedure to acquire darks 1.18e+02Hz or 0.009s.

    Current number of images = 5.

    Setting to 20 images.

  A 'deferred pause' has been requested. The RunEngine will pause at the next checkpoint. To pause immediately, hit Ctrl+C again in the next 10 seconds.                                             
Deferred pause acknowledged. Continuing to checkpoint.
^CPausing...
---------------------------------------------------------------------------
RunEngineInterrupted                      Traceback (most recent call last)
~/Beamline/Commissioning/2019_08/defs.py in <module>
----> 1 RE(bpp.pchain(ct_dark_all(20), ct([fccd],num=1,md={'purpose':'test und offset -24 & fast shutter & 2x1 bflslt'})))

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/run_engine.py in __call__(self, *args, **metadata_kw)
    754 
    755             if self._interrupted:
--> 756                 raise RunEngineInterrupted(self.pause_msg) from None
    757 
    758         return tuple(self._run_start_uids)

RunEngineInterrupted: 
Your RunEngine is entering a paused state. These are your options for changing
the state of the RunEngine:

RE.resume()    Resume the plan.
RE.abort()     Perform cleanup, then kill plan. Mark exit_stats='aborted'.
RE.stop()      Perform cleanup, then kill plan. Mark exit_status='success'.
RE.halt()      Emergency Stop: Do not perform cleanup --- just stop.

In [232]: RE.abort()                                                                                                                                                                                 
Aborting: running cleanup and marking exit_status as 'abort'...

Returning to intial conditions (pre-count).
        Total images per trigger are NOW:        5                                                                                                                                                   
    FCCD FCRIC gain value is NOW:        auto

[E 18:29:52.774 run_engine:1257] Run aborted
    Traceback (most recent call last):
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/run_engine.py", line 1150, in _run
        msg = self._plan_stack[-1].send(resp)
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 1294, in __call__
        return (yield from plan)
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 1149, in baseline_wrapper
        return (yield from plan_mutator(plan, insert_baseline))
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 169, in plan_mutator
        raise ex
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 122, in plan_mutator
        msg = plan_stack[-1].send(ret)
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 790, in monitor_during_wrapper
        return (yield from plan2)
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 169, in plan_mutator
        raise ex
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 122, in plan_mutator
        msg = plan_stack[-1].send(ret)
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 169, in plan_mutator
        raise ex
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 122, in plan_mutator
        msg = plan_stack[-1].send(ret)
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 848, in fly_during_wrapper
        return (yield from plan2)
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 169, in plan_mutator
        raise ex
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 122, in plan_mutator
        msg = plan_stack[-1].send(ret)
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 169, in plan_mutator
        raise ex
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 122, in plan_mutator
        msg = plan_stack[-1].send(ret)
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 267, in pchain
        rets.append((yield from p))
      File "/epics/xf/23id/ipython/profile_collection/startup/csx1/plans/dark_ff.py", line 171, in ct_dark_all
        yield from bps.mv(fccd.fccd1.capture_bgnd, 1)
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/plan_stubs.py", line 247, in mv
        yield Msg('wait', None, group=group)
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 194, in plan_mutator
        inner_ret = yield msg
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 194, in plan_mutator
        inner_ret = yield msg
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 194, in plan_mutator
        inner_ret = yield msg
      [Previous line repeated 2 more times]
    bluesky.utils.RequestAbort
Out[232]: []

In [233]: RE.abort()                                                                                                                                                                                 
---------------------------------------------------------------------------
TransitionError                           Traceback (most recent call last)
~/Beamline/Commissioning/2019_08/defs.py in <module>
----> 1 RE.abort()

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/run_engine.py in abort(self, reason)
   1007         """
   1008         if self.state.is_idle:
-> 1009             raise TransitionError("RunEngine is already idle.")
   1010         print("Aborting: running cleanup and marking "
   1011               "exit_status as 'abort'...")

TransitionError: RunEngine is already idle.

In [234]: RE(bpp.pchain(ct_dark_all(20), ct([fccd],num=1,md={'purpose':'test und offset -24 & fast shutter & 2x1 bflslt'})))                                                                         

Starting procedure to acquire darks 1.18e+02Hz or 0.009s.

    Current number of images = 5.

    Setting to 20 images.

Returning to intial conditions (pre-count).
        Total images per trigger are NOW:        5                                                                                                                                                   
    FCCD FCRIC gain value is NOW:        auto

[E 18:30:05.022 run_engine:1261] Run aborted
    Traceback (most recent call last):
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/run_engine.py", line 1150, in _run
        msg = self._plan_stack[-1].send(resp)
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 1294, in __call__
        return (yield from plan)
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 1149, in baseline_wrapper
        return (yield from plan_mutator(plan, insert_baseline))
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 169, in plan_mutator
        raise ex
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 122, in plan_mutator
        msg = plan_stack[-1].send(ret)
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 790, in monitor_during_wrapper
        return (yield from plan2)
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 169, in plan_mutator
        raise ex
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 122, in plan_mutator
        msg = plan_stack[-1].send(ret)
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 169, in plan_mutator
        raise ex
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 122, in plan_mutator
        msg = plan_stack[-1].send(ret)
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 848, in fly_during_wrapper
        return (yield from plan2)
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 169, in plan_mutator
        raise ex
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 122, in plan_mutator
        msg = plan_stack[-1].send(ret)
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 169, in plan_mutator
        raise ex
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 122, in plan_mutator
        msg = plan_stack[-1].send(ret)
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 267, in pchain
        rets.append((yield from p))
      File "/epics/xf/23id/ipython/profile_collection/startup/csx1/plans/dark_ff.py", line 171, in ct_dark_all
        yield from bps.mv(fccd.fccd1.capture_bgnd, 1)
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/plan_stubs.py", line 245, in mv
        ret = yield Msg('set', obj, val, group=group)
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 194, in plan_mutator
        inner_ret = yield msg
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 194, in plan_mutator
        inner_ret = yield msg
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 194, in plan_mutator
        inner_ret = yield msg
      [Previous line repeated 2 more times]
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/run_engine.py", line 1208, in _run
        new_response = yield from coro(msg)
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/asyncio/coroutines.py", line 120, in coro
        res = func(*args, **kw)
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/run_engine.py", line 1955, in _set
        ret = msg.obj.set(*msg.args, **kwargs)
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/ophyd/signal.py", line 941, in set
        return super().set(value, timeout=timeout, settle_time=settle_time)
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/ophyd/signal.py", line 185, in set
        raise RuntimeError('Another set() call is still in progress')
    RuntimeError: Another set() call is still in progress
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
~/Beamline/Commissioning/2019_08/defs.py in <module>
----> 1 RE(bpp.pchain(ct_dark_all(20), ct([fccd],num=1,md={'purpose':'test und offset -24 & fast shutter & 2x1 bflslt'})))

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/run_engine.py in __call__(self, *args, **metadata_kw)
    751                     # it (unless it is a canceled error)
    752                     if exc is not None:
--> 753                         raise exc
    754 
    755             if self._interrupted:

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/run_engine.py in _run(self)
   1260             self._reason = str(err)
   1261             self.log.exception("Run aborted")
-> 1262             raise err
   1263         finally:
   1264             # Some done_callbacks may still be alive in other threads.

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/run_engine.py in _run(self)
   1148                     else:
   1149                         try:
-> 1150                             msg = self._plan_stack[-1].send(resp)
   1151                         # We have exhausted the top generator
   1152                         except StopIteration:

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py in __call__(self, plan)
   1292         plan = monitor_during_wrapper(plan, self.monitors)
   1293         plan = baseline_wrapper(plan, self.baseline)
-> 1294         return (yield from plan)

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py in baseline_wrapper(plan, devices, name)
   1147         return (yield from plan)
   1148     else:
-> 1149         return (yield from plan_mutator(plan, insert_baseline))
   1150 
   1151 

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    167                     continue
    168                 else:
--> 169                     raise ex
    170         # if inserting / mutating, put new generator on the stack
    171         # and replace the current msg with the first element from the

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    120             ret = result_stack.pop()
    121             try:
--> 122                 msg = plan_stack[-1].send(ret)
    123             except StopIteration as e:
    124                 # discard the exhausted generator

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py in monitor_during_wrapper(plan, signals)
    788     plan1 = plan_mutator(plan, insert_after_open)
    789     plan2 = plan_mutator(plan1, insert_before_close)
--> 790     return (yield from plan2)
    791 
    792 

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    167                     continue
    168                 else:
--> 169                     raise ex
    170         # if inserting / mutating, put new generator on the stack
    171         # and replace the current msg with the first element from the

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    120             ret = result_stack.pop()
    121             try:
--> 122                 msg = plan_stack[-1].send(ret)
    123             except StopIteration as e:
    124                 # discard the exhausted generator

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    167                     continue
    168                 else:
--> 169                     raise ex
    170         # if inserting / mutating, put new generator on the stack
    171         # and replace the current msg with the first element from the

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    120             ret = result_stack.pop()
    121             try:
--> 122                 msg = plan_stack[-1].send(ret)
    123             except StopIteration as e:
    124                 # discard the exhausted generator

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py in fly_during_wrapper(plan, flyers)
    846     plan1 = plan_mutator(plan, insert_after_open)
    847     plan2 = plan_mutator(plan1, insert_before_close)
--> 848     return (yield from plan2)
    849 
    850 

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    167                     continue
    168                 else:
--> 169                     raise ex
    170         # if inserting / mutating, put new generator on the stack
    171         # and replace the current msg with the first element from the

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    120             ret = result_stack.pop()
    121             try:
--> 122                 msg = plan_stack[-1].send(ret)
    123             except StopIteration as e:
    124                 # discard the exhausted generator

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    167                     continue
    168                 else:
--> 169                     raise ex
    170         # if inserting / mutating, put new generator on the stack
    171         # and replace the current msg with the first element from the

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    120             ret = result_stack.pop()
    121             try:
--> 122                 msg = plan_stack[-1].send(ret)
    123             except StopIteration as e:
    124                 # discard the exhausted generator

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py in pchain(*args)
    265     rets = deque()
    266     for p in args:
--> 267         rets.append((yield from p))
    268     return tuple(rets)
    269 

/epics/xf/23id/ipython/profile_collection/startup/csx1/plans/dark_ff.py in ct_dark_all(numim, detectors)
    169         # SET TO 1 TO ARM FOR NEXT EVENT so that the FastCCD1 is
    170         # already bkg subt
--> 171         yield from bps.mv(fccd.fccd1.capture_bgnd, 1)
    172 
    173         # take darks

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/plan_stubs.py in mv(*args)
    243     step, = utils.merge_cycler(cyl)
    244     for obj, val in step.items():
--> 245         ret = yield Msg('set', obj, val, group=group)
    246         status_objects.append(ret)
    247     yield Msg('wait', None, group=group)

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    192         try:
    193             # yield out the 'current message' and collect the return
--> 194             inner_ret = yield msg
    195         except GeneratorExit:
    196             # special case GeneratorExit.  We must clean up all of our plans

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    192         try:
    193             # yield out the 'current message' and collect the return
--> 194             inner_ret = yield msg
    195         except GeneratorExit:
    196             # special case GeneratorExit.  We must clean up all of our plans

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    192         try:
    193             # yield out the 'current message' and collect the return
--> 194             inner_ret = yield msg
    195         except GeneratorExit:
    196             # special case GeneratorExit.  We must clean up all of our plans

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    192         try:
    193             # yield out the 'current message' and collect the return
--> 194             inner_ret = yield msg
    195         except GeneratorExit:
    196             # special case GeneratorExit.  We must clean up all of our plans

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    192         try:
    193             # yield out the 'current message' and collect the return
--> 194             inner_ret = yield msg
    195         except GeneratorExit:
    196             # special case GeneratorExit.  We must clean up all of our plans

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/run_engine.py in _run(self)
   1206                         # exceptions (coming in via throw) can be
   1207                         # raised
-> 1208                         new_response = yield from coro(msg)
   1209 
   1210                     # special case `CancelledError` and let the outer

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/asyncio/coroutines.py in coro(*args, **kw)
    118         @functools.wraps(func)
    119         def coro(*args, **kw):
--> 120             res = func(*args, **kw)
    121             if (base_futures.isfuture(res) or inspect.isgenerator(res) or
    122                     isinstance(res, CoroWrapper)):

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/run_engine.py in _set(self, msg)
   1953         group = kwargs.pop('group', None)
   1954         self._movable_objs_touched.add(msg.obj)
-> 1955         ret = msg.obj.set(*msg.args, **kwargs)
   1956         p_event = asyncio.Event(loop=self.loop)
   1957         pardon_failures = self._pardon_failures

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/ophyd/signal.py in set(self, value, timeout, settle_time)
    939         '''
    940         if not self._put_complete:
--> 941             return super().set(value, timeout=timeout, settle_time=settle_time)
    942 
    943         # using put completion:

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/ophyd/signal.py in set(self, value, timeout, settle_time)
    183 
    184         if self._set_thread is not None:
--> 185             raise RuntimeError('Another set() call is still in progress')
    186 
    187         st = Status(self)

RuntimeError: Another set() call is still in progress

In [235]: RE(bpp.pchain(ct_dark_all(20), ct([fccd],num=1,md={'purpose':'test und offset -24 & fast shutter & 2x1 bflslt'})))                                                                         

Starting procedure to acquire darks 1.18e+02Hz or 0.009s.

    Current number of images = 5.

    Setting to 20 images.

Returning to intial conditions (pre-count).
        Total images per trigger are NOW:        5                                                                                                                                                   
    FCCD FCRIC gain value is NOW:        auto

[E 18:30:09.994 run_engine:1261] Run aborted
    Traceback (most recent call last):
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/run_engine.py", line 1150, in _run
        msg = self._plan_stack[-1].send(resp)
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 1294, in __call__
        return (yield from plan)
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 1149, in baseline_wrapper
        return (yield from plan_mutator(plan, insert_baseline))
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 169, in plan_mutator
        raise ex
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 122, in plan_mutator
        msg = plan_stack[-1].send(ret)
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 790, in monitor_during_wrapper
        return (yield from plan2)
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 169, in plan_mutator
        raise ex
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 122, in plan_mutator
        msg = plan_stack[-1].send(ret)
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 169, in plan_mutator
        raise ex
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 122, in plan_mutator
        msg = plan_stack[-1].send(ret)
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 848, in fly_during_wrapper
        return (yield from plan2)
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 169, in plan_mutator
        raise ex
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 122, in plan_mutator
        msg = plan_stack[-1].send(ret)
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 169, in plan_mutator
        raise ex
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 122, in plan_mutator
        msg = plan_stack[-1].send(ret)
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 267, in pchain
        rets.append((yield from p))
      File "/epics/xf/23id/ipython/profile_collection/startup/csx1/plans/dark_ff.py", line 171, in ct_dark_all
        yield from bps.mv(fccd.fccd1.capture_bgnd, 1)
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/plan_stubs.py", line 245, in mv
        ret = yield Msg('set', obj, val, group=group)
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 194, in plan_mutator
        inner_ret = yield msg
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 194, in plan_mutator
        inner_ret = yield msg
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py", line 194, in plan_mutator
        inner_ret = yield msg
      [Previous line repeated 2 more times]
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/run_engine.py", line 1208, in _run
        new_response = yield from coro(msg)
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/asyncio/coroutines.py", line 120, in coro
        res = func(*args, **kw)
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/run_engine.py", line 1955, in _set
        ret = msg.obj.set(*msg.args, **kwargs)
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/ophyd/signal.py", line 941, in set
        return super().set(value, timeout=timeout, settle_time=settle_time)
      File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/ophyd/signal.py", line 185, in set
        raise RuntimeError('Another set() call is still in progress')
    RuntimeError: Another set() call is still in progress
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
~/Beamline/Commissioning/2019_08/defs.py in <module>
----> 1 RE(bpp.pchain(ct_dark_all(20), ct([fccd],num=1,md={'purpose':'test und offset -24 & fast shutter & 2x1 bflslt'})))

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/run_engine.py in __call__(self, *args, **metadata_kw)
    751                     # it (unless it is a canceled error)
    752                     if exc is not None:
--> 753                         raise exc
    754 
    755             if self._interrupted:

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/run_engine.py in _run(self)
   1260             self._reason = str(err)
   1261             self.log.exception("Run aborted")
-> 1262             raise err
   1263         finally:
   1264             # Some done_callbacks may still be alive in other threads.

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/run_engine.py in _run(self)
   1148                     else:
   1149                         try:
-> 1150                             msg = self._plan_stack[-1].send(resp)
   1151                         # We have exhausted the top generator
   1152                         except StopIteration:

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py in __call__(self, plan)
   1292         plan = monitor_during_wrapper(plan, self.monitors)
   1293         plan = baseline_wrapper(plan, self.baseline)
-> 1294         return (yield from plan)

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py in baseline_wrapper(plan, devices, name)
   1147         return (yield from plan)
   1148     else:
-> 1149         return (yield from plan_mutator(plan, insert_baseline))
   1150 
   1151 

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    167                     continue
    168                 else:
--> 169                     raise ex
    170         # if inserting / mutating, put new generator on the stack
    171         # and replace the current msg with the first element from the

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    120             ret = result_stack.pop()
    121             try:
--> 122                 msg = plan_stack[-1].send(ret)
    123             except StopIteration as e:
    124                 # discard the exhausted generator

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py in monitor_during_wrapper(plan, signals)
    788     plan1 = plan_mutator(plan, insert_after_open)
    789     plan2 = plan_mutator(plan1, insert_before_close)
--> 790     return (yield from plan2)
    791 
    792 

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    167                     continue
    168                 else:
--> 169                     raise ex
    170         # if inserting / mutating, put new generator on the stack
    171         # and replace the current msg with the first element from the

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    120             ret = result_stack.pop()
    121             try:
--> 122                 msg = plan_stack[-1].send(ret)
    123             except StopIteration as e:
    124                 # discard the exhausted generator

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    167                     continue
    168                 else:
--> 169                     raise ex
    170         # if inserting / mutating, put new generator on the stack
    171         # and replace the current msg with the first element from the

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    120             ret = result_stack.pop()
    121             try:
--> 122                 msg = plan_stack[-1].send(ret)
    123             except StopIteration as e:
    124                 # discard the exhausted generator

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py in fly_during_wrapper(plan, flyers)
    846     plan1 = plan_mutator(plan, insert_after_open)
    847     plan2 = plan_mutator(plan1, insert_before_close)
--> 848     return (yield from plan2)
    849 
    850 

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    167                     continue
    168                 else:
--> 169                     raise ex
    170         # if inserting / mutating, put new generator on the stack
    171         # and replace the current msg with the first element from the

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    120             ret = result_stack.pop()
    121             try:
--> 122                 msg = plan_stack[-1].send(ret)
    123             except StopIteration as e:
    124                 # discard the exhausted generator

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    167                     continue
    168                 else:
--> 169                     raise ex
    170         # if inserting / mutating, put new generator on the stack
    171         # and replace the current msg with the first element from the

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    120             ret = result_stack.pop()
    121             try:
--> 122                 msg = plan_stack[-1].send(ret)
    123             except StopIteration as e:
    124                 # discard the exhausted generator

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py in pchain(*args)
    265     rets = deque()
    266     for p in args:
--> 267         rets.append((yield from p))
    268     return tuple(rets)
    269 

/epics/xf/23id/ipython/profile_collection/startup/csx1/plans/dark_ff.py in ct_dark_all(numim, detectors)
    169         # SET TO 1 TO ARM FOR NEXT EVENT so that the FastCCD1 is
    170         # already bkg subt
--> 171         yield from bps.mv(fccd.fccd1.capture_bgnd, 1)
    172 
    173         # take darks

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/plan_stubs.py in mv(*args)
    243     step, = utils.merge_cycler(cyl)
    244     for obj, val in step.items():
--> 245         ret = yield Msg('set', obj, val, group=group)
    246         status_objects.append(ret)
    247     yield Msg('wait', None, group=group)

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    192         try:
    193             # yield out the 'current message' and collect the return
--> 194             inner_ret = yield msg
    195         except GeneratorExit:
    196             # special case GeneratorExit.  We must clean up all of our plans

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    192         try:
    193             # yield out the 'current message' and collect the return
--> 194             inner_ret = yield msg
    195         except GeneratorExit:
    196             # special case GeneratorExit.  We must clean up all of our plans

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    192         try:
    193             # yield out the 'current message' and collect the return
--> 194             inner_ret = yield msg
    195         except GeneratorExit:
    196             # special case GeneratorExit.  We must clean up all of our plans

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    192         try:
    193             # yield out the 'current message' and collect the return
--> 194             inner_ret = yield msg
    195         except GeneratorExit:
    196             # special case GeneratorExit.  We must clean up all of our plans

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    192         try:
    193             # yield out the 'current message' and collect the return
--> 194             inner_ret = yield msg
    195         except GeneratorExit:
    196             # special case GeneratorExit.  We must clean up all of our plans

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/run_engine.py in _run(self)
   1206                         # exceptions (coming in via throw) can be
   1207                         # raised
-> 1208                         new_response = yield from coro(msg)
   1209 
   1210                     # special case `CancelledError` and let the outer

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/asyncio/coroutines.py in coro(*args, **kw)
    118         @functools.wraps(func)
    119         def coro(*args, **kw):
--> 120             res = func(*args, **kw)
    121             if (base_futures.isfuture(res) or inspect.isgenerator(res) or
    122                     isinstance(res, CoroWrapper)):

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/run_engine.py in _set(self, msg)
   1953         group = kwargs.pop('group', None)
   1954         self._movable_objs_touched.add(msg.obj)
-> 1955         ret = msg.obj.set(*msg.args, **kwargs)
   1956         p_event = asyncio.Event(loop=self.loop)
   1957         pardon_failures = self._pardon_failures

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/ophyd/signal.py in set(self, value, timeout, settle_time)
    939         '''
    940         if not self._put_complete:
--> 941             return super().set(value, timeout=timeout, settle_time=settle_time)
    942 
    943         # using put completion:

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/ophyd/signal.py in set(self, value, timeout, settle_time)
    183 
    184         if self._set_thread is not None:
--> 185             raise RuntimeError('Another set() call is still in progress')
    186 
    187         st = Status(self)

RuntimeError: Another set() call is still in progress

In [236]: RE.abort()                                                                                                                                                                                 
---------------------------------------------------------------------------
TransitionError                           Traceback (most recent call last)
~/Beamline/Commissioning/2019_08/defs.py in <module>
----> 1 RE.abort()

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/bluesky/run_engine.py in abort(self, reason)
   1007         """
   1008         if self.state.is_idle:
-> 1009             raise TransitionError("RunEngine is already idle.")
   1010         print("Aborting: running cleanup and marking "
   1011               "exit_status as 'abort'...")

TransitionError: RunEngine is already idle.

In [237]:
ambarb commented 5 years ago

note that the exposure time is under .1 seconds for most of these failures. perhaps it is something related to this. Added 2 second sleep between setting the fccd.exposure and ct_dark_all()

happened again around 19:00 happened again around 22:42 happened again around 23:19 after resetart this time , noted error but things worked as they should.

Subscription value callback exception (EpicsSignal(read_pv='XF:23IDA-EPS{DP:1-Sh:1}Cmd:In-Cmd', name='inout_state1_cmd', parent='inout', value='None', timestamp=1565493340.553536, pv_kw={}, auto_monitor=False, string=True, write_pv='XF:23IDA-EPS{DP:1-Sh:1}Cmd:In-Cmd', limits=False, put_complete=False))
Traceback (most recent call last):
  File "/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/ophyd/ophydobj.py", line 269, in inner
    cb(*args, **kwargs)
  File "/epics/xf/23id/ipython/profile_collection/startup/csx1/devices/eps.py", line 55, in cmd_retry_cb
    time.sleep(.5)
NameError: name 'time' is not defined

and two more times before 1 am .

ambarb commented 5 years ago

This seems to happen only when having a short exposure time.

added sleep before setting gain and before launching count. see comments in dark_ff.py list as

    #adding because it gets  hung some where around here TODO
    #adding because it gets hung somewhere around here #8

if this works, then will remove one at a time to determine where the failure occurs

ambarb commented 5 years ago

Did not work. Issue in a sub-function.

In [6]: RE(myplan())                                                                                                                                           

Starting procedure to acquire darks 80.0Hz or 0.013s.

    Current number of images = 6000.

    Setting to 50 images.

/opt/conda_envs/collection-2019-3.0-csx/lib/python3.7/site-packages/epics/pv.py:110: UserWarning: The `context` kwarg for epics.get_pv() is deprecated. New PVs will _not_ be created in the requested context.
  'The `context` kwarg for epics.get_pv() is deprecated. New PVs '
Moving gains next.                                                                                                                                             

Gain bit set to 0 for a gain value of auto

Ready to begin data collection next.
Transient Scan ID: 120781     Time: 2019-08-12 03:00:19
Persistent Unique Scan ID: 'ea7fce8a-d6c4-461f-b307-08a4440efec0'

vs

Starting procedure to acquire darks 80.0Hz or 0.013s.                                                                                                          

    Current number of images = 250.

    Setting to 50 images.

  A 'deferred pause' has been requested. The RunEngine will pause at the next checkpoint. To pause immediately, hit Ctrl+C again in the next 10 seconds.       
Deferred pause acknowledged. Continuing to checkpoint.
^CPausing...
ambarb commented 4 years ago

@danielballan @mrakitin @jklynch i meant that this is the change from yesterday that we should try to merge and test from https://github.com/danielballan/xf23id1_profiles/commit/87001ee3f7b573b95f24694d23200d92763e8021 . It may address this issue, but I think it will fix another problem as well. I just need to find it.

ambarb commented 4 years ago

P.S. I am happy to test. I just don't know how to pick this very specific thing from Dan's repo. Willing to learn.

mrakitin commented 4 years ago

The following commands should help:

$ cd <you repo directory>
$ git remote -v
$ git remote add danielballan https://github.com/danielballan/xf23id1_profiles
$ git fetch danielballan
$ git checkout darkframes
mrakitin commented 4 years ago

Cherry-picking the commit from should then work (don't checkout to Dan's branch (last command above)):

# after you fetch Dan's remote in the previous commands, do the following:
$ git cherry-pick 2e57922

Same can be done with https://github.com/danielballan/xf23id1_profiles/commit/87001ee3f7b573b95f24694d23200d92763e8021.

ambarb commented 3 years ago

related to https://github.com/NSLS-II-CSX/xf23id1_profiles/issues/38

ambarb commented 1 year ago

This is duplicated in #38