error msg on abort and cannot use RE afterwards. future aborts marked as success eventhough state is running.

[ambarb]

had to exit bluesky to recover. no other changes made to epics after exiting and restarting bluesky to get a scan to work.

In [586]: RE(pchain(dscan(diag6_pid,-50,50,50)))
Transient Scan ID: 76126 @ 2017/03/11 08:33:36
Persistent Unique Scan ID: 'cdcb3029-5ce4-450d-9e25-28782d6239aa'
+-----------+------------+-------------------+----------------+--------------------+--------------------+-----------------------+-----------------------+-----------------------+
|   seq_num |       time | diag6_pid_actuate | diag6_pid_done | diag6_pid_readback | diag6_pid_setpoint | diag6_pid_stop_signal | dif_beam_stats2_total | dif_beam_stats3_total |
+-----------+------------+-------------------+----------------+--------------------+--------------------+-----------------------+-----------------------+-----------------------+
|         1 | 08:33:42.5 |                 1 |              1 |           1130.085 |           1130.085 |                     1 |              25571445 |              17367392 |
|         2 | 08:33:44.6 |                 1 |              1 |           1132.085 |           1132.085 |                     1 |              25685179 |              17457479 |
|         3 | 08:33:47.0 |                 1 |              1 |           1134.085 |           1134.085 |                     1 |              25197111 |              17052474 |
|         4 | 08:33:48.8 |                 1 |              1 |           1136.085 |           1136.085 |                     1 |              24154101 |              16220603 |
|         5 | 08:33:51.2 |                 1 |              1 |           1138.085 |           1138.085 |                     1 |              25239566 |              17093935 |
|         6 | 08:33:53.3 |                 1 |              1 |           1140.085 |           1140.085 |                     1 |              25523452 |              17328237 |
|         7 | 08:33:55.6 |                 1 |              1 |           1142.085 |           1142.085 |                     1 |              25787525 |              17531639 |
|         8 | 08:33:57.3 |                 1 |              1 |           1144.085 |           1144.085 |                     1 |              10614157 |               5710319 |
|         9 | 08:33:59.2 |                 1 |              1 |           1146.085 |           1146.085 |                     1 |               6308338 |               2487138 |
|        10 | 08:34:01.0 |                 1 |              1 |           1148.085 |           1148.085 |                     1 |               6557722 |               2663512 |
|        11 | 08:34:02.8 |                 1 |              1 |           1150.085 |           1150.085 |                     1 |               6749731 |               2811460 |
|        12 | 08:34:04.4 |                 1 |              1 |           1152.085 |           1152.085 |                     1 |               7324978 |               3238961 |
|        13 | 08:34:06.2 |                 1 |              1 |           1154.085 |           1154.085 |                     1 |               7866944 |               3650116 |
|        14 | 08:34:08.2 |                 1 |              1 |           1156.085 |           1156.085 |                     1 |               8243554 |               3925739 |
|        15 | 08:34:09.9 |                 1 |              1 |           1158.085 |           1158.085 |                     1 |               8769940 |               4313828 |
|        16 | 08:34:11.5 |                 1 |              1 |           1160.085 |           1160.085 |                     1 |               9667708 |               4990055 |
|        17 | 08:34:13.1 |                 1 |              1 |           1162.085 |           1162.085 |                     1 |              10812998 |               5864121 |
|        18 | 08:34:15.0 |                 1 |              1 |           1164.085 |           1164.085 |                     1 |              12422295 |               7113232 |
|        19 | 08:34:17.2 |                 1 |              1 |           1166.085 |           1166.085 |                     1 |              14659588 |               8831289 |
|        20 | 08:34:18.9 |                 1 |              1 |           1168.085 |           1168.085 |                     1 |              15306041 |               9325632 |
|        21 | 08:34:20.6 |                 1 |              1 |           1170.085 |           1170.085 |                     1 |              17760357 |              11249831 |
|        22 | 08:34:22.3 |                 1 |              1 |           1172.085 |           1172.085 |                     1 |              17100033 |              10740211 |
|        23 | 08:34:24.0 |                 1 |              1 |           1174.085 |           1174.085 |                     1 |              20562032 |              13446594 |
|        24 | 08:34:25.6 |                 1 |              1 |           1176.085 |           1176.085 |                     1 |              21439154 |              14129727 |
|        25 | 08:34:27.3 |                 1 |              1 |           1178.085 |           1178.085 |                     1 |              23560177 |              15803649 |
|        26 | 08:34:29.3 |                 1 |              1 |           1180.085 |           1180.085 |                     1 |              24133846 |              16253180 |
|        27 | 08:34:31.3 |                 1 |              1 |           1182.085 |           1182.085 |                     1 |              24509193 |              16523196 |
|        28 | 08:34:32.9 |                 1 |              1 |           1184.085 |           1184.085 |                     1 |              25157001 |              17040951 |
|        29 | 08:34:34.6 |                 1 |              1 |           1186.085 |           1186.085 |                     1 |              25616836 |              17398690 |
|        30 | 08:34:36.3 |                 1 |              1 |           1188.085 |           1188.085 |                     1 |              24503206 |              16470223 |
|        31 | 08:34:38.2 |                 1 |              1 |           1190.085 |           1190.085 |                     1 |              22551459 |              14904050 |
|        32 | 08:34:40.0 |                 1 |              1 |           1192.085 |           1192.085 |                     1 |              24011196 |              16052275 |
|        33 | 08:34:41.7 |                 1 |              1 |           1194.085 |           1194.085 |                     1 |              22898205 |              15155103 |
|        34 | 08:34:43.5 |                 1 |              1 |           1196.085 |           1196.085 |                     1 |              21455331 |              13986328 |
|        35 | 08:34:45.6 |                 1 |              1 |           1198.085 |           1198.085 |                     1 |              19483321 |              12396560 |
|        36 | 08:34:47.6 |                 1 |              1 |           1200.085 |           1200.085 |                     1 |              17511933 |              10838020 |
|        37 | 08:34:49.3 |                 1 |              1 |           1202.085 |           1202.085 |                     1 |              15656443 |               9361345 |
|        38 | 08:34:51.1 |                 1 |              1 |           1204.085 |           1204.085 |                     1 |              14732065 |               8666108 |
|        39 | 08:34:53.2 |                 1 |              1 |           1206.085 |           1206.085 |                     1 |              12620477 |               7031637 |
|        40 | 08:34:55.2 |                 1 |              1 |           1208.085 |           1208.085 |                     1 |              10463885 |               5408347 |
|        41 | 08:34:57.4 |                 1 |              1 |           1210.085 |           1210.085 |                     1 |               9598346 |               4748145 |
|        42 | 08:34:59.4 |                 1 |              1 |           1212.085 |           1212.085 |                     1 |               8736213 |               4124183 |
|        43 | 08:35:01.1 |                 1 |              1 |           1214.085 |           1214.085 |                     1 |               7918561 |               3521046 |
|        44 | 08:35:03.2 |                 1 |              1 |           1216.085 |           1216.085 |                     1 |               7042199 |               2901610 |
|        45 | 08:35:04.8 |                 1 |              1 |           1218.085 |           1218.085 |                     1 |               6499654 |               2517022 |
|        46 | 08:35:07.2 |                 1 |              1 |           1220.085 |           1220.085 |                     1 |               6116325 |               2263813 |
|        47 | 08:35:08.8 |                 1 |              1 |           1222.085 |           1222.085 |                     1 |               6178755 |               2311104 |
|        48 | 08:35:10.6 |                 1 |              1 |           1224.085 |           1224.085 |                     1 |               5617261 |               1923846 |
|        49 | 08:35:12.4 |                 1 |              1 |           1226.085 |           1226.085 |                     1 |               5638628 |               1936070 |
+-----------+------------+-------------------+----------------+--------------------+--------------------+-----------------------+-----------------------+-----------------------+
|   seq_num |       time | diag6_pid_actuate | diag6_pid_done | diag6_pid_readback | diag6_pid_setpoint | diag6_pid_stop_signal | dif_beam_stats2_total | dif_beam_stats3_total |
+-----------+------------+-------------------+----------------+--------------------+--------------------+-----------------------+-----------------------+-----------------------+
|        50 | 08:35:14.2 |                 1 |              1 |           1228.085 |           1228.085 |                     1 |               5420771 |               1799899 |
|        51 | 08:35:16.0 |                 1 |              1 |           1230.085 |           1230.085 |                     1 |               5246205 |               1682388 |
+-----------+------------+-------------------+----------------+--------------------+--------------------+-----------------------+-----------------------+-----------------------+
generator dscan ['cdcb30'] (scan num: 76126)
Out[586]: ['cdcb3029-5ce4-450d-9e25-28782d6239aa']

In [587]: olog('Scan #{} and {} at {:.2f} eV to set pid {:.1f} and epu gap {:.3f} respectively'.format(db[-1].start.scan_id,db[-1].start.scan_id-1,pgm_en.readback.value,pid_sp,epu_gap_max))

In [588]: RE(dscan(pgm_en,-5,10,45))
Transient Scan ID: 76127 @ 2017/03/11 08:37:55
Persistent Unique Scan ID: 'b0e35410-2816-40c4-8c4f-16cf32b65d4f'
^CA 'deferred pause' has been requested. The RunEngine will pause at the next checkpoint. To pause immediately, hit Ctrl+C again in the next 10 seconds.
Deferred pause acknowledged. Continuing to checkpoint.
^C
Your RunEngine is entering a paused state. These are your options for changing
the state of the RunEngine:

RE.resume()    Resume the plan.
RE.abort()     Perform cleanup, then kill plan. Mark exit_stats='aborted'.
RE.stop()      Perform cleanup, then kill plan. Mark exit_status='success'.
RE.halt()      Emergency Stop: Do not perform cleanup --- just stop.

Pausing...
Out[588]: ['b0e35410-2816-40c4-8c4f-16cf32b65d4f']

In [589]: RE.abort()
Aborting: running cleanup and marking exit_status as 'abort'...

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/epics/ca.py:1486: UserWarning: ca.get_timevars('XF:23ID1-OP{Mono}Enrgy-SP') timed out after 5.00 seconds.
  warnings.warn(msg % (name(chid), timeout))
/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/epics/ca.py:1486: UserWarning: ca.get_timevars('XF:23ID1-OP{Mono}Cmd:Stop-Cmd') timed out after 5.00 seconds.
  warnings.warn(msg % (name(chid), timeout))
---------------------------------------------------------------------------
FailedStatus                              Traceback (most recent call last)
/home/xf23id1/Beamline/Commissioning/2017_03/en_verify.py in <module>()
----> 1 RE.abort()

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/run_engine.py in abort(self, reason)
    802             task.cancel()
    803         if self.state == 'paused':
--> 804             self._resume_event_loop()
    805 
    806     def stop(self):

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/run_engine.py in _resume_event_loop(self)
    674                     # it (unless it is a canceled error)
    675                     if exc is not None:
--> 676                         raise exc
    677 
    678     def install_suspender(self, suspender):

/opt/conda_envs/collection-17Q1.0/lib/python3.5/asyncio/tasks.py in _step(***failed resolving arguments***)
    237                 # We use the `send` method directly, because coroutines
    238                 # don't have `__iter__` and `__next__` methods.
--> 239                 result = coro.send(None)
    240             else:
    241                 result = coro.throw(exc)

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/run_engine.py in _run(self)
   1002             self.log.error("Run aborted")
   1003             self.log.error("%r", err)
-> 1004             raise err
   1005         finally:
   1006             # Some done_callbacks may still be alive in other threads.

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/run_engine.py in _run(self)
    900                         resp = self._response_stack.pop()
    901                         try:
--> 902                             msg = self._plan_stack[-1].send(resp)
    903                         # We have exhausted the top generator
    904                         except StopIteration:

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/spec_api.py in dscan(motor, start, finish, intervals, time, md)
    330 
    331     return (yield from inner(gs.DETS, motor, start, finish, 1 + intervals,
--> 332                              md=md))
    333 gs.SUB_FACTORIES['dscan'] = [setup_livetable,
    334                              setup_plot,

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/plans.py in dec_inner(*inner_args, **inner_kwargs)
     45                 plan = gen_func(*inner_args, **inner_kwargs)
     46                 plan = wrapper(plan, *args, **kwargs)
---> 47                 return (yield from plan)
     48             return dec_inner
     49         return dec

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/plans.py in subs_wrapper(plan, subs)
    928 
    929     return (yield from finalize_wrapper(_inner_plan(),
--> 930                                         _unsubscribe()))
    931 
    932 

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/plans.py in finalize_wrapper(plan, final_plan, pause_for_debug)
   1023     cleanup = True
   1024     try:
-> 1025         ret = yield from plan
   1026     except GeneratorExit:
   1027         cleanup = False

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/plans.py in _inner_plan()
    925     def _inner_plan():
    926         yield from _subscribe()
--> 927         return (yield from plan)
    928 
    929     return (yield from finalize_wrapper(_inner_plan(),

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/plans.py in dec_inner(*inner_args, **inner_kwargs)
     45                 plan = gen_func(*inner_args, **inner_kwargs)
     46                 plan = wrapper(plan, *args, **kwargs)
---> 47                 return (yield from plan)
     48             return dec_inner
     49         return dec

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/plans.py in configure_count_time_wrapper(plan, time)
   1664     if time is None:
   1665         # no-op
-> 1666         return (yield from plan)
   1667     else:
   1668         return (yield from finalize_wrapper(plan_mutator(plan, insert_set),

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/plans.py in dec_inner(*inner_args, **inner_kwargs)
     45                 plan = gen_func(*inner_args, **inner_kwargs)
     46                 plan = wrapper(plan, *args, **kwargs)
---> 47                 return (yield from plan)
     48             return dec_inner
     49         return dec

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/plans.py in fly_during_wrapper(plan, flyers)
   1381     plan1 = plan_mutator(plan, insert_after_open)
   1382     plan2 = plan_mutator(plan1, insert_before_close)
-> 1383     return (yield from plan2)
   1384 
   1385 

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/plans.py in plan_mutator(plan, msg_proc)
    125             # if we have a stashed exception, pass it along
    126             try:
--> 127                 msg = plan_stack[-1].throw(exception)
    128             except Exception as e:
    129                 # if we catch an exception,

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/plans.py in plan_mutator(plan, msg_proc)
    125             # if we have a stashed exception, pass it along
    126             try:
--> 127                 msg = plan_stack[-1].throw(exception)
    128             except Exception as e:
    129                 # if we catch an exception,

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/plans.py in dec_inner(*inner_args, **inner_kwargs)
     45                 plan = gen_func(*inner_args, **inner_kwargs)
     46                 plan = wrapper(plan, *args, **kwargs)
---> 47                 return (yield from plan)
     48             return dec_inner
     49         return dec

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/plans.py in monitor_during_wrapper(plan, signals)
   1323     plan1 = plan_mutator(plan, insert_after_open)
   1324     plan2 = plan_mutator(plan1, insert_before_close)
-> 1325     return (yield from plan2)
   1326 
   1327 

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/plans.py in plan_mutator(plan, msg_proc)
    125             # if we have a stashed exception, pass it along
    126             try:
--> 127                 msg = plan_stack[-1].throw(exception)
    128             except Exception as e:
    129                 # if we catch an exception,

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/plans.py in plan_mutator(plan, msg_proc)
    125             # if we have a stashed exception, pass it along
    126             try:
--> 127                 msg = plan_stack[-1].throw(exception)
    128             except Exception as e:
    129                 # if we catch an exception,

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/plans.py in dec_inner(*inner_args, **inner_kwargs)
     45                 plan = gen_func(*inner_args, **inner_kwargs)
     46                 plan = wrapper(plan, *args, **kwargs)
---> 47                 return (yield from plan)
     48             return dec_inner
     49         return dec

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/plans.py in baseline_wrapper(plan, devices, name)
   1713         return (yield from plan)
   1714     else:
-> 1715         return (yield from plan_mutator(plan, insert_baseline))
   1716 
   1717 

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/plans.py in plan_mutator(plan, msg_proc)
    125             # if we have a stashed exception, pass it along
    126             try:
--> 127                 msg = plan_stack[-1].throw(exception)
    128             except Exception as e:
    129                 # if we catch an exception,

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/spec_api.py in inner_spec_plan(md, *args, **kwargs)
    243                    gs.MD_TIME_KEY: time}
    244             _md.update(md or {})
--> 245             return (yield from func(*args, md=_md, **kwargs))
    246         return inner_spec_plan
    247     return outer

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/plans.py in relative_scan(detectors, motor, start, stop, num, per_step, md)
   2183                                 num, per_step=per_step, md=_md))
   2184 
-> 2185     return (yield from inner_relative_scan())
   2186 
   2187 

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/plans.py in dec_inner(*inner_args, **inner_kwargs)
     45                 plan = gen_func(*inner_args, **inner_kwargs)
     46                 plan = wrapper(plan, *args, **kwargs)
---> 47                 return (yield from plan)
     48             return dec_inner
     49         return dec

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/plans.py in reset_positions_wrapper(plan, devices)
   1616 
   1617     return (yield from finalize_wrapper(plan_mutator(plan, insert_reads),
-> 1618                                         reset()))
   1619 
   1620 

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/plans.py in finalize_wrapper(plan, final_plan, pause_for_debug)
   1042         # https://docs.python.org/3/reference/expressions.html?#generator.close
   1043         if cleanup:
-> 1044             yield from ensure_generator(final_plan_instance)
   1045     return ret
   1046 

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/plans.py in reset()
   1612         blk_grp = 'reset-{}'.format(str(uuid.uuid4())[:6])
   1613         for k, v in initial_positions.items():
-> 1614             yield Msg('set', k, v, group=blk_grp)
   1615         yield Msg('wait', None, group=blk_grp)
   1616 

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/plans.py in plan_mutator(plan, msg_proc)
    213         try:
    214             # yield out the 'current message' and collect the return
--> 215             inner_ret = yield msg
    216         except GeneratorExit:
    217             # special case GeneratorExit.  We must clean up all of our plans

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/plans.py in plan_mutator(plan, msg_proc)
    213         try:
    214             # yield out the 'current message' and collect the return
--> 215             inner_ret = yield msg
    216         except GeneratorExit:
    217             # special case GeneratorExit.  We must clean up all of our plans

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/plans.py in plan_mutator(plan, msg_proc)
    213         try:
    214             # yield out the 'current message' and collect the return
--> 215             inner_ret = yield msg
    216         except GeneratorExit:
    217             # special case GeneratorExit.  We must clean up all of our plans

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/plans.py in plan_mutator(plan, msg_proc)
    213         try:
    214             # yield out the 'current message' and collect the return
--> 215             inner_ret = yield msg
    216         except GeneratorExit:
    217             # special case GeneratorExit.  We must clean up all of our plans

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/plans.py in plan_mutator(plan, msg_proc)
    213         try:
    214             # yield out the 'current message' and collect the return
--> 215             inner_ret = yield msg
    216         except GeneratorExit:
    217             # special case GeneratorExit.  We must clean up all of our plans

FailedStatus: MoveStatus(done=True, pos=pgm_en, elapsed=5.9, success=False, settle_time=0.0)

In [590]: 

In [590]: RE.abort()
---------------------------------------------------------------------------
TransitionError                           Traceback (most recent call last)
/home/xf23id1/Beamline/Commissioning/2017_03/en_verify.py in <module>()
----> 1 RE.abort()

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/run_engine.py in abort(self, reason)
    792         """
    793         if self.state.is_idle:
--> 794             raise TransitionError("RunEngine is already idle.")
    795         print("Aborting: running cleanup and marking "
    796               "exit_status as 'abort'...")

TransitionError: RunEngine is already idle.

In [591]: gs.DETS.remove(dif_beam)

In [592]: gs.TABLE_COLS = ['sclr_ch3','sclr_ch2','sclr_ch6']; gs.PLOT_Y = 'sclr_ch2'

In [593]: RE(dscan(pgm_en,-5,10,45))
/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/epics/ca.py:1435: UserWarning: ca.get_ctrlvars('XF:23ID1-OP{Mono}Enrgy-I') timed out after 5.00 seconds.
  warnings.warn(msg % (name(chid), timeout))
/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/epics/ca.py:1248: UserWarning: ca.get('XF:23ID1-OP{Mono}Enrgy-SP') timed out after 1.00 seconds.
  warnings.warn(msg % (name(chid), timeout))
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/home/xf23id1/Beamline/Commissioning/2017_03/en_verify.py in <module>()
----> 1 RE(dscan(pgm_en,-5,10,45))

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/run_engine.py in __call__(self, plan, subs, raise_if_interrupted, **metadata_kw)
    596                     # it (unless it is a canceled error)
    597                     if exc is not None:
--> 598                         raise exc
    599 
    600             if raise_if_interrupted and self._interrupted:

/opt/conda_envs/collection-17Q1.0/lib/python3.5/asyncio/tasks.py in _step(***failed resolving arguments***)
    237                 # We use the `send` method directly, because coroutines
    238                 # don't have `__iter__` and `__next__` methods.
--> 239                 result = coro.send(None)
    240             else:
    241                 result = coro.throw(exc)

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/run_engine.py in _run(self)
   1002             self.log.error("Run aborted")
   1003             self.log.error("%r", err)
-> 1004             raise err
   1005         finally:
   1006             # Some done_callbacks may still be alive in other threads.

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/run_engine.py in _run(self)
    900                         resp = self._response_stack.pop()
    901                         try:
--> 902                             msg = self._plan_stack[-1].send(resp)
    903                         # We have exhausted the top generator
    904                         except StopIteration:

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/spec_api.py in dscan(motor, start, finish, intervals, time, md)
    327     motors = [motor]
    328 
--> 329     inner = inner_spec_decorator('dscan', time, motors)(relative_scan)
    330 
    331     return (yield from inner(gs.DETS, motor, start, finish, 1 + intervals,

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/spec_api.py in inner_spec_decorator(plan_name, time, motors, **subs_kwargs)
    227 
    228 def inner_spec_decorator(plan_name, time, motors, **subs_kwargs):
--> 229     subs = construct_subs(plan_name, motors=motors, **subs_kwargs)
    230 
    231     # create the paramterized decorator

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/spec_api.py in construct_subs(plan_name, **kwargs)
    167 
    168             update_sub_lists(ret,
--> 169                              normalize_subs_input(factory(**fact_kwargs)))
    170 
    171     return ret

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/spec_api.py in setup_livetable(motors, gs)
     99     """Setup a LiveTable by inspecting motors and gs.
    100     """
--> 101     return LiveTable(motors + [gs.PLOT_Y] + gs.TABLE_COLS)
    102 
    103 

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/callbacks/core.py in __init__(self, fields, stream_name, print_header_interval, min_width, default_prec, extra_pad, logbook)
    503         self._header_interval = print_header_interval
    504         # expand objects
--> 505         self._fields = _get_obj_fields(fields)
    506         self._stream = stream_name
    507         self._start = None

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/callbacks/core.py in _get_obj_fields(fields)
    226         else:
    227             try:
--> 228                 field_list = sorted(field.describe().keys())
    229             except AttributeError:
    230                 raise ValueError("Fields must be strings or objects with a "

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/ophyd/device.py in describe(self)
    725         '''describe the read data keys' data types and other metadata'''
    726         res = super().describe()
--> 727         res.update(self._describe_attr_list(self.read_attrs))
    728         return res
    729 

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/ophyd/device.py in _describe_attr_list(self, attr_list, config)
    718                 desc.update(obj.describe_configuration())
    719 
--> 720             desc.update(obj.describe())
    721 
    722         return desc

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/ophyd/signal.py in describe(self)
    562 
    563         val = self.value
--> 564         desc['dtype'] = data_type(val)
    565         desc['shape'] = data_shape(val)
    566 

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/ophyd/utils/epics_pvs.py in data_type(val)
    462             return json_type
    463     # no legit type found...
--> 464     raise ValueError('{} not a valid type (int, float, ndarray, str)'.format(val))
    465 
    466 

ValueError: None not a valid type (int, float, ndarray, str)

In [594]: RE(dscan(pgm_en,-5,10,45))
Transient Scan ID: 76128 @ 2017/03/11 08:39:08
Persistent Unique Scan ID: '03a74b88-5e21-4938-b988-950ea55755ff'
/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/epics/ca.py:1486: UserWarning: ca.get_timevars('XF:23ID1-OP{Mono}Enrgy-SP') timed out after 5.00 seconds.
  warnings.warn(msg % (name(chid), timeout))

^CA 'deferred pause' has been requested. The RunEngine will pause at the next checkpoint. To pause immediately, hit Ctrl+C again in the next 10 seconds.
Deferred pause acknowledged. Continuing to checkpoint.
^C
Your RunEngine is entering a paused state. These are your options for changing
the state of the RunEngine:

RE.resume()    Resume the plan.
RE.abort()     Perform cleanup, then kill plan. Mark exit_stats='aborted'.
RE.stop()      Perform cleanup, then kill plan. Mark exit_status='success'.
RE.halt()      Emergency Stop: Do not perform cleanup --- just stop.

Pausing...
Out[594]: ['03a74b88-5e21-4938-b988-950ea55755ff']

In [595]: RE.stop()
Stopping: running cleanup and marking exit_status as 'success'...

^C^C^C
^C
^C
^C
^C
^C
^C
^C
^C---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
/home/xf23id1/Beamline/Commissioning/2017_03/en_verify.py in <module>()
----> 1 RE.stop()

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/run_engine.py in stop(self)
    816         self._task.cancel()
    817         if self.state == 'paused':
--> 818             self._resume_event_loop()
    819 
    820     def halt(self):

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/run_engine.py in _resume_event_loop(self)
    663                 return
    664             try:
--> 665                 self.loop.run_forever()
    666             finally:
    667                 if self._task.done():

/opt/conda_envs/collection-17Q1.0/lib/python3.5/asyncio/base_events.py in run_forever(self)
    293         try:
    294             while True:
--> 295                 self._run_once()
    296                 if self._stopping:
    297                     break

/opt/conda_envs/collection-17Q1.0/lib/python3.5/asyncio/base_events.py in _run_once(self)
   1216                            timeout * 1e3, dt * 1e3)
   1217         else:
-> 1218             event_list = self._selector.select(timeout)
   1219         self._process_events(event_list)
   1220 

/opt/conda_envs/collection-17Q1.0/lib/python3.5/selectors.py in select(self, timeout)
    430             ready = []
    431             try:
--> 432                 fd_event_list = self._epoll.poll(timeout, max_ev)
    433             except InterruptedError:
    434                 return ready

/opt/conda_envs/collection-17Q1.0/lib/python3.5/site-packages/bluesky/utils.py in handler(signum, frame)
    145                 orig_func = self.original_handler
    146                 self.release()
--> 147                 orig_func(signum, frame)
    148 
    149         signal.signal(self.sig, handler)

KeyboardInterrupt: 

In [596]: 

In [596]: RE.abort()
Aborting: running cleanup and marking exit_status as 'abort'...

In [597]: RE.state
Out[597]: 'running'

In [598]: RE.abort()
Aborting: running cleanup and marking exit_status as 'abort'...

In [599]: RE.state
Out[599]: 'running'

[tacaswell]

It looks like your PGM started to miss-behave (all of those CA timeout warnings). The exceptions that look like rraise ValueError('{} not a valid type (int, float, ndarray, str)'.format(val)) are from when reading a value from epics failed and we got back None instead of a number.

I suspect that when you hit ^C 10 times it was hung on one of those reads, which is an escape-hatch from the RE, otherwise you would have had to kill all of python to escape, but left RE in an inconsistent state....

Not clear to me why the PGM raised a failed status above though. Is there a timeout on it's moves?