Lambda acquisition stuck; 'Acquire' button not pressed

[qzhang234]

@prjemian The shutdown has officially started and Suresh has switched to Keenan's Lambda IOC without the previous .imm issue. When I ran AD_Acquire I get the following error:

Error from terminal output

``` In [17]: In [17]: AD_Acquire(lambdadet, 'A001_Test', 0.1, 0.11, 100, '/home/8ididata/2021-2/rigaku202107') Out[17]: In [18]: RE(AD_Acquire(lambdadet, 'A001_Test', 0.1, 0.11, 100, '/home/8ididata/2021-2/rigaku202107')) --------------------------------------------------------------------------- TimeoutError Traceback (most recent call last) ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/ophyd/signal.py in _get_with_timeout(self, pv, timeout, connection_timeout, as_string, form, use_monitor) 1112 try: -> 1113 self.wait_for_connection(timeout=connection_timeout) 1114 except TimeoutError as err: ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/ophyd/signal.py in wait_for_connection(self, timeout) 1489 '''Wait for the underlying signals to initialize or connect''' -> 1490 self._ensure_connected(self._read_pv, self._write_pv, timeout=timeout) 1491 ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/ophyd/signal.py in _ensure_connected(self, timeout, *pvs) 1045 if not pv.wait_for_connection(timeout=timeout): -> 1046 raise TimeoutError(f"{pv.pvname} could not connect within " 1047 f"{float(timeout):.3}-second timeout.") TimeoutError: 8LAMBDA1:IMMout:EnableCallbacks_RBV could not connect within 20.0-second timeout. The above exception was the direct cause of the following exception: ConnectionTimeoutError Traceback (most recent call last) in ----> 1 RE(AD_Acquire(lambdadet, 'A001_Test', 0.1, 0.11, 100, '/home/8ididata/2021-2/rigaku202107')) ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/run_engine.py in __call__(self, *args, **metadata_kw) 805 self._task_fut.add_done_callback(set_blocking_event) 806 --> 807 self._resume_task(init_func=_build_task) 808 809 if self._interrupted: ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/run_engine.py in _resume_task(self, init_func) 929 if (exc is not None 930 and not isinstance(exc, _RunEnginePanic)): --> 931 raise exc 932 933 def install_suspender(self, suspender): ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/run_engine.py in _run(self) 1498 exit_reason = str(err) 1499 self.log.exception("Run aborted") -> 1500 raise err 1501 finally: 1502 if not exit_reason: ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/run_engine.py in _run(self) 1363 else: 1364 try: -> 1365 msg = self._plan_stack[-1].send(resp) 1366 # We have exhausted the top generator 1367 except StopIteration: ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in __call__(self, plan) 1305 plan = monitor_during_wrapper(plan, self.monitors) 1306 plan = baseline_wrapper(plan, self.baseline) -> 1307 return (yield from plan) 1308 1309 ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in baseline_wrapper(plan, devices, name) 1160 return (yield from plan) 1161 else: -> 1162 return (yield from plan_mutator(plan, insert_baseline)) 1163 1164 ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc) 168 continue 169 else: --> 170 raise ex 171 # if inserting / mutating, put new generator on the stack 172 # and replace the current msg with the first element from the ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc) 121 ret = result_stack.pop() 122 try: --> 123 msg = plan_stack[-1].send(ret) 124 except StopIteration as e: 125 # discard the exhausted generator ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in monitor_during_wrapper(plan, signals) 801 plan1 = plan_mutator(plan, insert_after_open) 802 plan2 = plan_mutator(plan1, insert_before_close) --> 803 return (yield from plan2) 804 805 ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc) 168 continue 169 else: --> 170 raise ex 171 # if inserting / mutating, put new generator on the stack 172 # and replace the current msg with the first element from the ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc) 121 ret = result_stack.pop() 122 try: --> 123 msg = plan_stack[-1].send(ret) 124 except StopIteration as e: 125 # discard the exhausted generator ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc) 168 continue 169 else: --> 170 raise ex 171 # if inserting / mutating, put new generator on the stack 172 # and replace the current msg with the first element from the ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc) 121 ret = result_stack.pop() 122 try: --> 123 msg = plan_stack[-1].send(ret) 124 except StopIteration as e: 125 # discard the exhausted generator ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in fly_during_wrapper(plan, flyers) 859 plan1 = plan_mutator(plan, insert_after_open) 860 plan2 = plan_mutator(plan1, insert_before_close) --> 861 return (yield from plan2) 862 863 ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc) 168 continue 169 else: --> 170 raise ex 171 # if inserting / mutating, put new generator on the stack 172 # and replace the current msg with the first element from the ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc) 121 ret = result_stack.pop() 122 try: --> 123 msg = plan_stack[-1].send(ret) 124 except StopIteration as e: 125 # discard the exhausted generator ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc) 168 continue 169 else: --> 170 raise ex 171 # if inserting / mutating, put new generator on the stack 172 # and replace the current msg with the first element from the ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc) 121 ret = result_stack.pop() 122 try: --> 123 msg = plan_stack[-1].send(ret) 124 except StopIteration as e: 125 # discard the exhausted generator ~/.ipython-bluesky/profile_bluesky/startup/instrument/plans/xpcs_acquire.py in AD_Acquire(areadet, file_name, acquire_time, acquire_period, num_images, path, submit_xpcs_job, atten, md) 317 318 logger.info("calling full_acquire_procedure()") --> 319 return (yield from full_acquire_procedure(md=md)) ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/utils.py in dec_inner(*inner_args, **inner_kwargs) 1125 plan = gen_func(*inner_args, **inner_kwargs) 1126 plan = wrapper(plan, *args, **kwargs) -> 1127 return (yield from plan) 1128 return dec_inner 1129 return dec ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in stage_wrapper(plan, devices) 950 return (yield from plan) 951 --> 952 return (yield from finalize_wrapper(inner(), unstage_devices())) 953 954 ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in finalize_wrapper(plan, final_plan, pause_for_debug) 507 cleanup = True 508 try: --> 509 ret = yield from plan 510 except GeneratorExit: 511 cleanup = False ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in inner() 948 def inner(): 949 yield from stage_devices() --> 950 return (yield from plan) 951 952 return (yield from finalize_wrapper(inner(), unstage_devices())) ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/utils.py in dec_inner(*inner_args, **inner_kwargs) 1125 plan = gen_func(*inner_args, **inner_kwargs) 1126 plan = wrapper(plan, *args, **kwargs) -> 1127 return (yield from plan) 1128 return dec_inner 1129 return dec ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in monitor_during_wrapper(plan, signals) 801 plan1 = plan_mutator(plan, insert_after_open) 802 plan2 = plan_mutator(plan1, insert_before_close) --> 803 return (yield from plan2) 804 805 ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc) 76 # if we have a stashed exception, pass it along 77 try: ---> 78 msg = plan_stack[-1].throw(exception) 79 except StopIteration as e: 80 # discard the exhausted generator ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc) 76 # if we have a stashed exception, pass it along 77 try: ---> 78 msg = plan_stack[-1].throw(exception) 79 except StopIteration as e: 80 # discard the exhausted generator ~/.ipython-bluesky/profile_bluesky/startup/instrument/plans/xpcs_acquire.py in full_acquire_procedure(md) 285 logger.debug("before count()") 286 # yield from bp.count([areadet], md=md) --> 287 yield from inner_count([areadet], md=_md) 288 logger.debug("after count()") 289 ~/.ipython-bluesky/profile_bluesky/startup/instrument/plans/xpcs_acquire.py in inner_count(devices, md) 238 yield from bps.open_run(md=md) 239 for obj in devices: --> 240 yield from bps.stage(obj) 241 grp = bps._short_uid('trigger') 242 no_wait = True ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/plan_stubs.py in stage(obj) 694 :func:`bluesky.plan_stubs.unstage` 695 """ --> 696 return (yield Msg('stage', obj)) 697 698 ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc) 193 try: 194 # yield out the 'current message' and collect the return --> 195 inner_ret = yield msg 196 except GeneratorExit: 197 # special case GeneratorExit. We must clean up all of our plans ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc) 193 try: 194 # yield out the 'current message' and collect the return --> 195 inner_ret = yield msg 196 except GeneratorExit: 197 # special case GeneratorExit. We must clean up all of our plans ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc) 193 try: 194 # yield out the 'current message' and collect the return --> 195 inner_ret = yield msg 196 except GeneratorExit: 197 # special case GeneratorExit. We must clean up all of our plans ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc) 193 try: 194 # yield out the 'current message' and collect the return --> 195 inner_ret = yield msg 196 except GeneratorExit: 197 # special case GeneratorExit. We must clean up all of our plans ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc) 193 try: 194 # yield out the 'current message' and collect the return --> 195 inner_ret = yield msg 196 except GeneratorExit: 197 # special case GeneratorExit. We must clean up all of our plans ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc) 193 try: 194 # yield out the 'current message' and collect the return --> 195 inner_ret = yield msg 196 except GeneratorExit: 197 # special case GeneratorExit. We must clean up all of our plans ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc) 193 try: 194 # yield out the 'current message' and collect the return --> 195 inner_ret = yield msg 196 except GeneratorExit: 197 # special case GeneratorExit. We must clean up all of our plans ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/run_engine.py in _run(self) 1423 # exceptions (coming in via throw) can be 1424 # raised -> 1425 new_response = await coro(msg) 1426 1427 # special case `CancelledError` and let the outer ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/run_engine.py in _stage(self, msg) 2166 if not hasattr(obj, 'stage'): 2167 return [] -> 2168 result = obj.stage() 2169 self._staged.add(obj) # add first in case of failure below 2170 await self._reset_checkpoint_state_coro() ~/.ipython-bluesky/profile_bluesky/startup/instrument/devices/lambda_750k.py in stage(self) 403 404 def stage(self): --> 405 super().stage() 406 root = os.path.join("/", "home", "8-id-i/") 407 if self._file_path.startswith("/data/"): ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/ophyd/device.py in stage(self) 557 device = getattr(self, attr) 558 if hasattr(device, 'stage'): --> 559 device.stage() 560 devices_staged.append(device) 561 except Exception: ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/ophyd/device.py in stage(self) 535 536 # Read current values, to be restored by unstage() --> 537 original_vals = {sig: sig.get() for sig in stage_sigs} 538 539 # We will add signals and values from original_vals to ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/ophyd/device.py in (.0) 535 536 # Read current values, to be restored by unstage() --> 537 original_vals = {sig: sig.get() for sig in stage_sigs} 538 539 # We will add signals and values from original_vals to ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/ophyd/signal.py in get(self, as_string, timeout, connection_timeout, form, use_monitor, **kwargs) 1175 use_monitor = self._auto_monitor 1176 -> 1177 info = self._get_with_timeout( 1178 self._read_pv, timeout, connection_timeout, as_string, form, use_monitor 1179 ) ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/ophyd/signal.py in _get_with_timeout(self, pv, timeout, connection_timeout, as_string, form, use_monitor) 1113 self.wait_for_connection(timeout=connection_timeout) 1114 except TimeoutError as err: -> 1115 raise ConnectionTimeoutError( 1116 f"Failed to connect to {pv.pvname} " 1117 f"within {connection_timeout:.2f} sec") from err ConnectionTimeoutError: Failed to connect to 8LAMBDA1:IMMout:EnableCallbacks_RBV within 20.00 sec In [19]: ``` <\details>

[qzhang234]

Suresh mentioned that since the IOC has been rebuilt, there are discrepancies in some of the PV names. I did notice that although acquisition time and period seem to update upon running the Bluesky plan, the Acquire button is not pressed and the PV name 8LAMBDA1:cam1:Acquire is not defined anywhere in the current Bluesky version. Maybe lambda_750k.py in /instrument/device needs to be rebuilt?

[qzhang234]

Pete and I discussed the issue and we went over your suggestion of changing IMMout to imm1. The problem is, Lambda calls ad_imm_plugin.py, and if we want to implement the change, we need to change that file which will change the behavior for ALL detectors that use .imm.

@sureshnaps Is this the direction we want to go?

[qzhang234]

@prjemian I talked to Suresh earlier today and we at 8-ID decided to switch from IMMout to imm1 for all occurrences since Rigaku imm plugin is also using imm1. Could you provide some pointers on how we should proceed?

[prjemian]

There are at least two different types of edit to make. In one case, the IMMout plugin is named and the IMM1 plugin can replace. The other case is when various IMMxxx plugins are in a list. In that case, remove the IMMout from the list. Do this in a branch so we can look at it together.

[qzhang234]

This issue is not relevant anymore. Closing it now.