search

NSLS-II / Bug-Reports

Unified issue-tracker for bugs in the data acquisition, management, and analysis software at NSLS-II
BSD 3-Clause "New" or "Revised" License
2 stars 5 forks source link

Bluesky fails to operate Prosilica Camera #202

Closed bisogni closed 4 years ago

bisogni commented 4 years ago

Our "m3_diag_cam" has been so far enabled to be 'acquired' in Bluesky using 'count', for example. At the restart of operation as of yesterday, this command doesn't work anymore. We see the camera being staged and going into "Multiple" Image Mode, however then the timeout error comes out (see below). This camera is needed to perform standard beamline alignment, so it would be helpful to have a quick feedback on this issue, if possible.

'''' In [1]: RE(count([m3_diag_cam],num=1))

TimeoutError Traceback (most recent call last)

in ----> 1 RE(count([m3_diag_cam],num=1)) /opt/conda_envs/collection-2019-3.0-six/lib/python3.7/site-packages/bluesky/run_engine.py in __call__(self, *args, **metadata_kw) 751 # it (unless it is a canceled error) 752 if exc is not None: --> 753 raise exc 754 755 if self._interrupted: /opt/conda_envs/collection-2019-3.0-six/lib/python3.7/site-packages/bluesky/run_engine.py in _run(self) 1260 self._reason = str(err) 1261 self.log.exception("Run aborted") -> 1262 raise err 1263 finally: 1264 # Some done_callbacks may still be alive in other threads. /opt/conda_envs/collection-2019-3.0-six/lib/python3.7/site-packages/bluesky/run_engine.py in _run(self) 1148 else: 1149 try: -> 1150 msg = self._plan_stack[-1].send(resp) 1151 # We have exhausted the top generator 1152 except StopIteration: /opt/conda_envs/collection-2019-3.0-six/lib/python3.7/site-packages/bluesky/preprocessors.py in __call__(self, plan) 1292 plan = monitor_during_wrapper(plan, self.monitors) 1293 plan = baseline_wrapper(plan, self.baseline) -> 1294 return (yield from plan) /opt/conda_envs/collection-2019-3.0-six/lib/python3.7/site-packages/bluesky/preprocessors.py in baseline_wrapper(plan, devices, name) 1147 return (yield from plan) 1148 else: -> 1149 return (yield from plan_mutator(plan, insert_baseline)) 1150 1151 /opt/conda_envs/collection-2019-3.0-six/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc) 167 continue 168 else: --> 169 raise ex 170 # if inserting / mutating, put new generator on the stack 171 # and replace the current msg with the first element from the /opt/conda_envs/collection-2019-3.0-six/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc) 120 ret = result_stack.pop() 121 try: --> 122 msg = plan_stack[-1].send(ret) 123 except StopIteration as e: 124 # discard the exhausted generator /opt/conda_envs/collection-2019-3.0-six/lib/python3.7/site-packages/bluesky/preprocessors.py in monitor_during_wrapper(plan, signals) 788 plan1 = plan_mutator(plan, insert_after_open) 789 plan2 = plan_mutator(plan1, insert_before_close) --> 790 return (yield from plan2) 791 792 /opt/conda_envs/collection-2019-3.0-six/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc) 167 continue 168 else: --> 169 raise ex 170 # if inserting / mutating, put new generator on the stack 171 # and replace the current msg with the first element from the /opt/conda_envs/collection-2019-3.0-six/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc) 120 ret = result_stack.pop() 121 try: --> 122 msg = plan_stack[-1].send(ret) 123 except StopIteration as e: 124 # discard the exhausted generator /opt/conda_envs/collection-2019-3.0-six/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc) 167 continue 168 else: --> 169 raise ex 170 # if inserting / mutating, put new generator on the stack 171 # and replace the current msg with the first element from the /opt/conda_envs/collection-2019-3.0-six/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc) 120 ret = result_stack.pop() 121 try: --> 122 msg = plan_stack[-1].send(ret) 123 except StopIteration as e: 124 # discard the exhausted generator /opt/conda_envs/collection-2019-3.0-six/lib/python3.7/site-packages/bluesky/preprocessors.py in fly_during_wrapper(plan, flyers) 846 plan1 = plan_mutator(plan, insert_after_open) 847 plan2 = plan_mutator(plan1, insert_before_close) --> 848 return (yield from plan2) 849 850 /opt/conda_envs/collection-2019-3.0-six/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc) 167 continue 168 else: --> 169 raise ex 170 # if inserting / mutating, put new generator on the stack 171 # and replace the current msg with the first element from the /opt/conda_envs/collection-2019-3.0-six/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc) 120 ret = result_stack.pop() 121 try: --> 122 msg = plan_stack[-1].send(ret) 123 except StopIteration as e: 124 # discard the exhausted generator /opt/conda_envs/collection-2019-3.0-six/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc) 167 continue 168 else: --> 169 raise ex 170 # if inserting / mutating, put new generator on the stack 171 # and replace the current msg with the first element from the /opt/conda_envs/collection-2019-3.0-six/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc) 120 ret = result_stack.pop() 121 try: --> 122 msg = plan_stack[-1].send(ret) 123 except StopIteration as e: 124 # discard the exhausted generator /opt/conda_envs/collection-2019-3.0-six/lib/python3.7/site-packages/bluesky/plans.py in count(detectors, num, delay, md) 65 num=num, delay=delay)) 66 ---> 67 return (yield from inner_count()) 68 69 /opt/conda_envs/collection-2019-3.0-six/lib/python3.7/site-packages/bluesky/utils.py in dec_inner(*inner_args, **inner_kwargs) 1009 plan = gen_func(*inner_args, **inner_kwargs) 1010 plan = wrapper(plan, *args, **kwargs) -> 1011 return (yield from plan) 1012 return dec_inner 1013 return dec /opt/conda_envs/collection-2019-3.0-six/lib/python3.7/site-packages/bluesky/preprocessors.py in stage_wrapper(plan, devices) 937 return (yield from plan) 938 --> 939 return (yield from finalize_wrapper(inner(), unstage_devices())) 940 941 /opt/conda_envs/collection-2019-3.0-six/lib/python3.7/site-packages/bluesky/preprocessors.py in finalize_wrapper(plan, final_plan, pause_for_debug) 498 cleanup = True 499 try: --> 500 ret = yield from plan 501 except GeneratorExit: 502 cleanup = False /opt/conda_envs/collection-2019-3.0-six/lib/python3.7/site-packages/bluesky/preprocessors.py in inner() 934 935 def inner(): --> 936 yield from stage_devices() 937 return (yield from plan) 938 /opt/conda_envs/collection-2019-3.0-six/lib/python3.7/site-packages/bluesky/preprocessors.py in stage_devices() 927 def stage_devices(): 928 for d in devices: --> 929 yield Msg('stage', d) 930 931 def unstage_devices(): /opt/conda_envs/collection-2019-3.0-six/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc) 192 try: 193 # yield out the 'current message' and collect the return --> 194 inner_ret = yield msg 195 except GeneratorExit: 196 # special case GeneratorExit. We must clean up all of our plans /opt/conda_envs/collection-2019-3.0-six/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc) 192 try: 193 # yield out the 'current message' and collect the return --> 194 inner_ret = yield msg 195 except GeneratorExit: 196 # special case GeneratorExit. We must clean up all of our plans /opt/conda_envs/collection-2019-3.0-six/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc) 192 try: 193 # yield out the 'current message' and collect the return --> 194 inner_ret = yield msg 195 except GeneratorExit: 196 # special case GeneratorExit. We must clean up all of our plans /opt/conda_envs/collection-2019-3.0-six/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc) 192 try: 193 # yield out the 'current message' and collect the return --> 194 inner_ret = yield msg 195 except GeneratorExit: 196 # special case GeneratorExit. We must clean up all of our plans /opt/conda_envs/collection-2019-3.0-six/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc) 192 try: 193 # yield out the 'current message' and collect the return --> 194 inner_ret = yield msg 195 except GeneratorExit: 196 # special case GeneratorExit. We must clean up all of our plans /opt/conda_envs/collection-2019-3.0-six/lib/python3.7/site-packages/bluesky/run_engine.py in _run(self) 1206 # exceptions (coming in via throw) can be 1207 # raised -> 1208 new_response = yield from coro(msg) 1209 1210 # special case `CancelledError` and let the outer /opt/conda_envs/collection-2019-3.0-six/lib/python3.7/site-packages/bluesky/run_engine.py in _stage(self, msg) 2212 if not hasattr(obj, 'stage'): 2213 return [] -> 2214 result = obj.stage() 2215 self._staged.add(obj) # add first in case of failure below 2216 yield from self._reset_checkpoint_state_coro() ~/Repos/ophyd/ophyd/areadetector/trigger_mixins.py in stage(self) 119 def stage(self): 120 self._acquisition_signal.subscribe(self._acquire_changed) --> 121 super().stage() 122 123 def unstage(self): ~/Repos/ophyd/ophyd/areadetector/base.py in stage(self, *args, **kwargs) 156 157 def stage(self, *args, **kwargs): --> 158 ret = super().stage(*args, **kwargs) 159 try: 160 self.validate_asyn_ports() ~/Repos/ophyd/ophyd/device.py in stage(self) 605 device = getattr(self, attr) 606 if hasattr(device, 'stage'): --> 607 device.stage() 608 devices_staged.append(device) 609 except Exception: ~/Repos/ophyd/ophyd/areadetector/plugins.py in stage(self) 102 103 def stage(self): --> 104 super().stage() 105 106 def enable_on_stage(self): ~/Repos/ophyd/ophyd/areadetector/base.py in stage(self, *args, **kwargs) 156 157 def stage(self, *args, **kwargs): --> 158 ret = super().stage(*args, **kwargs) 159 try: 160 self.validate_asyn_ports() ~/Repos/ophyd/ophyd/areadetector/filestore_mixins.py in stage(self) 460 461 def stage(self): --> 462 super().stage() 463 res_kwargs = {'frame_per_point': self.get_frames_per_point()} 464 self._generate_resource(res_kwargs) ~/Repos/ophyd/ophyd/areadetector/filestore_mixins.py in stage(self) 433 set_and_wait(self.file_name, filename) 434 set_and_wait(self.file_number, 0) --> 435 super().stage() 436 437 # AD does this same templating in C, but we can't access it ~/Repos/ophyd/ophyd/areadetector/filestore_mixins.py in stage(self) 592 593 def stage(self): --> 594 super().stage() 595 self._point_counter = count() 596 ~/Repos/ophyd/ophyd/areadetector/filestore_mixins.py in stage(self) 295 self._locked_key_list = False 296 self._datum_uids.clear() --> 297 super().stage() 298 # Subclasses will assemble resource_kwargs and call 299 # self._generate_resource(resource_kwrags) ~/Repos/ophyd/ophyd/device.py in stage(self) 596 self.name, 597 val, original_vals[sig]) --> 598 set_and_wait(sig, val) 599 # It worked -- now add it to this list of sigs to unstage. 600 self._original_vals[sig] = original_vals[sig] ~/Repos/ophyd/ophyd/utils/epics_pvs.py in set_and_wait(signal, val, poll_time, timeout, rtol, atol) 264 raise TimeoutError("Attempted to set %r to value %r and timed " 265 "out after %r seconds. Current value is %r." % --> 266 (signal, val, timeout, current_value)) 267 268 TimeoutError: Attempted to set EpicsSignalWithRBV(read_pv='XF:02IDC-BI{Mir:3-Cam:13_U_1}HDF1:Capture_RBV', name='m3_diag_cam_hdf5_capture', parent='m3_diag_cam_hdf5', value=0, timestamp=1579033452.660428, pv_kw={}, auto_monitor=False, string=False, write_pv='XF:02IDC-BI{Mir:3-Cam:13_U_1}HDF1:Capture', limits=False, put_complete=False) to value 1 and timed out after 60 seconds. Current value is 0. ''''
tacaswell commented 4 years ago

This is a communication issue with the camera and/or an issue with the camera not an issue with the software. 

TimeoutError: Attempted to set EpicsSignalWithRBV(read_pv='XF:02IDC-BI{Mir:3-Cam:13_U_1}HDF1:Capture_RBV', name='m3_diag_cam_hdf5_capture', parent='m3_diag_cam_hdf5', value=0, timestamp=1579033452.660428, pv_kw={}, auto_monitor=False, string=False, write_pv='XF:02IDC-BI{Mir:3-Cam:13_U_1}HDF1:Capture', limits=False, put_complete=False) to value 1 and timed out after 60 seconds. Current value is 0.

If the camera has not started to acquire after a minute there is not much we can do on the software side.

bisogni commented 4 years ago

Hi Tom,

Thanks for the message. As far as I can see the camera works well, and I can correctly operate it in CSS. I of course tried to reboot it, but I still get the same error in Bluesky. I will reach out to John Sinsheimer, and let you know if he doesn’t find anything wrong from the camera/communication side. Unfortunately, I need to have this camera functional as soon as possible.

Thanks.

Valentina

From: Thomas A Caswell notifications@github.com Reply-To: NSLS-II/Bug-Reports reply@reply.github.com Date: Tuesday, January 14, 2020 at 7:30 PM To: NSLS-II/Bug-Reports Bug-Reports@noreply.github.com Cc: "Bisogni, Valentina" bisogni@bnl.gov, Author author@noreply.github.com Subject: Re: [NSLS-II/Bug-Reports] Bluesky fails to operate Prosilica Camera (#202)

This is a communication issue with the camera and/or an issue with the camera not an issue with the software.

TimeoutError: Attempted to set EpicsSignalWithRBV(read_pv='XF:02IDC-BI{Mir:3-Cam:13_U_1}HDF1:Capture_RBV', name='m3_diag_cam_hdf5_capture', parent='m3_diag_cam_hdf5', value=0, timestamp=1579033452.660428, pv_kw={}, auto_monitor=False, string=False, write_pv='XF:02IDC-BI{Mir:3-Cam:13_U_1}HDF1:Capture', limits=False, put_complete=False) to value 1 and timed out after 60 seconds. Current value is 0.

If the camera has not started to acquire after a minute there is not much we can do on the software side.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/NSLS-II/Bug-Reports/issues/202?email_source=notifications&email_token=AF2TZRX2YJK3BR5K4WT2FV3Q5ZKKRA5CNFSM4KGZCW4KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEI6UAQY#issuecomment-574439491, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AF2TZRS5LD4VWCAYVABDQCDQ5ZKKRANCNFSM4KGZCW4A.

tacaswell commented 4 years ago

The underlying issue is that the directories were missing and the timeout was a red herring. Closing this as we already know the AD code needs an overhaul.