search

DiamondLightSource / mx-bluesky

Bluesky plans, plan stubs, and utilities for MX beamlines
https://diamondlightsource.github.io/mx-bluesky/
Apache License 2.0
0 stars 2 forks source link

Failing to get all frames causes confusing errors #332

Open DominicOram opened 1 year ago

DominicOram commented 1 year ago

If odin fails to write the correct number of frames we get an error like:

Traceback (most recent call last):
  File "/dls_sw/i03/software/bluesky/hyperion_v7.0.0/hyperion/.venv/lib/python3.10/site-packages/bluesky/preprocessors.py", line 577, in contingency_wrapper
    ret = yield from plan
  File "/dls_sw/i03/software/bluesky/hyperion_v7.0.0/hyperion/src/hyperion/experiment_plans/pin_centre_then_xray_centre_plan.py", line 67, in pin_centre_then_xray_centre_plan
    yield from detect_grid_and_do_gridscan(
  File "/dls_sw/i03/software/bluesky/hyperion_v7.0.0/hyperion/src/hyperion/experiment_plans/grid_detect_then_xray_centre_plan.py", line 153, in detect_grid_and_do_gridscan
    yield from flyscan_xray_centre(flyscan_xray_centre_parameters)
  File "/dls_sw/i03/software/bluesky/hyperion_v7.0.0/hyperion/.venv/lib/python3.10/site-packages/bluesky/utils/__init__.py", line 1202, in dec_inner
    return (yield from plan)
  File "/dls_sw/i03/software/bluesky/hyperion_v7.0.0/hyperion/.venv/lib/python3.10/site-packages/bluesky/preprocessors.py", line 384, in subs_wrapper
    return (yield from finalize_wrapper(_inner_plan(),
  File "/dls_sw/i03/software/bluesky/hyperion_v7.0.0/hyperion/.venv/lib/python3.10/site-packages/bluesky/preprocessors.py", line 511, in finalize_wrapper
    ret = yield from plan
  File "/dls_sw/i03/software/bluesky/hyperion_v7.0.0/hyperion/.venv/lib/python3.10/site-packages/bluesky/preprocessors.py", line 382, in _inner_plan
    return (yield from plan)
  File "/dls_sw/i03/software/bluesky/hyperion_v7.0.0/hyperion/.venv/lib/python3.10/site-packages/bluesky/utils/__init__.py", line 1202, in dec_inner
    return (yield from plan)
  File "/dls_sw/i03/software/bluesky/hyperion_v7.0.0/hyperion/.venv/lib/python3.10/site-packages/bluesky/preprocessors.py", line 1349, in set_run_key_wrapper
    return (yield from msg_mutator(plan, _set_run_key))
  File "/dls_sw/i03/software/bluesky/hyperion_v7.0.0/hyperion/.venv/lib/python3.10/site-packages/bluesky/preprocessors.py", line 247, in msg_mutator
    ret = yield msg
  File "/dls_sw/i03/software/bluesky/hyperion_v7.0.0/hyperion/.venv/lib/python3.10/site-packages/bluesky/run_engine.py", line 1588, in _run
    new_response = await coro(msg)
  File "/dls_sw/i03/software/bluesky/hyperion_v7.0.0/hyperion/.venv/lib/python3.10/site-packages/bluesky/run_engine.py", line 2420, in _unstage
    ret = obj.unstage()
  File "/dls_sw/i03/software/bluesky/hyperion_v7.0.0/dodal/src/dodal/devices/eiger.py", line 120, in unstage
    self.stop_odin_when_all_frames_collected()
  File "/dls_sw/i03/software/bluesky/hyperion_v7.0.0/dodal/src/dodal/devices/eiger.py", line 109, in stop_odin_when_all_frames_collected
    ).wait(self.ALL_FRAMES_TIMEOUT)
  File "/dls_sw/i03/software/bluesky/hyperion_v7.0.0/hyperion/.venv/lib/python3.10/site-packages/ophyd/status.py", line 432, in wait
    raise WaitTimeoutError(f"Status {self!r} has not completed yet.")
ophyd.utils.errors.WaitTimeoutError: Status SubscriptionStatus(device=eiger_odin_file_writer_num_captured, done=False, success=False) has not completed yet.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/dls_sw/i03/software/bluesky/hyperion_v7.0.0/hyperion/.venv/lib/python3.10/site-packages/bluesky/run_engine.py", line 1668, in _run
    raise err
  File "/dls_sw/i03/software/bluesky/hyperion_v7.0.0/hyperion/.venv/lib/python3.10/site-packages/bluesky/run_engine.py", line 1503, in _run
    msg = self._plan_stack[-1].throw(stashed_exception or resp)
  File "/dls_sw/i03/software/bluesky/hyperion_v7.0.0/hyperion/src/hyperion/device_setup_plans/utils.py", line 27, in start_preparing_data_collection_then_do_plan
    yield from bpp.contingency_wrapper(
  File "/dls_sw/i03/software/bluesky/hyperion_v7.0.0/hyperion/.venv/lib/python3.10/site-packages/bluesky/preprocessors.py", line 587, in contingency_wrapper
    yield from except_plan(e)
  File "/dls_sw/i03/software/bluesky/hyperion_v7.0.0/hyperion/.venv/lib/python3.10/site-packages/bluesky/run_engine.py", line 1588, in _run
    new_response = await coro(msg)
  File "/dls_sw/i03/software/bluesky/hyperion_v7.0.0/hyperion/.venv/lib/python3.10/site-packages/bluesky/run_engine.py", line 2441, in _stop
    return await maybe_await(obj.stop())  # nominally, this returns None
  File "/dls_sw/i03/software/bluesky/hyperion_v7.0.0/dodal/src/dodal/devices/eiger.py", line 139, in stop
    self.disable_roi_mode()
  File "/dls_sw/i03/software/bluesky/hyperion_v7.0.0/dodal/src/dodal/devices/eiger.py", line 142, in disable_roi_mode
    self.change_roi_mode(False)
  File "/dls_sw/i03/software/bluesky/hyperion_v7.0.0/dodal/src/dodal/devices/eiger.py", line 152, in change_roi_mode
    status = self.cam.roi_mode.set(
  File "/dls_sw/i03/software/bluesky/hyperion_v7.0.0/hyperion/.venv/lib/python3.10/site-packages/ophyd/signal.py", line 2121, in set
    return super().set(value, timeout=timeout, settle_time=settle_time)
  File "/dls_sw/i03/software/bluesky/hyperion_v7.0.0/hyperion/.venv/lib/python3.10/site-packages/ophyd/signal.py", line 379, in set
    raise RuntimeError(
RuntimeError: Another set() call is still in progress for eiger_cam_roi_mode

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/dls_sw/i03/software/bluesky/hyperion_v7.0.0/hyperion/.venv/lib/python3.10/site-packages/bluesky/run_engine.py", line 1687, in _run
    obj.unstage()
  File "/dls_sw/i03/software/bluesky/hyperion_v7.0.0/dodal/src/dodal/devices/eiger.py", line 120, in unstage
    self.stop_odin_when_all_frames_collected()
  File "/dls_sw/i03/software/bluesky/hyperion_v7.0.0/dodal/src/dodal/devices/eiger.py", line 109, in stop_odin_when_all_frames_collected
    ).wait(self.ALL_FRAMES_TIMEOUT)
  File "/dls_sw/i03/software/bluesky/hyperion_v7.0.0/hyperion/.venv/lib/python3.10/site-packages/ophyd/status.py", line 432, in wait
    raise WaitTimeoutError(f"Status {self!r} has not completed yet.")
ophyd.utils.errors.WaitTimeoutError: Status SubscriptionStatus(device=eiger_odin_file_writer_num_captured, done=False, success=False) has not completed yet.

The underlying error of Status SubscriptionStatus(device=eiger_odin_file_writer_num_captured, done=False, success=False) is ok (but could be a bit more user friendly). It is also confused by the Another set() call is still in progress for eiger_cam_roi_mode, which is stop being called on the detector and failing. We should try and clean this up.

dperl-dls commented 12 months ago

At least it tells us which status is not completed yet now...