Testing v9.5.0

[dperl-dls]

• on first startup, stuck at init'ing dodal devices
  • hanging on `[2024-06-25 14:17:43,299] Hyperion context DEBUG: Looking for device panda of type <class 'ophyd_async.panda._hdf_panda.HDFPanda'> in context
  • trying to connect to panda by hand gives

    >>> await p.connect()
    Traceback (most recent call last):
    File "/dls_sw/apps/python/miniforge/4.10.0-0/envs/python3.11/lib/python3.11/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
       ^^^^^^^^^^^^^^^^^^^
    File "/dls_sw/apps/python/miniforge/4.10.0-0/envs/python3.11/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
    File "<console>", line 1, in <module>
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/ophyd_async/panda/_hdf_panda.py", line 44, in connect
    await fill_pvi_entries(self, self._prefix + "PVI", timeout=timeout, mock=mock)
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/ophyd_async/epics/pvi/pvi.py", line 298, in fill_pvi_entries
    await _get_pvi_entries(root_entry, timeout=timeout)
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/ophyd_async/epics/pvi/pvi.py", line 254, in _get_pvi_entries
    await _get_pvi_entries(sub_entry)
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/ophyd_async/epics/pvi/pvi.py", line 257, in _get_pvi_entries
    _verify_common_blocks(entry, entry.common_device_type)
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/ophyd_async/epics/pvi/pvi.py", line 99, in _verify_common_blocks
    raise RuntimeError(
    RuntimeError: sub device `datasets:<class 'typing._GenericAlias'>` was not provided by pvi

• after commenting out panda, sticks on robot
  • seems like above, instantiation errors not reported in hyperion logs and get stuck
  • creating robot by hand has error:

    r = i03.robot()
    ERROR:root:device `error_code` raised unexpected exception TypeError
    Traceback (most recent call last):
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/ophyd_async/core/signal.py", line 88, in connect
    await self._backend.connect(timeout=timeout)
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/ophyd_async/epics/_backend/_aioca.py", line 279, in connect
    self.converter = make_converter(self.datatype, self.initial_values)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/ophyd_async/epics/_backend/_aioca.py", line 228, in make_converter
    raise TypeError(
    TypeError: BL03I-MO-ROBOT-01:PRG_ERR_CODE has type float not int
    Traceback (most recent call last):
    File "/dls_sw/apps/python/miniforge/4.10.0-0/envs/python3.11/lib/python3.11/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
       ^^^^^^^^^^^^^^^^^^^
    File "/dls_sw/apps/python/miniforge/4.10.0-0/envs/python3.11/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
    File "/dls_sw/apps/python/miniforge/4.10.0-0/envs/python3.11/lib/python3.11/asyncio/__main__.py", line 34, in callback
    coro = func()
       ^^^^^^
    File "<console>", line 1, in <module>
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/dodal/src/dodal/beamlines/i03.py", line 461, in robot
    return device_instantiation(
       ^^^^^^^^^^^^^^^^^^^^^
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/dodal/src/dodal/utils.py", line 102, in wrapper
    return func(*args, **kwds)
       ^^^^^^^^^^^^^^^^^^^
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/dodal/src/dodal/common/beamlines/beamline_utils.py", line 112, in device_instantiation
    wait_for_connection(device_instance, mock=fake)
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/dodal/src/dodal/common/beamlines/beamline_utils.py", line 55, in wait_for_connection
    call_in_bluesky_event_loop(
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/run_engine.py", line 2782, in call_in_bluesky_event_loop
    return fut.result(timeout=timeout)
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/dls_sw/apps/python/miniforge/4.10.0-0/envs/python3.11/lib/python3.11/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
       ^^^^^^^^^^^^^^^^^^^
    File "/dls_sw/apps/python/miniforge/4.10.0-0/envs/python3.11/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/ophyd_async/core/utils.py", line 129, in wait_for_connection
    raise NotConnected(exceptions)
    ophyd_async.core.utils.NotConnected: 
    coros: NotConnected:
    error_code: TypeError: BL03I-MO-ROBOT-01:PRG_ERR_CODE has type float not int

  • worked around by changing dtype on the signal
  • Not waiting long enough for aperture to finish moving before reading it in:

    [2024-06-25 15:40:28,380] bluesky run_engine ERROR: Run aborted
    Traceback (most recent call last):
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/run_engine.py", line 1547, in _run
    msg = self._plan_stack[-1].send(resp)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/src/hyperion/device_setup_plans/utils.py", line 41, in start_preparing_data_collection_then_do_plan
    yield from bpp.contingency_wrapper(
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 606, in contingency_wrapper
    ret = yield from plan
      ^^^^^^^^^^^^^^^
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/src/hyperion/device_setup_plans/utils.py", line 39, in wrapped_plan
    yield from plan_to_run
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/src/hyperion/experiment_plans/pin_centre_then_xray_centre_plan.py", line 73, in pin_centre_then_xray_centre_plan
    yield from detect_grid_and_do_gridscan(
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/src/hyperion/experiment_plans/grid_detect_then_xray_centre_plan.py", line 113, in detect_grid_and_do_gridscan
    yield from ispyb_activation_wrapper(
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 351, in run_wrapper
    yield from contingency_wrapper(plan,
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 606, in contingency_wrapper
    ret = yield from plan
      ^^^^^^^^^^^^^^^
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/src/hyperion/experiment_plans/grid_detect_then_xray_centre_plan.py", line 192, in _detect_grid_and_do_gridscan
    yield from flyscan_xray_centre(
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/utils/__init__.py", line 1203, in dec_inner
    return (yield from plan)
        ^^^^^^^^^^^^^^^
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 1380, in set_run_key_wrapper
    return (yield from msg_mutator(plan, _set_run_key))
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 263, in msg_mutator
    msg = plan.send(_s)
      ^^^^^^^^^^^^^
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/utils/__init__.py", line 1203, in dec_inner
    return (yield from plan)
        ^^^^^^^^^^^^^^^
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 351, in run_wrapper
    yield from contingency_wrapper(plan,
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 606, in contingency_wrapper
    ret = yield from plan
      ^^^^^^^^^^^^^^^
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 675, in dec_inner
    ret = yield from plan
      ^^^^^^^^^^^^^^^
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/utils/__init__.py", line 1203, in dec_inner
    return (yield from plan)
        ^^^^^^^^^^^^^^^
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/src/hyperion/device_setup_plans/xbpm_feedback.py", line 81, in transmission_and_xbpm_feedback_for_collection_wrapper
    yield from finalize_wrapper(
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 535, in finalize_wrapper
    ret = yield from plan
      ^^^^^^^^^^^^^^^
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/src/hyperion/device_setup_plans/xbpm_feedback.py", line 78, in _inner_plan
    return (yield from plan)
        ^^^^^^^^^^^^^^^
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/src/hyperion/experiment_plans/flyscan_xray_centre_plan.py", line 382, in run_gridscan_and_move_and_tidy
    yield from run_gridscan_and_move(fgs_composite, params)
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/utils/__init__.py", line 1203, in dec_inner
    return (yield from plan)
        ^^^^^^^^^^^^^^^
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 1380, in set_run_key_wrapper
    return (yield from msg_mutator(plan, _set_run_key))
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 263, in msg_mutator
    msg = plan.send(_s)
      ^^^^^^^^^^^^^
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/utils/__init__.py", line 1203, in dec_inner
    return (yield from plan)
        ^^^^^^^^^^^^^^^
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 351, in run_wrapper
    yield from contingency_wrapper(plan,
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 606, in contingency_wrapper
    ret = yield from plan
      ^^^^^^^^^^^^^^^
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/src/hyperion/experiment_plans/flyscan_xray_centre_plan.py", line 299, in run_gridscan_and_move
    yield from run_gridscan(fgs_composite, parameters)
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/utils/__init__.py", line 1203, in dec_inner
    return (yield from plan)
        ^^^^^^^^^^^^^^^
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 1380, in set_run_key_wrapper
    return (yield from msg_mutator(plan, _set_run_key))
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 263, in msg_mutator
    msg = plan.send(_s)
      ^^^^^^^^^^^^^
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/utils/__init__.py", line 1203, in dec_inner
    return (yield from plan)
        ^^^^^^^^^^^^^^^
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 351, in run_wrapper
    yield from contingency_wrapper(plan,
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 606, in contingency_wrapper
    ret = yield from plan
      ^^^^^^^^^^^^^^^
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/src/hyperion/experiment_plans/flyscan_xray_centre_plan.py", line 236, in run_gridscan
    yield from read_hardware_for_ispyb_pre_collection(
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/src/hyperion/device_setup_plans/read_hardware_for_setup.py", line 35, in read_hardware_for_ispyb_pre_collection
    yield from bps.read(aperture_scatterguard)
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/plan_stubs.py", line 120, in read
    return (yield Msg('read', obj))
        ^^^^^^^^^^^^^^^^^^^^^^
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 251, in msg_mutator
    _s = yield msg
     ^^^^^^^^^
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 251, in msg_mutator
    _s = yield msg
     ^^^^^^^^^
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 251, in msg_mutator
    _s = yield msg
     ^^^^^^^^^
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/run_engine.py", line 1607, in _run
    new_response = await coro(msg)
               ^^^^^^^^^^^^^^^
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/run_engine.py", line 1889, in _read
    ret = await maybe_await(obj.read(*msg.args, **msg.kwargs))
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/utils/__init__.py", line 1852, in maybe_await
    return await ret
       ^^^^^^^^^
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/ophyd_async/core/standard_readable.py", line 96, in read
    return await merge_gathered_dicts([sig.read() for sig in self._readables])
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/ophyd_async/core/utils.py", line 174, in merge_gathered_dicts
    for result in await asyncio.gather(*coros):
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/ophyd_async/core/standard_readable.py", line 246, in read
    return await self.signal.read(cached=self.cached)
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/dodal/src/dodal/devices/aperturescatterguard.py", line 179, in read
    await self._backend.put(await self.parent._get_current_aperture_position())
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/dodal/src/dodal/devices/aperturescatterguard.py", line 252, in _get_current_aperture_position
    raise InvalidApertureMove(
    dodal.devices.aperturescatterguard.InvalidApertureMove: Current aperture/scatterguard state unrecognised at current_ap_y=34.4056

  • resolved by putting in a check for aperture y done moving before read_hardware_for_ispyb_pre_collection - should implement a tidier version
  • beamsize is wrong in ispyb by 1000x
  • seemed to run through several samples (~16:35-41) but not all registered in ispyb;
  • ran into waiting for eiger arm failure https://graylog.diamond.ac.uk/messages/graylog-daq-mx-hyperion_21/74776a40-3309-11ef-937c-2ae9b41e6ed4

[olliesilvester]

Looking at the panda issue, i03's PandablocksIOC is out of date with our ophyd_async since we were running the ioc from a local install from when main was broken. I can update it, and try and find out how to make sure it updates automatically

[DominicOram]

If we update it will it break old hyperion versions?

[olliesilvester]

It definitely could, but I couldn't say for certain without having a deep dive into all the panda server + ioc changes

It would only break panda related things though