search

aps-8id-dys / bluesky

XPCS bluesky instrument configuration
Other
2 stars 0 forks source link

Lambda2M timed out on `cam1:ArrayCallbacks_RBV` #7

Closed qzhang234 closed 1 year ago

qzhang234 commented 1 year ago

Lambda2M crashed after 82 hours. /home/8ididata has 9.9 TB of remaining storage.

Terminal Output ``` Transient Scan ID: 33823 Time: 2022-12-27 01:14:27 Persistent Unique Scan ID: '30d21812-88ae-48d9-9bd1-fb3406ce46a1' New stream: 'primary' +-----------+------------+ | seq_num | time | +-----------+------------+ | 1 | 01:15:09.3 | +-----------+------------+ generator count ['30d21812'] (scan num: 33823) 6716 Transient Scan ID: 33824 Time: 2022-12-27 01:15:09 Persistent Unique Scan ID: 'f6a79e6f-e959-4093-a439-a7cd1446e0dd' CA.Client.Exception............................................... Warning: "Virtual circuit unresponsive" Context: "lambda2m.xray.aps.anl.gov:5064" Source File: ../tcpiiu.cpp line 926 Current Time: Tue Dec 27 2022 01:17:32.412793541 .................................................................. New stream: 'primary' +-----------+------------+ | seq_num | time | +-----------+------------+ | 1 | 01:17:57.8 | lambda2M_cam_array_callbacks: _set_and_wait(value=1, timeout=60, atol=None, rtol=None, kwargs={}) Traceback (most recent call last): File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/ophyd/signal.py", line 332, in set_thread self._set_and_wait(value, timeout, **kwargs) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/ophyd/signal.py", line 303, in _set_and_wait return _set_and_wait( File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/ophyd/utils/epics_pvs.py", line 238, in _set_and_wait signal.put(val, **kwargs) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/ophyd/signal.py", line 2060, in put raise ReadOnlyError("No write access to underlying EPICS PV") ophyd.utils.errors.ReadOnlyError: No write access to underlying EPICS PV +-----------+------------+ generator count ['f6a79e6f'] (scan num: 33824) Run aborted Traceback (most recent call last): File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/run_engine.py", line 1498, in _run msg = self._plan_stack[-1].throw(stashed_exception or resp) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 1307, in __call__ return (yield from plan) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 1160, in baseline_wrapper return (yield from plan) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 803, in monitor_during_wrapper return (yield from plan2) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 78, in plan_mutator msg = plan_stack[-1].throw(exception) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 78, in plan_mutator msg = plan_stack[-1].throw(exception) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 861, in fly_during_wrapper return (yield from plan2) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 78, in plan_mutator msg = plan_stack[-1].throw(exception) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 78, in plan_mutator msg = plan_stack[-1].throw(exception) File "/home/beams10/8IDIUSER/bluesky/Lambda2M_Test_bps.py", line 21, in Rep_Acq yield from bp.count([lambda2M]) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/plans.py", line 77, in count return (yield from inner_count()) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/utils/__init__.py", line 1147, in dec_inner return (yield from plan) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 952, in stage_wrapper return (yield from finalize_wrapper(inner(), unstage_devices())) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 528, in finalize_wrapper yield from ensure_generator(final_plan_instance) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 946, in unstage_devices yield Msg('unstage', d) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 195, in plan_mutator inner_ret = yield msg File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 195, in plan_mutator inner_ret = yield msg File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 195, in plan_mutator inner_ret = yield msg [Previous line repeated 1 more time] File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/run_engine.py", line 1583, in _run new_response = await coro(msg) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/run_engine.py", line 2352, in _unstage result = obj.unstage() File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/ophyd/areadetector/trigger_mixins.py", line 133, in unstage super().unstage() File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/ophyd/device.py", line 724, in unstage device.unstage() File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/ophyd/device.py", line 730, in unstage sig.set(val).wait() File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/ophyd/status.py", line 434, in wait raise self._exception ophyd.utils.errors.UnknownStatusFailure: The status Status(obj=EpicsSignalWithRBV(read_pv='8idLambda2m:cam1:ArrayCallbacks_RBV', name='lambda2M_cam_array_callbacks', parent='lambda2M_cam', value=1, timestamp=1672125477.6894484, auto_monitor=True, string=False, write_pv='8idLambda2m:cam1:ArrayCallbacks', limits=False, put_complete=False), done=False, success=False) has failed. To obtain more specific, helpful errors in the future, update the Device to use set_exception(...) instead of _finished(success=False). lambda2M_cam_array_callbacks: _set_and_wait(value=1, timeout=60, atol=None, rtol=None, kwargs={}) Traceback (most recent call last): File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/ophyd/signal.py", line 332, in set_thread self._set_and_wait(value, timeout, **kwargs) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/ophyd/signal.py", line 303, in _set_and_wait return _set_and_wait( File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/ophyd/utils/epics_pvs.py", line 238, in _set_and_wait signal.put(val, **kwargs) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/ophyd/signal.py", line 2060, in put raise ReadOnlyError("No write access to underlying EPICS PV") ophyd.utils.errors.ReadOnlyError: No write access to underlying EPICS PV Failed to unstage Lambda2MDetector(prefix='8idLambda2m:', name='lambda2M', read_attrs=['hdf1'], configuration_attrs=['cam', 'cam.acquire_period', 'cam.acquire_time', 'cam.image_mode', 'cam.manufacturer', 'cam.model', 'cam.num_exposures', 'cam.num_images', 'cam.trigger_mode', 'hdf1']). Traceback (most recent call last): File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/run_engine.py", line 1663, in _run raise err File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/run_engine.py", line 1498, in _run msg = self._plan_stack[-1].throw(stashed_exception or resp) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 1307, in __call__ return (yield from plan) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 1160, in baseline_wrapper return (yield from plan) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 803, in monitor_during_wrapper return (yield from plan2) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 78, in plan_mutator msg = plan_stack[-1].throw(exception) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 78, in plan_mutator msg = plan_stack[-1].throw(exception) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 861, in fly_during_wrapper return (yield from plan2) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 78, in plan_mutator msg = plan_stack[-1].throw(exception) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 78, in plan_mutator msg = plan_stack[-1].throw(exception) File "/home/beams10/8IDIUSER/bluesky/Lambda2M_Test_bps.py", line 21, in Rep_Acq yield from bp.count([lambda2M]) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/plans.py", line 77, in count return (yield from inner_count()) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/utils/__init__.py", line 1147, in dec_inner return (yield from plan) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 952, in stage_wrapper return (yield from finalize_wrapper(inner(), unstage_devices())) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 528, in finalize_wrapper yield from ensure_generator(final_plan_instance) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 946, in unstage_devices yield Msg('unstage', d) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 195, in plan_mutator inner_ret = yield msg File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 195, in plan_mutator inner_ret = yield msg File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 195, in plan_mutator inner_ret = yield msg [Previous line repeated 1 more time] File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/run_engine.py", line 1583, in _run new_response = await coro(msg) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/run_engine.py", line 2352, in _unstage result = obj.unstage() File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/ophyd/areadetector/trigger_mixins.py", line 133, in unstage super().unstage() File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/ophyd/device.py", line 724, in unstage device.unstage() File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/ophyd/device.py", line 730, in unstage sig.set(val).wait() File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/ophyd/status.py", line 434, in wait raise self._exception ophyd.utils.errors.UnknownStatusFailure: The status Status(obj=EpicsSignalWithRBV(read_pv='8idLambda2m:cam1:ArrayCallbacks_RBV', name='lambda2M_cam_array_callbacks', parent='lambda2M_cam', value=1, timestamp=1672125477.6894484, auto_monitor=True, string=False, write_pv='8idLambda2m:cam1:ArrayCallbacks', limits=False, put_complete=False), done=False, success=False) has failed. To obtain more specific, helpful errors in the future, update the Device to use set_exception(...) instead of _finished(success=False). During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/run_engine.py", line 1682, in _run obj.unstage() File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/ophyd/areadetector/trigger_mixins.py", line 133, in unstage super().unstage() File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/ophyd/device.py", line 724, in unstage device.unstage() File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/ophyd/device.py", line 730, in unstage sig.set(val).wait() File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/ophyd/status.py", line 434, in wait raise self._exception ophyd.utils.errors.UnknownStatusFailure: The status Status(obj=EpicsSignalWithRBV(read_pv='8idLambda2m:cam1:ArrayCallbacks_RBV', name='lambda2M_cam_array_callbacks', parent='lambda2M_cam', value=1, timestamp=1672125477.6894484, auto_monitor=True, string=False, write_pv='8idLambda2m:cam1:ArrayCallbacks', limits=False, put_complete=False), done=False, success=False) has failed. To obtain more specific, helpful errors in the future, update the Device to use set_exception(...) instead of _finished(success=False). UnknownStatusFailure: The status Status(obj=EpicsSignalWithRBV(read_pv='8idLambda2m:cam1:ArrayCallbacks_RBV', name='lambda2M_cam_array_callbacks', parent='lambda2M_cam', value=1, timestamp=1672125477.6894484, auto_monitor=True, string=False, write_pv='8idLambda2m:cam1:ArrayCallbacks', limits=False, put_complete=False), done=False, success=False) has failed. To obtain more specific, helpful errors in the future, update the Device to use set_exception(...) instead of _finished(success=False). In [19]: ```
qzhang234 commented 1 year ago

Screenshots on EPICS:

image image

prjemian commented 1 year ago

Aha! Underlying problem caused by EPICS. ophyd.utils.errors.ReadOnlyError: No write access to underlying EPICS PV PV became unavailable. The early notice was the CA.Client.Exception.

CA.Client.Exception...............................................
    Warning: "Virtual circuit unresponsive"
    Context: "lambda2m.xray.aps.anl.gov:5064"
    Source File: ../tcpiiu.cpp line 926
    Current Time: Tue Dec 27 2022 01:17:32.412793541
..................................................................
New stream: 'primary'
+-----------+------------+
|   seq_num |       time |
+-----------+------------+
|         1 | 01:17:57.8 |
lambda2M_cam_array_callbacks: _set_and_wait(value=1, timeout=60, atol=None, rtol=None, kwargs={})
Traceback (most recent call last):
  File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/ophyd/signal.py", line 332, in set_thread
    self._set_and_wait(value, timeout, **kwargs)
  File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/ophyd/signal.py", line 303, in _set_and_wait
    return _set_and_wait(
  File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/ophyd/utils/epics_pvs.py", line 238, in _set_and_wait
    signal.put(val, **kwargs)
  File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/ophyd/signal.py", line 2060, in put
    raise ReadOnlyError("No write access to underlying EPICS PV")
ophyd.utils.errors.ReadOnlyError: No write access to underlying EPICS PV
prjemian commented 1 year ago

What happened with the IOC? Related to IT activity today? That started at/about 1 AM.

prjemian commented 1 year ago

It's all done now.

prjemian commented 1 year ago

I believe that might have been due to the planned IT maintenance. This event was at 1:17:57.8 this morning. Here's the note from APS IT

Major APS network maintenance will occur on December 27th starting at 1am.   All network switches will be rebooted to load the latest firmware.

then the all clear

Upgrade firmware on 213 network switches 27-Dec 9:00am, has been COMPLETED, with these exceptions due to power work ...