search

aps-8id-dys / bluesky

XPCS bluesky instrument configuration
Other
2 stars 0 forks source link

lambda2M:HDF1: plugin errors #4

Closed qzhang234 closed 1 year ago

qzhang234 commented 1 year ago

prepare_count timed out with the same error as during 12/19 afternoon meeting:

Terminal Output ``` In [1]: RE(prepare_count(lambda2M.hdf1, "test4", 0.01, 0.01, n_images=300, compr ...: ession="None", auto_save="No")) lambda2M_hdf1_file_name: _set_and_wait(value=test4, timeout=60, atol=None, rtol=None, kwargs={}) Run aborted Traceback (most recent call last): File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/run_engine.py", line 1498, in _run msg = self._plan_stack[-1].throw(stashed_exception or resp) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 1307, in __call__ return (yield from plan) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 1160, in baseline_wrapper return (yield from plan) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 803, in monitor_during_wrapper return (yield from plan2) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 78, in plan_mutator msg = plan_stack[-1].throw(exception) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 78, in plan_mutator msg = plan_stack[-1].throw(exception) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 861, in fly_during_wrapper return (yield from plan2) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 78, in plan_mutator msg = plan_stack[-1].throw(exception) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 78, in plan_mutator msg = plan_stack[-1].throw(exception) File "/home/beams10/8IDIUSER/bluesky/instrument/plans/detector_prep.py", line 35, in prepare_count yield from bps.mv( File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/plan_stubs.py", line 259, in mv yield Msg('wait', None, group=group) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 195, in plan_mutator inner_ret = yield msg File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 195, in plan_mutator inner_ret = yield msg File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 195, in plan_mutator inner_ret = yield msg [Previous line repeated 1 more time] bluesky.utils.FailedStatus: Status(obj=EpicsSignalWithRBV(read_pv='8idLambda2m:HDF1:FileName_RBV', name='lambda2M_hdf1_file_name', parent='lambda2M_hdf1', value='t', timestamp=1671592953.5621808, auto_monitor=True, string=True, write_pv='8idLambda2m:HDF1:FileName', limits=False, put_complete=False), done=True, success=False) FailedStatus: Status(obj=EpicsSignalWithRBV(read_pv='8idLambda2m:HDF1:FileName_RBV', name='lambda2M_hdf1_file_name', parent='lambda2M_hdf1', value='t', timestamp=1671592953.5621808, auto_monitor=True, string=True, write_pv='8idLambda2m:HDF1:FileName', limits=False, put_complete=False), done=True, success=False) In [2]: ```
qzhang234 commented 1 year ago

The issue doesn't occur all the time and I'm not quite sure what the condition is, however I thought I would log this for completeness

prjemian commented 1 year ago

Good to see this problem logged. It's trying to write with value='t' and that looks like a truncation test4 or such.

prjemian commented 1 year ago

Another very similar error (with same PV):

2023-03-04 09:13:34 ```py [I 2023-03-04 09:13:34,304 bluesky_queueserver.manager.manager] Starting the plan: {'name': 'bdp_acquire', 'args': [], 'kwargs': {'acq_rep': 1, 'file_name': 'BDPQS', 'method': 'file', 'md': {'title': 'BDP streaming demo with Bluesky Queueserver, 200 ' 'fps for ~10 s.'}}, 'user': 'Queue Server API User', 'user_group': 'primary', 'meta': {}, 'item_uid': 'cbff5f96-3ce2-4e6d-86a3-4b6db1d6c839'}. [I 2023-03-04 09:13:34,304 bluesky_queueserver.manager.worker] Starting execution of a plan ... [I 2023-03-04 09:13:34,304 bluesky_queueserver.manager.worker] Starting a plan 'bdp_acquire'. method='file' uses detector 'lambda2M' (connected=True) method='stream' uses detector 'lambda2Mpva' (connected=True) Selected detector 'lambda2M' with method 'file'. Staging setup det.stage_sigs=OrderedDict([('cam.acquire', 0), ('cam.image_mode', 1)]) Staging setup det.cam.stage_sigs=OrderedDict([('wait_for_plugins', 'Yes')]) [I 2023-03-04 09:13:34,566 bluesky_queueserver.manager.manager] Returning current queue and running plan ... [I 2023-03-04 09:13:34,569 bluesky_queueserver.manager.manager] Returning the list of runs for the running plan ... [I 2023-03-04 09:13:34,570 bluesky_queueserver.manager.manager] Returning plan history ... WARNING:ophyd.objects:lambda2M_hdf1_file_path: _set_and_wait(value=/home/8ididata/2023-1/bluesky202301, timeout=30, atol=None, rtol=None, kwargs={}) ERROR:bluesky:Run aborted Traceback (most recent call last): File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/run_engine.py", line 1498, in _run msg = self._plan_stack[-1].throw(stashed_exception or resp) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 1307, in __call__ return (yield from plan) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 1160, in baseline_wrapper return (yield from plan) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 803, in monitor_during_wrapper return (yield from plan2) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 78, in plan_mutator msg = plan_stack[-1].throw(exception) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 78, in plan_mutator msg = plan_stack[-1].throw(exception) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 861, in fly_during_wrapper return (yield from plan2) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 78, in plan_mutator msg = plan_stack[-1].throw(exception) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 78, in plan_mutator msg = plan_stack[-1].throw(exception) File "/home/beams10/8IDIUSER/bluesky/instrument/plans/acquire_plans.py", line 111, in bdp_acquire yield from prep_hdf_plugin(det.hdf1, n_images, file_path, file_name) File "/home/beams10/8IDIUSER/bluesky/instrument/plans/acquire_plans.py", line 52, in prep_hdf_plugin yield from bps.mv( File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/plan_stubs.py", line 259, in mv yield Msg('wait', None, group=group) File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 195, in plan_mutator inner_ret = yield msg File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 195, in plan_mutator inner_ret = yield msg File "/home/beams10/8IDIUSER/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/bluesky/preprocessors.py", line 195, in plan_mutator inner_ret = yield msg [Previous line repeated 1 more time] bluesky.utils.FailedStatus: Status(obj=EpicsPathSignal(read_pv='8idLambda2m:HDF1:FilePath_RBV', name='lambda2M_hdf1_file_path', parent='lambda2M_hdf1', value='/', timestamp=1677942814.5155315, auto_monitor=True, string=True, write_pv='8idLambda2m:HDF1:FilePath', limits=False, put_complete=False, path_semantics='posix'), done=True, success=False) ```
prjemian commented 1 year ago

This might be corrected by adjusting the order in which this plugin's components are set.

prjemian commented 1 year ago

This code needs to be tested: https://github.com/aps-8id-dys/bluesky/blob/fe4698ff82a789814b47c44b2ac1d60c68526452/instrument/plans/acquire_plans.py#L42-L67

prjemian commented 1 year ago

Aaack, again:

bluesky.utils.FailedStatus: Status(obj=EpicsSignalWithRBV(read_pv='8idLambda2m:HDF1:FileName_RBV', name='lambda2M_hdf1_file_name', parent='lambda2M_hdf1', value='B', timestamp=1678218339.5096989, auto_monitor=True, string=True, write_pv='8idLambda2m:HDF1:FileName', limits=False, put_complete=False), done=True, success=False)
[E 2023-03-07 13:46:09,673 bluesky_queueserver.manager.worker] The plan failed: Plan failed: Status(obj=EpicsSignalWithRBV(read_pv='8idLambda2m:HDF1:FileName_RBV', name='lambda2M_hdf1_file_name', parent='lambda2M_hdf1', value='B', timestamp=1678218339.5096989, auto_monitor=True, string=True, write_pv='8idLambda2m:HDF1:FileName', limits=False, put_complete=False), done=True, success=False)
[I 2023-03-07 13:46:10,841 bluesky_queueserver.manager.manager] Returning current queue and running plan ...
[I 2023-03-07 13:46:10,844 bluesky_queueserver.manager.manager] Returning the list of runs for the running plan ...
[I 2023-03-07 13:46:10,845 bluesky_queueserver.manager.manager] Returning plan history ...
prjemian commented 1 year ago

Ah, that result was from a stale RunEngine (started with code from the main branch, not this one). After restart of the RunEngine, 3 plans submitted with the method="file" kwarg and no errors observed.

prjemian commented 1 year ago

So, inadvertently, this sequence of fail..succeed testing verified that the branch has a fix for #4, as well as a solution for #10.