search

aps-8id-dys / ipython-8idiuser

8-ID-I ipython configuration for bluesky (and other)
1 stars 1 forks source link

Bluesky crashes upon startup #261

Closed qzhang234 closed 3 years ago

qzhang234 commented 3 years ago

Bluesky crashes upon startup with complaint on adrigaku

Terminal Output ``` kouga1g% cd ../2021-2/ /home/beams/8IDIUSER/bluesky_data/2021/2021-2 kouga1g% bluesky8IDI Python 3.8.2 (default, Mar 26 2020, 15:53:00) Type 'copyright', 'credits' or 'license' for more information IPython 7.19.0 -- An enhanced Interactive Python. Type '?' for help. IPython profile: bluesky Activating auto-logging. Current session state plus future input saved. Filename : /home/beams10/8IDIUSER/bluesky_data/2021/2021-2/.logs/ipython_console.log Mode : rotate Output logging : True Raw input log : False Timestamping : True State : active I Wed-00:31:06 - ############################################################ startup I Wed-00:31:06 - logging started I Wed-00:31:06 - logging level = 10 I Wed-00:31:06 - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/collection.py I Wed-00:31:06 - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/mpl/console.py I Wed-00:31:13 - bluesky framework I Wed-00:31:13 - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/framework/check_python.py I Wed-00:31:13 - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/framework/check_bluesky.py I Wed-00:31:21 - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/framework/initialize.py I Wed-00:31:33 - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/framework/user_dir.py I Wed-00:31:33 - User code directory: /home/beams/8IDIUSER/.ipython-bluesky/user I Wed-00:31:33 - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/framework/metadata.py I Wed-00:31:33 - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/framework/callbacks.py I Wed-00:31:34 - writing to SPEC file: /home/beams10/8IDIUSER/bluesky_data/2021/2021-2/20210707-003134.dat I Wed-00:31:34 - >>>> Using default SPEC file name <<<< I Wed-00:31:34 - file will be created when bluesky ends its next scan I Wed-00:31:34 - to change SPEC file, use command: newSpecFile('title') I Wed-00:31:34 - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/devices/aps_source.py I Wed-00:31:34 - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/devices/pss.py W Wed-00:31:34 - not is user operations, no suspender installed for D-station shutter I Wed-00:31:34 - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/devices/foe.py I Wed-00:31:34 - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/devices/monochromator.py I Wed-00:31:35 - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/devices/be_window.py I Wed-00:31:35 - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/devices/actuators.py I Wed-00:31:35 - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/devices/attenuators.py I Wed-00:31:35 - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/devices/crl.py I Wed-00:31:35 - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/devices/detector_stages.py I Wed-00:31:35 - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/devices/epid.py I Wed-00:31:35 - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/devices/fly.py I Wed-00:31:35 - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/devices/lakeshore.py I Wed-00:31:35 - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/devices/motors.py I Wed-00:31:35 - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/devices/preamps.py I Wed-00:31:35 - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/devices/sample_stage.py I Wed-00:31:35 - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/devices/data_management.py I Wed-00:31:38 - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/devices/scaler.py I Wed-00:31:40 - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/devices/shutters.py W Wed-00:31:40 - !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! W Wed-00:31:40 - Session started when APS not operating. W Wed-00:31:40 - Using simulator 'shutter'. W Wed-00:31:40 - !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! I Wed-00:31:40 - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/devices/slits.py I Wed-00:31:40 - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/devices/soft_glue_fpga.py I Wed-00:31:40 - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/devices/tables.py I Wed-00:31:41 - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/devices/lambda_750k.py I Wed-00:31:41 - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/devices/ad_acquire_detector_base.py I Wed-00:31:41 - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/devices/ad_imm_plugins.py I Wed-00:31:41 - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/devices/ad_rigaku_detector.py [TerminalIPythonApp] WARNING | Unknown error in handling startup files: --------------------------------------------------------------------------- KeyError Traceback (most recent call last) ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/ophyd/device.py in __get__(self, instance, owner) 222 try: --> 223 return instance._signals[self.attr] 224 except KeyError: KeyError: 'acquire' During handling of the above exception, another exception occurred: TimeoutError Traceback (most recent call last) ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/IPython/core/shellapp.py in _exec_file(self, fname, shell_futures) 375 else: 376 # default to python, even without extension --> 377 self.shell.safe_execfile(full_filename, 378 self.shell.user_ns, 379 shell_futures=shell_futures, ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/IPython/core/interactiveshell.py in safe_execfile(self, fname, exit_ignore, raise_exceptions, shell_futures, *where) 2738 try: 2739 glob, loc = (where + (None, ))[:2] -> 2740 py3compat.execfile( 2741 fname, glob, loc, 2742 self.compile if shell_futures else None) ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/IPython/utils/py3compat.py in execfile(fname, glob, loc, compiler) 166 with open(fname, 'rb') as f: 167 compiler = compiler or compile --> 168 exec(compiler(f.read(), fname, 'exec'), glob, loc) 169 170 # Refactor print statements in doctests. ~/.ipython-bluesky/profile_bluesky/startup/00-instrument.py in 4 """ 5 ----> 6 from instrument.collection import * 7 8 # show_ophyd_symbols() ~/.ipython-bluesky/profile_bluesky/startup/instrument/collection.py in 12 13 from .framework import * ---> 14 from .devices import * 15 from .plans import * 16 from .utils import * ~/.ipython-bluesky/profile_bluesky/startup/instrument/devices/__init__.py in 32 # from .lambda_2m import * 33 # from .rigaku_ufxc import * ---> 34 from .ad_rigaku_detector import * 35 36 # non-hardware support ~/.ipython-bluesky/profile_bluesky/startup/instrument/devices/ad_rigaku_detector.py in 265 266 --> 267 adrigaku = RigakuUfxcDetector(IOC_PREFIX, name="adrigaku") ~/.ipython-bluesky/profile_bluesky/startup/instrument/devices/ad_rigaku_detector.py in __init__(self, image_name, *args, **kwargs) 108 # Do their work here _after_ self.cam.acquire has been created. 109 ##### TriggerBase --> 110 if not hasattr(self.cam, "acquire"): 111 print("component_names :", self.component_names) 112 raise KeyError("No acquire attribute!") ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/ophyd/device.py in __get__(self, instance, owner) 223 return instance._signals[self.attr] 224 except KeyError: --> 225 return instance._instantiate_component(self.attr) 226 227 def __set__(self, instance, owner): ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/ophyd/device.py in _instantiate_component(self, attr) 1216 1217 try: -> 1218 self._signals[attr] = cpt.create_component(self) 1219 sig = self._signals[attr] 1220 for event_type, functions in cpt._subscriptions.items(): ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/ophyd/device.py in create_component(self, instance) 180 if self.lazy and hasattr(self.cls, 'wait_for_connection'): 181 if getattr(instance, 'lazy_wait_for_connection', True): --> 182 cpt_inst.wait_for_connection() 183 184 return cpt_inst ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/ophyd/signal.py in wait_for_connection(self, timeout) 1488 def wait_for_connection(self, timeout=1.0): 1489 '''Wait for the underlying signals to initialize or connect''' -> 1490 self._ensure_connected(self._read_pv, self._write_pv, timeout=timeout) 1491 1492 @property ~/.conda/envs/bluesky_2021_1/lib/python3.8/site-packages/ophyd/signal.py in _ensure_connected(self, timeout, *pvs) 1059 # @raise_if_disconnected can cause issues otherwise. 1060 if not self._signal_is_ready.wait(timeout): -> 1061 raise TimeoutError(f'Control layer {self.cl.name} failed to send connection and ' 1062 f'access rights information within {float(timeout):.1f} sec') 1063 TimeoutError: Control layer pyepics failed to send connection and access rights information within 1.0 sec In [1]: ```
prjemian commented 3 years ago

There is a PV connection TimeoutError associated with this code:

~/.ipython-bluesky/profile_bluesky/startup/instrument/devices/ad_rigaku_detector.py in __init__(self, image_name, *args, **kwargs)
    108         # Do their work here _after_ self.cam.acquire has been created.
    109         ##### TriggerBase
--> 110         if not hasattr(self.cam, "acquire"):
    111             print("component_names :", self.component_names)
    112             raise KeyError("No acquire attribute!")

Are the detector PVs accessible from the linux command line using caget? Perhaps this work cannot be done until the python object is connected.

qzhang234 commented 3 years ago

@prjemian We are currently running the Rigaku detector and DM workflow in Spec, so I assume caget works on Rigaku PVs

prjemian commented 3 years ago

After some testing, we discovered that the detector would connect fully from an interactive session but not when run as part of the startup sequence. By adding a delay before creating the detector object (simulating the interactive experience), the detector connects without fault. There must be a backlog of PVs to connect. By waiting a short time (1 second delay is too short, 2 seconds or more was OK), the detector connects reliably.