implement bluesky-queueserver

prjemian commented 2 years ago

Close #10
Close #11
Close #16
Close #17
Close #20

prjemian commented 2 years ago

Looks as if the instrument package will need some changes before the queueserver can run it. In these next steps, the queueserver cannot even find the instrument package.

Try the server

Starting from a local repository clone (in the root directory of the clone):

run server in a console

cd ./qserver
conda activate queue_server
start-re-manager --startup-dir ./

shows this output in the console:

(queue_server) prjemian@zap:~/.../bdp_controls/qserver$ start-re-manager --startup-dir ./
[W 2022-03-21 17:35:45,209 bluesky_queueserver.manager.start_manager] The file with the list of allowed plans and devices ('/home/prjemian/Documents/projects/BCDA-APS/bdp_controls/qserver/existing_plans_and_devices.yaml') does not exist. The manager will be started with empty list. The list will be populated after RE worker environment is opened the first time.
[I 2022-03-21 17:35:45,214 bluesky_queueserver.manager.manager] Starting ZMQ server at 'tcp://*:60615'
[I 2022-03-21 17:35:45,214 bluesky_queueserver.manager.manager] ZMQ control channels: encryption disabled
[I 2022-03-21 17:35:45,216 bluesky_queueserver.manager.manager] Starting RE Manager process
[I 2022-03-21 17:35:45,229 bluesky_queueserver.manager.manager] Loading the lists of allowed plans and devices ...
[W 2022-03-21 17:35:45,229 bluesky_queueserver.manager.profile_ops] List of plans and devices is not loaded. File '/home/prjemian/Documents/projects/BCDA-APS/bdp_controls/qserver/existing_plans_and_devices.yaml' does not exist.
[I 2022-03-21 17:35:45,230 bluesky_queueserver.manager.manager] Starting ZeroMQ server ...
[I 2022-03-21 17:35:45,231 bluesky_queueserver.manager.manager] ZeroMQ server is waiting on tcp://*:60615

Tell qserver to open an environment (in different console)

cd ./qserver
conda activate queue_server
qserver environment open

shows this output in the console:

Arguments: ['environment', 'open']
17:36:34 - MESSAGE: 
{'msg': '', 'success': True}

New output in first console

This is the console where start-re-manager is running:

[I 2022-03-21 17:36:34,715 bluesky_queueserver.manager.manager] Opening the new RE environment ...
[I 2022-03-21 17:36:34,716 bluesky_queueserver.manager.start_manager] Starting RE Worker ...
[I 2022-03-21 17:36:34,718 bluesky_queueserver.manager.manager] Waiting for RE worker to start ...
[I 2022-03-21 17:36:34,954 bluesky_queueserver.manager.profile_ops] Loading RE Worker startup code from directory '/home/prjemian/Documents/projects/BCDA-APS/bdp_controls/qserver' ...
[I 2022-03-21 17:36:34,954 bluesky_queueserver.manager.profile_ops] Startup directory: '/home/prjemian/Documents/projects/BCDA-APS/bdp_controls/qserver'
[I 2022-03-21 17:36:34,955 bluesky_queueserver.manager.profile_ops] Loading startup file '/home/prjemian/Documents/projects/BCDA-APS/bdp_controls/qserver/starter.py' ...
sys.path[-1] = '/tmp/bluesky'
[E 2022-03-21 17:36:34,957 bluesky_queueserver.manager.worker] Failed to start RE Worker environment. Error while loading startup code: No module named 'instrument'.
Traceback (most recent call last):
  File "/home/prjemian/.local/lib/python3.8/site-packages/bluesky_queueserver/manager/worker.py", line 954, in run
    self._re_namespace = load_worker_startup_code(
  File "/home/prjemian/.local/lib/python3.8/site-packages/bluesky_queueserver/manager/profile_ops.py", line 390, in load_worker_startup_code
    nspace = load_profile_collection(startup_dir, keep_re=keep_re)
  File "/home/prjemian/.local/lib/python3.8/site-packages/bluesky_queueserver/manager/profile_ops.py", line 252, in load_profile_collection
    raise exc_info[1].with_traceback(exc_info[2])
  File "/tmp/qserver/profile_temp.py", line 22, in <module>
    import instrument
ModuleNotFoundError: No module named 'instrument'
[I 2022-03-21 17:36:34,957 bluesky_queueserver.manager.worker] Environment is waiting to be closed ...
[I 2022-03-21 17:36:35,320 bluesky_queueserver.manager.manager] Waiting for exit confirmation from RE worker ...
[I 2022-03-21 17:36:35,321 bluesky_queueserver.manager.worker] Run Engine environment was closed successfully
[I 2022-03-21 17:36:35,321 bluesky_queueserver.manager.manager] Wait for RE Worker process to close (join)
[I 2022-03-21 17:36:35,321 bluesky_queueserver.manager.start_manager] Joining RE Worker ...
[E 2022-03-21 17:36:35,324 bluesky_queueserver.manager.manager] Error occurred while opening RE Worker environment.

Load `instrument` into regular Python session

Test that the instrument can be loaded by the starter.py:

(queue_server) prjemian@zap:~/.../bdp_controls/qserver$ python ./starter.py 
sys.path[-1] = '/home/prjemian/Documents/projects/BCDA-APS/bdp_controls/bluesky'
instrument = <module 'instrument' from '/home/prjemian/Documents/projects/BCDA-APS/bdp_controls/bluesky/instrument/__init__.py'>
I Mon-17:42:30 - ############################################################ startup
I Mon-17:42:30 - logging started
I Mon-17:42:30 - logging level = 10
I Mon-17:42:30 - /home/prjemian/Documents/projects/BCDA-APS/bdp_controls/bluesky/instrument/collection.py
I Mon-17:42:30 - /home/prjemian/Documents/projects/BCDA-APS/bdp_controls/bluesky/instrument/mpl/console.py
I Mon-17:42:31 - #### Bluesky Framework ####
I Mon-17:42:31 - /home/prjemian/Documents/projects/BCDA-APS/bdp_controls/bluesky/instrument/framework/check_python.py
I Mon-17:42:31 - /home/prjemian/Documents/projects/BCDA-APS/bdp_controls/bluesky/instrument/utils/configuration.py
I Mon-17:42:31 - /home/prjemian/Documents/projects/BCDA-APS/bdp_controls/bluesky/instrument/framework/check_bluesky.py
I Mon-17:42:32 - /home/prjemian/Documents/projects/BCDA-APS/bdp_controls/bluesky/instrument/framework/initialize.py
I Mon-17:42:32 - using databroker catalog 'bdp2022'
I Mon-17:42:32 - /home/prjemian/Documents/projects/BCDA-APS/bdp_controls/bluesky/instrument/framework/metadata.py
I Mon-17:42:32 - #### Devices ####
I Mon-17:42:32 - /home/prjemian/Documents/projects/BCDA-APS/bdp_controls/bluesky/instrument/devices/aps_source.py
I Mon-17:42:32 - /home/prjemian/Documents/projects/BCDA-APS/bdp_controls/bluesky/instrument/devices/calculation_records.py
I Mon-17:42:33 - /home/prjemian/Documents/projects/BCDA-APS/bdp_controls/bluesky/instrument/devices/coarse_xy_stage_device.py
I Mon-17:42:33 - /home/prjemian/Documents/projects/BCDA-APS/bdp_controls/bluesky/instrument/devices/feedback_api.py
I Mon-17:42:33 - /home/prjemian/Documents/projects/BCDA-APS/bdp_controls/bluesky/instrument/devices/fine_xy_stage_device.py
I Mon-17:42:33 - /home/prjemian/Documents/projects/BCDA-APS/bdp_controls/bluesky/instrument/devices/ioc_stats.py
I Mon-17:42:33 - /home/prjemian/Documents/projects/BCDA-APS/bdp_controls/bluesky/instrument/devices/simulated_beam.py
I Mon-17:42:33 - /home/prjemian/Documents/projects/BCDA-APS/bdp_controls/bluesky/instrument/devices/simulated_shutter.py
I Mon-17:42:33 - /home/prjemian/Documents/projects/BCDA-APS/bdp_controls/bluesky/instrument/devices/area_detector.py
I Mon-17:42:34 - #### Callbacks ####
I Mon-17:42:34 - #### Plans ####
I Mon-17:42:34 - /home/prjemian/Documents/projects/BCDA-APS/bdp_controls/bluesky/instrument/plans/bpd_simulation.py
I Mon-17:42:34 - #### Utilities ####
I Mon-17:42:34 - #### Startup is complete. ####
==== =============== ======== ========
name ophyd structure EPICS PV label(s)
==== =============== ======== ========
==== =============== ======== ========

prjemian commented 2 years ago

The queueserver is in early development and may not be prepared for how we load the instrument package. Also, the instrument package needs to separate the steps that run the framework (since this will be handled by the queueserver) and the steps that define the devices and plans. As it stands now, those steps are too intermingled in the instrument package.

Concerns to be investigated:

handling of supplementary data (sd)
calls to get_ipython()
IPython magics
plan access to the databroker catalog need to be localized to the plan's code
namespace concerns for plans that import device objects
instrument package might need a setup.py file for local installation via pip install -e .

prjemian commented 2 years ago

Note that the new instrument/configuration.yml file is very helpful!

prjemian commented 2 years ago

Refactor

Confirmed it necessary to reconfigure the instrument package, extracting all IPython references and making a local cat (databroker) object as needed.

Was not necessary to make an installable package but note that instrument cannot be located in queueserver by manipulation of the sys.path. Rather, a starter.py file is needed in the same directory as the instrument package to be imported. (Might have implications how the standard instrument package should be changed.)

`start-re-manager` terminal

``` (queue_server) prjemian@zap:~/.../bdp_controls/qserver$ start-re-manager --startup-dir ./ [I 2022-03-22 10:19:03,674 bluesky_queueserver.manager.manager] Starting ZMQ server at 'tcp://*:60615' [I 2022-03-22 10:19:03,675 bluesky_queueserver.manager.manager] ZMQ control channels: encryption disabled [I 2022-03-22 10:19:03,677 bluesky_queueserver.manager.manager] Starting RE Manager process [I 2022-03-22 10:19:03,689 bluesky_queueserver.manager.manager] Loading the lists of allowed plans and devices ... [I 2022-03-22 10:19:03,843 bluesky_queueserver.manager.manager] Starting ZeroMQ server ... [I 2022-03-22 10:19:03,844 bluesky_queueserver.manager.manager] ZeroMQ server is waiting on tcp://*:60615 [I 2022-03-22 10:19:10,505 bluesky_queueserver.manager.manager] Opening the new RE environment ... [I 2022-03-22 10:19:10,506 bluesky_queueserver.manager.start_manager] Starting RE Worker ... [I 2022-03-22 10:19:10,508 bluesky_queueserver.manager.manager] Waiting for RE worker to start ... [I 2022-03-22 10:19:10,733 bluesky_queueserver.manager.profile_ops] Loading RE Worker startup code from directory '/home/prjemian/Documents/projects/BCDA-APS/bdp_controls/qserver' ... [I 2022-03-22 10:19:10,733 bluesky_queueserver.manager.profile_ops] Startup directory: '/home/prjemian/Documents/projects/BCDA-APS/bdp_controls/qserver' [I 2022-03-22 10:19:10,734 bluesky_queueserver.manager.profile_ops] Loading startup file '/home/prjemian/Documents/projects/BCDA-APS/bdp_controls/qserver/starter.py' ... /home/prjemian/Documents/projects/BCDA-APS/bdp_controls/qserver/instrument/__init__.py /home/prjemian/Documents/projects/BCDA-APS/bdp_controls/qserver/instrument/iconfig_dict.py /home/prjemian/Documents/projects/BCDA-APS/bdp_controls/qserver/instrument/qserver.py /home/prjemian/Documents/projects/BCDA-APS/bdp_controls/qserver/instrument/devices/default_timeouts.py /home/prjemian/Documents/projects/BCDA-APS/bdp_controls/qserver/instrument/devices/calculation_records.py /home/prjemian/Documents/projects/BCDA-APS/bdp_controls/qserver/instrument/devices/feedback_api.py /home/prjemian/Documents/projects/BCDA-APS/bdp_controls/qserver/instrument/devices/ioc_stats.py /home/prjemian/Documents/projects/BCDA-APS/bdp_controls/qserver/instrument/devices/samplexy_stage.py /home/prjemian/Documents/projects/BCDA-APS/bdp_controls/qserver/instrument/devices/simulated_beam.py /home/prjemian/Documents/projects/BCDA-APS/bdp_controls/qserver/instrument/devices/simulated_shutter.py /home/prjemian/Documents/projects/BCDA-APS/bdp_controls/qserver/instrument/devices/area_detector.py /home/prjemian/Documents/projects/BCDA-APS/bdp_controls/qserver/instrument/plans/bpd_simulation.py iconfig = {'DATABROKER_CATALOG': 'bdp2022', 'beamline_id': 'BDP', 'instrument_name': 'APS-U Beamline Data Pipelines project in 2022', 'proposal_id': 'bdp2022', 'ADSIM_IOC_PREFIX': 'ad:', 'GP_IOC_PREFIX': 'gp:', 'XY_STAGE_X_PV': 'gp:m9', 'XY_STAGE_Y_PV': 'gp:m10', 'PZT_X_PV': 'gp:gp:float1', 'PZT_Y_PV': 'gp:gp:float2', 'ADSIM_X_CALC_PV': 'gp:userCalc9', 'ADSIM_Y_CALC_PV': 'gp:userCalc10', 'INCIDENT_BEAM_SIMULATOR_CALC_PV': 'gp:userCalc2', 'HDF5_FILE_NAME_PV': 'gp:gp:longtext1', 'ALLOW_AREA_DETECTOR_WARMUP': False, 'ENABLE_AREA_DETECTOR_IMAGE_PLUGIN': True, 'APS_IN_BASELINE': False, 'ENABLE_CALCS': True, 'USE_PROGRESS_BAR': False, 'PV_TIMEOUT': 60, 'PV_WRITE_TIMEOUT': 60, 'PV_CONNECTION_TIMEOUT': 60, 'MINIMUM_PYTHON_VERSION': [3, 7], 'MINIMUM_BLUESKY_VERSION': [1, 8], 'MINIMUM_OPHYD_VERSION': [1, 6], 'MINIMUM_DATABROKER_VERSION': [1, 2]} ================== ========== device/object connected? ================== ========== ad_x_calc True ad_y_calc True adsimdet True gp_stats True image_file_created True incident_beam_calc True incident_beam True samplexy True shutter True ================== ========== [I 2022-03-22 10:19:21,454 bluesky_queueserver.manager.worker] Startup code loading was completed [I 2022-03-22 10:19:21,454 bluesky_queueserver.manager.worker] Generating lists of allowed plans and devices [I 2022-03-22 10:19:21,489 bluesky_queueserver.manager.worker] List of allowed plans and devices was successfully generated [I 2022-03-22 10:19:21,631 bluesky_queueserver.manager.profile_ops] Updating the list of existing plans and devices in the file '/home/prjemian/Documents/projects/BCDA-APS/bdp_controls/qserver/existing_plans_and_devices.yaml' [I 2022-03-22 10:19:21,759 bluesky_queueserver.manager.manager] Downloading the lists of existing plans and devices from the worker environment [I 2022-03-22 10:19:21,799 bluesky_queueserver.manager.worker] Instantiating and configuring Run Engine ... [I 2022-03-22 10:19:21,799 bluesky_queueserver.manager.worker] RE Environment is ready [I 2022-03-22 10:19:22,262 bluesky_queueserver.manager.manager] Worker started successfully. ```

prjemian commented 2 years ago

Also removed any calls to setup supplementary data, such as baseline signals.

prjemian commented 2 years ago

note: https://blueskyproject.io/bluesky-queueserver/cli_tools.html#location-of-startup-code

prjemian commented 2 years ago

This works! Queueing these jobs:

qserver queue add plan '{"name": "move_coarse_positioner", "args": [2.71, 3.14]}'
qserver queue add plan '{"name": "move_fine_positioner", "args": [.123, -0.456]}'
qserver queue add plan '{"name": "take_image", "kwargs": {"md": {"task": "use the qserver"}}}'
qserver queue add plan front '{"name": "set_acquire_time", "args": [0.5]}'
qserver queue start

shows this qserver console output:

[I 2022-03-22 14:28:54,925 bluesky_queueserver.manager.manager] Starting queue processing ...
[I 2022-03-22 14:28:54,925 bluesky_queueserver.manager.manager] Processing the next queue item: 4 plans are left in the queue.
[I 2022-03-22 14:28:54,927 bluesky_queueserver.manager.manager] Starting the plan:
{'args': [0.5],
 'item_uid': '1429629a-3e53-4c08-9f0a-343b0107dacd',
 'kwargs': {},
 'meta': {},
 'name': 'set_acquire_time',
 'user': 'qserver-cli',
 'user_group': 'admin'}.
[I 2022-03-22 14:28:54,927 bluesky_queueserver.manager.worker] Starting execution of a plan ...
[I 2022-03-22 14:28:54,927 bluesky_queueserver.manager.worker] Starting a plan 'set_acquire_time'.
[I 2022-03-22 14:28:55,309 bluesky_queueserver.manager.manager] Processing the next queue item: 3 plans are left in the queue.
[I 2022-03-22 14:28:55,310 bluesky_queueserver.manager.manager] Starting the plan:
{'args': [2.71, 3.14],
 'item_uid': 'd6c86650-7bd4-4ae5-a929-a8605def5614',
 'kwargs': {},
 'meta': {},
 'name': 'move_coarse_positioner',
 'user': 'qserver-cli',
 'user_group': 'admin'}.
[I 2022-03-22 14:28:55,310 bluesky_queueserver.manager.worker] Starting execution of a plan ...
[I 2022-03-22 14:28:55,310 bluesky_queueserver.manager.worker] Starting a plan 'move_coarse_positioner'.
[I 2022-03-22 14:28:58,824 bluesky_queueserver.manager.manager] Processing the next queue item: 2 plans are left in the queue.
[I 2022-03-22 14:28:58,826 bluesky_queueserver.manager.manager] Starting the plan:
{'args': [0.123, -0.456],
 'item_uid': '4993eeff-9045-425d-8b99-3b8f6bd645b8',
 'kwargs': {},
 'meta': {},
 'name': 'move_fine_positioner',
 'user': 'qserver-cli',
 'user_group': 'admin'}.
[I 2022-03-22 14:28:58,826 bluesky_queueserver.manager.worker] Starting execution of a plan ...
[I 2022-03-22 14:28:58,826 bluesky_queueserver.manager.worker] Starting a plan 'move_fine_positioner'.
[I 2022-03-22 14:28:59,327 bluesky_queueserver.manager.manager] Processing the next queue item: 1 plans are left in the queue.
[I 2022-03-22 14:28:59,330 bluesky_queueserver.manager.manager] Starting the plan:
{'args': [],
 'item_uid': 'f7c6c5f2-12c8-4a6c-b562-b77bb36c30a9',
 'kwargs': {'md': {'task': 'use the qserver'}},
 'meta': {},
 'name': 'take_image',
 'user': 'qserver-cli',
 'user_group': 'admin'}.
[I 2022-03-22 14:28:59,330 bluesky_queueserver.manager.worker] Starting execution of a plan ...
[I 2022-03-22 14:28:59,330 bluesky_queueserver.manager.worker] Starting a plan 'take_image'.

Transient Scan ID: 3     Time: 2022-03-22 14:29:00
Persistent Unique Scan ID: '4c74553b-75e2-455a-a780-5af3cda66a7f'
[I 2022-03-22 14:29:00,282 bluesky_queueserver.manager.plan_monitoring] New run was open: '4c74553b-75e2-455a-a780-5af3cda66a7f'
New stream: 'primary'
+-----------+------------+
|   seq_num |       time |
+-----------+------------+
|         1 | 14:29:00.4 |
+-----------+------------+
generator take_image ['4c74553b'] (scan num: 3)

Run was closed: '4c74553b-75e2-455a-a780-5af3cda66a7f'
[I 2022-03-22 14:29:01,839 bluesky_queueserver.manager.manager] No items are left in the queue.
[I 2022-03-22 14:29:01,839 bluesky_queueserver.manager.manager] Queue is empty.

prjemian commented 2 years ago

Working now.

prjemian commented 2 years ago

Started running a GUI for the queueserver today and decided it would be easier if the API for take_image() was simpler. Specifically, the arguments that could be None or tuple are very difficult to provide reliably through the qserver. Separate the jobs from inside the take_image() plan so they can be executed individually. But, combine the acquisition time into the take_image() API.

prjemian commented 2 years ago

Might be nice to run the queueserver in a screen session, like the feedback IOC is run. Sounds like an enhancement feature.

prjemian commented 2 years ago

@keenanlang @tguruswamy : This is ready for review. Demo at tomorrow's meeting.

prjemian commented 2 years ago

@keenanlang @tguruswamy : Today's demo was a test of this PR. I'd like to merge it today so I can move forward on other issues. If no objections, I'll merge later this evening.

BCDA-APS / bdp_controls