ENH: Auto-start queue as non-default option

maffettone commented 1 year ago

Description

This extends the enum for MState, adds an attribute to the REManager, and some methods for interfacing with that attribute via ZMQ. It provides the user an option for an empty queue, that is otherwise uninterrupted, to remain "hot" and auto-start when new items are added. This retains the default behavior of the REManager, but creates a new type of 'idle' state beyond MState.IDLE. All other interruptions (i.e. stopping the queue, an error in a plan, etc.) will stop the queue in an IDLE state, that will require user intervention to restart the queue. This will for example, allow a user to ask for a stop to the queue during the final running plan, and not have any new added plans automatically run.

Motivation and Context

Beamline scientists have been begging for this. It reflects the behavior that they are accustomed to with the RunEngine (prompt execution of plans), and enables remote agents to drive an experiment starting from an empty queue. While there is already functionality for prompt execution of plans, this would enable the addition of a plan or set of plans from multiple sources, with automatic processing from the queue. I broadly agree that this should not be the default behavior, but should be an option.

Summary of Changes for Release Notes

Adds an optional autostart state to REManager, so that an empty queue can begin processing plans automatically as they are added.

Fixed

Adjusts the _start_plan_task to first check for a queue stop, then check if there are pending plans. This ordering wasn't an issue when both no pending plans and a queue stop triggered similar behavior.

Added

Additional state to MState
Activate and deactivate functions for autostart, with handlers

Changed

Caveat to some checks for IDLE to optionally change to AUTOSTART instead.
Removed

How Has This Been Tested?

Written unit tests following the same syntactical approach in test_scenarios.py and test_zmq_api.py.

maffettone commented 1 year ago

This is no longer failing on: ~~bluesky_queueserver/manager/tests/test_scenarios.py::test_large_datasets_03 bluesky_queueserver/manager/tests/test_zmq_api.py::test_zmq_api_queue_item_add_batch_1 bluesky_queueserver/manager/tests/test_zmq_api.py::test_zmq_api_queue_item_add_batch_2 bluesky_queueserver/manager/tests/test_zmq_api.py::test_zmq_api_queue_item_add_batch_3 bluesky_queueserver/manager/tests/test_zmq_api.py::test_zmq_api_queue_item_add_batch_4_fail bluesky_queueserver/manager/tests/test_zmq_api.py::test_zmq_api_lock_6~~

I fixed an issue below, and added an additional test for batch addition.

dmgav commented 1 year ago

At this point I am not sure it's a good idea to make 'autostart' a separate manager state, because it does not reflect what is the manager is currently doing. It looks more like a queue mode. I will think where to put it when I get back from vacation.

dmgav commented 1 year ago

Closing in favor of PR https://github.com/bluesky/bluesky-queueserver/pull/269

bluesky / bluesky-queueserver