This PR is a significant extension of the Queue Server functionality. There are no significant changes to the way how Queue Server operates in normal mode. All the existing API are expected to function identically in the normal 'Python' and in the extended 'IPython' modes. There are no meaningful changes to the API. (re_pause API requests are now accepted by the server whenever Run Engine is in the running state, independently on the states of the manager or the worker).
The new features are still in the development stage. Unit tests for IPython-specific features and updates to documentation will be implemented in the following PRs.
Description
If start-re-manager is started with --use-ipython-kernel=ON parameter, opening the environment starts an IPython kernel in the worker process. The startup code is loaded during kernel initialization from the specified profile (--ipython-dir and --startup-profile parameters). Optionally the code may be loaded from a module (--startup-module) or a script (--startup-script), but the startup code from the specified profile (or default IPython profile) is always loaded. The startup code is not patched and loaded as is. The code may contain IPython-specific elements, like magics. Matplotlib backend may be passed using --ipython-matplotlib parameter (accepts same values as --matplotlib parameter of IPython, e.g. --ipython-matplotlib=qt5. If run on a beamline workstation, the IPython-based worker is expected to support standard live plotting designed for IPython workflow.
Once the environment is open and the IPython kernel is running, it can be connected with Jupyter console. On the same workstation the kernel may be accessed as jupyter console --existing. It should be possible to access the kernel from a different workstation by feeding it the connection information (it is expected that API to do it conveniently). Note, that currently the 0MQ ports used by the kernel are temporarily fixed, but it is expected that they will be randomly generated in the future (this is default behavior of IPython kernel that we want to replicate). The kernel is single-threaded, i.e. the manager and the console are blocking each other. Jupyter console is going to be unresponsive when the manager is running a plan, and the manager can not start a plan when there is a task started from console. Closing the Jupyter console does not stop the task, so a plan started from the console will continue running and the output will be displayed in the manager output. It is possible to start a plan in the manager, pause it, then resume from the console. If a plan started from the manager is resumed and completed from the console, the respective entry is added to history (no traceback is going to be added if the plan failed). Plans started from the console can not currently be resumed from the manager.
Note: Plans running in the console can not be paused using Ctrl-C (it just temporarily disrupts output to the console). Use qserver re pause to pause the plan.
Motivation and Context
Summary of Changes for Release Notes
Added
New CLI parameters for 'start-re-manager' (and matching parameters for 'qserver-list-plans-devices'): --use-python-kernel (values ON/OFF, config file parameterworker/use_ipython_kernel, boolean, env. variableQSERVER_USE_IPYTHON_KERNEL, boolean);--ipython-dir(IPython directory, overrides default directory or IPYTHONDIR, config file parameterstartup/ipython_dir); --ipython-matplotlib (accepts the same values as --matplotlib parameter of IPython, config file parameter worker/ipython_matplotlib, used only in the IPython kernel mode);
--ignore-invalid-plans (values ON/OFF, boolean config file parameter startup/ignore_invalid_plans); --device-max-depth (integer, config file parameter startup/device_max_depth, restricts maximum depth for device components included in the list of existing devices).
An option to start IPython kernel in the worker process. The option is selected by starting RE Manager with the option --use-ipython-kernel=ON, setting config file parameter worker/use_ipython_kernel=True or environment variable QSERVER_USE_IPYTHON_KERNEL=true. Users may connect to the kernel directly using Jupyter console. The mode may be useful when transitioning from the existing IPython workflow or for debugging purposes.
New RE Manager status parameters: ip_kernel_state (None - environment is closed, disabled - IP kernel is not started or not used in the current mode, starting - startup in progress, idle, busy); ip_kernel_captured (None - environment is closed or the kernel is not used, True/False - indicates if the kernel is running tasks started by RE Manager, clients may connect to IP kernel directly and start tasks only when the kernel is not captured);
New worker environment state (env_state status parameter): failed. Indicates that the environment failed to start and will be closed. The state is used internally and is unlikely to be reported.
is_ipython_mode() function may be used in startup code to detect if the code is executed in IPython environment. The function returns correct result even if the code is running in the Python-based worker environment with monkeypatched IPython package. The use cases are similar is_re_worker_active().
register_plan and register_device functions for using in startup code. Currently, register_plan allows to explicitly exclude a given plan from processing by Queue Server (may be useful for problematic plans) and register_device allows to exclude a given device or set maximum depth for the device. This is experimental feature. Functionality may be added in the future.
Changed
re_pause API calls are now accepted whenever Run Engine is in the running state. For example, the API may be used to pause the plan that was started in IPython kernel directly using Jupyter console and not managed by RE Manager.
How Has This Been Tested?
While unit tests for some of the new features were implemented, there are no tests for IPython specific behavior. Developing IPython-specific tests requires significant work and will be a subject of next PRs. RE Manager with IPython-based worker passes all the existing tests (~1800), which is the reason to assume it is fully compatible with the original Python kernel. The IPython specific behavior was tested manually on with simulated plans and devices and on SRX and HXN beamlines at NSLS-II.
This PR is a significant extension of the Queue Server functionality. There are no significant changes to the way how Queue Server operates in normal mode. All the existing API are expected to function identically in the normal 'Python' and in the extended 'IPython' modes. There are no meaningful changes to the API. (
re_pauseAPI requests are now accepted by the server whenever Run Engine is in therunningstate, independently on the states of the manager or the worker).The new features are still in the development stage. Unit tests for IPython-specific features and updates to documentation will be implemented in the following PRs.
Description
If
start-re-manageris started with--use-ipython-kernel=ONparameter, opening the environment starts an IPython kernel in the worker process. The startup code is loaded during kernel initialization from the specified profile (--ipython-dirand--startup-profileparameters). Optionally the code may be loaded from a module (--startup-module) or a script (--startup-script), but the startup code from the specified profile (or default IPython profile) is always loaded. The startup code is not patched and loadedas is. The code may contain IPython-specific elements, like magics. Matplotlib backend may be passed using--ipython-matplotlibparameter (accepts same values as--matplotlibparameter of IPython, e.g.--ipython-matplotlib=qt5. If run on a beamline workstation, the IPython-based worker is expected to support standard live plotting designed for IPython workflow.Once the environment is open and the IPython kernel is running, it can be connected with Jupyter console. On the same workstation the kernel may be accessed as
jupyter console --existing. It should be possible to access the kernel from a different workstation by feeding it the connection information (it is expected that API to do it conveniently). Note, that currently the 0MQ ports used by the kernel are temporarily fixed, but it is expected that they will be randomly generated in the future (this is default behavior of IPython kernel that we want to replicate). The kernel is single-threaded, i.e. the manager and the console are blocking each other. Jupyter console is going to be unresponsive when the manager is running a plan, and the manager can not start a plan when there is a task started from console. Closing the Jupyter console does not stop the task, so a plan started from the console will continue running and the output will be displayed in the manager output. It is possible to start a plan in the manager, pause it, then resume from the console. If a plan started from the manager is resumed and completed from the console, the respective entry is added to history (no traceback is going to be added if the plan failed). Plans started from the console can not currently be resumed from the manager.Note: Plans running in the console can not be paused using
Ctrl-C(it just temporarily disrupts output to the console). Useqserver re pauseto pause the plan.Motivation and Context
Summary of Changes for Release Notes
Added
New CLI parameters for 'start-re-manager' (and matching parameters for 'qserver-list-plans-devices'):
--use-python-kernel (values ON/OFF, config file parameterworker/use_ipython_kernel, boolean, env. variableQSERVER_USE_IPYTHON_KERNEL, boolean);--ipython-dir(IPython directory, overrides default directory or IPYTHONDIR, config file parameterstartup/ipython_dir);--ipython-matplotlib(accepts the same values as --matplotlib parameter of IPython, config file parameterworker/ipython_matplotlib, used only in the IPython kernel mode); --ignore-invalid-plans (values ON/OFF, boolean config file parameterstartup/ignore_invalid_plans); --device-max-depth (integer, config file parameterstartup/device_max_depth, restricts maximum depth for device components included in the list of existing devices).An option to start IPython kernel in the worker process. The option is selected by starting RE Manager with the option
--use-ipython-kernel=ON, setting config file parameterworker/use_ipython_kernel=Trueor environment variableQSERVER_USE_IPYTHON_KERNEL=true. Users may connect to the kernel directly using Jupyter console. The mode may be useful when transitioning from the existing IPython workflow or for debugging purposes.New RE Manager status parameters:
ip_kernel_state(None - environment is closed,disabled- IP kernel is not started or not used in the current mode,starting- startup in progress,idle,busy);ip_kernel_captured(None - environment is closed or the kernel is not used, True/False - indicates if the kernel is running tasks started by RE Manager, clients may connect to IP kernel directly and start tasks only when the kernel is not captured);New worker environment state (
env_statestatus parameter):failed. Indicates that the environment failed to start and will be closed. The state is used internally and is unlikely to be reported.is_ipython_mode()function may be used in startup code to detect if the code is executed in IPython environment. The function returns correct result even if the code is running in the Python-based worker environment with monkeypatched IPython package. The use cases are similaris_re_worker_active().register_planandregister_devicefunctions for using in startup code. Currently,register_planallows to explicitly exclude a given plan from processing by Queue Server (may be useful for problematic plans) andregister_deviceallows to exclude a given device or set maximum depth for the device. This is experimental feature. Functionality may be added in the future.Changed
re_pauseAPI calls are now accepted whenever Run Engine is in therunningstate. For example, the API may be used to pause the plan that was started in IPython kernel directly using Jupyter console and not managed by RE Manager.How Has This Been Tested?
While unit tests for some of the new features were implemented, there are no tests for IPython specific behavior. Developing IPython-specific tests requires significant work and will be a subject of next PRs. RE Manager with IPython-based worker passes all the existing tests (~1800), which is the reason to assume it is fully compatible with the original Python kernel. The IPython specific behavior was tested manually on with simulated plans and devices and on SRX and HXN beamlines at NSLS-II.