[Feature] Progress tracking and scheduling

Fixes

Related to https://github.com/FCP-INDI/C-PAC/pull/1363 by @sgiavasis

Description

This PR creates a C-PAC (or virtually anything) scheduler, with an API interface, that allows running containerized images and checking its progress. It has a CLI, so the user can start up and configure the scheduler, and an API interface, to communicate with the C-PAC GUI project mainly.

Technical details

The implementation relies heavily on the asyncio API, to simplify concurrency. However, it is not a parallel API, meaning that everything is executed in the same thread (and there is no race condition) and the different tasks that are being executed concurrently must not block the asyncio execution (e.g. it can have an asyncio.sleep in a task, or an IO function). Considering this past detail, all the feature implementations must have this in mind, which is why it is hard to leverage all the asyncio potential in projects that were not thought to work this way (e.g. nipype → pydra). The good thing is that, given it is single threaded, it eases a lot to handle different moving parts, while on parallel setups one would have to use different mechanisms of communication to avoid race conditions.

That said, we have # main parts on this implementation: Scheduler, Backend, Shedule (and its children), Message, Result, and the API. Beggining with the Schedule. Schedule is a abstraction of the task it should be executed. For C-PAC, we have three tasks:

DataSettings: A task to generate data configs from a provided data settings;
DataConfig: A task to schedule a pipeline for the subjects from a data config, spawning new tasks for each participant;
ParticipantPipeline: A task to execute a pipeline for a single subject.

Supposedly, it handles the logic aspects of it in terms of the abstract task they are performing. More technical aspects, such as running containers, are handled by a specialization of the Schedule class: BackendSchedule. The BackendSchedules are specific to a Backend, an interface between Python & the softwares of a specific backend (e.g. singularity binaries). The Backend must contains the parameters required for the BackendSchedules to properly communicate with the underlying softwares, such as the Docker image to be used or the SSH connection to access a SLURM cluster.

The Scheduler is the central part of this implementation, and maybe the most simpler. It stores the Schedules into a tree-like structure, since Schedules can spawn new Schedules, and manage the Messages received by each Schedule, together with the callbacks associated to a Schedule Message type. When a Schedule is scheduled, the Scheduler will send the Schedule to its Backend, and the Backend will specialize this "naive" Schedule into a BackendSchedule for that Backend:

ParticipantPipelineSchedule + DockerBackend = DockerParticipantPipelineSchedule

This "backend-aware" Schedule (from the superclass BackendSchedule) will then be executed by the Scheduler. The BackendSchedule behave as a Python generator, so the Scheduler simply iterate this object, and the items of this iteration are defined as Messages. The Messages are data classes (i.e. only store data, not methods), to give information for the Scheduler about the execution. The Messages are relayed to Scheduler watchers, which are external agents that provide a callback function for the Scheduler to call when it receives a specific type of Message. For the Spawn Message, the Scheduler schedules a new Schedule, with the parameters contained in the Spawn message.

Specifics of the Docker and Singularity containers are actually the same: they share the same base code for container execution, only differing in the container creation. When the container is created, three tasks run concurrently for this Schedule: container status, log listener, and file listener. The first yields Messages of type Status, as a ping, so we know the container is running fine. The second connects to the websocket server running in the container, to capture which nodes it has run so far, and yield Messages of type Log. The last one looks in the output directory for logs and crashes, storing the files as Results in the Schedule, and yielding Messages of type Result. Only the ParticipantPipeline has the second and the third, the others have just the container status Messages.

For SLURM, it starts connecting to the cluster via SSH. It uses the SSH multiplexing connection feature, so the authentication process happens only once, which is a good idea for connections that has a multi-factor authentication layer. After connecting to a cluster, the Backend allocates nodes to execute the Schedules and install a Miniconda & CPACpy. By using the API provided by CPACpy, the local CPACpy communicates with the node CPACpy (yes, via HTTP & WS) to run the Schedules. It uses the same API to gather the results and keep the local Schedule state updated. By default, the Singularity Backend is used by the node CPACpy to run the Schedules.

The Results are basically files in which it would be too much to transfer via WS. The API to gather the Results allow to slice the content using HTTP headers (Content-Range). It is essential for results that will be incremented during the execution (i.e. logs). Using the slice, one do not need to request for the whole file again, but only the part it does not have:

/result/cpac_pipeline.log from bytes 0-    # Returns 200 bytes

# The file has some increments from the nipype log

/result/cpac_pipeline.log from bytes 200-  # Returns 100 bytes

# The file has some increments from the nipype log

/result/cpac_pipeline.log from bytes 300-  # Returns 10 bytes

Tests

Screenshots

I mean, I can show some code, I guess...

Checklist

[ ] My pull request has a descriptive title (not a vague title like Update index.md).
[ ] My pull request targets the develop branch of the repository.
[ ] My commit messages follow best practices.
[ ] My code follows the established code style of the repository.
[ ] I added tests for the changes I made (if applicable).
[ ] I updated the changelog.
[ ] I added or updated documentation (if applicable).
[ ] I tried running the project locally and verified that there are no visible errors.

Developer Certificate of Origin

``` Developer Certificate of Origin Version 1.1 Copyright (C) 2004, 2006 The Linux Foundation and its contributors. 1 Letterman Drive Suite D4700 San Francisco, CA, 94129 Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Developer's Certificate of Origin 1.1 By making a contribution to this project, I certify that: (a) The contribution was created in whole or in part by me and I have the right to submit it under the open source license indicated in the file; or (b) The contribution is based upon previous work that, to the best of my knowledge, is covered under an appropriate open source license and I have the right under that license to submit that work with modifications, whether created in whole or in part by me, under the same open source license (unless I am permitted to submit under a different license), as indicated in the file; or (c) The contribution was provided directly to me by some other person who certified (a), (b) or (c) and I have not modified it. (d) I understand and agree that this project and the contribution are public and that a record of the contribution (including all personal information I submit with it, including my sign-off) is maintained indefinitely and may be redistributed consistent with this project or the open source license(s) involved. ```

FCP-INDI / cpac