Refactor for multiple deployment scenarios

[dylanmcreynolds]

As part of the IRI project, we need to be able to launch jobs centrally and have them run on NERSC, ALCF and OLCF. The current design launches jobs as containers locally. We need to be able to support multiple forms of job dispatching (e.g. NERSC SFAPI or however we can do it at the other facilities).

This is a large task. Some of the things that we will need to complete it:

• [ ] Currently, the computing api launches a docker image from within a docker. This is insufficient for several reasons. Docker is not a universally-supported orchestration system and is specifically no supported in HPC.
• [ ] The computing API needs to be able to be able to launch and monitor jobs that are on remote systems.
• [ ] The remote systems will have a variety of communication and queueing mechanisms.
• [ ] Configurations about each facilities job run needs to be stored centrally. For example, when using the NERSC API, a slurm script will be saved and sent over the SFAPI, with certain variables (e.g. file path) plugged into the script before being sent.

The design of this refactor could go in many ways.

We could potentially plug Prefect in and have it manage jobs. Then, agents at different facilities could poll for jobs. In this case, the compute api would be enhanced to launch flows on a prefect server.

[dylanmcreynolds]

Just to add to the first point about HPC. The dependence on using the docker to launch docker is not supported in a variety of systems. It won't work in userspace-only mode in docker/podman, etc.