XNAT Container Service Plugin - k8s Extension

[dean-taylor]

This epic covers the extension of the XNAT Container Service to run pipelines on kubernetes. The purpose is to be able to run both XNAT and all containers on a kubernetes cluster, i.e. AIS node.

Features

• [] https://github.com/Australian-Imaging-Service/pipelines-core/issues/19
• [] https://github.com/Australian-Imaging-Service/pipelines-core/issues/20

Requirements to be turned into features:

• We need a mechanism to flag which containers require GPU resources (and probably required resources in general). We can insert this information into the command.json when we build the images, but the container service will need to pick it up and provide the appropriate flags to k8s
• Some pipelines (e.g. Freesurfer->fMRIPrep) require software licences to be mounted into their containers. Not sure of the best way to do this (currently just building them into the images). Maybe licenses could be installed somewhere on the XNAT host and use a flag in the command.json to mount them in
• Important for us to be able to spawn containers from within containers, i.e. have a master pipeline that controls the execution of separate child containers for each pipeline node.
• Nice to have access subject-level data via direct mounts when running project-level pipelines
• Important to be able to allow special characters to be used in container launch UI fields (maybe blacklisted characters could be set in the command JSON)
• Nice to be able to customise the container launch UI with dropdown lists and expandable fields (e.g. click a '+' sign to create a new row of options)

[dean-taylor]

Interim solution during team discussion was to enable a standardised connection to a Docker API for the container service plugin.

[dean-taylor]

Need to document how to expose a Docker API to and external client.

[dean-taylor]

1. Service object
2. Endpoint object
3. Config either in existing Secret or another ConfigMap

[RyanPSullivan7]

Met with Flywheel this morning. John Flavin, the original author of the Container Service Plugin, has joined Flywheel from Washington University. He has an alpha version of kubernetes support for the CS Plugin and discussed with @fxusyd some details on namespace.