containerized version

[gdevenyi]

People want container versions for reproducibility and because most don't have the skill/ability to build and install software from sources.

Containers pose a problem for tightly integrated pipelines such as MAGeTbrain because passing jobs out of container, into a job and then back into another container is very challenging.

In general, the design for such a system seems to be

• launch the pipeline proper inside a control container using a special launcher
• launcher monitors a special directory for work todo
• generate command files for pipline in that special directory
• launcher parses jobfiles, wraps them in a new container launcher and submits them to the cluster to be run

This poses some problems for the existing MAGeTbrain implementation

• we need to pass out the commands themselves
• we need to pass out the metadata about the command (walltime, memory etc)
• magetbrain should know something about the cluster, but it's inside a container

Solutions will require a new "mode" for qbatch, for now called "submitter and requester"

• in requester mode, qbatch dumps a given job as a "jobname.joblist" and "jobname.meta" inside a given directory (which the above mentioned launcher will monitor)
• the submitter will be invoked whenever a new job is discovered in the workdir, it will grab the joblist, wrap it in the container invocation command, parse the metadata and submit the job

Questions: Can I use header-commands and footer-commands to wrap the actual joblist in a container launch (bash heredoc?)

[gdevenyi]

Changes to qbatch can be minimal

• add footer command
• Create a batch mode which dumps the commands into a file and the qbatch command-line into another file Getting the command line of qbatch:

  import sys
  print(sys.argv)

Until ITK fixes the MINC vs NIFTI loading issues, we will need to detect the input files and and use the appropriate atlases.

[gdevenyi]

Got it working, hurrah.