Containerized tasks

[tcompa]

Notes from a preliminary discussion with @mfranzon :

Current task.command

/some/path/python /some/path/task.py -j /some/path/args.json --metadiff /some/path/meta.json

Prototype of Docker-based solution

We would have two images.

The first one would be only built once, and then used from the cache

/some/path/Dockerfile.fractal-tasks-core-0.1.3

Pseudocode:

FROM python-slim
RUN pip install fractal-tasks-core[fractal-tasks]

Then there would be N (N=number of relevant tasks) other images, and each one of them would be re-built any time a task is run.

/some/path/Dockerfile.fractal-tasks-core-0.1.3-run-cellpose

Pseudocode:

FROM fractal-tasks-core-0.1.3
ARG ARGSPATH=/default     # To be overridden during build phase
ARG METAPATH=/default     # To be overridden during build phase
ARG SHAREDVOLUME=/default # To be overridden during build phase
MKDIR $SHAREDVOLUME
CMD /some/path/python /some/path/task.py -j $ARGSPATH --metadiff $METAPATH

and then there would be run script, like

/some/path/run_cellpose_docker.sh

# TASK_EXECUTION_ID is unique for each execution of a given task, meaning it
# changes for different jobs, workflowtasks, parallel components...
docker build -t fractal-tasks-core-0.1.3-run-cellpose-${TASK_EXECUTION_ID} \
    --build-args ARGSPATH=/some/path/args.json \
    --build-args METAPATH=/some/path/metadiff.json \
    --build-args SHAREDVOLUME=/data/active \
    /some/path/Dockerfile.fractal-tasks-core-0.1.3-run-cellpose \

docker run fractal-tasks-core-0.1.3-run-cellpose-${TASK_EXECUTION_ID} \
    --volume /data/active:/data/active

The new task.command would be

/some/path/run_cellpose_docker.sh -j /some/path/args.json --metadiff /some/path/meta.json

Data lifecycle

This is a very broad question, let's re-discuss it. Here is just one possible way of doing things.

1. There exists a shared volume where all users have RW access, e.g. /data/active. NOTE: this cannot have conflicts with typical filesystem paths, e.g. you would not be able to use /home/ (but it would be possible to mount /home/myuser, provided that it has broad permissions).
2. All paths that are used for I/O within tasks are in that shared volume (i.e. images, ome-zarrs, pre-trained models, metadata files, ...).
3. When the container is run, it writes files back to /data/active, and these files are owned by docker user.
4. Files are accessible to all users (this could be restricted by using e.g. groups).
5. If a certain ome-zarr needs to have restricted access, this can only be achieved at the end of processing. The user would run something like

   chown -R myuser:myuser /data/active/myuser/my.zarr
   chmod 755 /data/active/myuser/my.zarr

6. If point 5 is executed, then the file cannot be processed any more through containers, cause it would not be accessible to the docker user.

There are multiple ways to extend this possible scheme, and their relevance also depends on how we can impersonate users. If we are happy with either sudo -u or other impersonation strategies (e.g. SSH keys), then we could run Singularity containers as user processes. This may mitigate the data-access issues, as the container would have the same permissions as the user.

[tcompa]

Small addendum: the file-access issue would also be very relevant e.g. for /some/path/metadiff.json, which goes back to the question of how impersonating users.

The proof-of-concept should be clearly explored with the single-user local runner.

[tcompa]

Note: in what we described above, containers are units of computation. Another approach (ref https://www.nextflow.io/docs/latest/docker.html) is to use containers only to provide the environment. Quote:

In practice Nextflow will automatically wrap your processes and run them by executing the docker run

How to handle the file access in this approach? To be explored (ref https://github.com/nextflow-io/nextflow/discussions/4260). Relevant quote:

Nextflow automatically manages the file system mounts each time a container is launched depending on the process input files.

[tcompa]

One take-home message from our current discussion:

• Using containers to provide a portable environment is "easy"
• Using containers to actually perform computation (and then I/O) in a general way is clearly less flexible, and there is the risk that you need one container for each workflow manager (e.g. nextflow requires some specific attributes, to decide which paths must be mounted)
• One possible way forward is that fractal-tasks-core offers base images with just the appropriate environment, and then these are specialized to any specific workflow orchestrator which supports containerized execution

[jluethi]

Great summary! Another tool to look at (according to Jonas Windhager): Podman, running in user space

[jwindhager]

... and without requiring a daemon 😇