worker: Configurable executor backends

[buchanae]

We have had a few use cases and requests for different, non-docker executor backends:

• a couple cases of wanting to run tasks without a container, just standard os/exec
• udocker, singularity, gvisor and other container runtimes are often more acceptable and easier to adopt in large, university HPC environments.

We've already discussed how we might support this, including having a configurable template string.

[adamstruck]

Funnel can now support anything that implements the docker CLI. Funnel uses exec to make calls to whatever docker is in your path so to use something like udocker just wrap it in a script called docker and put it in your PATH.

[kfox1111]

Looking to do the same. Will look into a shell wrapper for now. Does anyone have a singularity wrapper already and willing to share? :)

[illusional]

Hey @kfox1111, did you have any luck with Singularity? We're considering Rootless Docker as a method to get around running containers in a protected environment.

[kfox1111]

I have not gotten a chance to look at funnel further. But in general, Singularity is working pretty well on our HPC clusters. Recently I've started using podman/buildah on my centos8 boxes and that too is working pretty well for rootless containers. Centos8 comes out of the box with all the functionality for rootless podman preconfigured. I believe podman is docker cli compatible enough aliasing podman=docker might work for funnel. HPC systems tend to move slower though so singlularity support would still be very useful.

[adamstruck]

@illusional I alias docker to https://github.com/indigo-dc/udocker on our internal slurm HPC cluster. Podman (https://github.com/containers/libpod) looks like another promising option, but I have not tested it myself.

[elenimath]

@adamstruck would it be possible to provide the wrapper script you are using? I tried wrapping both udocker and singularity as docker but in both cases when running Funnel in our HPC I get an Exec error (unknown flag: --read-only)

[adamstruck]

@elenimath the command is end up looking something like: docker run --rm -i --read-only -w [workdir] -v [bindings] [imageName] [cmd].

https://github.com/ohsu-comp-bio/funnel/blob/master/worker/docker.go#L73

[kfox1111]

Podman takes those same arguments and works in rootless mode. Making the command executed configurable so that s/docker/podman/ can be ran would be interesting.