consider maintaining a singularity container image along with a docker one

[sergpolly]

I got a little tired of manually rebuilding our docker image into a singularity one. Our cluster cannot pull things from docker, as we have no rights to rebuild it it seems like ...

So, why wouldn't try to convert the existing Dockerfile into a singularity recipe: https://groups.google.com/a/lbl.gov/d/msg/singularity/JkW0tQd6G8M/BJHSHYPfAwAJ and "publish" it on Singularity-hub ... I still need to try if it stores images or just recipes - and if an image requires rebuilding every time it gets pulled from shub .

The motivation is to minimize configuration of distiller every time we run it on the cluster. Ideally, we just clone it as is, swap a project file and run - instead of re-configuring imge location, mount point etc

[sergpolly]

Just to expand confirm about singularity on our ghpcc:

• we cannot singularity pull docker://anything because of squashfs issue
• we can easily singularity pull shub://something (side note - having java inside of a container, containerized nextflow failed)
• crazy part: we can singularity pull docker://anything from a singularity container puller from docker-hub!?
• waiting for a response from our hpcc-admins about installing this squashfs thing, so that we could do singularity pull docker://anything
• checked some of the nf-core pipelines, everyone supports singularity , but no one maintains an image on singularity-hub ... everyone seems to be fine with singularity pull docker://anything on their clusters ...

[sergpolly]

Save some singularity instructions from the nextlfow.config here instead, to clear it a bit:

// specify local singularity image when modifying
// distiller_env docker image. Example:
// 1. grab docker and rebuild in a modifiable mode:
// sudo singularity build --sandbox tmp.img docker://mirnylab/distiller_env:${version}
// 2. modify it, e.g. add mounting point /my_new_mount:
// sudo singularity exec --writable tmp.img mkdir /my_new_mount
// 3. rebuild into read-only image for production:
// sudo singularity build production_distiller.img tmp.img
// use process.container = "production_distiller.img" on your cluster.

[sergpolly]

more info about mounting drives:

    // manual mounting of a shared network drive,
    // very typical for computer clusters.
    // replace /nearline with the name of shared
    // space on your cluster, and make sure the
    // distiller image is modified to include a
    // mounting point /mount.
    runOptions = "--bind /nearline:/mount:rw"
    // shared cluster space can be auto-mounted
    // when this feature is supported, disabled by default
    autoMounts = false

[sergpolly]

Singularity is amazing: https://github.com/sylabs/singularity/releases/tag/v3.0.0

somehow - this one supports mounting network drives without manually creating corresponding folder inside of the images ... and fully supports, current distiller_env image from dockerhub

apparently there is no need for separate singularity image

todo: rethink how we ran distiller - we are getting closer and closer to running it the same way people do on local machines with docker ...