search

nextflow-io / nf-nomad

Hashicorp Nomad executor plugin for Nextflow
https://nextflow-io.github.io/nf-nomad/
Apache License 2.0
2 stars 2 forks source link

Allow storage config using nomad `volume` blocks #13

Closed tomiles closed 4 months ago

tomiles commented 4 months ago

While docker volumes is a quick way to prototype adding storage to a job/task it is not considered best practice in nomad. Since it is very driver specific approach and bypasses any native storage (Volume) handling in Nomad. Also most production clusters will lock down mounting volumes this way for security reasons. So we should find a user friendly way to do this using the nomad volume block.

_Originally posted by @tomiles in https://github.com/nextflow-io/nf-nomad/pull/12#discussion_r1493728335_

tomiles commented 4 months ago 
Yes, I agree
by the moment the alternative/workaround is to use an s3 + fusion (and let `dockerVolume` empty)

_Originally posted by @jagedn in https://github.com/nextflow-io/nf-nomad/pull/12#discussion_r1493729101_

jagedn commented 4 months ago

One quick question:

in your experience, mount multiple volumes is a must or only one (as working dir) is enough ?

Currently if dockerVolume is specified it's mounted as working dir ( https://github.com/nextflow-io/nf-nomad/blob/4370f7b1d81873dc54159053f74bfc00ea7f0bd4/plugins/nf-nomad/src/main/nextflow/nomad/executor/NomadService.groovy#L100-L106)

I'm designing a new config value to allow mount a volume (and probably deprecate dockerVolume) and I wonder if multiple volumes is a typical scenario or by the moment only one is ok.

My current approach is something similar as

nomad{
    jobs{
          // only one?
          volume type:'csi', name: 'the-volume'
          volume type:'host', name: 'another-volume'
          volume type:'docker', name: 'a-nfs-volume'
    }
}

What do you think?