Feature: add support for `spread` JobOpt

[matthdsm]

Hi,

Can we add support for the spread block in the JobOpts? This would enable us to spread the nf-nomad load more evenly among the nodes in the cluster.

Cheers M

https://developer.hashicorp.com/nomad/docs/job-specification/spread

[matthdsm]

Or is this already possible through the jobConstraints ? If so, could you add an example?

[abhi18av]

Hi @matthdsm ,

Good point! As of now, we don't support that particular block but it can be added via job.spread field.

But that being said, that with some iterations with constraints it should be possible to achieve the desired effect (maybe via distinct_hosts ?) https://developer.hashicorp.com/nomad/docs/job-specification/constraint#constraint-parameters

Could you please tell us a bit more about the exact use-case of spread in your scenario?

[abhi18av]

Just came across another possibility, I'd like to confirm with you guys spread scheduler-algorithm https://developer.hashicorp.com/nomad/docs/commands/operator/scheduler/set-config#scheduler-algorithm

[matthdsm]

Hey Abhi,

Our cluster is configured by default to binpack jobs (better for services), but we'd like the nextflow jobs to be spread out (better for batch jobs). Hence the needs for the spread block in the nomad job definitions from nextflow

[abhi18av]

Interesting scenario!

Let me give a closer look and perhaps we can test this out on an edge-release soon.

[jagedn]

This syntax is a first attempt to cover this feature, what do you think? is "nextflow-centric" ?

jobs {
        deleteOnCompletion = false
        volume = { type "host" name "scratchdir" }

        spreads = {
            spreadOver 'node.datacenter' weight 100 targets ['us-east1':70, 'us-east2':30]
            spreadOver 'node.datacenter' weight 50 
        }
    }

[jagedn]

another option can be to use a similar syntax as in constraints

jobs{
   spreads = {
        spread = name:'node.datacenter', weight: 50, targets : ['us-east1':70, 'us-east2':30] 
   }
}