Dynamic label directive in a process definition

[jgolob]

New feature

I would propose allowing dynamic definition of the label directive in a process, to allow the following to be a valid directive in a process definition:

process SomeName {    
    label { task.attempt > 1 ? 'mem_super' : 'mem_high' }
    errorStrategy { task.exitStatus in 137..140 ? 'retry' : 'terminate' }
    maxRetries 1
.....
}

Usage scenario

It would support the very common use case when needing more memory than is available in one queue to switch not just the memory requested but also the queue (and potentially even engine). Specific examples would be:

• Academic SLURM-based HPC that have a 'regular' and 'high-memory' queue
• Hybrid use of 'local' commodity compute nodes and then dispatching to AWS / GCS when the local nodes have insufficient per-node memory.

• This would require the appropriate entries in the config file, but allow for this to be done once for all processes and workflows.

Suggest implementation

I am no wizard at groovy or java and have no specific suggestion. I apologize.

[bentsherman]

Note that since a label is essentially a shorthand for a group of process directives, you can accomplish the same by making those process directives dynamic. For example since you mentioned memory and queue:

process {
  memory = { task.attempt > 1 ? 16.GB : 8.GB }
  queue = { task.attempt > 1 ? 'high-memory' : 'regular' }
}

[jgolob]

@bentsherman Absolutely.

But with dynamic labels my hope is to keep the details of an HPC / computational environment abstracted away in the config file, and not hard-wired into each process definition.

For example, queue names will not have a regular pattern (particularly academic / private clouds).

[bentsherman]

You can already do that, the example I gave is what you would put in the config file