Change behavior of `process.cpus` back to `--cpus` when using Docker with a local executor

[mriffle]

New feature

Recently Nextflow changed the behavior of process.cpus to use --cpu-shares instead of --cpus when using Docker. I understand the benefits of this when using a cloud or HPC environment, but this doesn't make much sense for a local executor when the use of this parameter is to limit the number of CPUs this process will use.

This change effectively removed any ability to limit the impact on the system cpu-wise when running on a local machine. This change results in degradation of the functionality of Nextflow in this case.

Suggest implementation

I suggest changing it back to --cpus passed to Docker for the local executor.

[bentsherman]

I guess the problem is that if Nextflow wasn't also launched in a docker container with --cpu-shares then it does not have the same CPU shares guarantee as the containerized tasks, so it can be starved

You can set NXF_DOCKER_LEGACY=true in your environment to revert to the old behavior

[mriffle]

Hmm, I'd think that when most people run using a local executor, they aren't running Nextflow, itself, in a container. So anything that starts up can eat up 100% of the CPU, regardless of what process.cpus is set at.

[pditommaso]

The same can also happen when using hpc or k8s cluster, isn't it? The rationale was the cpu limit is very inefficient, and likely this is the same also when using Docker because, very likely, this depends on cgroups.

However, this bring me to the point if we should add support for explicit limits to cpus directive (both for docker and k8s)

[bentsherman]

We could add a docker.cpuLimits config option just like k8s.cpuLimits

[pditommaso]

@mriffle would work you adding an option e.g. docker.cpuLimits in the config to use --cpus instead of cpu-shares ?

[mriffle]

I think that is a great idea.

[bentsherman]

You can also use the NXF_DOCKER_LEGACY environment var for now, should have the same effect