running nextfow pipeline in slurm using srun and mpi support

Hi,

Using a nextflow pipeline (developed in 22.10.6), I want to run ~4000 jobs; these jobs each can take 1-2 days depending on the coverage. I intend to run this workflow on the Linux clusters of our university. This HPC has a standard set-up: a login node should only be used to submit jobs to one of the several clusters and partitions. Now depending on the queue time and limitations associated with policies of the HPC administration, the pipeline completion can take at least a few weeks. Further, except for the login node, all the other queues support the job spawning with mpi or srun from the sbatch file. In this pipeline, I am using executor as "slurm" (in nextflow config file) with computational resources mentioned in base.config. In the past, I have run this pipeline (with 50-100 jobs) by directly running nextflow command in "screen" from the login node (though it is not recommended way in our HPC system).

I would highly appreciate if anyone can answer these questions: (1) how can I wrap "nextflow run" command in a shell script and submit it as any other job, (2) can I run nextflow job with mpi and srun; based on my extensive search in the past issues of this github repository, I came to know that it was possible to do so in the past releases of nextflow (20.XX), however, current version does not support it anymore.

Thanks.

nextflow-io / nextflow

running nextfow pipeline in slurm using srun and mpi support #3646