How to run hybpiper.nf pipeline script using a Platform LSF cluster

[FuXiaogang-KIB]

Hi, Chris I wanna to submit my task using "bsub" in LSF cluster. How can I modify the config file in hybpiper-nf ?

[chrisjackson-pellicle]

Hi @FuXiaogang-KIB,

I haven't used LSF myself, so I can't speak from direct experience, but Nextflow does support it out of the box. So, it's likely just a matter of modifying one of the existing profiles (e.g. slurm_singularity) by replacing:

executor {
            name = 'slurm'
        }

with:

executor {
            name = 'lsf'
        }

Then, add any details specific to your cluster, such as adding the correct queue, clusterOptions and required modules to each (see e.g. processes in the gadi profile e.g.:

withName: ASSEMBLE_PAIRED_END {
                cpus = { 10 * task.attempt }
                memory = { 10.GB * task.attempt }
                errorStrategy  = { task.exitStatus in 137..141 ? 'retry' : 'terminate' }
                maxRetries = 3
                time = '24h'
                queue = 'normal'
                clusterOptions = "-l wd -l storage=gdata/nm31+scratch/nm31 -P nm31"
                module = 'singularity'
            }

All of the above is native Nextflow functionality - it's not specific to the hybpiper.nf pipeline.

Cheers,

Chris

[FuXiaogang-KIB]

Hi, Chris

Thanks for your responses and kind reply. I tried to modify the config file following your advice, but I failed. See the attachment files. Could you take a look at it when you have time? Thanks very much!

Best wishes, X.G

付小刚 @.***

 

------------------ 原始邮件 ------------------ 发件人: "chrisjackson-pellicle/hybpiper-nf" @.>; 发送时间: 2024年5月31日(星期五) 上午9:04 @.>; @.**@.>; 主题: Re: [chrisjackson-pellicle/hybpiper-nf] How to run hybpiper.nf pipeline script using a Platform LSF cluster (Issue #15)

Hi @FuXiaogang-KIB,

I haven't used LSF myself, so I can't speak from direct experience, but Nextflow does support it out of the box. So, it's likely just a matter of modifying one of the existing profiles (e.g. slurm_singularity) by replacing: executor { name = 'slurm' }
with: executor { name = 'lsf' }
Then, add any details specific to your cluster, such as adding the correct queue, clusterOptions and required modules to each (see e.g. processes in the gadi profile e.g.: withName: ASSEMBLE_PAIRED_END { cpus = { 10 task.attempt } memory = { 10.GB task.attempt } errorStrategy = { task.exitStatus in 137..141 ? 'retry' : 'terminate' } maxRetries = 3 time = '24h' queue = 'normal' clusterOptions = "-l wd -l storage=gdata/nm31+scratch/nm31 -P nm31" module = 'singularity' }
All of the above is native Nextflow functionality - it's not specific to the hybpiper.nf pipeline.

Cheers,

Chris

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

[chrisjackson-pellicle]

Hi @FuXiaogang-KIB - you'll need to attach the files directly via the repo website. Files attached when you reply via email do not appear here.

Cheers,

Chris

[FuXiaogang-KIB]

config.txt output_fxg.435690.txt Hi, Chris! Thanks a lot. Here is the attached files. Best wishes

[chrisjackson-pellicle]

Hi @FuXiaogang-KIB ,

In your output file you can see the following error:

ERROR ~ Error executing process > 'assemble:assemble_main:TRIMMOMATIC_PAIRED (4)'

Caused by:
  Failed to submit process to grid scheduler for execution

Command executed:

  cat << 'LAUNCH_COMMAND_EOF' | bsub
  #!/bin/bash
  #BSUB -o /nfs/fuxiaogang/2024_Urticaceae/1_YTS_Rosales/work/95/91f6d07d955a462bfb5bb378d847d3/.command.log
  #BSUB -q Q104C512G_X4
  #BSUB -n 10
  #BSUB -R "span[hosts=1]"
  #BSUB -W 24:00
  #BSUB -M 1048576
  #BSUB -R "select[mem>=10485760] rusage[mem=10240]"
  #BSUB -J nf-assemble_assemble_main_TRIMMOMATIC_PAIRED_(4)
  #BSUB -J hybpiper_nf
  #BSUB -n 40
  #BSUB -M 32000
  #BSUB -W 24:00
  NXF_CHDIR=/nfs/fuxiaogang/2024_Urticaceae/1_YTS_Rosales/work/95/91f6d07d955a462bfb5bb378d847d3
  # NEXTFLOW TASK: assemble:assemble_main:TRIMMOMATIC_PAIRED (4)
  set -e
  set -u
  NXF_DEBUG=${NXF_DEBUG:=0}; [[ $NXF_DEBUG > 1 ]] && set -x
  NXF_ENTRY=${1:-nxf_main}

...
...
...

Command exit status:
  255

Command output:
  Invalid syntax; the -R option was used more than once.
  . Job not submitted.

So, the issue appears to be multiple use of the -R option in the bsub submission script that Nextflow creates, i.e. the lines:

#BSUB -R "span[hosts=1]"

...and

 #BSUB -R "select[mem>=10485760] rusage[mem=10240]"

You can see how Nextflow creates these lines in the source code here (lines 85 and 102).

The LSF scheduler allows multiple usages of the -R option. However, I see from your output file that your HPC is using an openlava fork of LSF. Unfortunately, it looks like the openlava fork doesn't allow this (see e.g. here).

This means that you won't be able to use Nextflow (and hence hybpiper.nf) to submit jobs on your HPC while specifying memory and CPU requirements. Perhaps you could ask your HPC admin to patch the corresponding code in the Nextflow LsfExecutor.groovy file (as linked above) to concatenate multiple -R parameters in to a single string.

Cheers,

Chris

[FuXiaogang-KIB]

Hi, Chris. Got it. Thank you very much!Best Wishes-------- 原始郵件 --------寄件者： Chris Jackson @.>日期： 2024年6月4日 週二 09:26收件人： chrisjackson-pellicle/hybpiper-nf @.>副本： FuXiaogang-KIB @.>, Mention @.>主旨： Re: [chrisjackson-pellicle/hybpiper-nf] How to run hybpiper.nf pipeline script using a Platform LSF cluster (Issue #15) Hi @FuXiaogang-KIB , In your output file you can see the following error: ERROR ~ Error executing process > 'assemble:assemble_main:TRIMMOMATIC_PAIRED (4)'

Caused by: Failed to submit process to grid scheduler for execution

Command executed:

cat << 'LAUNCH_COMMAND_EOF' | bsub

!/bin/bash

BSUB -o /nfs/fuxiaogang/2024_Urticaceae/1_YTS_Rosales/work/95/91f6d07d955a462bfb5bb378d847d3/.command.log

BSUB -q Q104C512G_X4

BSUB -n 10

BSUB -R "span[hosts=1]"

BSUB -W 24:00

BSUB -M 1048576

BSUB -R "select[mem>=10485760] rusage[mem=10240]"

BSUB -J nf-assemble_assemble_main_TRIMMOMATICPAIRED(4)

BSUB -J hybpiper_nf

BSUB -n 40

BSUB -M 32000

BSUB -W 24:00

NXF_CHDIR=/nfs/fuxiaogang/2024_Urticaceae/1_YTS_Rosales/work/95/91f6d07d955a462bfb5bb378d847d3

NEXTFLOW TASK: assemble:assemble_main:TRIMMOMATIC_PAIRED (4)

set -e set -u NXF_DEBUG=${NXF_DEBUG:=0}; [[ $NXF_DEBUG > 1 ]] && set -x NXF_ENTRY=${1:-nxf_main}

... ... ...

Command exit status: 255

Command output: Invalid syntax; the -R option was used more than once. . Job not submitted.

So, the issue appears to be multiple use of the -R option in the bsub submission script that Nextflow creates, i.e. the lines:

BSUB -R "span[hosts=1]"

...and

BSUB -R "select[mem>=10485760] rusage[mem=10240]"

You can see how Nextflow creates these lines in the source code here (lines 85 and 102). The LSF scheduler allows multiple usages of the -R option. However, I see from your output file that your HPC is using an openlava fork of LSF. Unfortunately, it looks like the openlava fork doesn't allow this (see e.g. here). This means that you won't be able to use Nextflow (and hence hybpiper.nf) to submit jobs on your HPC while specifying memory and CPU requirements. Perhaps you could ask your HPC admin to patch the corresponding code in the Nextflow LsfExecutor.groovy file (as linked above) to concatenate multiple -R parameters in to a single string using the && operator. Cheers, Chris

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>