Process exceeded running time limit

[mpdoane2]

I am attempting to run eDNAFlow and am running into this error. The runtime can not exceed 4 hours and appears to stop at that point. You can see from the file % complete for the command that they get to approximately 77% completion and require an addition 1 hour and 8 minutes to complete this step.

executor >  local (1)
[3c/f53f14] process > 00_fastQC (1)               [100%] 1 of 1, cached: 1 ✔
[78/b5e6e8] process > 01_a_quality_Filtering (1)  [100%] 1 of 1, cached: 1 ✔
[fd/6e0044] process > 01_b_fastQC (1)             [100%] 1 of 1, cached: 1 ✔
[46/34aff4] process > 02_assigned_dmux (1)        [100%] 1 of 1, failed: 1 ✘
[-        ] process > 03_Length_filtered          -
[-        ] process > 04_splitSamples_relabel_Cat -
[-        ] process > 05_relabel_Cat              -
[-        ] process > 06_Uniques_ZOTUs            -
[-        ] process > 07_blast                    -
[-        ] process > 08_lulu                     -
[-        ] process > 09_taxonomyAssigned         -
Error executing process > '02_assigned_dmux (1)'

Caused by:
  Process exceeded running time limit (4h)

Command executed:

  ngsfilter -t 16s_bc.txt -e 2 -u "orphan.fastq" EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq > "EFMSRun199-Elib180_S1_L001_R1_001_16s_bc_QF_Dmux.fastq"

Command exit status:
  -

Command output:
  (empty)

Command error:
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.5 % |######################################\            ] remain : 01:08:59
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.5 % |######################################|            ] remain : 01:08:59
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.5 % |######################################/            ] remain : 01:08:58
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.5 % |######################################-            ] remain : 01:08:58
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.5 % |######################################\            ] remain : 01:08:58
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.5 % |######################################|            ] remain : 01:08:58
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.5 % |######################################/            ] remain : 01:08:57
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.5 % |######################################-            ] remain : 01:08:57
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.5 % |######################################\            ] remain : 01:08:57
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.6 % |######################################|            ] remain : 01:08:56
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.6 % |######################################/            ] remain : 01:08:56
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.6 % |######################################-            ] remain : 01:08:56
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.6 % |######################################\            ] remain : 01:08:56
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.6 % |######################################|            ] remain : 01:08:55
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.6 % |######################################/            ] remain : 01:08:55
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.6 % |######################################-            ] remain : 01:08:55
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.6 % |######################################\            ] remain : 01:08:55
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.6 % |######################################|            ] remain : 01:08:54
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.6 % |######################################/            ] remain : 01:08:54
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.6 % |######################################-            ] remain : 01:08:54
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.6 % |######################################\            ] remain : 01:08:54
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.6 % |######################################|            ] remain : 01:08:53
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.6 % |######################################/            ] remain : 01:08:53
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.6 % |######################################-            ] remain : 01:08:53
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.6 % |######################################\            ] remain : 01:08:53
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.6 % |######################################|            ] remain : 01:08:52
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.6 % |######################################/            ] remain : 01:08:52
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.6 % |######################################-            ] remain : 01:08:52
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.6 % |######################################\            ] remain : 01:08:51
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.6 % |######################################|            ] remain : 01:08:51
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.6 % |######################################/            ] remain : 01:08:51
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.6 % |######################################-            ] remain : 01:08:51
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.6 % |######################################\            ] remain : 01:08:50
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.6 % |######################################|            ] remain : 01:08:50
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.6 % |######################################/            ] remain : 01:08:50
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.6 % |######################################-            ] remain : 01:08:50
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.6 % |######################################\            ] remain : 01:08:49
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.6 % |######################################|            ] remain : 01:08:49
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.6 % |######################################/            ] remain : 01:08:49
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.6 % |######################################-            ] remain : 01:08:48
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.6 % |######################################\            ] remain : 01:08:48
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.6 % |######################################|            ] remain : 01:08:48
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.6 % |######################################/            ] remain : 01:08:48
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.6 % |######################################-            ] remain : 01:08:47
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.6 % |######################################\            ] remain : 01:08:47
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.6 % |######################################|            ] remain : 01:08:47
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.6 % |######################################/            ] remain : 01:08:47
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.6 % |######################################-            ] remain : 01:08:46
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.6 % |######################################\            ] remain : 01:08:46
  EFMSRun199-Elib180_S1_L001_R1_001_QF.fastq  77.6 % |######################################|            ] remain : 01:08:46

Work dir:
  /home/doan0033/eDNAFlow/work/46/34aff46e683f56c8cc834c9673a244

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`

Can you please help with this problem? We are not familiar with Nextflow and Singularity (rather we use Snakemake and Docker) and so are at a loss for how to bridge this problem.

Thank you

[sebrauschert]

Hi @mpdoane2 , the conf folder contains the file base.config:

singularity {
      enabled = true
      autoMounts = true
      String parameters = ''
      if (params.bindDir!='') {
    for (a in params.bindDir.split()) {
       parameters = "${parameters} -B ${a}"
        }
    runOptions = parameters.substring(1)
      }
      if (params.singularityDir!='') {
        cacheDir = "${params.singularityDir}"
      }
    }

  process {

      withLabel: 'fastqc'                   { container = 'docker://biocontainers/fastqc:v0.11.9_cv7' }
      withLabel: 'adapterRemoval'           { container = 'docker://biocontainers/adapterremoval:v2.2.0-1-deb_cv1' }
      withLabel: 'obitools'                 { container = 'quay.io/biocontainers/obitools:1.2.11--py27_1' }
      withLabel: 'usearch'                  { container = 'docker://sunqiangkun/usearch:v1' }
      withLabel: 'blast'                    { container = 'docker://ncbi/blast:2.10.1'  }
      withLabel: 'lulu'                     { container = 'docker://index.docker.io/mahsamousavi/lulu:2019' }
      withLabel: 'lca_python3'              { container = 'docker://python:3.6'}

      cache = 'lenient' 

      cpus = { check_max( 1 * task.attempt, 'cpus' ) }
      memory = { check_max( 6.GB * task.attempt, 'memory' ) }
      time = { check_max( 4.h * task.attempt, 'time' ) }
}

change the time parameter from 4.h to something like 10.h (or more, if you still receive the error), and it should work.

[mahsa-mousavi]

Hi @mpdoane2, Sorry I missed seeing your enquiry on time. But excellent solution by @Hobbeist (Thanks :) ) Also you can increase cpus and memory in base.config if you have access to larger computational resources.