Some processes were skipped during resume

[fengqing-dong]

Problem: the workflow run only can run correctly at the frist time ; it will be failed when with -stub or -resume. Some process will be skipped.

why and how to fix it? Can you give me some advice?

The detail informaton as flow:

• CMD

  # fail
  nextflow run /mnt/Taurus/private/fengqing/pipelines/panel/main.nf --sample_info sample_info.cfg -resume 
  nextflow run /mnt/Taurus/private/fengqing/pipelines/panel/main.nf --sample_info sample_info.cfg -resume 
  # success
  nextflow run /mnt/Taurus/private/fengqing/pipelines/panel/main.nf --sample_info sample_info.cfg

• LOG

  executor >  local (5)
  [f3/c05c29] process > CFG:CFG_WRITE_SAMPLE_INFO                   [100%] 1 of 1 ✔
  [cb/80b87f] process > TRIM:TRIM_ADAPTER_FASTP (T430V208)          [100%] 2 of 2, cached: 2 ✔
  [38/6ce494] process > ALN:ALN_BWA_MEM (T430V206)                  [100%] 2 of 2, cached: 2 ✔
  [88/44ebb1] process > ALN:ALN_PICARD_MARK_DUP (T430V208)          [100%] 2 of 2, cached: 2 ✔
  [06/70cd5d] process > ALN:ALN_GATK_BQSR (T430V208)                [100%] 2 of 2, cached: 2 ✔
  [11/b5b4c0] process > ALN:ALN_ABRA_REALIGN (T430V206)             [100%] 2 of 2, cached: 2 ✔
  [-        ] process > QC:QC_PICARD                          -
  [6a/b3a927] process > SNVINDEL:SNVINDEL_CALLER_VARDICT (T430V208) [100%] 1 of 1, cached: 1 ✔
  [79/793325] process > SNVINDEL:SNVINDEL_ANNO (T430V208)           [100%] 1 of 1, cached: 1 ✔
  [91/de2b49] process > SNVINDEL:SNVINDEL_FILTER (T430V208)         [100%] 1 of 1 ✔
  [83/67c275] process > SNVINDEL:REPORT_SNVINDEL (T430V208)         [100%] 1 of 1 ✔
  [da/8aee08] process > CNV:CNV_CALLER_CNVKIT (T430V208)            [100%] 1 of 1, cached: 1 ✔
  [b8/82da6f] process > CNV:CNV_CNVKIT_ANNO_FILTER (T430V208)       [100%] 1 of 1, cached: 1 ✔
  [49/d9431b] process > CNV:REPORT_CNV (T430V208)                   [100%] 1 of 1, cached: 1 ✔
  [d6/1b75dc] process > SV:SV_CALLER_DELLY (T430V208)               [100%] 1 of 1, cached: 1 ✔
  [d4/833f43] process > SV:SV_ANNO (T430V208)                       [100%] 1 of 1, cached: 1 ✔
  [50/5ae3bc] process > SV:SV_FILTER (T430V208)                     [100%] 1 of 1 ✔
  [cc/130a73] process > SV:REPORT_SV (T430V208)                     [100%] 1 of 1 ✔
  [-        ] process > SUMMARY_COPY_FILE                           -
  [-        ] process > INSERT_SQL 

  the processes of QC:QC_PICARD 、SUMMARY_COPY_FILE and INSERT_SQL were skipped!!! and not any error or warnning was throwed.

• SCRIPT

  workflow {
  all_meta = Channel.fromList(sampleParser(params.sample_info))
  TRIM(all_meta)
  ALN(TRIM.out.clean_fastq)
  ch_qc_params  = ALN.out.join(TRIM.out.fastp_json, by: [0])
  QC(ch_qc_params)
  tumor_bam = ALN.out.filter{ it[0].is_paired == false}.map{ [it[1],it[2]] }
  pair_bam  = ALN.out.filter{ it[0].is_paired == true}.map{ [it[1],it[2]] }
  sample_bam = all_meta.take(1).concat(tumor_bam.concat(pair_bam).collect()).collect(flat:false)
  SNVINDEL(sample_bam)
  CNV(sample_bam)
  SV(sample_bam)
  SUMMARY_COPY_FILE(params.meta, 
                  QC.out.collect(), 
                  SNVINDEL.out.snvindel_somatic_txt, 
                  SNVINDEL.out.snvindel_germline_txt,
                  SNVINDEL.out.snvindel_hotspot_txt,
                  CNV.out.cnv_outpath,
                  SV.out.sv_outpath,
                  hrr_hrd_result
                  )
  INSERT_SQL(params.meta, SUMMARY_COPY_FILE.out.report_dir)
  }

• INFO

  all_meta =  Channel.of([ [sample_id:T430V208, product_id:P00H03, sample_type:FFPE, library_method:hybrid, sample_fastq_path:[/mnt/Taurus/private/fengqing/run/panel/HRD/test_fastq/T430V208.R1.fastq, /mnt/Taurus/private/fengqing/run/panel/HRD/test_fastq/T430V208.R2.fastq], is_paired:false],
  [sample_id:T430V206, product_id:P00H03, sample_type:FFPE, library_method:hybrid, sample_fastq_path:[/mnt/Taurus/private/fengqing/run/panel/HRD/test_fastq/T430V206.R1.fastq, /mnt/Taurus/private/fengqing/run/panel/HRD/test_fastq/T430V206.R2.fastq], is_paired:true]]

• version: 23.04.4 build 5881

[bentsherman]

Can you share the complete log file? And is your pipeline repo publicly available? If not, it would be good to see the source code for the processes that were skipped.

[fengqing-dong]

Due to confidentiality, I am so sorry for that the full process code cannot be provided. Here's the streamlined process.

• pipeline:main.nf.txt
• log file: nextflow.log

[bentsherman]

Thank you for sharing what you could. From your pipeline code and log, the key issue is that QC:QC_PICARD was skipped, which caused the other processes to be skipped because they depend on QC.out.collect(). QC_PICARD depends on ALN which was not included, but clearly ALN was executed. So my guess is that something is wrong with the outputs of the ALN workflow. I would look there.

Another helpful debugging tool might be the nf-prov plugin. Why don't you use it with the dag format to generate a task DAG for the initial and resumed runs? Then you can compare them, and it might reveal exactly where the chain was broken.