-resume goes all the way back to SplitIntervals, make_uBAM and Bwa

[mantczakaus]

Hi, I was wondering if there is anything that could be done to improve Nextflow's -resume functionality in the pipeline. I have resumed a run that failed on pVACseq but still the pipeline went all the way back to SplitIntervals, make_uBAM and Bwa. It used cached tasks only for some quality control processes and HLA typing (see below). Happy to provide more details if necessary. Best wishes, Magda

[-        ] process > merge_fastq                    -
[3e/eb6cb3] process > RegionsBedToIntervalList (R... [100%] 1 of 1, cached: 1 ✔
[6f/a525c5] process > BaitsBedToIntervalList (Bai... [100%] 1 of 1, cached: 1 ✔
[4f/0880a3] process > preprocessIntervalList (pre... [100%] 1 of 1, cached: 1 ✔
[3f/f24619] process > SplitIntervals (SplitInterv... [100%] 1 of 1 ✔
[ff/b46504] process > IntervalListToBed (BedFromI... [100%] 1 of 1, cached: 1 ✔
[99/4b767f] process > ScatteredIntervalListToBed ... [100%] 40 of 40 ✔
[e6/87b50e] process > FastQC (patient1 : tumor_RNA)  [100%] 3 of 3, cached: 3 ✔
[a3/c6d8da] process > fastp (patient1 : tumor_DNA)   [100%] 2 of 2, cached: 2 ✔
[ab/6e40c4] process > FastQC_trimmed (patient1 : ... [100%] 2 of 2, cached: 2 ✔
[56/7cbddd] process > make_uBAM (patient1 : norma... [  0%] 0 of 2
[e8/2a5ab0] process > Bwa (patient1 : normal_DNA)    [  0%] 0 of 2
[-        ] process > merge_uBAM_BAM                 -
[-        ] process > MarkDuplicates                 -
[-        ] process > alignmentMetrics               -
[-        ] process > scatterBaseRecalGATK4          -
[-        ] process > gatherGATK4scsatteredBQSRta... -
[-        ] process > scatterGATK4applyBQSRS         -
[-        ] process > GatherRecalBamFiles            -
[-        ] process > GetPileup                      -
[c9/8805c3] process > install_conda_GATK3 (instal... [100%] 1 of 1, cached: 1 ✔
[-        ] process > Mutect2                        -
[-        ] process > gatherMutect2VCFs              -
[-        ] process > FilterMutect2                  -
[-        ] process > HaploTypeCaller                -
[-        ] process > CNNScoreVariants               -
[-        ] process > MergeHaploTypeCallerGermlin... -
[-        ] process > FilterGermlineVariantTranches  -
[-        ] process > IndelRealignerIntervals        -
[-        ] process > GatherRealignedBamFiles        -
[-        ] process > VarscanSomaticScattered        -
[-        ] process > gatherVarscanVCFs              -
[-        ] process > ProcessVarscan                 -
[-        ] process > FilterVarscan                  -
[-        ] process > MergeAndRenameSamplesInVars... -
[-        ] process > MantaSomaticIndels             -
[-        ] process > StrelkaSomatic                 -
[-        ] process > finalizeStrelkaVCF             -
[-        ] process > mkHCsomaticVCF                 -
[-        ] process > VepTab                         -
[-        ] process > mkCombinedVCF                  -
[-        ] process > VEPvcf                         -
[-        ] process > ReadBackedphasing              -
[-        ] process > AlleleCounter                  -
[-        ] process > ConvertAlleleCounts            -
[-        ] process > Ascat                          -
[-        ] process > SequenzaUtils                  -
[-        ] process > gatherSequenzaInput            -
[-        ] process > Sequenza                       -
[4e/66fcfd] process > make_CNVkit_access_file (mk... [100%] 1 of 1, cached: 1 ✔
[-        ] process > Clonality                      -
[-        ] process > MutationalBurden               -
[-        ] process > MutationalBurdenCoding         -
[-        ] process > mhc_extract                    -
[-        ] process > pre_map_hla                    -
[-        ] process > OptiType                       -
[77/556919] process > pre_map_hla_RNA (patient1)     [100%] 1 of 1, cached: 1 ✔
[14/000db6] process > OptiType_RNA (patient1)        [100%] 1 of 1, cached: 1 ✔
[-        ] process > run_hla_hd                     -
[-        ] process > get_vhla                       -
[-        ] process > Neofuse                        -
[-        ] process > publish_NeoFuse                -
[-        ] process > add_geneID                     -
[-        ] process > gene_annotator                 -
[-        ] process > pVACseq                        -
[-        ] process > concat_pVACseq_files           -
[-        ] process > aggregated_reports             -
[-        ] process > pVACtools_generate_protein_seq -
[-        ] process > pepare_mixMHC2_seq             -
[-        ] process > mixMHC2pred                    -
[-        ] process > addCCF                         -
[-        ] process > make_epitopes_fasta            -
[-        ] process > blast_epitopes                 -
[-        ] process > add_blast_hits                 -
[-        ] process > csin                           -
[-        ] process > immunogenicity_scoring         -
[-        ] process > collectSampleInfo              -
[-        ] process > multiQC                        -

[riederd]

There are certain shared filesystems that seem to have inconsistent files timestamps, this is screwing up the caching invalidation of nextflow.

You could try to set the following parameter in conf/process.config around https://github.com/icbi-lab/nextNEOpi/blob/aac260dbd5da701d22a09846045f3f42981cd4a1/conf/process.config#L10

     // default caching
     cache='lenient'

Let me know if this works

[mantczakaus]

Setting cache to lenient worked - it now uses over 500 tasks from the cache instead of 15 - thank you!

[riederd]

resolved by 3f34b81da8c155e9b0f905a95aaca9c06710bb4b