mapped BAM index not produced in high-throughput (ht) mode

[lauren-tjoeka]

Description of the bug

Hi! I was expecting a mapped BAM index to be output alongside the final mapped BAM when running fastquorum in high-throughput mode. Perhaps I'm missing something from my params file (see below). If I'm not missing anything, it would be useful if this was implemented in a future release. Many thanks!

Command used and terminal output

# run command
nextflow run nf-core/fastquorum \
-r dev \
-profile singularity \
-c fastquorum_ht.config \
-params-file fastquorum_params.yaml \
-with-trace trace.txt \
-with-report report.html \
--input samplesheet.csv \
--outdir /scratch/fastquorum

Relevant files

# parameters file: fastquorum

## Main options
mode: "ht"

## fgbio options
duplex_seq: true

## References
fasta: "/GATK.GRCh37/human_g1k_v37_decoy.fasta"
bwa: "/GATK.GRCh37/BWAIndex"
dict: "/GATK.GRCh37/human_g1k_v37_decoy.dict"
fasta_fai: "/GATK.GRCh37/human_g1k_v37_decoy.fasta.fai"
save_reference: false

# config file

// Institute HPC nf-core configuration profile
params {
    config_profile_description = 'Institute HPC cluster profile'
    config_profile_contact     = 'user'
    config_profile_url         = "https://institute.org/"
    monochromeLogs = false
    max_memory = '300.GB'
    max_cpus   = 40
    max_time   = '336.h'
}

// Queue
executor {
    queueSize         = 100
    queueStatInterval = '1 min'
    pollInterval      = '1 min'
    submitRateLimit   = '20 min'
}

// SLURM
process {
    executor = 'slurm'
    cache    = 'lenient'
    beforeScript = 'module load singularity'
    stageInMode = 'symlink'
    queue = { task.time < 2.h ? 'prod_short' : task.time < 24.h ? 'prod_med' : 'prod' } 
    errorStrategy = { task.exitStatus in [1] ? 'retry' : 'finish' }
    withLabel:error_retry {
        errorStrategy = 'retry'
        maxRetries    = 4
    }
    withName: 'NFCORE_FASTQUORUM:FASTQUORUM:FASTQC' {
    cpus = 4 
    memory = '15.GB'
    time = '1h'
    }
    withName: 'NFCORE_FASTQUORUM:FASTQUORUM:FASTQTOBAM' {
    cpus = 2
    memory = '5.GB'
    time = '1h 55m'
    }
    withName: 'NFCORE_FASTQUORUM:FASTQUORUM:ALIGN_RAW_BAM' {
    cpus = 19
    memory = '45.GB'
    time = '1h 55m'
    }
    withName: 'NFCORE_FASTQUORUM:FASTQUORUM:MERGE_BAM' {
    cpus = 4
    memory = '2.GB'
    time = '1h'
    }
    withName: 'NFCORE_FASTQUORUM:FASTQUORUM:GROUPREADSBYUMI' {
    cpus = 4
    memory = '5.GB'
    time = '1h 55m'
    }
    withName: 'NFCORE_FASTQUORUM:FASTQUORUM:COLLECTDUPLEXSEQMETRICS' {
    cpus = 2
    memory = '5.GB'
    time = '1h 55m'
    }
    withName: 'NFCORE_FASTQUORUM:FASTQUORUM:CALLANDFILTERDUPLEXCONSENSUSREADS' {
    cpus = 5
    memory = '20.GB'
    time = '3h'
    }
    withName: 'NFCORE_FASTQUORUM:FASTQUORUM:ALIGN_CONSENSUS_BAM' {
    cpus = 10
    memory = '35.GB'
    time = '4h'
    }

    withName: 'NFCORE_FASTQUORUM:FASTQUORUM:MULTIQC' {
    cpus = 1
    memory = '1.GB'
    time = '1h'
    }

    withName: 'NFCORE_FASTQUORUM:FASTQUORUM:SORTBAM' {
    cpus = 2
    memory = '5.GB'
    time = '1h 55m'
    }
}

// SINGULARITY
// If using custom files using ext.args, you need to mount the folder
singularity {
    enabled = true
    cacheDir = "/path/nextflow"
}

// Overwrite when resuming 
trace {
    overwrite = true
}

report {
    overwrite = true
}

System information

Nextflow version: nextflow/23.04.1 Hardware: HPC Executor: slurm Container engine: Singularity OS: CentOS 7 (rip) Version of nf-core/fastquorum (from MultiQC Report html): nf-core/fastquorum v1.0.1

[SPPearce]

I think it may be being made, but not exported. Can you take a look in the working directory for ALIGN_CONSENSUS_BAM?

[lauren-tjoeka]

thank you! yes it is there

[SPPearce]

This is still an issue that needs fixing. For now, you can change the publishDir rule for the process to ensure it does get published.

[SPPearce]

Try adding this as a seperate config file for now:

    withName: '.*ALIGN_CONSENSUS_BAM' {
        publishDir = [
            path: { "${params.outdir}/filtering/align_consensus_bam/${meta.id}" },
            mode: params.publish_dir_mode,
            pattern: '*.mapped.bam*'
        ]
    }

[SPPearce]

This should be fixed in the new dev version, so that the index file is actually exported.

[lauren-tjoeka]

The index files weren't exported.

I got the latest version of the dev branch: nextflow pull nf-core/fastquorum -r dev

And ran git log dev -1 --format=%cd #Tue Sep 24 13:02:58 2024

[znorgaard]

@lauren-tjoeka. Odd the tests were passing. I was able to replicate. Looks like it's because of a space in the glob pattern.

https://github.com/nf-core/fastquorum/pull/80 should resolve this.

[SPPearce]

Ah, didn't notice the space there. Can you test it is fixed now?

[lauren-tjoeka]

works great now! thank you!