gwforg / gwf

A flexible, pragmatic workflow tool.
https://gwf.app/
GNU General Public License v3.0
31 stars 12 forks source link

After update to 1.8.0: Targets ending in "Shouldrun" even though they finish correctly #397

Closed simondrue closed 1 year ago

simondrue commented 2 years ago

After the update to 1.8.0 I have some target that have worked previously that end up in "Shouldrun". The output files are correct, the spec have not changed (I've tried running with the "gwf config set use_spec_hashes no" but did not help). An example of a templete that gives a target with this behaviour:

def minimap2_alignment(fastq_files_list, reference_genome, output_bam, read_group, cores=8, memory="64g", walltime="12:00:00"):
"""A template for aligning fastq files to the reference genome"""

inputs = fastq_files_list + [reference_genome]
outputs = [output_bam, f"{output_bam}.bai"]

fastq_files_str = " ".join(fastq_files_list)

spec = f"""

set -e

start_time=`date +%s`

TEMP_DIR=$(mktemp -d --tmpdir=/scratch/$SLURM_JOBID)

# Note: Minimap uses 1 threads for I/O opeartions (thus cores - 1)
minimap2 -ax map-ont -t {cores - 1} --MD {reference_genome} {fastq_files_str} \
| samtools sort \
    -@ {cores} \
    -T ${{TEMP_DIR}} \
    -O "bam" \
> ${{TEMP_DIR}}/sorted.bam

samtools addreplacerg \
    -@ {cores} \
    -r "@RG\tID:{read_group}" \
    -o ${{TEMP_DIR}}/tagged.bam \
    ${{TEMP_DIR}}/sorted.bam

samtools index -@ {cores} ${{TEMP_DIR}}/tagged.bam

mv ${{TEMP_DIR}}/tagged.bam {output_bam}
mv ${{TEMP_DIR}}/tagged.bam.bai {output_bam}.bai

end_time=`date +%s`

runtime=$((end_time-start_time))

printf 'runtime: %02d:%02d:%02d\n' $((runtime/3600)) $((runtime%3600/60)) $((runtime%60)) >&2

"""

return AnonymousTarget(inputs=inputs, outputs=outputs, options={"cores": cores, "memory": memory, "walltime": walltime}, spec=spec)
dansondergaard commented 2 years ago

Yes, something (wrong) happened to the scheduling logic. When running a workflow it works fine, but the status output is wrong. I haven't had time to look at it yet, unfortunately.

dansondergaard commented 1 year ago

It's fixed now and will be part of the next release (hopefully soon).