search

langmead-lab / monorail-external

examples to run monorail externally
MIT License
13 stars 5 forks source link

STAR: Bus error #22

Closed daria-dc closed 2 years ago

daria-dc commented 2 years ago

Hi,

I am trying to process a couple of single end fastq files, however, STAR aligner is throwing a Bus error

/bin/bash: line 16: 99885 Bus error               STAR --runMode alignReads --runThreadN 10 --genomeDir /container-mounts/recount/ref/hg38/star_idx --readFilesIn ${READ_FILES} --twopassMode None --genomeLoad ${genome_load} --outTmpDir ${TMP} --outReadsUnmapped Fastx --outMultimapperOrder Old_2.4 --outSAMreadID ${readnames} --outSAMtype BAM Unsorted --outSAMmode NoQS --outSAMattributes NH MD --chimOutType Junctions SeparateSAMold --chimOutJunctionFormat 1 --chimSegmentReadGapMax 3 --chimJunctionOverhangMin 12 --chimSegmentMin 12 2>&1
     99886 Done                    | tee -a /container-mounts/recount/output/DEKFOMUC00002!LOCAL_STUDY!hg38!local.align.log

This is the complete log of the align rule

Error in rule align:
    jobid: 1
    output: /container-mounts/recount/temp_big/DEKFOMUC00002!LOCAL_STUDY!hg38!local.bam, /container-mounts/recount/output/DEKFOMUC00002!LOCAL_STUDY!hg38!local.sjout.zst, /container-mounts/recount/output/DEKFOMUC00002!LOCAL_STUDY!hg38!local.Chimeric.out.junction.zst, /container-mounts/recount/output/DEKFOMUC00002!LOCAL_STUDY!hg38!local.Chimeric.out.sam.zst, /container-mounts/recount/temp/DEKFOMUC00002!LOCAL_STUDY!hg38!local_0.unmappedfastq, /container-mounts/recount/temp/DEKFOMUC00002!LOCAL_STUDY!hg38!local_1.unmappedfastq, /container-mounts/recount/temp/DEKFOMUC00002!LOCAL_STUDY!hg38!local_2.unmappedfastq
    log: /container-mounts/recount/output/DEKFOMUC00002!LOCAL_STUDY!hg38!local.align.log

RuleException:
CalledProcessError in line 864 of /Snakefile:
Command ' set -euo pipefail;  
        genome_load="LoadAndRemove"
        if [[ "0" == "1" ]]; then
            genome_load="NoSharedMemory"
        fi

        READ_FILES="/container-mounts/recount/temp_big/DEKFOMUC00002!LOCAL_STUDY!hg38!local_0.fastq"
        if [[ -s /container-mounts/recount/temp_big/DEKFOMUC00002!LOCAL_STUDY!hg38!local_2.fastq ]] ; then
            READ_FILES="/container-mounts/recount/temp_big/DEKFOMUC00002!LOCAL_STUDY!hg38!local_1.fastq /container-mounts/recount/temp_big/DEKFOMUC00002!LOCAL_STUDY!hg38!local_2.fastq"
        fi
        readnames='Number'
        if [[ 0 == "1" ]] ; then
            readnames='Standard'
        fi
        TMP="/container-mounts/recount/temp_big/align_temp.DEKFOMUC00002"
        rm -rf ${TMP}
        time STAR                          --runMode alignReads             --runThreadN 10             --genomeDir /container-mounts/recount/ref/hg38/star_idx             --readFilesIn ${READ_FILES}             --twopassMode None             --genomeLoad ${genome_load}             --outTmpDir ${TMP}             --outReadsUnmapped Fastx             --outMultimapperOrder Old_2.4             --outSAMreadID ${readnames}             --outSAMtype BAM Unsorted             --outSAMmode NoQS             --outSAMattributes NH MD             --chimOutType Junctions SeparateSAMold             --chimOutJunctionFormat 1             --chimSegmentReadGapMax 3             --chimJunctionOverhangMin 12             --chimSegmentMin 12 2>&1 | tee -a /container-mounts/recount/output/DEKFOMUC00002!LOCAL_STUDY!hg38!local.align.log

        # Full set of output files:
        #
        # Aligned.out.bam
        # Chimeric.out.junction
        # Log.final.out
        # Log.out
        # Log.progress.out
        # SJ.out.tab
        # Unmapped.out.mate1
        # Unmapped.out.mate2 (if any reads were paired-end)

        #
        # Logs
        #
        rm -rf ${TMP}
        cat Log.out >> /container-mounts/recount/output/DEKFOMUC00002!LOCAL_STUDY!hg38!local.align.log
        cat Log.final.out >> /container-mounts/recount/output/DEKFOMUC00002!LOCAL_STUDY!hg38!local.align.log
        rm -f Log*.out

        #
        # Junctions
        #
        test -f SJ.out.tab
        time zstd SJ.out.tab -o /container-mounts/recount/output/DEKFOMUC00002!LOCAL_STUDY!hg38!local.sjout.zst 2>&1 | tee -a /container-mounts/recount/output/DEKFOMUC00002!LOCAL_STUDY!hg38!local.align.log
        rm -f SJ.out.tab
        size=$(wc -c < /container-mounts/recount/output/DEKFOMUC00002!LOCAL_STUDY!hg38!local.sjout.zst)
        echo "COUNT_CompressedJxBytes ${size}"

        #
        # Chimerics
        #
        test -f Chimeric.out.junction
        test -s Chimeric.out.junction
        sort -k1,1 -k2,2n Chimeric.out.junction > Chimeric.out.junction.sorted
        time zstd Chimeric.out.junction.sorted -o /container-mounts/recount/output/DEKFOMUC00002!LOCAL_STUDY!hg38!local.Chimeric.out.junction.zst 2>&1 | tee -a /container-mounts/recount/output/DEKFOMUC00002!LOCAL_STUDY!hg38!local.align.log
        rm -f Chimeric.out.junction Chimeric.out.junction.sorted
        size=$(wc -c < /container-mounts/recount/output/DEKFOMUC00002!LOCAL_STUDY!hg38!local.Chimeric.out.junction.zst)
        echo "COUNT_ChimericBytes ${size}"

        test -f Chimeric.out.sam
        test -s Chimeric.out.sam
        time zstd Chimeric.out.sam -o /container-mounts/recount/output/DEKFOMUC00002!LOCAL_STUDY!hg38!local.Chimeric.out.sam.zst 2>&1 | tee -a /container-mounts/recount/output/DEKFOMUC00002!LOCAL_STUDY!hg38!local.align.log
        rm -f Chimeric.out.sam
        size=$(wc -c < /container-mounts/recount/output/DEKFOMUC00002!LOCAL_STUDY!hg38!local.Chimeric.out.sam.zst)
        echo "COUNT_ChimericSAMBytes ${size}"

        #
        # Unmapped
        #
        touch /container-mounts/recount/temp/DEKFOMUC00002!LOCAL_STUDY!hg38!local_2.unmappedfastq
        test -f Unmapped.out.mate1
        mv Unmapped.out.mate1 /container-mounts/recount/temp/DEKFOMUC00002!LOCAL_STUDY!hg38!local_1.unmappedfastq
        if [[ -f Unmapped.out.mate2 ]] ; then
            mv Unmapped.out.mate2 /container-mounts/recount/temp/DEKFOMUC00002!LOCAL_STUDY!hg38!local_2.unmappedfastq
        fi

        #
        # Alignments
        #
        size=$(wc -c < Aligned.out.bam)
        echo "COUNT_BAMBytes ${size}"
        mv Aligned.out.bam /container-mounts/recount/temp_big/DEKFOMUC00002!LOCAL_STUDY!hg38!local.bam

        echo "COUNT_AlignComplete 1"

        if [[ -s /container-mounts/recount/temp_big/DEKFOMUC00002!LOCAL_STUDY!hg38!local_2.fastq && -s /container-mounts/recount/temp_big/DEKFOMUC00002!LOCAL_STUDY!hg38!local_0.fastq ]] ; then
            READ_FILES="/container-mounts/recount/temp_big/DEKFOMUC00002!LOCAL_STUDY!hg38!local_0.fastq"
            TMP="/container-mounts/recount/temp_big/align_temp.DEKFOMUC00002.0"
            rm -rf ${TMP}
            time STAR                                  --runMode alignReads                 --runThreadN 10                 --genomeDir /container-mounts/recount/ref/hg38/star_idx                 --readFilesIn ${READ_FILES}                 --twopassMode None                 --genomeLoad ${genome_load}                 --outTmpDir ${TMP}                 --outReadsUnmapped Fastx                 --outMultimapperOrder Old_2.4                 --outSAMreadID ${readnames}                 --outSAMtype BAM Unsorted                 --outSAMmode NoQS                 --outSAMattributes NH MD                 --chimOutType Junctions SeparateSAMold                 --chimOutJunctionFormat 1                 --chimSegmentReadGapMax 3                 --chimJunctionOverhangMin 12                 --chimSegmentMin 12 2>&1 | tee -a /container-mounts/recount/output/DEKFOMUC00002!LOCAL_STUDY!hg38!local.align.log

            # Full set of output files:
            #
            # Aligned.out.bam
            # Chimeric.out.junction
            # Log.final.out
            # Log.out
            # Log.progress.out
            # SJ.out.tab
            # Unmapped.out.mate1
            # Unmapped.out.mate2 (if any reads were paired-end)

            #
            # Logs
            #
            rm -rf ${TMP}
            cat Log.out >> /container-mounts/recount/output/DEKFOMUC00002!LOCAL_STUDY!hg38!local.align.log
            cat Log.final.out >> /container-mounts/recount/output/DEKFOMUC00002!LOCAL_STUDY!hg38!local.align.log
            rm -f Log*.out

            #
            # Junctions
            #
            test -f SJ.out.tab
            if [[ -s /container-mounts/recount/output/DEKFOMUC00002!LOCAL_STUDY!hg38!local.sjout.zst ]] ; then
                cat <(zstd -cd /container-mounts/recount/output/DEKFOMUC00002!LOCAL_STUDY!hg38!local.sjout.zst) SJ.out.tab | sort -k1,1 -k2,2n -k3,3n -k4,4n -k5,5n -k6,6n | perl -ne 'chomp; $f=$_; @f=split(/ /,$f); ($c,$s,$e,$o,$motif,$annot,$nu,$nmm,$max_anchor)=@f; $k=join("   ",($c,$s,$e,$o,$motif,$annot)); if($pk) { if($pk eq $k) { $pnu+=$nu; $pnmm+=$nmm; $pmax_anchor = $max_anchor > $pmax_anchor?$max_anchor:$pmax_anchor; next; } print "$pk    $pnu    $pnmm   $pmax_anchor
"; } $pk=$k; $pnu=$nu; $pnmm=$nmm; $pmax_anchor=$max_anchor; END { if($pk) { print "$pk $pnu    $pnmm   $pmax_anchor
"; } }' > sjout.all.merged
                time zstd -f sjout.all.merged -o /container-mounts/recount/output/DEKFOMUC00002!LOCAL_STUDY!hg38!local.sjout.zst 2>&1 | tee -a /container-mounts/recount/output/DEKFOMUC00002!LOCAL_STUDY!hg38!local.align.log
                rm -f SJ.out.tab sjout.all.merged
            else
                time zstd -f SJ.out.tab -o /container-mounts/recount/output/DEKFOMUC00002!LOCAL_STUDY!hg38!local.sjout.zst 2>&1 | tee -a /container-mounts/recount/output/DEKFOMUC00002!LOCAL_STUDY!hg38!local.align.log
            fi
            size=$(wc -c < /container-mounts/recount/output/DEKFOMUC00002!LOCAL_STUDY!hg38!local.sjout.zst)
            echo "COUNT_CompressedJxBytes_both ${size}"

            #
            # Chimerics
            #
            test -f Chimeric.out.junction
            test -s Chimeric.out.junction
            #now join them with the first run chimerics
            if [[ -s /container-mounts/recount/output/DEKFOMUC00002!LOCAL_STUDY!hg38!local.Chimeric.out.junction.zst ]] ; then
                cat <(zstd -cd /container-mounts/recount/output/DEKFOMUC00002!LOCAL_STUDY!hg38!local.Chimeric.out.junction.zst) Chimeric.out.junction | sort -k1,1 -k2,2n > Chimeric.out.junction.sorted
            else
                cat Chimeric.out.junction | sort -k1,1 -k2,2n > Chimeric.out.junction.sorted
            fi
            time zstd -f Chimeric.out.junction.sorted -o /container-mounts/recount/output/DEKFOMUC00002!LOCAL_STUDY!hg38!local.Chimeric.out.junction.zst 2>&1 | tee -a /container-mounts/recount/output/DEKFOMUC00002!LOCAL_STUDY!hg38!local.align.log
            rm -f Chimeric.out.junction Chimeric.out.junction.sorted
            size=$(wc -c < /container-mounts/recount/output/DEKFOMUC00002!LOCAL_STUDY!hg38!local.Chimeric.out.junction.zst)
            echo "COUNT_ChimericBytes_both ${size}"

            test -f Chimeric.out.sam
            test -s Chimeric.out.sam
            #now join them with the first run chimerics sam
            if [[ -s /container-mounts/recount/output/DEKFOMUC00002!LOCAL_STUDY!hg38!local.Chimeric.out.sam.zst ]] ; then
                cat <(zstd -cd /container-mounts/recount/output/DEKFOMUC00002!LOCAL_STUDY!hg38!local.Chimeric.out.sam.zst) <(samtools view Chimeric.out.sam) > Chimeric.out.sam.sorted
            else
                mv Chimeric.out.sam Chimeric.out.sam.sorted
            fi
            time zstd -f Chimeric.out.sam.sorted -o /container-mounts/recount/output/DEKFOMUC00002!LOCAL_STUDY!hg38!local.Chimeric.out.sam.zst 2>&1 | tee -a /container-mounts/recount/output/DEKFOMUC00002!LOCAL_STUDY!hg38!local.align.log
            rm -f Chimeric.out.sam Chimeric.out.sam.sorted
            size=$(wc -c < /container-mounts/recount/output/DEKFOMUC00002!LOCAL_STUDY!hg38!local.Chimeric.out.sam.zst)
            echo "COUNT_ChimericSAMBytes_both ${size}"

            #
            # Unmapped
            #
            test -f Unmapped.out.mate1
            mv Unmapped.out.mate1 /container-mounts/recount/temp/DEKFOMUC00002!LOCAL_STUDY!hg38!local_0.unmappedfastq

            #
            # Alignments
            #
            size=$(wc -c < Aligned.out.bam)
            echo "COUNT_BAMBytes_split3 ${size}"
            mv Aligned.out.bam /container-mounts/recount/temp_big/DEKFOMUC00002!LOCAL_STUDY!hg38!local.bam.0

            echo "COUNT_AlignComplete_split3 1"

            #now cat the 2 BAMs
            samtools cat /container-mounts/recount/temp_big/DEKFOMUC00002!LOCAL_STUDY!hg38!local.bam /container-mounts/recount/temp_big/DEKFOMUC00002!LOCAL_STUDY!hg38!local.bam.0 -o /container-mounts/recount/temp_big/DEKFOMUC00002!LOCAL_STUDY!hg38!local.bam.2
            mv /container-mounts/recount/temp_big/DEKFOMUC00002!LOCAL_STUDY!hg38!local.bam.2 /container-mounts/recount/temp_big/DEKFOMUC00002!LOCAL_STUDY!hg38!local.bam
            rm /container-mounts/recount/temp_big/DEKFOMUC00002!LOCAL_STUDY!hg38!local.bam.0
        else
            touch /container-mounts/recount/temp/DEKFOMUC00002!LOCAL_STUDY!hg38!local_0.unmappedfastq
        fi ' returned non-zero exit status 135.
  File "/Snakefile", line 864, in __rule_align
  File "/opt/conda/envs/recount/lib/python3.6/concurrent/futures/thread.py", line 56, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /container-mounts/recount/temp_big/snakemake-wd/.snakemake/log/2022-08-22T122951.005808.snakemake.log

Please let me know if you need more information than that. Any help is appreciated!

Daria

ChristopherWilks commented 2 years ago

Hi @daria-dc,

A bus error is fairly unusual, my informed guess would be an issue with the interaction between STAR and the OS/architecture of the machine this is run on.

A couple of questions might help further debug this: 1) which version of the the monorail pump Docker image are you using 2) are you using Docker or Singularity? 3) what's the underlying architecture of the system and OS name/version (e.g. x86_64 on Ubuntu 20.04)?

Thanks, Chris

daria-dc commented 2 years ago

Hi Chris,

  1. version 1.0.6
  2. Singularity
  3. x86_64 on CentOS Linux 7 (Core)

Thanks, Daria

daria-dc commented 2 years ago

Hi Chris,

I managed to do the processing of my samples on another machine, and STAR runs just fine, so closing this issue.

Daria

ChristopherWilks commented 2 years ago

thanks for the update @daria-dc