search

icbi-lab / nextNEOpi

nextNEOpi: a comprehensive pipeline for computational neoantigen prediction
Other
67 stars 24 forks source link

how to install all required tools？ #6

Closed sunsong95 closed 2 years ago

sunsong95 commented 2 years ago

Thank you for providing such convenient software. I saw in the readme it was written:

The pipeline will install almost all required tools via Singularity images or conda environments.

But I didn't find a way or requirement.txt to install the dependent tools. Do I need to use conda to install all the dependent tools by myself?

Thanks for your help！

riederd commented 2 years ago

Hi, thanks for your interest in nextNEOpi.

As stated in the README.md, you need to have the following software installed on your system:

all the other dependencies are containerized in to singularity containers, which will be fetched automatically.

After installing the software mentioned above and downloading the reference files (see README.md) you can run nextNEOpi, using the option -profile singularity as shown in the example in the README.md.

sunsong95 commented 2 years ago

Hi, I tried to install by the following command:

../nextflow run nextNEOpi.nf --batchFile example_my_BAM.csv -profile singularity -config conf/params.config

An error occurred：

Error executing process > 'mixcr (sample1 : normal_DNA)'

Caused by:
  Process `mixcr (sample1 : normal_DNA)` terminated with an error exit status (1)

Command executed:

  mixcr analyze shotgun \
      --threads 16 \
      --species hs \
      --starting-material dna \
      --only-productive \
      sample1_normal_DNA_R1.fastq.gz sample1_normal_DNA_R2.fastq.gz \
      sample1_normal_DNA_mixcr

Command exit status:
  1

Command output:
  (empty)

Command error:
  INFO:    Converting SIF file to temporary sandbox...
  Exception in thread "main" java.lang.ExceptionInInitializerError
        at com.milaboratory.mixcr.cli.Main.main(Main.java)
        at com.milaboratory.mixcr.cli.Main1.main(Main1.java:61)
  Caused by: java.lang.RuntimeException: java.nio.file.FileSystemException: /home/myname/.mi: Read-only file system
        at com.milaboratory.milm.KVFilesInFolderStore.forClass(KVFilesInFolderStore.java:90)
        at com.milaboratory.milm.LM.<init>(LM.java:78)
        at com.milaboratory.milm.LM.<init>(LM.java:63)
        at com.milaboratory.milm.MiXCRMain.<clinit>(MiXCRMain.java:11)
        ... 2 more
  Caused by: java.nio.file.FileSystemException: /home/myname/.mi: Read-only file system
        at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
        at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389)
        at java.base/java.nio.file.Files.createDirectory(Files.java:689)
        at java.base/java.nio.file.Files.createAndCheckIsDirectory(Files.java:796)
        at java.base/java.nio.file.Files.createDirectories(Files.java:782)
        at com.milaboratory.milm.KVFilesInFolderStore.forClass(KVFilesInFolderStore.java:88)
        ... 5 more
  INFO:    Cleaning up image...

Work dir:
  /home/myname/software/nextNEOpi/work/58/f70e2dd86ece6d1f39f4316e183a45

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

I did chmod -R 777 in /home/myname/.mi but the error persists. Do you have any suggestion？

riederd commented 2 years ago

Hi, this is strange, I wonder where the following line is coming from:

INFO: Converting SIF file to temporary sandbox...

Can you post/attach the following file:

/home/myname/software/nextNEOpi/work/58/f70e2dd86ece6d1f39f4316e183a45/.command.run

Thanks

sunsong95 commented 2 years ago

This is it:

#!/bin/bash
# NEXTFLOW TASK: mixcr (sample1 : normal_DNA)
set -e
set -u
NXF_DEBUG=${NXF_DEBUG:=0}; [[ $NXF_DEBUG > 1 ]] && set -x
NXF_ENTRY=${1:-nxf_main}

nxf_tree() {
    local pid=$1

    declare -a ALL_CHILDREN
    while read P PP;do
        ALL_CHILDREN[$PP]+=" $P"
    done < <(ps -e -o pid= -o ppid=)

    pstat() {
        local x_pid=$1
        local STATUS=$(2> /dev/null < /proc/$1/status egrep 'Vm|ctxt')

        if [ $? = 0 ]; then
        local  x_vsz=$(echo "$STATUS" | grep VmSize | awk '{print $2}' || echo -n '0')
        local  x_rss=$(echo "$STATUS" | grep VmRSS | awk '{print $2}' || echo -n '0')
        local x_peak=$(echo "$STATUS" | egrep 'VmPeak|VmHWM' | sed 's/^.*:\s*//' | sed 's/[\sa-zA-Z]*$//' | tr '\n' ' ' || echo -n '0 0')
        local x_pmem=$(awk -v rss=$x_rss -v mem_tot=$mem_tot 'BEGIN {printf "%.0f", rss/mem_tot*100*10}' || echo -n '0')
        local vol_ctxt=$(echo "$STATUS" | grep '\bvoluntary_ctxt_switches' | awk '{print $2}' || echo -n '0')
        local inv_ctxt=$(echo "$STATUS" | grep '\bnonvoluntary_ctxt_switches' | awk '{print $2}' || echo -n '0')
        cpu_stat[x_pid]="$x_pid $x_pmem $x_vsz $x_rss $x_peak $vol_ctxt $inv_ctxt"
        fi
    }

    pwalk() {
        pstat $1
        for i in ${ALL_CHILDREN[$1]:=}; do pwalk $i; done
    }

    pwalk $1
}

nxf_stat() {
    cpu_stat=()
    nxf_tree $1

    declare -a sum=(0 0 0 0 0 0 0 0)
    local pid
    local i
    for pid in "${!cpu_stat[@]}"; do
        local row=(${cpu_stat[pid]})
        [ $NXF_DEBUG = 1 ] && echo "++ stat mem=${row[*]}"
        for i in "${!row[@]}"; do
        if [ $i != 0 ]; then
            sum[i]=$((sum[i]+row[i]))
        fi
        done
    done

    [ $NXF_DEBUG = 1 ] && echo -e "++ stat SUM=${sum[*]}"

    for i in {1..7}; do
        if [ ${sum[i]} -lt ${cpu_peak[i]} ]; then
            sum[i]=${cpu_peak[i]}
        else
            cpu_peak[i]=${sum[i]}
        fi
    done

    [ $NXF_DEBUG = 1 ] && echo -e "++ stat PEAK=${sum[*]}\n"
    nxf_stat_ret=(${sum[*]})
}

nxf_mem_watch() {
    set -o pipefail
    local pid=$1
    local trace_file=.command.trace
    local count=0;
    declare -a cpu_stat=(0 0 0 0 0 0 0 0)
    declare -a cpu_peak=(0 0 0 0 0 0 0 0)
    local mem_tot=$(< /proc/meminfo grep MemTotal | awk '{print $2}')
    local timeout
    local DONE
    local STOP=''

    [ $NXF_DEBUG = 1 ] && nxf_sleep 0.2 && ps fx

    while true; do
        nxf_stat $pid
        if [ $count -lt 10 ]; then timeout=1;
        elif [ $count -lt 120 ]; then timeout=5;
        else timeout=30;
        fi
        read -t $timeout -r DONE || true
        [[ $DONE ]] && break
        if [ ! -e /proc/$pid ]; then
            [ ! $STOP ] && STOP=$(nxf_date)
            [ $(($(nxf_date)-STOP)) -gt 10000 ] && break
        fi
        count=$((count+1))
    done

    echo "%mem=${nxf_stat_ret[1]}"      >> $trace_file
    echo "vmem=${nxf_stat_ret[2]}"      >> $trace_file
    echo "rss=${nxf_stat_ret[3]}"       >> $trace_file
    echo "peak_vmem=${nxf_stat_ret[4]}" >> $trace_file
    echo "peak_rss=${nxf_stat_ret[5]}"  >> $trace_file
    echo "vol_ctxt=${nxf_stat_ret[6]}"  >> $trace_file
    echo "inv_ctxt=${nxf_stat_ret[7]}"  >> $trace_file
}

nxf_write_trace() {
    echo "nextflow.trace/v2"           > $trace_file
    echo "realtime=$wall_time"         >> $trace_file
    echo "%cpu=$ucpu"                  >> $trace_file
    echo "rchar=${io_stat1[0]}"        >> $trace_file
    echo "wchar=${io_stat1[1]}"        >> $trace_file
    echo "syscr=${io_stat1[2]}"        >> $trace_file
    echo "syscw=${io_stat1[3]}"        >> $trace_file
    echo "read_bytes=${io_stat1[4]}"   >> $trace_file
    echo "write_bytes=${io_stat1[5]}"  >> $trace_file
}

nxf_trace_mac() {
    local start_millis=$(nxf_date)

    /bin/bash -ue /home/myname/software/nextNEOpi/work/58/f70e2dd86ece6d1f39f4316e183a45/.command.sh

    local end_millis=$(nxf_date)
    local wall_time=$((end_millis-start_millis))
    local ucpu=''
    local io_stat1=('' '' '' '' '' '')
    nxf_write_trace
}

nxf_fd() {
    local FD=11
    while [ -e /proc/$$/fd/$FD ]; do FD=$((FD+1)); done
    echo $FD
}

nxf_trace_linux() {
    local pid=$$
    command -v ps &>/dev/null || { >&2 echo "Command 'ps' required by nextflow to collect task metrics cannot be found"; exit 1; }
    local num_cpus=$(< /proc/cpuinfo grep '^processor' -c)
    local tot_time0=$(grep '^cpu ' /proc/stat | awk '{sum=$2+$3+$4+$5+$6+$7+$8+$9; printf "%.0f",sum}')
    local cpu_time0=$(2> /dev/null < /proc/$pid/stat awk '{printf "%.0f", ($16+$17)*10 }' || echo -n 'X')
    local io_stat0=($(2> /dev/null < /proc/$pid/io sed 's/^.*:\s*//' | head -n 6 | tr '\n' ' ' || echo -n '0 0 0 0 0 0'))
    local start_millis=$(nxf_date)
    trap 'kill $mem_proc' ERR

    /bin/bash -ue /home/myname/software/nextNEOpi/work/58/f70e2dd86ece6d1f39f4316e183a45/.command.sh &
    local task=$!

    mem_fd=$(nxf_fd)
    eval "exec $mem_fd> >(nxf_mem_watch $task)"
    local mem_proc=$!

    wait $task

    local end_millis=$(nxf_date)
    local tot_time1=$(grep '^cpu ' /proc/stat | awk '{sum=$2+$3+$4+$5+$6+$7+$8+$9; printf "%.0f",sum}')
    local cpu_time1=$(2> /dev/null < /proc/$pid/stat awk '{printf "%.0f", ($16+$17)*10 }' || echo -n 'X')
    local ucpu=$(awk -v p1=$cpu_time1 -v p0=$cpu_time0 -v t1=$tot_time1 -v t0=$tot_time0 -v n=$num_cpus 'BEGIN { pct=(p1-p0)/(t1-t0)*100*n; printf("%.0f", pct>0 ? pct : 0) }' )

    local io_stat1=($(2> /dev/null < /proc/$pid/io sed 's/^.*:\s*//' | head -n 6 | tr '\n' ' ' || echo -n '0 0 0 0 0 0'))
    local i
    for i in {0..5}; do
        io_stat1[i]=$((io_stat1[i]-io_stat0[i]))
    done

    local wall_time=$((end_millis-start_millis))
    [ $NXF_DEBUG = 1 ] && echo "+++ STATS %CPU=$ucpu TIME=$wall_time I/O=${io_stat1[*]}"

    echo "nextflow.trace/v2"           > $trace_file
    echo "realtime=$wall_time"         >> $trace_file
    echo "%cpu=$ucpu"                  >> $trace_file
    echo "rchar=${io_stat1[0]}"        >> $trace_file
    echo "wchar=${io_stat1[1]}"        >> $trace_file
    echo "syscr=${io_stat1[2]}"        >> $trace_file
    echo "syscw=${io_stat1[3]}"        >> $trace_file
    echo "read_bytes=${io_stat1[4]}"   >> $trace_file
    echo "write_bytes=${io_stat1[5]}"  >> $trace_file

    [ -e /proc/$mem_proc ] && eval "echo 'DONE' >&$mem_fd" || true
    wait $mem_proc 2>/dev/null || true
    while [ -e /proc/$mem_proc ]; do nxf_sleep 0.1; done
}

nxf_trace() {
    local trace_file=.command.trace
    touch $trace_file
    if [[ $(uname) = Darwin ]]; then
        nxf_trace_mac
    else
        nxf_trace_linux
    fi
}
nxf_container_env() {
cat << EOF
export PATH="/home/myname/software/nextNEOpi/bin:\$PATH"
EOF
}

nxf_sleep() {
  sleep $1 2>/dev/null || sleep 1;
}

nxf_date() {
    local ts=$(date +%s%3N);
    if [[ ${#ts} == 10 ]]; then echo ${ts}000
    elif [[ $ts == *%3N ]]; then echo ${ts/\%3N/000}
    elif [[ $ts == *3N ]]; then echo ${ts/3N/000}
    elif [[ ${#ts} == 13 ]]; then echo $ts
    else echo "Unexpected timestamp value: $ts"; exit 1
    fi
}

nxf_env() {
    echo '============= task environment ============='
    env | sort | sed "s/\(.*\)AWS\(.*\)=\(.\{6\}\).*/\1AWS\2=\3xxxxxxxxxxxxx/"
    echo '============= task output =================='
}

nxf_kill() {
    declare -a children
    while read P PP;do
        children[$PP]+=" $P"
    done < <(ps -e -o pid= -o ppid=)

    kill_all() {
        [[ $1 != $$ ]] && kill $1 2>/dev/null || true
        for i in ${children[$1]:=}; do kill_all $i; done
    }

    kill_all $1
}

nxf_mktemp() {
    local base=${1:-/tmp}
    if [[ $(uname) = Darwin ]]; then mktemp -d $base/nxf.XXXXXXXXXX
    else TMPDIR="$base" mktemp -d -t nxf.XXXXXXXXXX
    fi
}

nxf_fs_copy() {
  local source=$1
  local target=$2
  local basedir=$(dirname $1)
  mkdir -p $target/$basedir
  cp -fRL $source $target/$basedir
}

nxf_fs_move() {
  local source=$1
  local target=$2
  local basedir=$(dirname $1)
  mkdir -p $target/$basedir
  mv -f $source $target/$basedir
}

nxf_fs_rsync() {
  rsync -rRl $1 $2
}

on_exit() {
    exit_status=${nxf_main_ret:=$?}
    printf $exit_status > /home/myname/software/nextNEOpi/work/58/f70e2dd86ece6d1f39f4316e183a45/.exitcode
    set +u
    [[ "$tee1" ]] && kill $tee1 2>/dev/null
    [[ "$tee2" ]] && kill $tee2 2>/dev/null
    [[ "$ctmp" ]] && rm -rf $ctmp || true
    exit $exit_status
}

on_term() {
    set +e
    [[ "$pid" ]] && kill $pid 2>/dev/null
}

nxf_launch() {
    set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} SINGULARITYENV_NXF_DEBUG=${NXF_DEBUG:=0} singularity exec -B /home/myname/software/nextNEOpi -B "$PWD" --no-home -B /home/myname/software/nextNEOpi/assets -B /tmp -B /home/myname/software/nextNEOpi/resources -B /home/myname/software/nextNEOpi/resources/databases/iedb:/opt/iedb -B /home/myname/software/nextNEOpi/resources/databases/mhcflurry_data:/opt/mhcflurry_data /home/myname/software/nextNEOpi/work/singularity/apps-01.i-med.ac.at-images-singularity-nextNEOpi_1.3.0_6fc098c3.sif /bin/bash -c "cd $PWD; eval $(nxf_container_env); /bin/bash /home/myname/software/nextNEOpi/work/58/f70e2dd86ece6d1f39f4316e183a45/.command.run nxf_trace"
}

nxf_stage() {
    true
    # stage input files
    rm -f sample1_normal_DNA_R1.fastq.gz
    rm -f sample1_normal_DNA_R2.fastq.gz
    rm -f .mixcr_install_ok.chck
    ln -s /home/myname/software/nextNEOpi/work/d3/68b940da3f5350280a2ea9a8c4f541/sample1_normal_DNA_R1.fastq.gz sample1_normal_DNA_R1.fastq.gz
    ln -s /home/myname/software/nextNEOpi/work/d3/68b940da3f5350280a2ea9a8c4f541/sample1_normal_DNA_R2.fastq.gz sample1_normal_DNA_R2.fastq.gz
    ln -s /home/myname/software/nextNEOpi/bin/.mixcr_install_ok.chck .mixcr_install_ok.chck
}

nxf_unstage() {
    true
    [[ ${nxf_main_ret:=0} != 0 ]] && return
}

nxf_main() {
    trap on_exit EXIT
    trap on_term TERM INT USR2
    trap '' USR1

    [[ "${NXF_CHDIR:-}" ]] && cd "$NXF_CHDIR"
    export NXF_BOXID="nxf-$(dd bs=18 count=1 if=/dev/urandom 2>/dev/null | base64 | tr +/ 0A)"
    NXF_SCRATCH=''
    [[ $NXF_DEBUG > 0 ]] && nxf_env
    touch /home/myname/software/nextNEOpi/work/58/f70e2dd86ece6d1f39f4316e183a45/.command.begin
    set +u
    set -u
    [[ $NXF_SCRATCH ]] && echo "nxf-scratch-dir $HOSTNAME:$NXF_SCRATCH" && cd $NXF_SCRATCH
    nxf_stage

    set +e
    ctmp=$(set +u; nxf_mktemp /dev/shm 2>/dev/null || nxf_mktemp $TMPDIR)
    local cout=$ctmp/.command.out; mkfifo $cout
    local cerr=$ctmp/.command.err; mkfifo $cerr
    tee .command.out < $cout &
    tee1=$!
    tee .command.err < $cerr >&2 &
    tee2=$!
    ( nxf_launch ) >$cout 2>$cerr &
    pid=$!
    wait $pid || nxf_main_ret=$?
    wait $tee1 $tee2
    nxf_unstage
}

$NXF_ENTRY
riederd commented 2 years ago

Thanks so much. Can you try to comment (prepend //) line nr 6247 in nextNEOpi.nf

label 'nextNEOpiENV' should get: // label 'nextNEOpiENV'

and resume, with

$ nextflow run nextNEOpi.nf --batchFile example_my_BAM.csv -profile singularity -config conf/params.config -resume
sunsong95 commented 2 years ago

Thanks. It seems to be solved, now it needs a license for mixcr, I have just applied for it. I would like to know how to add the license to the installation process？

Execution cancelled -- Finishing pending tasks before exit
Error executing process > 'mixcr (sample1 : tumor_DNA)'

Caused by:
  Process `mixcr (sample1 : tumor_DNA)` terminated with an error exit status (22)

Command executed:

  mixcr analyze shotgun \
      --threads 16 \
      --species hs \
      --starting-material dna \
      --only-productive \
      sample1_tumor_DNA_R1.fastq.gz sample1_tumor_DNA_R2.fastq.gz \
      sample1_tumor_DNA_mixcr

Command exit status:
  22

Command output:
  (empty)

Command error:
  === No License ===

  To use MiXCR, please, provide a valid license.

  If you already have a license, activate it by calling:
    mixcr activate-license

  You can also activate the license via a special file, environment
  variable or other means, please check the docs at
    https://github.com/milaboratory/mixcr/wiki/Using-license

  Academic users can quickly get a license at
    https://licensing.milaboratories.com.

  Commercial trial license may be requested at
    https://licensing.milaboratories.com
  or by email to
    licensing@milaboratories.com.

Work dir:
  /home/myname/software/nextNEOpi/work/77/29bf8718ae5734d15917361273c38d

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`
riederd commented 2 years ago

MiXCR v3.0.13, which should be used with nextNEOpi 1.3.x, should not need the license file, and is free for academic/non profit use, however you should check if you fall in to this user group.

I'll implement the mixcr 4.0 licensing mode in the next release of nextNEOpi

sunsong95 commented 2 years ago

I downloaded nextNEOpi 1.3.1 through git, where can I modify the installation script to download the old version of MiXCR? Thank you！

riederd commented 2 years ago

The the mixcr url is given in conf/params.conf, see:

https://github.com/icbi-lab/nextNEOpi/blob/4f9f4e595f58533996e43d5b7cf13547d62f6d55/conf/params.config#L113

This should be fine. However I just realized, that the mixcr team changed the old version v3.0.13 from 2020 on June 16 2022, so that it requires a license now. I was not aware of this.

See the two md5 sums of the jars:

old v30.13: 60c92b2b6c72fef57b063e2b45425683 new v3.0.13: 64047c6a90dc7b719d119f946b480e2c

But I think it should be sufficient if you copy your license file to the bin directory of nextNEOpi, in your case:

$ cp mi.license /home/myname/software/nextNEOpi/bin

Let me know if this works.

sunsong95 commented 2 years ago

Great, I followed the method you said and now this error is gone,

[86/44dc62] process > mixcr (sample1 : tumor_DNA) [100%] 2 of 2, cached: 2 ✔

but I have a new problem, My network is not good and I can't pull gatk3 through singularity. I modified the default path of gatk /home/myname/miniconda3/bin/gatk3 in params.config. https://github.com/icbi-lab/nextNEOpi/blob/5c28e3de65d4e8ee72c0fcc222eafd1cdb3b362a/conf/params.config#L97-L100

After rerunning the command, I found that it is still pulling from https://depot.galaxyproject.org/singularity/gatk:3.8--hdfd78af_11

And I also installed cnvkit via conda, but I didn't find cnvit settings and paths in params.config

riederd commented 2 years ago

This would only work if you do not use singularity or conda at all, which means that you would need to manually install all dependencies and make the tools available in the in your program $PATH.

Before doing this I'd suggest that you try to set the timeout for pulling the singularity images to a higher value, e.g. 60 min. You can do this in conf/profiles.config near line 75:

   singularity {
          pullTimeout = "60 min"

Another possibility would be that you manually download the images to a singularity cache directory and specify this cache directory either in the conf/profiles.config (a) or as environment variable (b)

a) conf/profiles.config near line 75

   singularity {
          cacheDir = "/path/to/your/shared/singulariy_cache_dir"

b) environment variable

export NXF_SINGULARITY_CACHEDIR=/path/to/your/shared/singulariy_cache_dir
echo "export NXF_SINGULARITY_CACHEDIR=/path/to/your/shared/singulariy_cache_dir" >> ~/.bashrc

The downloaded images need to have the following naming scheme: server[-directory1-directory2-[...]-image e.g. depot.galaxyproject.org-singularity-gatk-3.8--hdfd78af_11.img apps-01.i-med.ac.at-images-singularity-nextNEOpi_1.3.0_6fc098c3.sif

sunsong95 commented 2 years ago

Sorry to bother you again, the following error occurred during the operation, I don't know what caused it.

Error executing process > 'make_uBAM (sample2 : normal_DNA)'

Caused by:
  Process `make_uBAM (sample2 : normal_DNA)` terminated with an error exit status (247)

Command executed:

  mkdir -p /tmp/myname/nextNEOpi
  gatk --java-options "-Xmx64G -XX:ParallelGCThreads=1" FastqToSam \
      --TMP_DIR /tmp/myname/nextNEOpi \
      --MAX_RECORDS_IN_RAM 4194304 \
      -F1 normal_1.fastq.gz -F2 normal_2.fastq.gz \
      --READ_GROUP_NAME sample2_normal \
      --SAMPLE_NAME sample2_normal \
      --LIBRARY_NAME sample2_normal \
      --PLATFORM ILLUMINA \
      -O sample2_normal_DNA_unaligned.bam

Command exit status:
  247

Command output:
  (empty)

Command error:
  INFO  2022-07-06 07:45:46     FastqToSam      Processed   147,000,000 records.  Elapsed time: 00:41:40s.  Time for last 1,000,000:   30s.  Last read position: */*
  INFO  2022-07-06 07:45:52     FastqToSam      Processed   148,000,000 records.  Elapsed time: 00:41:46s.  Time for last 1,000,000:    5s.  Last read position: */*
  INFO  2022-07-06 07:45:56     FastqToSam      Processed   149,000,000 records.  Elapsed time: 00:41:50s.  Time for last 1,000,000:    4s.  Last read position: */*
  INFO  2022-07-06 07:46:02     FastqToSam      Processed   150,000,000 records.  Elapsed time: 00:41:56s.  Time for last 1,000,000:    5s.  Last read position: */*
  INFO  2022-07-06 07:46:29     FastqToSam      Processed   151,000,000 records.  Elapsed time: 00:42:23s.  Time for last 1,000,000:   26s.  Last read position: */*
  INFO  2022-07-06 07:46:35     FastqToSam      Processed   152,000,000 records.  Elapsed time: 00:42:28s.  Time for last 1,000,000:    5s.  Last read position: */*
  INFO  2022-07-06 07:46:39     FastqToSam      Processed   153,000,000 records.  Elapsed time: 00:42:33s.  Time for last 1,000,000:    4s.  Last read position: */*
  INFO  2022-07-06 07:46:43     FastqToSam      Processed   154,000,000 records.  Elapsed time: 00:42:37s.  Time for last 1,000,000:    4s.  Last read position: */*
  INFO  2022-07-06 07:46:51     FastqToSam      Processed   155,000,000 records.  Elapsed time: 00:42:44s.  Time for last 1,000,000:    7s.  Last read position: */*
  INFO  2022-07-06 07:47:22     FastqToSam      Processed   156,000,000 records.  Elapsed time: 00:43:16s.  Time for last 1,000,000:   31s.  Last read position: */*
  INFO  2022-07-06 07:47:26     FastqToSam      Processed   157,000,000 records.  Elapsed time: 00:43:19s.  Time for last 1,000,000:    3s.  Last read position: */*
  INFO  2022-07-06 07:47:30     FastqToSam      Processed   158,000,000 records.  Elapsed time: 00:43:24s.  Time for last 1,000,000:    4s.  Last read position: */*
  INFO  2022-07-06 07:47:36     FastqToSam      Processed   159,000,000 records.  Elapsed time: 00:43:29s.  Time for last 1,000,000:    5s.  Last read position: */*
  INFO  2022-07-06 07:48:00     FastqToSam      Processed   160,000,000 records.  Elapsed time: 00:43:54s.  Time for last 1,000,000:   24s.  Last read position: */*
  INFO  2022-07-06 07:48:04     FastqToSam      Processed   161,000,000 records.  Elapsed time: 00:43:58s.  Time for last 1,000,000:    3s.  Last read position: */*
  INFO  2022-07-06 07:48:08     FastqToSam      Processed   162,000,000 records.  Elapsed time: 00:44:02s.  Time for last 1,000,000:    3s.  Last read position: */*
  INFO  2022-07-06 07:48:14     FastqToSam      Processed   163,000,000 records.  Elapsed time: 00:44:07s.  Time for last 1,000,000:    5s.  Last read position: */*
  INFO  2022-07-06 07:48:37     FastqToSam      Processed   164,000,000 records.  Elapsed time: 00:44:31s.  Time for last 1,000,000:   23s.  Last read position: */*
  INFO  2022-07-06 07:48:41     FastqToSam      Processed   165,000,000 records.  Elapsed time: 00:44:34s.  Time for last 1,000,000:    3s.  Last read position: */*
  INFO  2022-07-06 07:48:45     FastqToSam      Processed   166,000,000 records.  Elapsed time: 00:44:39s.  Time for last 1,000,000:    4s.  Last read position: */*
  INFO  2022-07-06 07:48:57     FastqToSam      Processed   167,000,000 records.  Elapsed time: 00:44:51s.  Time for last 1,000,000:   12s.  Last read position: */*
  INFO  2022-07-06 07:49:09     FastqToSam      Processed   168,000,000 records.  Elapsed time: 00:45:03s.  Time for last 1,000,000:   12s.  Last read position: */*
  INFO  2022-07-06 07:49:15     FastqToSam      Processed   169,000,000 records.  Elapsed time: 00:45:09s.  Time for last 1,000,000:    5s.  Last read position: */*
  INFO  2022-07-06 07:49:20     FastqToSam      Processed   170,000,000 records.  Elapsed time: 00:45:14s.  Time for last 1,000,000:    4s.  Last read position: */*
  INFO  2022-07-06 07:49:27     FastqToSam      Processed   171,000,000 records.  Elapsed time: 00:45:20s.  Time for last 1,000,000:    6s.  Last read position: */*
  INFO  2022-07-06 07:49:51     FastqToSam      Processed   172,000,000 records.  Elapsed time: 00:45:44s.  Time for last 1,000,000:   24s.  Last read position: */*
  INFO  2022-07-06 07:49:56     FastqToSam      Processed   173,000,000 records.  Elapsed time: 00:45:50s.  Time for last 1,000,000:    5s.  Last read position: */*
  INFO  2022-07-06 07:50:01     FastqToSam      Processed   174,000,000 records.  Elapsed time: 00:45:54s.  Time for last 1,000,000:    4s.  Last read position: */*
  INFO  2022-07-06 07:50:07     FastqToSam      Processed   175,000,000 records.  Elapsed time: 00:46:00s.  Time for last 1,000,000:    5s.  Last read position: */*
  INFO  2022-07-06 07:50:11     FastqToSam      Processed   176,000,000 records.  Elapsed time: 00:46:05s.  Time for last 1,000,000:    4s.  Last read position: */*
  INFO  2022-07-06 07:50:36     FastqToSam      Processed   177,000,000 records.  Elapsed time: 00:46:30s.  Time for last 1,000,000:   24s.  Last read position: */*
  INFO  2022-07-06 07:50:40     FastqToSam      Processed   178,000,000 records.  Elapsed time: 00:46:34s.  Time for last 1,000,000:    3s.  Last read position: */*
  INFO  2022-07-06 07:50:47     FastqToSam      Processed   179,000,000 records.  Elapsed time: 00:46:41s.  Time for last 1,000,000:    7s.  Last read position: */*
  INFO  2022-07-06 07:50:52     FastqToSam      Processed   180,000,000 records.  Elapsed time: 00:46:46s.  Time for last 1,000,000:    4s.  Last read position: */*
  INFO  2022-07-06 07:51:15     FastqToSam      Processed   181,000,000 records.  Elapsed time: 00:47:09s.  Time for last 1,000,000:   23s.  Last read position: */*
  INFO  2022-07-06 07:51:19     FastqToSam      Processed   182,000,000 records.  Elapsed time: 00:47:13s.  Time for last 1,000,000:    4s.  Last read position: */*
  INFO  2022-07-06 07:51:27     FastqToSam      Processed   183,000,000 records.  Elapsed time: 00:47:21s.  Time for last 1,000,000:    7s.  Last read position: */*
  INFO  2022-07-06 07:51:31     FastqToSam      Processed   184,000,000 records.  Elapsed time: 00:47:25s.  Time for last 1,000,000:    3s.  Last read position: */*
  INFO  2022-07-06 07:51:56     FastqToSam      Processed   185,000,000 records.  Elapsed time: 00:47:50s.  Time for last 1,000,000:   24s.  Last read position: */*
  INFO  2022-07-06 07:52:00     FastqToSam      Processed   186,000,000 records.  Elapsed time: 00:47:54s.  Time for last 1,000,000:    4s.  Last read position: */*
  INFO  2022-07-06 07:52:03     FastqToSam      Processed   187,000,000 records.  Elapsed time: 00:47:57s.  Time for last 1,000,000:    2s.  Last read position: */*
  INFO  2022-07-06 07:52:11     FastqToSam      Processed   188,000,000 records.  Elapsed time: 00:48:05s.  Time for last 1,000,000:    8s.  Last read position: */*
  INFO  2022-07-06 07:52:34     FastqToSam      Processed   189,000,000 records.  Elapsed time: 00:48:28s.  Time for last 1,000,000:   22s.  Last read position: */*
  INFO  2022-07-06 07:52:40     FastqToSam      Processed   190,000,000 records.  Elapsed time: 00:48:34s.  Time for last 1,000,000:    6s.  Last read position: */*
  INFO  2022-07-06 07:52:45     FastqToSam      Processed   191,000,000 records.  Elapsed time: 00:48:39s.  Time for last 1,000,000:    5s.  Last read position: */*
  INFO  2022-07-06 07:52:52     FastqToSam      Processed   192,000,000 records.  Elapsed time: 00:48:46s.  Time for last 1,000,000:    6s.  Last read position: */*
  Using GATK jar /opt/conda/share/gatk4-4.2.5.0-0/gatk-package-4.2.5.0-local.jar
  Running:
      java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -Xmx64G -XX:ParallelGCThreads=1 -jar /opt/conda/share/gatk4-4.2.5.0-0/gatk-package-4.2.5.0-local.jar FastqToSam --TMP_DIR /tmp/myname/nextNEOpi --MAX_RECORDS_IN_RAM 4194304 -F1 normal_1.fastq.gz -F2 normal_2.fastq.gz --READ_GROUP_NAME sample2_normal --SAMPLE_NAME sample2_normal --LIBRARY_NAME sample2_normal --PLATFORM ILLUMINA -O sample2_normal_DNA_unaligned.bam
  INFO:    Cleaning up image...

Work dir:
  /home/myname/software/nextNEOpi/work/b0/1c43c425c06a7f87a1aef5927195bb

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`
riederd commented 2 years ago

Hi,

Command exit status:
  247

not sure but, might be related to a memory issue. How much memory does your system have?

sunsong95 commented 2 years ago

my system have 125G memory

riederd commented 2 years ago

Hmm, I'd expect that is is enough. Can you try to run the failed process manually, to see if it is temporary, by doing:

$ cd  /home/myname/software/nextNEOpi/work/b0/1c43c425c06a7f87a1aef5927195bb
$ bash .command.run

if it fails again, try to edit the file .command.sh in that directory and change the line with

    gatk --java-options "-Xmx64G -XX:ParallelGCThreads=1" FastqToSam \

to

    gatk --java-options "-Xmx32G -XX:ParallelGCThreads=1" FastqToSam \

then run again:

$ bash .command.run

If that works, you should change the JAVA_Xmx = "-Xmx64G" to JAVA_Xmx = "-Xmx32G" in conf/params.config or use --JAVA_Xmx "-Xmx32G" -resume when rerunning the pipeline

riederd commented 2 years ago

No feedback since a while. Assuming that the issue is resolved. Closing it for now, feel free to reopen.