search

ablab / spades

SPAdes Genome Assembler
http://ablab.github.io/spades/
Other
737 stars 134 forks source link

rnaSPAdes has paused without error. Is there a way to restart from where it left off? #252

Closed jolespin closed 5 years ago

jolespin commented 5 years ago

Here's my spades.log file:

Let me know if there is any other information you need and I will update accordingly. 

-bash-4.1$ cat rnaspades_output/frozen/spades.log
Command line: /usr/local/projdata/9043/projects/MGX/localenv/envs/spades/bin/rnaspades.py   -s  /local/ifs2_projdata/0568/projects/PLANKTON/illumina_aallen/jespinoz/Dropseq/rap_unmapped/seq_raw/frozen_unmapped.fastq -o  /local/ifs2_projdata/0568/projects/PLANKTON/illumina_aallen/jespinoz/Dropseq/rap_unmapped/rnaspades_output/frozen   -t  16  -m  300

System information:
  SPAdes version: 3.11.1
  Python version: 3.6.2
  OS: Linux-2.6.32-696.18.7.el6.x86_64-x86_64-with-centos-6.4-Final

Output dir: /local/ifs2_projdata/0568/projects/PLANKTON/illumina_aallen/jespinoz/Dropseq/rap_unmapped/rnaspades_output/frozen
Mode: read error correction and assembling
Debug mode is turned OFF

Dataset parameters:
  RNA-seq mode
  Reads:
    Library number: 1, library type: single
      left reads: not specified
      right reads: not specified
      interlaced reads: not specified
      single reads: ['/local/ifs2_projdata/0568/projects/PLANKTON/illumina_aallen/jespinoz/Dropseq/rap_unmapped/seq_raw/frozen_unmapped.fastq']
Read error correction parameters:
  Iterations: 1
  PHRED offset will be auto-detected
  Corrected reads will be compressed (with gzip)
Assembly parameters:
  k: 29
  Repeat resolution is enabled
  Mismatch careful mode is turned OFF
  MismatchCorrector will be SKIPPED
  Coverage cutoff is turned OFF
Other parameters:
  Dir for temp files: /local/ifs2_projdata/0568/projects/PLANKTON/illumina_aallen/jespinoz/Dropseq/rap_unmapped/rnaspades_output/frozen/tmp
  Threads: 16
  Memory limit (in Gb): 300

======= SPAdes pipeline started. Log can be found here: /local/ifs2_projdata/0568/projects/PLANKTON/illumina_aallen/jespinoz/Dropseq/rap_unmapped/rnaspades_output/frozen/spades.log

===== Read error correction started.

== Running read error correction tool: /local/ifs2_projdata/9043/projects/MGX/packages/SPAdes-3.11.1-Linux/bin/hammer /local/ifs2_projdata/0568/projects/PLANKTON/illumina_aallen/jespinoz/Dropseq/rap_unmapped/rnaspades_output/frozen/corrected/configs/config.info

  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  83)   Starting BayesHammer, built from N/A, git revision N/A
  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  84)   Loading config from /local/ifs2_projdata/0568/projects/PLANKTON/illumina_aallen/jespinoz/Dropseq/rap_unmapped/rnaspades_output/frozen/corrected/configs/config.info
  0:00:00.001     4M / 4M    INFO    General                 (memory_limit.hpp          :  51)   Memory limit set to 300 Gb
  0:00:00.001     4M / 4M    INFO    General                 (main.cpp                  :  93)   Trying to determine PHRED offset
  0:00:00.001     4M / 4M    INFO    General                 (main.cpp                  :  99)   Determined value is 33
  0:00:00.002     4M / 4M    INFO    General                 (hammer_tools.cpp          :  36)   Hamming graph threshold tau=1, k=21, subkmer positions = [ 0 10 ]
  0:00:00.002     4M / 4M    INFO    General                 (main.cpp                  : 120)   Size of aux. kmer data 24 bytes
     === ITERATION 0 begins ===
  0:00:00.004     4M / 4M    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 289)   Building kmer index
  0:00:00.004     4M / 4M    INFO    General                 (kmer_index_builder.hpp    : 106)   Splitting kmer instances into 256 buckets using 16 threads. This might take a while.
  0:00:00.004     4M / 4M    INFO    General                 (file_limit.hpp            :  32)   Open file limit set to 1024
  0:00:00.004     4M / 4M    INFO    General                 (kmer_splitters.hpp        :  74)   Memory available for splitting buffers: 6.24992 Gb
  0:00:00.004     4M / 4M    INFO    General                 (kmer_splitters.hpp        :  82)   Using cell size of 262144
  0:00:01.828     8G / 8G    INFO   K-mer Splitting          (kmer_data.cpp             :  96)   Processing /local/ifs2_projdata/0568/projects/PLANKTON/illumina_aallen/jespinoz/Dropseq/rap_unmapped/seq_raw/frozen_unmapped.fastq
  0:00:18.662     8G / 9G    INFO   K-mer Splitting          (kmer_data.cpp             : 106)   Processed 5679116 reads
  0:00:32.814     8G / 9G    INFO   K-mer Splitting          (kmer_data.cpp             : 106)   Processed 10507267 reads
  0:00:47.340     8G / 9G    INFO   K-mer Splitting          (kmer_data.cpp             : 106)   Processed 15147891 reads
  0:01:01.774     8G / 9G    INFO   K-mer Splitting          (kmer_data.cpp             : 106)   Processed 19913229 reads
  0:01:17.902     8G / 9G    INFO   K-mer Splitting          (kmer_data.cpp             : 106)   Processed 24503993 reads
  0:01:34.996     8G / 9G    INFO   K-mer Splitting          (kmer_data.cpp             : 106)   Processed 30070833 reads
  0:01:49.698     8G / 9G    INFO   K-mer Splitting          (kmer_data.cpp             : 106)   Processed 34951853 reads
  0:02:03.229     8G / 9G    INFO   K-mer Splitting          (kmer_data.cpp             : 106)   Processed 39385326 reads
  0:02:19.590     8G / 9G    INFO   K-mer Splitting          (kmer_data.cpp             : 106)   Processed 44625901 reads
  0:02:35.101     8G / 9G    INFO   K-mer Splitting          (kmer_data.cpp             : 106)   Processed 49421729 reads
  0:02:49.763     8G / 9G    INFO   K-mer Splitting          (kmer_data.cpp             : 106)   Processed 55038420 reads
  0:03:31.558     8G / 9G    INFO   K-mer Splitting          (kmer_data.cpp             : 106)   Processed 69152666 reads
  0:06:48.325     8G / 9G    INFO   K-mer Splitting          (kmer_data.cpp             : 106)   Processed 137336577 reads
  0:13:25.077     8G / 9G    INFO   K-mer Splitting          (kmer_data.cpp             : 106)   Processed 269897514 reads
  0:21:24.934     8G / 9G    INFO   K-mer Splitting          (kmer_data.cpp             : 111)   Total 411654768 reads processed
  0:21:25.952    64M / 9G    INFO    General                 (kmer_index_builder.hpp    : 109)   Starting k-mer counting.
  0:25:05.397    64M / 9G    INFO    General                 (kmer_index_builder.hpp    : 115)   K-mer counting done. There are 6976758448 kmers in total.
  0:25:05.397    64M / 9G    INFO    General                 (kmer_index_builder.hpp    : 121)   Merging temporary buckets.
  0:28:35.152    64M / 9G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 298)   Building perfect hash indices
  0:34:16.097     3G / 9G    INFO    General                 (kmer_index_builder.hpp    : 137)   Merging final buckets.
  0:36:21.989     3G / 9G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 320)   Index built. Total 3235178908 bytes occupied (3.70966 bits per kmer).
  0:36:21.992     3G / 9G    INFO   K-mer Counting           (kmer_data.cpp             : 357)   Arranging kmers in hash map order
  0:50:59.058   107G / 107G  INFO    General                 (main.cpp                  : 155)   Clustering Hamming graph.
  3:26:20.237   107G / 107G  INFO    General                 (main.cpp                  : 162)   Extracting clusters
asl commented 5 years ago

Hello

This step is I/O-bound, so please double check that everything is ok here.

PS: The latest version of SPAdes is 3.13.0 and it contains some updates especially for rnaSPAdes.

jolespin commented 5 years ago

This is good to know. I will request that we update to 3.13.0 on our servers. I ran rnaSPAdes for a similar dataset and it ran perfectly.

This step is I/O-bound, so please double check that everything is ok here.

Is there any file in particular I can check to make sure this is ok?

Thanks for getting back to me so quickly btw.

asl commented 5 years ago

Is there any file in particular I can check to make sure this is ok?

Well, you may want to check in the system logs, you may want to check the process status via "top", etc.

jolespin commented 5 years ago

I'm running on the grid: 

qsub -cwd -P 0749 -pe threaded $N_JOBS -M jespinoz@jcvi.org -m e -N frozen_rnaspades "/usr/local/projdata/9043/projects/MGX/localenv/envs/spades/bin/rnaspades.py -s seq_raw/frozen_unmapped.fastq -o rnaspades_output/frozen/ -t $N_JOBS"

Here's my qstat:

job-ID     prior   name       user         state submit/start at     queue                          jclass                         slots ja-task-ID
------------------------------------------------------------------------------------------------------------------------------------------------
   9434455 0.56506 frozen_rna jespinoz     r     02/28/2019 01:19:49 default.q@dell-2.jcvi.org                                        16

I'm not sure if top captures those processes. Is there a way for me to kill the job via qdel 9434455 and resume using the files already created?

asl commented 5 years ago

Well, then I would suggest you to ask your system administrator, how the things could be troubleshooted on the cluster.

Sadly, there is no way to restart from this stage.

jolespin commented 5 years ago

Oh ok, thanks for the responses. I'll kill the job and try again with the newest version w/ a fresh conda env.

conda create --name spades_env -c bioconda spades b/c it looks like the version is v3.13.0 https://anaconda.org/bioconda/spades

asl commented 5 years ago

There seems to be conda version, yes, though we do not maintain it.