search

ablab / spades

SPAdes Genome Assembler
http://ablab.github.io/spades/
Other
713 stars 131 forks source link

spades-4.0.0 :: some test failure #1312

Closed EricDeveaud closed 1 month ago

EricDeveaud commented 1 month ago

Description of bug

while running the test suite from spades_test/run_spades.sh

some tests fail with different errors. here are the output of both failure (commented fail one and run iteratively)

NB spades.py --test ran smoothly

context: SPAdes-4.0.0 built from source (https://github.com/ablab/spades/archive/refs/tags/v4.0.0.tar.gz) using gcc/9.2.0 boost/1.81.0 cmake/3.27.7 with -DSPADES_ENABLE_PROJECTS="all" -DSPADES_USE_NCBISDK=ON

sh -x run_spades.sh 2>&1 | tee log
+ set -e
+ true
+ /opt/gensoft/exe/SPAdes/4.0.0/bin/spades-hammer /opt/gensoft/src/SPAdes/spades-4.0.0/spades_test/corrected/configs/config.info
  0:00:00.000     1M / 7M    INFO    General                 (main.cpp                  :  76)   Starting BayesHammer, built from N/A, git revision N/A
  0:00:00.020     1M / 7M    INFO    General                 (main.cpp                  :  77)   Loading config from "/opt/gensoft/src/SPAdes/spades-4.0.0/spades_test/corrected/configs/config.info"
  0:00:00.027     1M / 8M    INFO    General                 (main.cpp                  :  79)   Maximum # of threads to use (adjusted due to OMP capabilities): 16
  0:00:00.033     1M / 8M    INFO    General                 (memory_limit.cpp          :  55)   Memory limit set to 250 Gb
  0:00:00.033     1M / 8M    INFO    General                 (main.cpp                  :  87)   Trying to determine PHRED offset
  0:00:00.034     1M / 10M   INFO    General                 (main.cpp                  :  93)   Determined value is 33
  0:00:00.035     1M / 10M   INFO    General                 (hammer_tools.cpp          :  40)   Hamming graph threshold tau=1, k=21, subkmer positions = [ 0 10 ]
  0:00:00.035     1M / 10M   INFO    General                 (main.cpp                  : 114)   Size of aux. kmer data 24 bytes

     === ITERATION 0 begins ===
Exception caught Cannot create temporary dir /opt/gensoft/src/SPAdes/spades-4.0.0/spades_test/tmp/hammer_yamyaegj/kmer_splitter_XXXXXX
+ /opt/gensoft/exe/SPAdes/4.0.0/bin/spades-core /opt/gensoft/src/SPAdes/spades-4.0.0/spades_test/K21/configs/config.info
  0:00:00.000     1M / 10M   INFO    General                 (main.cpp                  :  94)   Loaded config from "/opt/gensoft/src/SPAdes/spades-4.0.0/spades_test/K21/configs/config.info"
  0:00:00.001     1M / 10M   INFO    General                 (memory_limit.cpp          :  55)   Memory limit set to 250 Gb
  0:00:00.001     1M / 10M   INFO    General                 (main.cpp                  : 102)   Starting SPAdes, built from N/A, git revision N/A
  0:00:00.001     1M / 10M   INFO    General                 (main.cpp                  : 103)   Maximum k-mer length: 128
  0:00:00.001     1M / 10M   INFO    General                 (main.cpp                  : 104)   Assembling dataset ("/opt/gensoft/src/SPAdes/spades-4.0.0/spades_test/dataset.info") with K=21
  0:00:00.001     1M / 10M   INFO    General                 (main.cpp                  : 105)   Maximum # of threads to use (adjusted due to OMP capabilities): 16
  0:00:00.001     1M / 10M   INFO    General                 (pipeline.cpp              : 212)   SPAdes started
  0:00:00.001     1M / 10M   INFO    General                 (pipeline.cpp              : 225)   Starting from stage: read_conversion
  0:00:00.001     1M / 10M   INFO    General                 (pipeline.cpp              : 234)   Two-step repeat resolution disabled
  0:00:00.001     1M / 10M   INFO   GraphCore                (graph_core.hpp            : 689)   Graph created, vertex min_id: 3, edge min_id: 3
  0:00:00.001     1M / 10M   INFO   GraphCore                (graph_core.hpp            : 690)   Vertex size: 48, edge size: 40
  0:00:00.001     1M / 10M   INFO    General                 (edge_index.hpp            : 132)   Size of edge index entries: 12/8
  0:00:00.001     1M / 10M   INFO   StageManager             (stage.cpp                 : 189)   STAGE == Binary Read Conversion (id: read_conversion)
  0:00:00.002     1M / 11M   INFO    General                 (read_converter.cpp        :  78)   Converting reads to binary format for library #0 (takes a while)
  0:00:00.002     1M / 11M   INFO    General                 (read_converter.cpp        :  99)   Converting paired reads
  0:00:00.007    81M / 81M   ERROR   General                 (paired_readers.cpp        :  42)   The number of left read-pairs is larger than the number of right read-pairs
  0:00:00.007    81M / 81M   ERROR   General                 (paired_readers.cpp        :  44)   Unequal number of read-pairs detected in the following files: "/opt/gensoft/src/SPAdes/spades-4.0.0/spades_test/corrected/ecoli_1K_1.fq00.0_0.cor.fastq.gz.gz"  "/opt/gensoft/src/SPAdes/spades-4.0.0/spades_test/corrected/ecoli_1K_2.fq00.0_0.cor.fastq.gz.gz"
=== Stack Trace ===
/opt/gensoft/exe/SPAdes/4.0.0/bin/spades-core() [0x903eac]
/opt/gensoft/exe/SPAdes/4.0.0/bin/spades-core() [0x9037f2]
/opt/gensoft/exe/SPAdes/4.0.0/bin/spades-core() [0x910efc]
/opt/gensoft/exe/SPAdes/4.0.0/bin/spades-core() [0x908c94]
/opt/gensoft/exe/SPAdes/4.0.0/bin/spades-core() [0x8f35cc]
/opt/gensoft/exe/SPAdes/4.0.0/bin/spades-core() [0x8f4284]
/opt/gensoft/exe/SPAdes/4.0.0/bin/spades-core() [0x6a7a9d]
/opt/gensoft/exe/SPAdes/4.0.0/bin/spades-core() [0x7a5c1e]
/opt/gensoft/exe/SPAdes/4.0.0/bin/spades-core() [0x4522b2]
/opt/gensoft/exe/SPAdes/4.0.0/bin/spades-core() [0x44b614]
/lib64/libc.so.6(__libc_start_main+0xe5) [0x7f0228e5ed85]
/opt/gensoft/exe/SPAdes/4.0.0/bin/spades-core() [0x4501ae]
+ /opt/gensoft/exe/SPAdes/4.0.0/bin/spades-core /opt/gensoft/src/SPAdes/spades-4.0.0/spades_test/K33/configs/config.info
  0:00:00.000     1M / 10M   INFO    General                 (main.cpp                  :  94)   Loaded config from "/opt/gensoft/src/SPAdes/spades-4.0.0/spades_test/K33/configs/config.info"
  0:00:00.001     1M / 10M   INFO    General                 (memory_limit.cpp          :  55)   Memory limit set to 250 Gb
  0:00:00.001     1M / 10M   INFO    General                 (main.cpp                  : 102)   Starting SPAdes, built from N/A, git revision N/A
  0:00:00.001     1M / 10M   INFO    General                 (main.cpp                  : 103)   Maximum k-mer length: 128
  0:00:00.001     1M / 10M   INFO    General                 (main.cpp                  : 104)   Assembling dataset ("/opt/gensoft/src/SPAdes/spades-4.0.0/spades_test/dataset.info") with K=33
  0:00:00.001     1M / 10M   INFO    General                 (main.cpp                  : 105)   Maximum # of threads to use (adjusted due to OMP capabilities): 16
  0:00:00.001     1M / 10M   INFO    General                 (pipeline.cpp              : 212)   SPAdes started
  0:00:00.001     1M / 10M   INFO    General                 (pipeline.cpp              : 225)   Starting from stage: read_conversion
  0:00:00.001     1M / 10M   INFO    General                 (pipeline.cpp              : 234)   Two-step repeat resolution disabled
  0:00:00.001     1M / 10M   INFO   GraphCore                (graph_core.hpp            : 689)   Graph created, vertex min_id: 3, edge min_id: 3
  0:00:00.001     1M / 10M   INFO   GraphCore                (graph_core.hpp            : 690)   Vertex size: 48, edge size: 40
  0:00:00.001     1M / 10M   INFO    General                 (edge_index.hpp            : 132)   Size of edge index entries: 12/8
  0:00:00.001     1M / 10M   INFO   StageManager             (stage.cpp                 : 189)   STAGE == Binary Read Conversion (id: read_conversion)
  0:00:00.002     1M / 10M   INFO    General                 (read_converter.cpp        :  78)   Converting reads to binary format for library #0 (takes a while)
  0:00:00.002     1M / 10M   INFO    General                 (read_converter.cpp        :  99)   Converting paired reads
  0:00:00.007    81M / 81M   ERROR   General                 (paired_readers.cpp        :  42)   The number of left read-pairs is larger than the number of right read-pairs
  0:00:00.007    81M / 81M   ERROR   General                 (paired_readers.cpp        :  44)   Unequal number of read-pairs detected in the following files: "/opt/gensoft/src/SPAdes/spades-4.0.0/spades_test/corrected/ecoli_1K_1.fq00.0_0.cor.fastq.gz.gz"  "/opt/gensoft/src/SPAdes/spades-4.0.0/spades_test/corrected/ecoli_1K_2.fq00.0_0.cor.fastq.gz.gz"
=== Stack Trace ===
/opt/gensoft/exe/SPAdes/4.0.0/bin/spades-core() [0x903eac]
/opt/gensoft/exe/SPAdes/4.0.0/bin/spades-core() [0x9037f2]
/opt/gensoft/exe/SPAdes/4.0.0/bin/spades-core() [0x910efc]
/opt/gensoft/exe/SPAdes/4.0.0/bin/spades-core() [0x908c94]
/opt/gensoft/exe/SPAdes/4.0.0/bin/spades-core() [0x8f35cc]
/opt/gensoft/exe/SPAdes/4.0.0/bin/spades-core() [0x8f4284]
/opt/gensoft/exe/SPAdes/4.0.0/bin/spades-core() [0x6a7a9d]
/opt/gensoft/exe/SPAdes/4.0.0/bin/spades-core() [0x7a5c1e]
/opt/gensoft/exe/SPAdes/4.0.0/bin/spades-core() [0x4522b2]
/opt/gensoft/exe/SPAdes/4.0.0/bin/spades-core() [0x44b614]
/lib64/libc.so.6(__libc_start_main+0xe5) [0x7f2bf00cad85]
/opt/gensoft/exe/SPAdes/4.0.0/bin/spades-core() [0x4501ae]

same kind of error with K55 test data set

best regards

Eric

spades.log

non relevant, no log generated

params.txt

non relevant

SPAdes version

spades-4.0.0

Operating System

cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="8.8 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.8 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.8
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.8"

Python Version

python-3.8.1

Method of SPAdes installation

manual

No errors reported in spades.log

asl commented 1 month ago

The log reads:

Exception caught Cannot create temporary dir /opt/gensoft/src/SPAdes/spades-4.0.0/spades_test/tmp/hammer_yamyaegj/kmer_splitter_XXXXXX

plus:

  0:00:00.007    81M / 81M   ERROR   General                 (paired_readers.cpp        :  42)   The number of left read-pairs is larger than the number of right read-pairs
  0:00:00.007    81M / 81M   ERROR   General                 (paired_readers.cpp        :  44)   Unequal number of read-pairs detected in the following files: "/opt/gensoft/src/SPAdes/spades-4.0.0/spades_test/corrected/ecoli_1K_1.fq00.0_0.cor.fastq.gz.gz"  "/opt/gensoft/src/SPAdes/spades-4.0.0/spades_test/corrected/ecoli_1K_2.fq00.0_0.cor.fastq.gz.gz"

This smells like an I/O issue. I'd suggest you to check with your system administrator

PS: I have no idea what run_spades.sh is, it is not a part of SPAdes. And spades.log is always generated

EricDeveaud commented 1 month ago

thanks for the comment.

spades_test/run_spades.sh is generated from spades --test

IO error of course. but 

spades [rpm]:SPAdes/spades-4.0.0 > cd spades_test/
spades [rpm]:spades-4.0.0/spades_test > ls -ld tmp/
drwxrwsr-x 2 gensoft gensoft 10 Jun  4 17:31 tmp/
spades [rpm]:spades-4.0.0/spades_test > id
uid=207(gensoft) gid=207(gensoft) groups=207(gensoft),10(wheel)

I can provide spades.log generated fro spades --test of course

but spades-hammer nor spades-core does not ;-)

EricDeveaud commented 1 month ago

OK got it.....

run_spades.sh is generated by spades --test hardcode the temp directory in config.info

and this one no longer exists.

I was looking for a command line to test spades-hammer. as it does provide help menu.

asl commented 1 month ago

I was looking for a command line to test spades-hammer. as it does provide help menu.

All binaries are internal unless explicitly specified. So, you'd always run spades.py.

EricDeveaud commented 1 month ago

in this case, maybee they should land to PREFIX/libexec instead of PREFIX/bin

as packager for our cluster, I tend to always test individual programs installed.