(Hybrid) metaSPAdes finished abnormally, OS return value: 1

Description of bug

Dear developer

I am trying to create hybrid assembly using metaSPAdes module. This is not the first time that we are doing this, however we never experienced any problems. metaSPAdes starts by read correction for Illumina reads . The software never achieved a stage to use Nanopore reads and it crashed before utilizing the Nanopore reads. As a results no assembly file / graphs was created. the content for log file and params is provided.

i appreciate your help

spades.log

Command line: /home/me/Downloads/SPAdes-4.0.0-Linux/bin/metaspades.py -t 70 --nanopore /datastore/me_HiAS/Reads/SQK-RBK114-24_barcode01.fastq.gz --pe1-1 /datastore/me_HiAS/Reads/D1A_EDME220001852-1a_H223VDSX3_L3_1.fq.gz --pe1-2 /datastore/me_HiAS/Reads/D1A_EDME220001852-1a_H223VDSX3_L3_2.fq.gz -o /datastore/me_HiAS/Hybrid_SPAdes

System information: SPAdes version: 4.0.0 Python version: 3.12.4 OS: Linux-3.10.0-1160.119.1.el7.x86_64-x86_64-with-glibc2.17

Output dir: /datastore/me_HiAS/Hybrid_SPAdes Mode: read error correction and assembling Debug mode is turned OFF

Dataset parameters: Metagenomic mode Reads: Library number: 1, library type: nanopore left reads: not specified right reads: not specified interlaced reads: not specified single reads: ['/datastore/me_HiAS/Reads/SQK-RBK114-24_barcode01.fastq.gz'] merged reads: not specified Library number: 2, library type: paired-end orientation: fr left reads: ['/datastore/me_HiAS/Reads/D1A_EDME220001852-1a_H223VDSX3_L3_1.fq.gz'] right reads: ['/datastore/me_HiAS/Reads/D1A_EDME220001852-1a_H223VDSX3_L3_2.fq.gz'] interlaced reads: not specified single reads: not specified merged reads: not specified Read error correction parameters: Iterations: 1 PHRED offset will be auto-detected Corrected reads will be compressed Assembly parameters: k: [21, 33, 55] Repeat resolution is enabled Mismatch careful mode is turned OFF MismatchCorrector will be SKIPPED Coverage cutoff is turned OFF Assembly graph output will use GFA v1.2 format Other parameters: Dir for temp files: /datastore/me_HiAS/Hybrid_SPAdes/tmp Threads: 70 Memory limit (in Gb): 250

======= SPAdes pipeline started. Log can be found here: /datastore/me_HiAS/Hybrid_SPAdes/spades.log

/datastore/me_HiAS/Reads/D1A_EDME220001852-1a_H223VDSX3_L3_1.fq.gz: max reads length: 150 /datastore/me_HiAS/Reads/D1A_EDME220001852-1a_H223VDSX3_L3_2.fq.gz: max reads length: 150

Reads length: 150

===== Before start started.

===== Read error correction started.

== Running: /home/me/Downloads/SPAdes-4.0.0-Linux/bin/spades-hammer /datastore/me_HiAS/Hybrid_SPAdes/corrected/configs/config.info

0:00:00.000 1M / 15M INFO General (main.cpp : 76) Starting BayesHammer, built from N/A, git revision 38af83de44403f833a32073fd47ba469d998efd1 0:00:00.005 1M / 15M INFO General (main.cpp : 77) Loading config from "/datastore/me_HiAS/Hybrid_SPAdes/corrected/configs/config.info" 0:00:00.010 1M / 15M INFO General (main.cpp : 79) Maximum # of threads to use (adjusted due to OMP capabilities): 70 0:00:00.013 1M / 15M INFO General (memory_limit.cpp : 55) Memory limit set to 250 Gb 0:00:00.015 1M / 15M INFO General (main.cpp : 87) Trying to determine PHRED offset 0:00:00.015 1M / 15M INFO General (main.cpp : 93) Determined value is 33 0:00:00.015 1M / 15M INFO General (hammer_tools.cpp : 40) Hamming graph threshold tau=1, k=21, subkmer positions = [ 0 10 ] 0:00:00.016 1M / 15M INFO General (main.cpp : 114) Size of aux. kmer data 24 bytes === ITERATION 0 begins === 0:00:00.016 1M / 15M INFO K-mer Counting (kmer_data.cpp : 284) Estimating k-mer count 0:00:00.623 1121M / 1128M INFO K-mer Counting (kmer_data.cpp : 289) Processing "/datastore/me_HiAS/Reads/D1A_EDME220001852-1a_H223VDSX3_L3_1.fq.gz" 0:01:10.690 1121M / 1135M INFO K-mer Counting (kmer_data.cpp : 298) Processed 19790636 reads 0:01:10.695 1121M / 1135M INFO K-mer Counting (kmer_data.cpp : 289) Processing "/datastore/me_HiAS/Reads/D1A_EDME220001852-1a_H223VDSX3_L3_2.fq.gz" 0:02:18.093 1121M / 1135M INFO K-mer Counting (kmer_data.cpp : 298) Processed 39581272 reads 0:02:18.098 1121M / 1135M INFO K-mer Counting (kmer_data.cpp : 303) Total 39581272 reads processed 0:02:19.778 1121M / 1135M INFO K-mer Counting (kmer_data.cpp : 306) Estimated 4374102497 distinct kmers 0:02:19.780 1M / 1135M INFO K-mer Counting (kmer_data.cpp : 310) Filtering singleton k-mers mimalloc: warning: unable to allocate aligned OS memory directly, fall back to over-allocation (21751660544 bytes, address: 0x7f6c45843000, alignment: 67108864, commit: 1) 0:02:31.246 21G / 21G INFO K-mer Counting (kmer_data.cpp : 316) Processing "/datastore/me_HiAS/Reads/D1A_EDME220001852-1a_H223VDSX3_L3_1.fq.gz" 0:23:36.626 21G / 21G INFO K-mer Counting (kmer_data.cpp : 325) Processed 19790636 reads 0:23:36.626 21G / 21G INFO K-mer Counting (kmer_data.cpp : 316) Processing "/datastore/me_HiAS/Reads/D1A_EDME220001852-1a_H223VDSX3_L3_2.fq.gz" 0:43:22.096 21G / 21G INFO K-mer Counting (kmer_data.cpp : 325) Processed 39581272 reads 0:43:22.097 21G / 21G INFO K-mer Counting (kmer_data.cpp : 330) Total 39581272 reads processed 0:43:22.099 21G / 21G INFO General (kmer_index_builder.hpp : 258) Splitting kmer instances into 16 files using 70 threads. This might take a while. 0:43:22.099 21G / 21G INFO General (file_limit.hpp : 43) Open file limit set to 1024 0:43:22.099 21G / 21G INFO General (kmer_splitter.hpp : 94) Memory available for splitting buffers: 1.09395 Gb 0:43:22.100 21G / 21G INFO General (kmer_splitter.hpp : 102) Using cell size of 4194304 0:43:22.124 60G / 60G INFO K-mer Splitting (kmer_data.cpp : 98) Processing "/datastore/me_HiAS/Reads/D1A_EDME220001852-1a_H223VDSX3_L3_1.fq.gz" mimalloc: warning: thread 0x7f717387d700: mimalloc: warning: thread 0x7f7178887700: mimalloc: warning: thread 0x7f7179088700: unable to allocate aligned OS memory directly, fall back to over-allocation (2042626048 bytes, address: 0x7f6b50800000, alignment: 67108864, commit: 1) mimalloc: warning: thread 0x7f7175881700: mimalloc: warning: thread 0x7f7177084700: mimalloc: warning: thread 0x7f7177885700: mimalloc: warning: thread 0x7f717287b700: mimalloc: warning: thread 0x7f7175080700: mimalloc: warning: thread 0x7f7179889700: mimalloc: warning: thread 0x7f7176082700: mimalloc: warning: thread 0x7f717487f700: mimalloc: warning: thread 0x7f7178086700: mimalloc: warning: thread 0x7f717407e700: unable to allocate aligned OS memory directly, fall back to over-allocation (2042626048 bytes, address: 0x7f6b50800000, alignment: 67108864, commit: 1) unable to allocate aligned OS memory directly, fall back to over-allocation (2042626048 bytes, address: 0x7f6bca400000, alignment: 67108864, commit: 1) mimalloc: warning: thread 0x7f7176883700: unable to allocate aligned OS memory directly, fall back to over-allocation (2042626048 bytes, address: 0x7f6bca400000, alignment: 67108864, commit: 1) unable to allocate aligned OS memory directly, fall back to over-allocation (2042626048 bytes, address: 0x7f6bca400000, alignment: 67108864, commit: 1) unable to allocate aligned OS memory directly, fall back to over-allocation (2042626048 bytes, address: 0x7f6b50800000, alignment: 67108864, commit: 1) unable to allocate aligned OS memory directly, fall back to over-allocation (2042626048 bytes, address: 0x7f6a5d000000, alignment: 67108864, commit: 1) mimalloc: warning: thread 0x7f717307c700: unable to allocate aligned OS memory directly, fall back to over-allocation (2042626048 bytes, address: 0x7f69e3400000, alignment: 67108864, commit: 1) unable to allocate aligned OS memory directly, fall back to over-allocation (2042626048 bytes, address: 0x7f6bca400000, alignment: 67108864, commit: 1) unable to allocate aligned OS memory directly, fall back to over-allocation (2042626048 bytes, address: 0x7f6ad6c00000, alignment: 67108864, commit: 1) unable to allocate aligned OS memory directly, fall back to over-allocation (2042626048 bytes, address: 0x7f6bca400000, alignment: 67108864, commit: 1) unable to allocate aligned OS memory directly, fall back to over-allocation (2042626048 bytes, address: 0x7f6bca400000, alignment: 67108864, commit: 1) unable to allocate aligned OS memory directly, fall back to over-allocation (2042626048 bytes, address: 0x7f6bca400000, alignment: 67108864, commit: 1) mimalloc: warning: unable to allocate aligned OS memory directly, fall back to over-allocation (2042626048 bytes, address: 0x7f6a5d000000, alignment: 67108864, commit: 1) unable to allocate aligned OS memory directly, fall back to over-allocation (2042626048 bytes, address: 0x7f6ad6c00000, alignment: 67108864, commit: 1) unable to allocate aligned OS memory directly, fall back to over-allocation (2042626048 bytes, address: 0x7f6bca400000, alignment: 67108864, commit: 1) 0:45:00.085 60G / 82G INFO K-mer Splitting (kmer_data.cpp : 108) Processed 19790636 reads 0:45:00.087 60G / 82G INFO K-mer Splitting (kmer_data.cpp : 98) Processing "/datastore/me_HiAS/Reads/D1A_EDME220001852-1a_H223VDSX3_L3_2.fq.gz" 0:46:37.713 60G / 82G INFO K-mer Splitting (kmer_data.cpp : 108) Processed 39581272 reads 0:46:37.718 60G / 82G INFO K-mer Splitting (kmer_data.cpp : 113) Total 39581272 reads processed 0:46:42.305 21G / 82G INFO General (kmer_index_builder.hpp : 264) Starting k-mer counting. 0:46:50.105 21G / 82G INFO General (kmer_index_builder.hpp : 275) K-mer counting done. There are 1615482512 kmers in total. 0:46:53.260 18M / 82G INFO K-mer Index Building (kmer_index_builder.hpp : 410) Building perfect hash indices 0:48:01.515 1206M / 82G INFO K-mer Index Building (kmer_index_builder.hpp : 446) Index built. Total 1615482512 kmers, 1166816800 bytes occupied (5.77817 bits per kmer). 0:48:01.518 1206M / 82G INFO K-mer Counting (kmer_data.cpp : 355) Arranging kmers in hash map order 0:48:47.258 26G / 82G INFO General (main.cpp : 149) Clustering Hamming graph. 0:56:33.859 26G / 82G INFO General (main.cpp : 156) Extracting clusters: 0:56:33.859 26G / 82G INFO General (concurrent_dsu.cpp : 19) Connecting to root 0:56:34.832 26G / 82G INFO General (concurrent_dsu.cpp : 35) Calculating counts 1:11:42.787 60G / 82G INFO General (concurrent_dsu.cpp : 64) Writing down entries 1:28:14.985 26G / 82G INFO General (main.cpp : 168) Clustering done. Total clusters: 1335860462 1:28:16.120 14G / 82G INFO K-mer Counting (kmer_data.cpp : 372) Collecting K-mer information, this takes a while. 1:28:37.946 50G / 82G INFO K-mer Counting (kmer_data.cpp : 378) Processing "/datastore/me_HiAS/Reads/D1A_EDME220001852-1a_H223VDSX3_L3_1.fq.gz" 1:30:15.907 50G / 82G INFO K-mer Counting (kmer_data.cpp : 378) Processing "/datastore/me_HiAS/Reads/D1A_EDME220001852-1a_H223VDSX3_L3_2.fq.gz" 1:31:45.899 50G / 82G INFO K-mer Counting (kmer_data.cpp : 385) Collection done, postprocessing. 1:31:54.115 50G / 82G INFO K-mer Counting (kmer_data.cpp : 398) There are 1615482512 kmers in total. Among them 2139770 (0.132454%) are singletons. 1:31:54.116 50G / 82G INFO General (main.cpp : 174) Subclustering Hamming graph 1:46:27.064 50G / 82G INFO Hamming Subclustering (kmer_cluster.cpp : 637) Subclustering done. Total 103730 non-read kmers were generated. 1:46:27.087 50G / 82G INFO Hamming Subclustering (kmer_cluster.cpp : 638) Subclustering statistics: 1:46:27.088 50G / 82G INFO Hamming Subclustering (kmer_cluster.cpp : 639) Total singleton hamming clusters: 1215187044. Among them 1136333366 (93.511%) are good 1:46:27.088 50G / 82G INFO Hamming Subclustering (kmer_cluster.cpp : 640) Total singleton subclusters: 23516775. Among them 23500663 (99.9315%) are good 1:46:27.088 50G / 82G INFO Hamming Subclustering (kmer_cluster.cpp : 641) Total non-singleton subcluster centers: 132219255. Among them 128461735 (97.1581%) are good 1:46:27.089 50G / 82G INFO Hamming Subclustering (kmer_cluster.cpp : 642) Average size of non-trivial subcluster: 3.0284 kmers 1:46:27.089 50G / 82G INFO Hamming Subclustering (kmer_cluster.cpp : 643) Average number of sub-clusters per non-singleton cluster: 1.29056 1:46:27.089 50G / 82G INFO Hamming Subclustering (kmer_cluster.cpp : 644) Total solid k-mers: 1288295764 1:46:27.089 50G / 82G INFO Hamming Subclustering (kmer_cluster.cpp : 645) Substitution probabilities: ( 0.918805 0.0258449 0.033439 0.0219107 ) ( 0.0147723 0.926822 0.0299558 0.0284497 ) ( 0.024833 0.0271687 0.931005 0.0169931 ) ( 0.0188366 0.0279715 0.0226885 0.930503 ) 1:46:29.174 50G / 82G INFO General (main.cpp : 179) Finished clustering. 1:46:29.174 50G / 82G INFO General (main.cpp : 198) Starting solid k-mers expansion in 70 threads. 1:48:45.043 50G / 82G INFO General (main.cpp : 219) Solid k-mers iteration 0 produced 64165362 new k-mers. 1:50:58.779 50G / 82G INFO General (main.cpp : 219) Solid k-mers iteration 1 produced 1473861 new k-mers. 1:53:12.206 50G / 82G INFO General (main.cpp : 219) Solid k-mers iteration 2 produced 71433 new k-mers. 1:55:26.357 50G / 82G INFO General (main.cpp : 219) Solid k-mers iteration 3 produced 18195 new k-mers. 1:57:38.853 50G / 82G INFO General (main.cpp : 219) Solid k-mers iteration 4 produced 7471 new k-mers. 1:59:52.143 50G / 82G INFO General (main.cpp : 219) Solid k-mers iteration 5 produced 3361 new k-mers. 2:02:06.100 50G / 82G INFO General (main.cpp : 219) Solid k-mers iteration 6 produced 1609 new k-mers. 2:04:18.947 50G / 82G INFO General (main.cpp : 219) Solid k-mers iteration 7 produced 859 new k-mers. 2:06:31.970 50G / 82G INFO General (main.cpp : 219) Solid k-mers iteration 8 produced 354 new k-mers. 2:08:44.973 50G / 82G INFO General (main.cpp : 219) Solid k-mers iteration 9 produced 56 new k-mers. 2:10:57.544 50G / 82G INFO General (main.cpp : 219) Solid k-mers iteration 10 produced 13 new k-mers. 2:13:10.262 50G / 82G INFO General (main.cpp : 219) Solid k-mers iteration 11 produced 0 new k-mers. 2:13:10.266 50G / 82G INFO General (main.cpp : 223) Solid k-mers finalized 2:13:10.270 50G / 82G INFO General (hammer_tools.cpp : 226) Starting read correction in 70 threads. 2:13:10.275 50G / 82G INFO General (hammer_tools.cpp : 239) Correcting pair of reads: "/datastore/me_HiAS/Reads/D1A_EDME220001852-1a_H223VDSX3_L3_1.fq.gz" and "/datastore/me_HiAS/Reads/D1A_EDME220001852-1a_H223VDSX3_L3_2.fq.gz" 2:13:45.479 56G / 82G INFO General (hammer_tools.cpp : 173) Prepared batch 0 of 7000000 reads. 2:14:45.210 56G / 82G INFO General (hammer_tools.cpp : 180) Processed batch 0 2:15:02.151 56G / 82G INFO General (hammer_tools.cpp : 190) Written batch 0 2:15:32.979 56G / 82G INFO General (hammer_tools.cpp : 173) Prepared batch 1 of 7000000 reads. 2:16:41.016 57G / 82G INFO General (hammer_tools.cpp : 180) Processed batch 1 2:16:57.325 57G / 82G INFO General (hammer_tools.cpp : 190) Written batch 1 2:17:23.962 57G / 82G INFO General (hammer_tools.cpp : 173) Prepared batch 2 of 5790636 reads. 2:18:17.186 57G / 82G INFO General (hammer_tools.cpp : 180) Processed batch 2 2:18:29.741 57G / 82G INFO General (hammer_tools.cpp : 190) Written batch 2 2:18:35.275 50G / 82G INFO General (hammer_tools.cpp : 280) Correction done. Changed 18337433 bases in 8554955 reads. 2:18:35.477 50G / 82G INFO General (hammer_tools.cpp : 281) Failed to correct 258130 bases out of 5937190790. 2:18:43.457 18M / 82G INFO General (main.cpp : 256) Saving corrected dataset description to /datastore/me_HiAS/Hybrid_SPAdes/corrected/corrected.yaml 2:18:43.484 18M / 82G INFO General (main.cpp : 263) All done. Exiting.

===== Read error correction finished.

===== corrected reads compression started.

== Running: /home/me/anaconda3/bin/python3 /home/me/Downloads/SPAdes-4.0.0-Linux/share/spades/spades_pipeline/scripts/compress_all.py --input_file /datastore/me_HiAS/Hybrid_SPAdes/corrected/corrected.yaml --ext_python_modules_home /home/me/Downloads/SPAdes-4.0.0-Linux/share/spades --max_threads 70 --output_dir /datastore/me_HiAS/Hybrid_SPAdes/corrected --not_used_yaml_file /datastore/me_HiAS/Hybrid_SPAdes/corrected/dont_correct.yaml --gzip_output

== Compressing corrected reads (with gzip) == Files to compress: ['/datastore/me_HiAS/Hybrid_SPAdes/corrected/D1A_EDME220001852-1a_H223VDSX3_L3_1.fq00.0_0.cor.fastq', '/datastore/me_HiAS/Hybrid_SPAdes/corrected/D1A_EDME220001852-1a_H223VDSX3_L3_2.fq00.0_0.cor.fastq', '/datastore/me_HiAS/Hybrid_SPAdes/corrected/D1A_EDME220001852-1a_H223VDSX3_L3__unpaired00.0_0.cor.fastq'] == Files compression is finished == Info about datasets not used in error correction stage is loaded == Dataset yaml file is updated

===== corrected reads compression finished.

===== Read error correction finished.

===== Assembling started.

===== K21 started.

== Running: /home/me/Downloads/SPAdes-4.0.0-Linux/bin/spades-core /datastore/me_HiAS/Hybrid_SPAdes/K21/configs/config.info /datastore/me_HiAS/Hybrid_SPAdes/K21/configs/mda_mode.info /datastore/me_HiAS/Hybrid_SPAdes/K21/configs/meta_mode.info

/home/me/Downloads/SPAdes-4.0.0-Linux/bin/spades-core: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /home/me/Downloads/SPAdes-4.0.0-Linux/bin/spades-core)

== Error == system call for: "['/home/me/Downloads/SPAdes-4.0.0-Linux/bin/spades-core', '/datastore/me_HiAS/Hybrid_SPAdes/K21/configs/config.info', '/datastore/me_HiAS/Hybrid_SPAdes/K21/configs/mda_mode.info', '/datastore/me_HiAS/Hybrid_SPAdes/K21/configs/meta_mode.info']" finished abnormally, OS return value: 1 None

params.txt

System information: SPAdes version: 4.0.0 Python version: 3.12.4 OS: Linux-3.10.0-1160.119.1.el7.x86_64-x86_64-with-glibc2.17

Output dir: /datastore/me_HiAS/Hybrid_SPAdes Mode: read error correction and assembling Debug mode is turned OFF

SPAdes version

4.0.0

Operating System

Red Hat Enterprise Linux Server 7.9

Python Version

3.12.4

Method of SPAdes installation

SPAdes Linux binaries

No errors reported in spades.log

[X] Yes

ablab / spades