search

jtamames / SqueezeMeta

A complete pipeline for metagenomic analysis
GNU General Public License v3.0
357 stars 78 forks source link

Error occurs when reading inputs #659

Closed lisiruisusan closed 1 year ago

lisiruisusan commented 1 year ago

Dear author,

I run a total of 24 samples which were sequenced and provided by the same company. 12 of them were successful while others failed in step 1. The system log wrote that error occurs when reading inputs. What shall I do to deal with the problem?

Syslog

Run started Fri Mar 31 16:02:07 2023 in coassembly mode

SqueezeMeta v1.6.2, March 2023 - (c) J. Tamames, F. Puente-Sánchez CNB-CSIC, Madrid, SPAIN

Please cite: Tamames & Puente-Sanchez, Frontiers in Microbiology 10.3389 (2019). doi: https://doi.org/10.3389/fmicb.2018.03349

Run started for ZYG180325, Fri Mar 31 16:02:07 2023 Project: ZYG180325 Map file: /database/users/zzzhou/huichen/shuju/mulu/ZYG180325/ZYG180325.txt Fastq directory: /database/users/zzzhou/huichen/shuju/data Command: /database/users/zzzhou/miniconda3/envs/SqueezeMeta/bin/SqueezeMeta.pl -m coassembly -p /database/users/zzzhou/huichen/shuju/mulu/ZYG180325/ZYG180325 -s /database/users/zzzhou/huichen/shuju/mulu/ZYG180325/ZYG180325.txt -f /database/users/zzzhou/huichen/shuju/data [0 seconds]: STEP0 -> SqueezeMeta.pl COGS; KEGG; PFAM;

[0 seconds]: STEP1 -> 01.run_all_assemblies.pl (megahit) Stopping in STEP1 -> 01.run_all_assemblies.pl. Program finished abnormally

System information:

Tree for the project: 25/ZYG180325/data/raw_fastq/par1.fastq.gz Preparing files for pair2: cat /database/users/zzzhou/huichen/shuju/data/ZYG180325_2.fastq.gz > /database/users/zzzhou/huichen/shuju/mulu/ZYG180325/ZYG180325/data/raw_fastq/par2.fastq.gz Running assembly with megahit: perl /database/users/zzzhou/miniconda3/envs/SqueezeMeta/SqueezeMeta/lib/SqueezeMeta/assembly_megahit.pl /database/users/zzzhou/huichen/shuju/mulu/ZYG180325/ZYG180325 ZYG180325 /database/users/zzzhou/huichen/shuju/mulu/ZYG180325/ZYG180325/data/raw_fastq/par1.fastq.gz /database/users/zzzhou/huichen/shuju/mulu/ZYG180325/ZYG180325/data/raw_fastq/par2.fastq.gz 2023-03-31 16:02:14 - MEGAHIT v1.2.9 2023-03-31 16:02:14 - Using megahit_core with POPCNT and BMI2 support 2023-03-31 16:02:14 - Convert reads to binary library 2023-03-31 16:06:58 - b'INFO sequence/io/sequence_lib.cpp : 77 - Lib 0 (/database/users/zzzhou/huichen/shuju/mulu/ZYG180325/ZYG180325/data/raw_fastq/par1.fastq.gz,/database/users/zzzhou/huichen/shuju/mulu/ZYG180325/ZYG180325/data/raw_fastq/par2.fastq.gz): pe, 62654668 reads, 170 max length' 2023-03-31 16:06:58 - b'INFO utils/utils.h : 152 - Real: 284.0187\tuser: 118.4028\tsys: 44.1144\tmaxrss: 245584' 2023-03-31 16:06:58 - Error occurs when reading inputs Linux dell-PowerEdge-R740 5.4.0-59-generic #65-Ubuntu SMP Thu Dec 10 12:01:51 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux [4.0K Mar 31 16:02] /database/users/zzzhou/huichen/shuju/mulu/ZYG180325/ZYG180325 ├── [ 31 Mar 31 16:02] creator.txt ├── [4.0K Mar 31 16:02] data │   ├── [ 74 Mar 31 16:02] 00.ZYG180325.samples │   ├── [4.0K Mar 31 16:02] megahit │   │   ├── [ 7 Mar 31 16:02] checkpoints.txt │   │   ├── [4.0K Mar 31 16:02] intermediate_contigs │   │   ├── [ 948 Mar 31 16:06] log │   │   ├── [1.0K Mar 31 16:02] options.json │   │   └── [4.0K Mar 31 16:06] tmp │   │   ├── [ 369 Mar 31 16:02] reads.lib │   │   ├── [2.6G Mar 31 16:06] reads.lib.bin │   │   └── [ 219 Mar 31 16:06] reads.lib.lib_info │   └── [4.0K Mar 31 16:02] raw_fastq │   ├── [2.2G Mar 31 16:02] par1.fastq.gz │   └── [2.3G Mar 31 16:02] par2.fastq.gz ├── [4.0K Mar 31 16:02] ext_tables ├── [4.0K Mar 31 16:02] intermediate │   └── [4.0K Mar 31 16:02] binners ├── [ 117 Mar 31 16:02] methods.txt ├── [3.1K Mar 31 16:02] parameters.pl ├── [ 37 Mar 31 16:02] progress ├── [4.0K Mar 31 16:02] results ├── [8.3K Mar 31 16:02] SqueezeMeta_conf.pl ├── [2.3K Mar 31 16:06] syslog └── [4.0K Mar 31 16:02] temp

11 directories, 15 files

(SqueezeMeta) zzzhou@dell-PowerEdge-R740:~/huichen$ SqueezeMeta.pl -m coassembly -p /database/users/zzzhou/huichen/shuju/mulu/ZYG180325/ZYG180325 -s /database/users/zzzhou/huichen/shuju/mulu/ZYG180325/ZYG180325.txt -f /database/users/zzzhou/huichen/shuju/data

SqueezeMeta v1.6.2, March 2023 - (c) J. Tamames, F. Puente-Sánchez CNB-CSIC, Madrid, SPAIN

Please cite: Tamames & Puente-Sanchez, Frontiers in Microbiology 9, 3349 (2019). doi: https://doi.org/10.3389/fmicb.2018.03349

Run started Fri Mar 31 16:02:07 2023 in coassembly mode 1 metagenomes found: ZYG180325

Now creating directories Reading configuration from /database/users/zzzhou/huichen/shuju/mulu/ZYG180325/ZYG180325/SqueezeMeta_conf.pl [0 seconds]: STEP1 -> RUNNING ASSEMBLY: 01.run_all_assemblies.pl (megahit) Concatenating all samples: ZYG180325_1.fastq.gz ZYG180325_2.fastq.gz pair1: cat /database/users/zzzhou/huichen/shuju/data/ZYG180325_1.fastq.gz > /database/users/zzzhou/huichen/shuju/mulu/ZYG180325/ZYG180325/data/raw_fastq/par1.fastq.gz pair2: cat /database/users/zzzhou/huichen/shuju/data/ZYG180325_2.fastq.gz > /database/users/zzzhou/huichen/shuju/mulu/ZYG180325/ZYG180325/data/raw_fastq/par2.fastq.gz Running assembly with megahit: perl /database/users/zzzhou/miniconda3/envs/SqueezeMeta/SqueezeMeta/lib/SqueezeMeta/assembly_megahit.pl /database/users/zzzhou/huichen/shuju/mulu/ZYG180325/ZYG180325 ZYG180325 /database/users/zzzhou/huichen/shuju/mulu/ZYG180325/ZYG180325/data/raw_fastq/par1.fastq.gz /database/users/zzzhou/huichen/shuju/mulu/ZYG180325/ZYG180325/data/raw_fastq/par2.fastq.gz Error running command: /database/users/zzzhou/miniconda3/envs/SqueezeMeta/SqueezeMeta/bin/megahit/megahit -1 /database/users/zzzhou/huichen/shuju/mulu/ZYG180325/ZYG180325/data/raw_fastq/par1.fastq.gz -2 /database/users/zzzhou/huichen/shuju/mulu/ZYG180325/ZYG180325/data/raw_fastq/par2.fastq.gz -t 12 -o /database/users/zzzhou/huichen/shuju/mulu/ZYG180325/ZYG180325/data/megahit >> /database/users/zzzhou/huichen/shuju/mulu/ZYG180325/ZYG180325/syslog 2>&1 at /database/users/zzzhou/miniconda3/envs/SqueezeMeta/SqueezeMeta/lib/SqueezeMeta/assembly_megahit.pl line 36. Assembly not present in /database/users/zzzhou/huichen/shuju/mulu/ZYG180325/ZYG180325/results/01.ZYG180325.fasta, exiting Stopping in STEP1 -> 01.run_all_assemblies.pl. Program finished abnormally

If you don't know what went wrong or want further advice, please look for similar issues in https://github.com/jtamames/SqueezeMeta/issues Feel free to open a new issue if you don't find the answer there. Please add a brief description of the problem and upload the /database/users/zzzhou/huichen/shuju/mulu/ZYG180325/ZYG180325/syslog file (zip it first) Died at /database/users/zzzhou/miniconda3/envs/SqueezeMeta/bin/SqueezeMeta.pl line 941. (SqueezeMeta) zzzhou@dell-PowerEdge-R740:~/huichen$

Yours sincerely,

Li

March 31st 2023

euduca commented 1 year ago

Hi.

I have the same problem, but with SPAdes (my server: 24 cores, and 754GB RAM).

I tried it with Megahit and it works without any errors.

Running assembly with spades: perl /data03/duca/bioinfo/local/anaconda3/2022/10/envs/SqueezeMeta/SqueezeMeta/lib/SqueezeMeta/assembly_spades.pl /data03/duca/bioinfo/projetos/ufmg/glen/petro_shotgun/run/sm_petrobras_shotgun sm_petrobras_shotgun /data03/duca/bioinfo/projetos/ufmg/glen/petro_shotgun/run/sm_petrobras_shotgun/data/raw_fastq/par1.fastq.gz /data03/duca/bioinfo/projetos/ufmg/glen/petro_shotgun/run/sm_petrobras_shotgun/data/raw_fastq/par2.fastq.gz
Error running command:    /data03/duca/bioinfo/local/anaconda3/2022/10/envs/SqueezeMeta/SqueezeMeta/bin/SPAdes/spades.py --meta --pe1-1 /data03/duca/bioinfo/projetos/ufmg/glen/petro_shotgun/run/sm_petrobras_shotgun/data/raw_fastq/par1.fastq.gz --pe1-2 /data03/duca/bioinfo/projetos/ufmg/glen/petro_shotgun/run/sm_petrobras_shotgun/data/raw_fastq/par2.fastq.gz -k 21,33,55,77,99,127  -t 12 -o /data03/duca/bioinfo/projetos/ufmg/glen/petro_shotgun/run/sm_petrobras_shotgun/data/spades >> /data03/duca/bioinfo/projetos/ufmg/glen/petro_shotgun/run/sm_petrobras_shotgun/syslog 2>&1 at /data03/duca/bioinfo/local/anaconda3/2022/10/envs/SqueezeMeta/SqueezeMeta/lib/SqueezeMeta/assembly_spades.pl line 39.
Assembly not present in /data03/duca/bioinfo/projetos/ufmg/glen/petro_shotgun/run/sm_petrobras_shotgun/results/01.sm_petrobras_shotgun.fasta, exiting
Stopping in STEP1 -> 01.run_all_assemblies.pl. Program finished abnormally
If you don't know what went wrong or want further advice, please look for similar issues in https://github.com/jtamames/SqueezeMeta/issues
Feel free to open a new issue if you don't find the answer there. Please add a brief description of the problem and upload the /data03/duca/bioinfo/projetos/ufmg/glen/petro_shotgun/run/sm_petrobras_shotgun/syslog file (zip it first).

I found this error in syslog file:

12:17:05.060 41G / 51G ERROR General (mmapped_reader.hpp : 52) mmap(2) failed. Reason: Cannot allocate memory. Error code: 12

The SPAdes params.txt file indicates the 250 Gb limit by default.
Other parameters:
Dir for temp files: /data03/duca/bioinfo/projetos/ufmg/glen/petro_shotgun/run/sm_petrobras_shotgun/data/spades/tmp
Threads: 12
Memory limit (in Gb): 250

I followed the entire analysis with the help of Zabbix and the max memory used on the server was 78GB. So I don't know why this error.

Best regards.

fpusan commented 1 year ago

This is a different error. In your case you are running out of memory. MEGAHIT works in your case because it is more memory-efficient than SPAdes.