search

jtamames / SqueezeMeta

A complete pipeline for metagenomic analysis
GNU General Public License v3.0
374 stars 80 forks source link

Step 1.5 seqmerge crash #458

Closed phagelab closed 2 years ago

phagelab commented 2 years ago

Hello. I tried running SQM for 50 samples in seqmerge mode. Step 01 seems to run ok and then then it crashes at 1.5 during merge 17.

Error message below:

Error running command: /home/plzrja/miniconda3/envs/SqueezeMeta/SqueezeMeta/bin/AMOS/minimus2_mod /home/plzrja/shigella/20220401_shigella_seqmerge/temp/mergedassemblies.20220401_shigella_seqmerge -D OVERLAP=100 -D MINID=95 -D THREADS=12 > /dev/null 2>&1 at /home/plzrja/miniconda3/envs/SqueezeMeta/SqueezeMeta/scripts/01.merge_sequential.pl line 102. Stopping in STEP1.5 -> 01.merge_sequential.pl. Program finished abnormally

If you don't know what went wrong or want further advice, please look for similar issues in https://github.com/jtamames/SqueezeMeta/issues Feel free to open a new issue if you don't find the answer there. Please add a brief description of the problem and upload the /home/plzrja/shigella/20220401_shigella_seqmerge/syslog file (zip it first) Died at /home/plzrja/miniconda3/envs/SqueezeMeta/bin/SqueezeMeta.pl line 1381.

syslog attached. Any help appreciated. syslog.txt

jtamames commented 2 years ago

Hello! Minimus 2 is crashing. Could you run the command: /home/plzrja/miniconda3/envs/SqueezeMeta/SqueezeMeta/bin/AMOS/minimus2_mod /home/plzrja/shigella/20220401_shigella_seqmerge/temp/mergedassemblies.20220401_shigella_seqmerge -D OVERLAP=100 -D MINID=95 -D THREADS=12 and paste here the output? Best, J

phagelab commented 2 years ago

mergedassemblies.20220401_shigella_seqmerge.runAmos.log

jtamames commented 2 years ago

Looks like it runs out of memory. Could you monitor the memory usage while the command is running?

phagelab commented 2 years ago

Do you mean the command you pasted above?

jtamames commented 2 years ago

Yes sorry, while Minimus2 is running

phagelab commented 2 years ago

I uploaded the .runAmos.log file and memory.log.

mergedassemblies.20220401_shigella_seqmerge.runAmos.log memory.log

jtamames commented 2 years ago

Let's see if we can make it work relaxing the alignment error rate. Please run: /home/plzrja/miniconda3/envs/SqueezeMeta/SqueezeMeta/bin/AMOS/make-consensus -B -e 0.15 -b /home/plzrja/shigella/20220401_shigella_seqmerge/temp/mergedassemblies.20220401_shigella_seqmerge.bnk -w 15 And let me know if it finishes correctly

jtamames commented 2 years ago

Please also set -v 5, that controls verbosity, let's see if we can make it more expressive

phagelab commented 2 years ago

/home/plzrja/miniconda3/envs/SqueezeMeta/SqueezeMeta/bin/AMOS/make-consensus -B -e 0.15 -b /home/plzrja/shigella/20220401_shigella_seqmerge/temp/mergedassemblies.20220401_shigella_seqmerge.bnk -w 15 -v 5 Starting on Fri Apr 8 08:01:02 2022

Read bank is /home/plzrja/shigella/20220401_shigella_seqmerge/temp/mergedassemblies.20220401_shigella_seqmerge.bnk Alignment error rate is 0.15 Minimum overlap bases is 5 Output will be written to the bank AMOS Exception WHAT: Could not open bank for reading, locked by 'w 1001 plzrja' LINE: 1164 FILE: Bank_AMOS.cc

jtamames commented 2 years ago

What? This looks like a permissions issue. But why at this point and not before? Could you please ls -l /home/plzrja/shigella/20220401_shigella_seqmerge/temp/mergedassemblies.20220401_shigella_seqmerge.bnk? What do you think @fpusan?

phagelab commented 2 years ago

ls -l /home/plzrja/shigella/20220401_shigella_seqmerge/temp/mergedassemblies.20220401_shigella_seqmerge.bnk total 4 -rw-r--r-- 1 plzrja plzrja 0 Apr 8 07:59 CTG.0.0.fix -rw-r--r-- 1 plzrja plzrja 0 Apr 8 07:59 CTG.0.0.var -rw-r--r-- 1 plzrja plzrja 0 Apr 8 07:59 CTG.0.map -rw-r--r-- 1 plzrja plzrja 183 Apr 8 07:59 CTG.ifo

fpusan commented 2 years ago

From https://sourceforge.net/p/amos/mailman/amos-help/thread/11372E21-A30D-4EF9-A9C2-A6B6CCBC6C6B%40ebc.uu.se/#msg24676867

You can use the following command to unlock your bank manually: bank-unlock

However we don't have bank unlock in SqueezeMeta (maybe it wasn't included in minimus2?) and trying to compile AMOS from source is failing in my Ubuntu20 machine

phagelab commented 2 years ago

Also tried compiling AMOS from source, but no luck so far...

fpusan commented 2 years ago

Any update on this? I found no solution for your AMOS problem. Does it reproduce if you try again with a new project? As an alternative, maybe you can run several smaller coassemblies and then combine them inside SQMtools...

phagelab commented 2 years ago

No success doing this so far. As a backup plan I've run all of the samples using sequential mode which seemed to work without any issues. I may revisit it and try the smaller coassemblies suggestion though and see if that works.

pedres commented 2 years ago

Hi,

I am not sure but I think I have the same problem. I am processing 12 paired samples using the merged mode on a desktop computer with 24 processors and 48Gb of ram. Error running command: /home/fulgencio/miniconda3/envs/SqueezeMeta/SqueezeMeta/bin/AMOS/minimus2 /media/fulgencio/DATOS/grapes_vcs_merged/temp/mergedassemblies.grapes_vcs_merged.99 -D OVERLAP=100 -D MINID=95 -D THREADS=20 >> /media/fulgencio/DATOS/grapes_vcs_merged/syslog 2>&1 at /home/fulgencio/miniconda3/envs/SqueezeMeta/SqueezeMeta/scripts/01.merge_assemblies.pl line 70. Stopping in STEP1.5 -> 01.merge_assemblies.pl. Program finished abnormally

The runAmos.log said that "exited with status: 137" so it should be a memory issue? !

syslog.txt

mergedassemblies.grapes_vcs_merged.99.runAmos.log

jtamames commented 2 years ago

Hello Very likely. Try running the conflicting program by itself, to see if you can get more information: LD_LIBRARY_PATH=/home/fulgencio/miniconda3/envs/SqueezeMeta/SqueezeMeta/bin/AMOS/../../lib/mummer /home/fulgencio/miniconda3/envs/SqueezeMeta/SqueezeMeta/bin/AMOS/../mummer/nucmer --maxmatch --threads 20 -c 100 /media/fulgencio/DATOS/grapes_vcs_merged/temp/mergedassemblies.grapes_vcs_merged.99.ref.seq /media/fulgencio/DATOS/grapes_vcs_merged/temp/mergedassemblies.grapes_vcs_merged.99.qry.seq -p /media/fulgencio/DATOS/grapes_vcs_merged/temp/mergedassemblies.grapes_vcs_merged.99

fpusan commented 2 years ago

Closing due to lack of activity, feel free to reopen

oyac commented 9 months ago

Hello! I am running 12 samples in SqueezeMeta and I think I had the same kind of trouble as above: I was running the samples, assembly with Spades, in merged mode, but I got this

Command: /home/nioo/ohanac/.conda/envs/squeezemeta1.6/SqueezeMeta/bin/AMOS/make-consensus -B -e 0.06 -b /home/nioo/ohanac/squeezem/analyses/Han/Han_meta/temp/mergedassemblies.Han_meta.99.bnk -w 15 exited with status: 139

I thought it could be related to memory or so, then I switched to seqmerge mode, since the individual samples were assembled, but I got something similar

Error running command: /home/nioo/ohanac/.conda/envs/squeezemeta1.6/SqueezeMeta/bin/AMOS/minimus2_mod /home/nioo/ohanac/squeezem/analyses/Han/Han_meta/temp/mergedassemblies.Han_meta -D OVERLAP=100 -D MINID=95 -D THREADS=12 > /dev/null 2>&1 at /home/nioo/ohanac/.conda/envs/squeezemeta1.6/bin/01.merge_sequential.pl line 102.

I tried running the commands alone, and it was the same I tried to look for answers here, but I was not able to find them, and I got this:

(squeezemeta1.6) ohanac@nioo0003:~/squeezem/analyses/Han$ /home/nioo/ohanac/.conda/envs/squeezemeta1.6/SqueezeMeta/bin/AMOS/make-consensus -B -e 0.15 -b /home /nioo/ohanac/squeezem/analyses/Han/Han_meta/temp/mergedassemblies.Han_meta.bnk -w 15 Starting on Tue Dec 19 15:04:12 2023

Read bank is /home/nioo/ohanac/squeezem/analyses/Han/Han_meta/temp/mergedassemblies.Han_meta.bnk Alignment error rate is 0.15 Minimum overlap bases is 5 Output will be written to the bank AMOS Exception WHAT: Could not open bank for reading, locked by 'w 27366 ohanac' LINE: 1164 FILE: Bank_AMOS.cc

Somewhere else I also saw that could be a problem with naming, since my samples have a few underscores, like this: COM1_5_1

Can you help me, please? Here are the log files

mergedassemblies.Han_meta.runAmos.log syslog.txt

fpusan commented 9 months ago

ecode 139 looks like a segmentation fault by AMOS. Is this also happening when running the seqmerge mode with the test dataset?

oyac commented 9 months ago

Hi, I just run the test data and it was ok, it did the merging normally in the seqmerge mode.

oyac commented 9 months ago

Hello! Do you have any idea on how to solve my problem above? the test dataset runs normally, but my samples are stuck with the AMOS error. Thanks!

fpusan commented 9 months ago

Then it's hard to solve. If AMOS also failed with the test dataset then maybe recompiling It from source would help. But since It works there I'd say you ran into an AMOS-specific bug, and since It just causes a segfault then it's hard to know what is happening or where. I'd recommend to use the coassembly mode (if possible) or sequential mode.