search

soedinglab / MMseqs2

MMseqs2: ultra fast and sensitive search and clustering suite
https://mmseqs.com
MIT License
1.46k stars 198 forks source link

Illegal instruction (core dumped) #469

Closed zhaoxvwahaha closed 3 years ago

zhaoxvwahaha commented 3 years ago

Hi, Thanks for the wonderful MMseqs2, I run it on my 1k+ sample parallelly on computer cluster, most of samples failed with the following errors, only 20% samples succussed:

/opt/gridengine/default/spool/cngb-compute-m16-18/job_scripts/7307636: line 1: 4653 Illegal instruction (core dumped)/MMseqs2/build/bin/mmseqs created contigs.fa result/R2011005973_0_20201225/R2011005973_0_20201225.DB /opt/gridengine/default/spool/cngb-compute-m16-18/job_scripts/7307636: line 2: 4658 Illegal instruction (core dumped)MMseqs2/build/bin/mmseqs search -a -s 7 --num-iterations 2 --search-type 2 result/R2011005973_0_20201225/R2011005973_0_20201225.DB DB/segmentNucDB result/R2011005973_0_20201225/R2011005973_0_20201225.result result/R2011005973_0_20201225/R2011005973_0_20201225.tmp /opt/gridengine/default/spool/cngb-compute-m16-18/job_scripts/7307636: line 3: 4662 Illegal instruction (core dumped)/MMseqs2/build/bin/mmseqs convertalis result/R2011005973_0_20201225/R2011005973_0_20201225.DB DB/segmentNucDB result/R2011005973_0_20201225/R2011005973_0_20201225.result result/R2011005973_0_20201225/R2011005973_0_20201225.result.m8

MMseqs2 Version: 24f6b52a38cd8cf66d10ce00bf37dc815fef986e-MPI

Anyone could give some advice? Thanks in advance.

milot-mirdita commented 3 years ago

This sound like your compute nodes have very different CPUs to the node where you compiled MMseqs2 on. Please either compile MMseqs2 on the compute node or choose one of the cmake flags that most closely fits to your compute nodes CPUs (see https://github.com/soedinglab/MMseqs2/wiki#customizing-compilation-through-cmake -DHAVE_AVX2/-DHAVE_SSE4_1).

zhaoxvwahaha commented 3 years ago

Hi @milot-mirdita Thank you for your helpful reply, I change the node to run MMseqs2, and now, there is no "core dumped" problem. But, a new error exists: I still run many jobs parallelly, and I assigned the tmp directory for each job (mmseqs search -a -s 7 --num-iterations 2 --search-type 2 sample/sample.DB DB/segmentNucDB sample/sample.result sample/sample.tmp), but some of jobs failed with the following error:

**A call to mkdir was unable to create the desired directory:

Directory: /tmp/6072204.1.st.q/openmpi-sessions-36903@cngb-compute-e04-2_0 Error: No space left on device

Please check to ensure you have adequate permissions to perform the desired operation.**

It's strange that, I assigned the tmp directory, why MMseqs2 will mkdir a directory in the root diretory /tmp, is it still related to my compute node?

Looking forward to your reply

milot-mirdita commented 3 years ago

That seems like an unrelated MPI issue: I found this link that might help: https://www.open-mpi.org/faq/?category=sm#where-sm-file

You should be able to set a different OpenMPI temp directory by passing --mca orte_tmpdir_base TMP_DIR to the mpirun call.

e.g something like:

export RUNNER="mpirun --mca orte_tmpdir_base /new_tmp"
mmseqs search ...
zhaoxvwahaha commented 3 years ago

Hi @milot-mirdita ,

Thank you, I installed the mpi version, but I used just the normal mode, let me try your advice. And, I also get an another error: /opt/gridengine/default/spool/cngb-compute-e05-9/job_scripts/6073613: line 1: /MMseqs2/build/bin/mmseqs: Unknown error 521 Do you have any idea about this unknown error 521?

milot-mirdita commented 3 years ago

No idea at all. Googling seems to indicate some issue with NFS?