search

soedinglab / MMseqs2

MMseqs2: ultra fast and sensitive search and clustering suite
https://mmseqs.com
MIT License
1.44k stars 197 forks source link

Example run segmentation fault - MPI compilation #679

Closed Rridley7 closed 1 year ago

Rridley7 commented 1 year ago

Current Behavior

Hello, thanks for developing this very useful tool! I have used mmseqs via conda installation without error on a linux system, however when looking to compile from source for use of MPI, I encounter a segmentation fault with signal 11. This test was not run with an additional RUNNER variable, so I'm guessing it relates to the initial installation?

Steps to Reproduce (for bugs)

Please make sure to execute the reproduction steps with newly recreated and empty tmp folders. I installed mmseqs as instructed in the manual, with added MPI support

git clone https://github.com/soedinglab/MMseqs2.git
mv MMseqs2 mmseqs2 && cd mmseqs2/
mkdir build-mpi && cd build-mpi
cmake -DHAVE_MPI=1 -DCMAKE_INSTALL_PREFIX=. -DCMAKE_BUILD_TYPE=Release ..
make
make install

Following this, I got the example DB.fasta file from the examples folder, renamed exp.fa.

mmseqs easy-cluster exp.fa test tmp2

MMseqs Output (for bugs)

Please make sure to also post the complete output of MMseqs. You can use gist.github.com for large output.

Create directory tmp2
easy-cluster exp.fa test tmp2

MMseqs Version:                         6b9388469a41935813f5d207d6d149767feb4c42-MPI
Substitution matrix                     aa:blosum62.out,nucl:nucleotide.out
Seed substitution matrix                aa:VTML80.out,nucl:nucleotide.out
Sensitivity                             4
k-mer length                            0
k-score                                 seq:2147483647,prof:2147483647
Alphabet size                           aa:21,nucl:5
Max sequence length                     65535
Max results per query                   20
Split database                          0
Split mode                              2
Split memory limit                      0
Coverage threshold                      0.8
Coverage mode                           0
Compositional bias                      1
Compositional bias                      1
Diagonal scoring                        true
Exact k-mer matching                    0
Mask residues                           1
Mask residues probability               0.9
Mask lower case residues                0
Minimum diagonal score                  15
Selected taxa
Include identical seq. id.              false
Spaced k-mers                           1
Preload mode                            0
Pseudo count a                          substitution:1.100,context:1.400
Pseudo count b                          substitution:4.100,context:5.800
Spaced k-mer pattern
Local temporary path
Threads                                 24
Compressed                              0
Verbosity                               3
Add backtrace                           false
Alignment mode                          3
Alignment mode                          0
Allow wrapped scoring                   false
E-value threshold                       0.001
Seq. id. threshold                      0
Min alignment length                    0
Seq. id. mode                           0
Alternative alignments                  0
Max reject                              2147483647
Max accept                              2147483647
Score bias                              0
Realign hits                            false
Realign score bias                      -0.2
Realign max seqs                        2147483647
Correlation score weight                0
Gap open cost                           aa:11,nucl:5
Gap extension cost                      aa:1,nucl:2
Zdrop                                   40
Rescore mode                            0
Remove hits by seq. id. and coverage    false
Sort results                            0
Cluster mode                            0
Max connected component depth           1000
Similarity type                         2
Weight file name
Cluster Weight threshold                0.9
Single step clustering                  false
Cascaded clustering steps               3
Cluster reassign                        false
Remove temporary files                  true
Force restart with latest tmp           false
MPI runner
k-mers per sequence                     21
Scale k-mers per sequence               aa:0.000,nucl:0.200
Adjust k-mer length                     false
Shift hash                              67
Include only extendable                 false
Skip repeating k-mers                   false
Database type                           0
Shuffle input database                  true
Createdb mode                           1
Write lookup file                       0
Offset of numeric ids                   0

createdb exp.fa tmp2/6664721939282577716/input --dbtype 0 --shuffle 1 --createdb-mode 1 --write-lookup 0 --id-offset 0 --compressed 0 -v 3

Shuffle database cannot be combined with --createdb-mode 0
We recompute with --shuffle 0
Converting sequences
[19999] 0s 24ms
Time for merging to input_h: 0h 0m 0s 2ms
Time for merging to input: 0h 0m 0s 1ms
Database type: Aminoacid
Time for processing: 0h 0m 0s 33ms
Create directory tmp2/6664721939282577716/clu_tmp
cluster tmp2/6664721939282577716/input tmp2/6664721939282577716/clu tmp2/6664721939282577716/clu_tmp --max-seqs 20 -c 0.8 --spaced-kmer-mode 1 --alignment-mode 3 -e 0.001 --remove-tmp-files 1

Set cluster sensitivity to -s 6.000000
Set cluster mode SET COVER
Set cluster iterations to 3
linclust tmp2/6664721939282577716/input tmp2/6664721939282577716/clu_tmp/1194185237556116514/clu_redundancy tmp2/6664721939282577716/clu_tmp/1194185237556116514/linclust --cluster-mode 0 --max-iterations 1000 --similarity-type 2 --threads 24 --compressed 0 -v 3 --cluster-weight-threshold 0.9 --sub-mat 'aa:blosum62.out,nucl:nucleotide.out' -a 0 --alignment-mode 3 --alignment-output-mode 0 --wrapped-scoring 0 -e 0.001 --min-seq-id 0 --min-aln-len 0 --seq-id-mode 0 --alt-ali 0 -c 0.8 --cov-mode 0 --max-seq-len 65535 --comp-bias-corr 1 --comp-bias-corr-scale 1 --max-rejected 2147483647 --max-accept 2147483647 --add-self-matches 0 --db-load-mode 0 --pca substitution:1.100,context:1.400 --pcb substitution:4.100,context:5.800 --score-bias 0 --realign 0 --realign-score-bias -0.2 --realign-max-seqs 2147483647 --corr-score-weight 0 --gap-open aa:11,nucl:5 --gap-extend aa:1,nucl:2 --zdrop 40 --alph-size aa:13,nucl:5 --kmer-per-seq 21 --spaced-kmer-mode 1 --kmer-per-seq-scale aa:0.000,nucl:0.200 --adjust-kmer-len 0 --mask 0 --mask-prob 0.9 --mask-lower-case 0 -k 0 --hash-shift 67 --split-memory-limit 0 --include-only-extendable 0 --ignore-multi-kmer 0 --rescore-mode 0 --filter-hits 0 --sort-results 0 --remove-tmp-files 1 --force-reuse 0

[][error_sighandler] Caught error: Segmentation fault (signal 11)
tmp2/6664721939282577716/clu_tmp/1194185237556116514/linclust/18444629910076322402/linclust.sh: line 26: 124298 Segmentation fault      (core dumped) $RUNNER "$MMSEQS" kmermatcher "$INPUT" "${TMP_PATH}/pref" ${KMERMATCHER_PAR}
Error: kmermatcher died
Error: linclust died
Error: Search died

Context

Providing context helps us come up with a solution and improve our documentation for the future.

Your Environment

Include as many relevant details about the environment you experienced the bug in.

milot-mirdita commented 1 year ago

Please make sure you submit one MPI job per node. Thread parallelization still happens through OpenMP not MPI. Also how are you calling mmseqs2? Are you passing the relevant mpirun with the RUNNER env variable?

Rridley7 commented 1 year ago

Hi, thanks for the quick response! Currently I'm not running any MP, and simply calling mmseqs directly after exporting the bin directory to my path. When testing the MP mode, I used srun --ntasks-per-node 1, and recieved a similar error, however running it without any MP made me think the problem is likely upstream of this.

milot-mirdita commented 1 year ago

Can you upload your input sequences?

Rridley7 commented 1 year ago

The file used here is DB.fasta , I simply renamed to exp.fa as I was testing in a directory with other similarly named files.

milot-mirdita commented 1 year ago

Okay I am not sure how to reproduce the issue. It doesn't crash here. Does it crash consistently?

Can you please repeat only the linclust call with the same tmp files:

mmseqs linclust tmp2/6664721939282577716/input tmp2/6664721939282577716/clu_tmp/1194185237556116514/clu_redundancy tmp2/6664721939282577716/clu_tmp/1194185237556116514/linclust --cluster-mode 0 --max-iterations 1000 --similarity-type 2 --threads 24 --compressed 0 -v 3 --cluster-weight-threshold 0.9 --sub-mat 'aa:blosum62.out,nucl:nucleotide.out' -a 0 --alignment-mode 3 --alignment-output-mode 0 --wrapped-scoring 0 -e 0.001 --min-seq-id 0 --min-aln-len 0 --seq-id-mode 0 --alt-ali 0 -c 0.8 --cov-mode 0 --max-seq-len 65535 --comp-bias-corr 1 --comp-bias-corr-scale 1 --max-rejected 2147483647 --max-accept 2147483647 --add-self-matches 0 --db-load-mode 0 --pca substitution:1.100,context:1.400 --pcb substitution:4.100,context:5.800 --score-bias 0 --realign 0 --realign-score-bias -0.2 --realign-max-seqs 2147483647 --corr-score-weight 0 --gap-open aa:11,nucl:5 --gap-extend aa:1,nucl:2 --zdrop 40 --alph-size aa:13,nucl:5 --kmer-per-seq 21 --spaced-kmer-mode 1 --kmer-per-seq-scale aa:0.000,nucl:0.200 --adjust-kmer-len 0 --mask 0 --mask-prob 0.9 --mask-lower-case 0 -k 0 --hash-shift 67 --split-memory-limit 0 --include-only-extendable 0 --ignore-multi-kmer 0 --rescore-mode 0 --filter-hits 0 --sort-results 0 --remove-tmp-files 1 --force-reuse 0
Rridley7 commented 1 year ago

Yes, I've tried recompiling several times as well, and receive the same error after successful compilation. When running the command you gave, I get the following error:

Command: 

~/data/dir/mmseqs2/build-mpi/bin/mmseqs linclust tmp2/6664721939282577716/input tmp2/6664721939282577716/clu_tmp/1194185237556116514/clu_redundancy tmp2/6664721939282577716/clu_tmp/1194185237556116514/linclust --cluster-mode 0 --max-iterations 1000 --similarity-type 2 --threads 24 --compressed 0 -v 3 --cluster-weight-threshold 0.9 --sub-mat 'aa:blosum62.out,nucl:nucleotide.out' -a 0 --alignment-mode 3 --alignment-output-mode 0 --wrapped-scoring 0 -e 0.001 --min-seq-id 0 --min-aln-len 0 --seq-id-mode 0 --alt-ali 0 -c 0.8 --cov-mode 0 --max-seq-len 65535 --comp-bias-corr 1 --comp-bias-corr-scale 1 --max-rejected 2147483647 --max-accept 2147483647 --add-self-matches 0 --db-load-mode 0 --pca substitution:1.100,context:1.400 --pcb substitution:4.100,context:5.800 --score-bias 0 --realign 0 --realign-score-bias -0.2 --realign-max-seqs 2147483647 --corr-score-weight 0 --gap-open aa:11,nucl:5 --gap-extend aa:1,nucl:2 --zdrop 40 --alph-size aa:13,nucl:5 --kmer-per-seq 21 --spaced-kmer-mode 1 --kmer-per-seq-scale aa:0.000,nucl:0.200 --adjust-kmer-len 0 --mask 0 --mask-prob 0.9 --mask-lower-case 0 -k 0 --hash-shift 67 --split-memory-limit 0 --include-only-extendable 0 --ignore-multi-kmer 0 --rescore-mode 0 --filter-hits 0 --sort-results 0 --remove-tmp-files 1 --force-reuse 0

Output:

linclust tmp2/6664721939282577716/input tmp2/6664721939282577716/clu_tmp/1194185237556116514/clu_redundancy tmp2/6664721939282577716/clu_tmp/1194185237556116514/linclust --cluster-mode 0 --max-iterations 1000 --similarity-type 2 --threads 24 --compressed 0 -v 3 --cluster-weight-threshold 0.9 --sub-mat aa:blosum62.out,nucl:nucleotide.out -a 0 --alignment-mode 3 --alignment-output-mode 0 --wrapped-scoring 0 -e 0.001 --min-seq-id 0 --min-aln-len 0 --seq-id-mode 0 --alt-ali 0 -c 0.8 --cov-mode 0 --max-seq-len 65535 --comp-bias-corr 1 --comp-bias-corr-scale 1 --max-rejected 2147483647 --max-accept 2147483647 --add-self-matches 0 --db-load-mode 0 --pca substitution:1.100,context:1.400 --pcb substitution:4.100,context:5.800 --score-bias 0 --realign 0 --realign-score-bias -0.2 --realign-max-seqs 2147483647 --corr-score-weight 0 --gap-open aa:11,nucl:5 --gap-extend aa:1,nucl:2 --zdrop 40 --alph-size aa:13,nucl:5 --kmer-per-seq 21 --spaced-kmer-mode 1 --kmer-per-seq-scale aa:0.000,nucl:0.200 --adjust-kmer-len 0 --mask 0 --mask-prob 0.9 --mask-lower-case 0 -k 0 --hash-shift 67 --split-memory-limit 0 --include-only-extendable 0 --ignore-multi-kmer 0 --rescore-mode 0 --filter-hits 0 --sort-results 0 --remove-tmp-files 1 --force-reuse 0

MMseqs Version:                         6b9388469a41935813f5d207d6d149767feb4c42-MPI
Cluster mode                            0
Max connected component depth           1000
Similarity type                         2
Threads                                 24
Compressed                              0
Verbosity                               3
Weight file name
Cluster Weight threshold                0.9
Substitution matrix                     aa:blosum62.out,nucl:nucleotide.out
Add backtrace                           false
Alignment mode                          3
Alignment mode                          0
Allow wrapped scoring                   false
E-value threshold                       0.001
Seq. id. threshold                      0
Min alignment length                    0
Seq. id. mode                           0
Alternative alignments                  0
Coverage threshold                      0.8
Coverage mode                           0
Max sequence length                     65535
Compositional bias                      1
Compositional bias                      1
Max reject                              2147483647
Max accept                              2147483647
Include identical seq. id.              false
Preload mode                            0
Pseudo count a                          substitution:1.100,context:1.400
Pseudo count b                          substitution:4.100,context:5.800
Score bias                              0
Realign hits                            false
Realign score bias                      -0.2
Realign max seqs                        2147483647
Correlation score weight                0
Gap open cost                           aa:11,nucl:5
Gap extension cost                      aa:1,nucl:2
Zdrop                                   40
Alphabet size                           aa:13,nucl:5
k-mers per sequence                     21
Spaced k-mers                           1
Spaced k-mer pattern
Scale k-mers per sequence               aa:0.000,nucl:0.200
Adjust k-mer length                     false
Mask residues                           0
Mask residues probability               0.9
Mask lower case residues                0
k-mer length                            0
Shift hash                              67
Split memory limit                      0
Include only extendable                 false
Skip repeating k-mers                   false
Rescore mode                            0
Remove hits by seq. id. and coverage    false
Sort results                            0
Remove temporary files                  true
Force restart with latest tmp           false
MPI runner

[][error_sighandler] Caught error: Segmentation fault (signal 11)
tmp2/6664721939282577716/clu_tmp/1194185237556116514/linclust/18444629910076322402/linclust.sh: line 26: 63437 Segmentation fault      $RUNNER "$MMSEQS" kmermatcher "$INPUT" "${TMP_PATH}/pref" ${KMERMATCHER_PAR}
Error: kmermatcher died
milot-mirdita commented 1 year ago

It seems to be crashing very early in the MPI init phase. This happens at the very beginning before anything else is printed. It calls the following and then nothing else MPI related:

MPI_Init(&argc, const_cast<char ***>(&argv));
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &numProc);

What MPI backend does your cluster use? Can you please paste the output of the cmake call? We have only tried OpenMPI.

Rridley7 commented 1 year ago

Output of cmake call:

-- The CXX compiler identification is GNU 10.3.0
-- The C compiler identification is GNU 10.3.0
-- Check for working CXX compiler: /usr/local/pace-apps/spack/packages/linux-rhel7-x86_64/gcc-4.8.5/gcc-10.3.0-o57x6h2gubo7bzh7evmy4mvibdqrlghr/bin/c++
-- Check for working CXX compiler: /usr/local/pace-apps/spack/packages/linux-rhel7-x86_64/gcc-4.8.5/gcc-10.3.0-o57x6h2gubo7bzh7evmy4mvibdqrlghr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working C compiler: /usr/local/pace-apps/spack/packages/linux-rhel7-x86_64/gcc-4.8.5/gcc-10.3.0-o57x6h2gubo7bzh7evmy4mvibdqrlghr/bin/gcc
-- Check for working C compiler: /usr/local/pace-apps/spack/packages/linux-rhel7-x86_64/gcc-4.8.5/gcc-10.3.0-o57x6h2gubo7bzh7evmy4mvibdqrlghr/bin/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Source Directory: /storage/home/hcoda1/6/rridley3/data/dir/mmseqs2
-- Project Directory: /storage/home/hcoda1/6/rridley3/data/dir/mmseqs2
-- Compiler is GNU
-- Performing Test HAVE_AVX2_EXTENSIONS
-- Performing Test HAVE_AVX2_EXTENSIONS - Success
-- Performing Test HAVE_AVX_EXTENSIONS
-- Performing Test HAVE_AVX_EXTENSIONS - Success
-- Performing Test HAVE_SSE4_2_EXTENSIONS
-- Performing Test HAVE_SSE4_2_EXTENSIONS - Success
-- Performing Test HAVE_SSE4_1_EXTENSIONS
-- Performing Test HAVE_SSE4_1_EXTENSIONS - Success
-- Performing Test HAVE_SSE3_EXTENSIONS
-- Performing Test HAVE_SSE3_EXTENSIONS - Success
-- Performing Test HAVE_SSE2_EXTENSIONS
-- Performing Test HAVE_SSE2_EXTENSIONS - Success
-- Performing Test HAVE_SSE_EXTENSIONS
-- Performing Test HAVE_SSE_EXTENSIONS - Success
-- ZSTD VERSION 1.3.8
-- ShellCheck disabled
-- Performing Test HAVE_POSIX_FADVISE
-- Performing Test HAVE_POSIX_FADVISE - Success
-- Performing Test HAVE_POSIX_MADVISE
-- Performing Test HAVE_POSIX_MADVISE - Success
-- Performing Test ATOMIC_LIBRARY_NATIVE
-- Performing Test ATOMIC_LIBRARY_NATIVE - Failed
-- Performing Test ATOMIC_LIBRARY_LIB
-- Performing Test ATOMIC_LIBRARY_LIB - Success
-- Found Atomic: atomic
-- IPS4O sorting works
-- Found ZLIB
-- Performing Test HAVE_ZLIB_CHECK
-- Performing Test HAVE_ZLIB_CHECK - Success
-- ZLIB works
-- Looking for BZ2_bzCompressInit in /usr/lib64/libbz2.so
-- Looking for BZ2_bzCompressInit in /usr/lib64/libbz2.so - found
-- Found BZLIB
-- Performing Test HAVE_BZLIB_CHECK
-- Performing Test HAVE_BZLIB_CHECK - Success
-- BZLIB works
-- Found MPI_C: /usr/local/pace-apps/spack/packages/linux-rhel7-x86_64/gcc-10.3.0/mvapich2-2.3.6-ouywalrqjnakjlhjxgunwqssb3iongrc/lib/libmpi.so
-- Found MPI_CXX: /usr/local/pace-apps/spack/packages/linux-rhel7-x86_64/gcc-10.3.0/mvapich2-2.3.6-ouywalrqjnakjlhjxgunwqssb3iongrc/lib/libmpicxx.so;/usr/local/pace-apps/spack/packages/linux-rhel7-x86_64/gcc-10.3.0/mvapich2-2.3.6-ouywalrqjnakjlhjxgunwqssb3iongrc/lib/libmpi.so
-- Found MPI
-- Try OpenMP C flag = [-fopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Success
-- Try OpenMP CXX flag = [-fopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Success
-- Found OpenMP
-- Configuring done
-- Generating done
-- Build files have been written to: /storage/home/hcoda1/6/rridley3/data/dir/mmseqs2/build-mpi

Output of following make:

Scanning dependencies of target microtar
[  1%] Building C object lib/microtar/CMakeFiles/microtar.dir/microtar.c.o
Linking C static library libmicrotar.a
[  1%] Built target microtar
Scanning dependencies of target cacode
[  2%] Building CXX object lib/cacode/CMakeFiles/cacode.dir/lambda_calculator.cpp.o
[  2%] Building CXX object lib/cacode/CMakeFiles/cacode.dir/lubksb.cpp.o
[  2%] Building CXX object lib/cacode/CMakeFiles/cacode.dir/ludcmp.cpp.o
[  3%] Building CXX object lib/cacode/CMakeFiles/cacode.dir/nrutil.cpp.o
[  3%] Built target cacode
Scanning dependencies of target alp
[  3%] Building CXX object lib/alp/CMakeFiles/alp.dir/njn_dynprogprob.cpp.o
[  3%] Building CXX object lib/alp/CMakeFiles/alp.dir/njn_random.cpp.o
[  4%] Building CXX object lib/alp/CMakeFiles/alp.dir/njn_dynprogprobproto.cpp.o
[  4%] Building CXX object lib/alp/CMakeFiles/alp.dir/njn_localmaxstat.cpp.o
[  4%] Building CXX object lib/alp/CMakeFiles/alp.dir/njn_localmaxstatmatrix.cpp.o
[  5%] Building CXX object lib/alp/CMakeFiles/alp.dir/sls_alp_sim.cpp.o
[  5%] Building CXX object lib/alp/CMakeFiles/alp.dir/sls_pvalues.cpp.o
[  6%] Building CXX object lib/alp/CMakeFiles/alp.dir/njn_dynprogproblim.cpp.o
[  6%] Building CXX object lib/alp/CMakeFiles/alp.dir/sls_alignment_evaluer.cpp.o
[  6%] Building CXX object lib/alp/CMakeFiles/alp.dir/sls_alp_regression.cpp.o
[  7%] Building CXX object lib/alp/CMakeFiles/alp.dir/sls_alp.cpp.o
[  7%] Building CXX object lib/alp/CMakeFiles/alp.dir/njn_localmaxstatutil.cpp.o
[  7%] Building CXX object lib/alp/CMakeFiles/alp.dir/sls_basic.cpp.o
[  8%] Building CXX object lib/alp/CMakeFiles/alp.dir/sls_alp_data.cpp.o
[  8%] Building CXX object lib/alp/CMakeFiles/alp.dir/njn_ioutil.cpp.o
[  8%] Built target alp
Scanning dependencies of target ksw2
[  8%] Building CXX object lib/ksw2/CMakeFiles/ksw2.dir/ksw2_extz2_sse.cpp.o
[  8%] Built target ksw2
Scanning dependencies of target generated
[  8%] Generating ../generated/VTML80.out.h
[  9%] Generating ../generated/VTML40.out.h
[  9%] Generating ../generated/nucleotide.out.h
[  9%] Generating ../generated/blosum62.out.h
[ 10%] Generating ../generated/PAM30.out.h
[ 10%] Generating ../generated/CovSeqidQscPercMinDiag.lib.h
[ 10%] Generating ../generated/CovSeqidQscPercMinDiagTargetCov.lib.h
[ 11%] Generating ../generated/ExpOpt3_8_polished.cs32.lib.h
[ 11%] Generating ../generated/Library255_may17.lib.h
[ 12%] Generating ../generated/libPolished_8.lib.h
[ 12%] Generating ../generated/cs219.lib.h
[ 12%] Generating ../generated/krona_prelude.html.h
[ 13%] Generating ../generated/result_viz_prelude.html.zst.h
[ 13%] Generating ../generated/K4000.crf.h
[ 13%] Generating ../generated/easysearch.sh.h
[ 14%] Generating ../generated/easycluster.sh.h
[ 14%] Generating ../generated/easytaxonomy.sh.h
[ 15%] Generating ../generated/easyrbh.sh.h
[ 15%] Generating ../generated/blastp.sh.h
[ 15%] Generating ../generated/blastpgp.sh.h
[ 16%] Generating ../generated/map.sh.h
[ 16%] Generating ../generated/rbh.sh.h
[ 17%] Generating ../generated/linclust.sh.h
[ 17%] Generating ../generated/clustering.sh.h
[ 17%] Generating ../generated/cascaded_clustering.sh.h
[ 18%] Generating ../generated/update_clustering.sh.h
[ 18%] Generating ../generated/searchtargetprofile.sh.h
[ 18%] Generating ../generated/createindex.sh.h
[ 19%] Generating ../generated/createtaxdb.sh.h
[ 19%] Generating ../generated/translated_search.sh.h
[ 20%] Generating ../generated/taxpercontig.sh.h
[ 20%] Generating ../generated/multihitdb.sh.h
[ 20%] Generating ../generated/multihitsearch.sh.h
[ 21%] Generating ../generated/enrich.sh.h
[ 21%] Generating ../generated/blastn.sh.h
[ 21%] Generating ../generated/searchslicedtargetprofile.sh.h
[ 22%] Generating ../generated/taxonomy.sh.h
[ 22%] Generating ../generated/linsearch.sh.h
[ 23%] Generating ../generated/databases.sh.h
[ 23%] Generating ../generated/nucleotide_clustering.sh.h
[ 23%] Generating ../generated/iterativepp.sh.h
[ 24%] Generating ../generated/tsv2exprofiledb.sh.h
[ 24%] Built target generated
Scanning dependencies of target version
[ 25%] Building CXX object src/version/CMakeFiles/version.dir/Version.cpp.o
Linking CXX static library libversion.a
[ 25%] Built target version
Scanning dependencies of target libzstd_static
[ 25%] Building C object lib/zstd/build/cmake/lib/CMakeFiles/libzstd_static.dir/__/__/__/lib/common/entropy_common.c.o
[ 25%] Building C object lib/zstd/build/cmake/lib/CMakeFiles/libzstd_static.dir/__/__/__/lib/common/fse_decompress.c.o
[ 26%] Building C object lib/zstd/build/cmake/lib/CMakeFiles/libzstd_static.dir/__/__/__/lib/common/threading.c.o
[ 26%] Building C object lib/zstd/build/cmake/lib/CMakeFiles/libzstd_static.dir/__/__/__/lib/common/pool.c.o
[ 26%] Building C object lib/zstd/build/cmake/lib/CMakeFiles/libzstd_static.dir/__/__/__/lib/common/zstd_common.c.o
[ 27%] Building C object lib/zstd/build/cmake/lib/CMakeFiles/libzstd_static.dir/__/__/__/lib/common/error_private.c.o
[ 27%] Building C object lib/zstd/build/cmake/lib/CMakeFiles/libzstd_static.dir/__/__/__/lib/common/xxhash.c.o
[ 28%] Building C object lib/zstd/build/cmake/lib/CMakeFiles/libzstd_static.dir/__/__/__/lib/compress/hist.c.o
[ 28%] Building C object lib/zstd/build/cmake/lib/CMakeFiles/libzstd_static.dir/__/__/__/lib/compress/fse_compress.c.o
[ 28%] Building C object lib/zstd/build/cmake/lib/CMakeFiles/libzstd_static.dir/__/__/__/lib/compress/huf_compress.c.o
[ 29%] Building C object lib/zstd/build/cmake/lib/CMakeFiles/libzstd_static.dir/__/__/__/lib/compress/zstd_compress.c.o
[ 29%] Building C object lib/zstd/build/cmake/lib/CMakeFiles/libzstd_static.dir/__/__/__/lib/compress/zstdmt_compress.c.o
[ 30%] Building C object lib/zstd/build/cmake/lib/CMakeFiles/libzstd_static.dir/__/__/__/lib/compress/zstd_fast.c.o
[ 30%] Building C object lib/zstd/build/cmake/lib/CMakeFiles/libzstd_static.dir/__/__/__/lib/compress/zstd_double_fast.c.o
[ 30%] Building C object lib/zstd/build/cmake/lib/CMakeFiles/libzstd_static.dir/__/__/__/lib/compress/zstd_lazy.c.o
[ 31%] Building C object lib/zstd/build/cmake/lib/CMakeFiles/libzstd_static.dir/__/__/__/lib/compress/zstd_opt.c.o
[ 31%] Building C object lib/zstd/build/cmake/lib/CMakeFiles/libzstd_static.dir/__/__/__/lib/compress/zstd_ldm.c.o
[ 31%] Building C object lib/zstd/build/cmake/lib/CMakeFiles/libzstd_static.dir/__/__/__/lib/decompress/huf_decompress.c.o
[ 32%] Building C object lib/zstd/build/cmake/lib/CMakeFiles/libzstd_static.dir/__/__/__/lib/decompress/zstd_decompress.c.o
[ 32%] Building C object lib/zstd/build/cmake/lib/CMakeFiles/libzstd_static.dir/__/__/__/lib/decompress/zstd_decompress_block.c.o
[ 33%] Building C object lib/zstd/build/cmake/lib/CMakeFiles/libzstd_static.dir/__/__/__/lib/decompress/zstd_ddict.c.o
[ 33%] Building C object lib/zstd/build/cmake/lib/CMakeFiles/libzstd_static.dir/__/__/__/lib/dictBuilder/cover.c.o
[ 33%] Building C object lib/zstd/build/cmake/lib/CMakeFiles/libzstd_static.dir/__/__/__/lib/dictBuilder/fastcover.c.o
[ 34%] Building C object lib/zstd/build/cmake/lib/CMakeFiles/libzstd_static.dir/__/__/__/lib/dictBuilder/divsufsort.c.o
[ 34%] Building C object lib/zstd/build/cmake/lib/CMakeFiles/libzstd_static.dir/__/__/__/lib/dictBuilder/zdict.c.o
[ 34%] Building C object lib/zstd/build/cmake/lib/CMakeFiles/libzstd_static.dir/__/__/__/lib/deprecated/zbuff_common.c.o
[ 35%] Building C object lib/zstd/build/cmake/lib/CMakeFiles/libzstd_static.dir/__/__/__/lib/deprecated/zbuff_compress.c.o
[ 35%] Building C object lib/zstd/build/cmake/lib/CMakeFiles/libzstd_static.dir/__/__/__/lib/deprecated/zbuff_decompress.c.o
Linking C static library libzstd.a
[ 35%] Built target libzstd_static
Scanning dependencies of target tinyexpr
[ 35%] Building C object lib/tinyexpr/CMakeFiles/tinyexpr.dir/tinyexpr.c.o
Linking C static library libtinyexpr.a
[ 35%] Built target tinyexpr
Scanning dependencies of target mmseqs-framework
[ 35%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/alignment/Alignment.cpp.o
[ 36%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/alignment/CompressedA3M.cpp.o
[ 36%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/alignment/Main.cpp.o
[ 37%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/alignment/Matcher.cpp.o
[ 37%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/alignment/MsaFilter.cpp.o
[ 37%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/alignment/MultipleAlignment.cpp.o
[ 38%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/alignment/PSSMCalculator.cpp.o
[ 38%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/alignment/StripedSmithWaterman.cpp.o
[ 38%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/alignment/BandedNucleotideAligner.cpp.o
[ 39%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/alignment/rescorediagonal.cpp.o
[ 39%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/clustering/Main.cpp.o
[ 40%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/clustering/AlignmentSymmetry.cpp.o
[ 40%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/clustering/Clustering.cpp.o
[ 40%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/clustering/ClusteringAlgorithms.cpp.o
[ 41%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/commons/CSProfile.cpp.o
[ 41%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/commons/A3MReader.cpp.o
[ 41%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/commons/Application.cpp.o
[ 42%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/commons/BaseMatrix.cpp.o
[ 42%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/commons/Command.cpp.o
[ 43%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/commons/CommandCaller.cpp.o
[ 43%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/commons/DBConcat.cpp.o
[ 43%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/commons/DBReader.cpp.o
[ 44%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/commons/DBWriter.cpp.o
[ 44%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/commons/Debug.cpp.o
[ 45%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/commons/ExpressionParser.cpp.o
[ 45%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/commons/FileUtil.cpp.o
[ 45%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/commons/HeaderSummarizer.cpp.o
[ 46%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/commons/KSeqWrapper.cpp.o
[ 46%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/commons/MemoryMapped.cpp.o
[ 46%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/commons/MemoryTracker.cpp.o
[ 47%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/commons/MMseqsMPI.cpp.o
[ 47%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/commons/MultiParam.cpp.o
[ 48%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/commons/NucleotideMatrix.cpp.o
[ 48%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/commons/Orf.cpp.o
[ 48%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/commons/Parameters.cpp.o
[ 49%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/commons/ProfileStates.cpp.o
[ 49%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/commons/LibraryReader.cpp.o
[ 50%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/commons/Sequence.cpp.o
[ 50%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/commons/SequenceWeights.cpp.o
[ 50%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/commons/SubstitutionMatrix.cpp.o
[ 51%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/commons/tantan.cpp.o
[ 51%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/commons/UniprotKB.cpp.o
[ 51%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/commons/Util.cpp.o
[ 52%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/prefiltering/CacheFriendlyOperations.cpp.o
[ 52%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/prefiltering/ExtendedSubstitutionMatrix.cpp.o
[ 53%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/prefiltering/Indexer.cpp.o
[ 53%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/prefiltering/IndexBuilder.cpp.o
[ 53%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/prefiltering/KmerGenerator.cpp.o
[ 54%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/prefiltering/Main.cpp.o
[ 54%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/prefiltering/Prefiltering.cpp.o
[ 54%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/prefiltering/PrefilteringIndexReader.cpp.o
[ 55%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/prefiltering/QueryMatcher.cpp.o
[ 55%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/prefiltering/ReducedMatrix.cpp.o
[ 56%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/prefiltering/SequenceLookup.cpp.o
[ 56%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/prefiltering/UngappedAlignment.cpp.o
[ 56%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/prefiltering/ungappedprefilter.cpp.o
[ 57%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/multihit/besthitperset.cpp.o
[ 57%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/multihit/combinepvalperset.cpp.o
[ 58%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/multihit/Aggregation.cpp.o
[ 58%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/multihit/MultiHitDb.cpp.o
[ 58%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/multihit/MultiHitSearch.cpp.o
[ 59%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/taxonomy/lca.cpp.o
[ 59%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/taxonomy/addtaxonomy.cpp.o
[ 59%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/taxonomy/NcbiTaxonomy.cpp.o
[ 60%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/taxonomy/filtertaxdb.cpp.o
[ 60%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/taxonomy/filtertaxseqdb.cpp.o
[ 61%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/taxonomy/aggregatetax.cpp.o
[ 61%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/taxonomy/createtaxdb.cpp.o
[ 61%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/taxonomy/createbintaxonomy.cpp.o
[ 62%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/taxonomy/createbintaxmapping.cpp.o
[ 62%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/taxonomy/taxonomyreport.cpp.o
[ 62%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/linclust/kmermatcher.cpp.o
[ 63%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/linclust/kmerindexdb.cpp.o
[ 63%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/linclust/kmersearch.cpp.o
[ 64%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/linclust/LinsearchIndexReader.cpp.o
[ 64%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/alignall.cpp.o
[ 64%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/alignbykmer.cpp.o
[ 65%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/appenddbtoindex.cpp.o
[ 65%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/apply.cpp.o
[ 66%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/clusthash.cpp.o
[ 66%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/compress.cpp.o
[ 66%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/convert2fasta.cpp.o
[ 67%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/convertalignments.cpp.o
[ 67%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/convertca3m.cpp.o
[ 67%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/convertkb.cpp.o
[ 68%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/convertmsa.cpp.o
[ 68%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/convertprofiledb.cpp.o
[ 69%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/createdb.cpp.o
[ 69%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/dbtype.cpp.o
[ 69%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/db2tar.cpp.o
[ 70%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/indexdb.cpp.o
[ 70%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/offsetalignment.cpp.o
[ 70%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/createseqfiledb.cpp.o
[ 71%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/createsubdb.cpp.o
[ 71%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/view.cpp.o
[ 72%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/createtsv.cpp.o
[ 72%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/diffseqdbs.cpp.o
[ 72%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/expandaln.cpp.o
[ 73%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/extractalignedregion.cpp.o
[ 73%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/extractdomains.cpp.o
[ 74%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/extractorfs.cpp.o
[ 74%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/orftocontig.cpp.o
[ 74%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/touchdb.cpp.o
[ 75%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/filterdb.cpp.o
[ 75%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/gff2db.cpp.o
[ 75%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/renamedbkeys.cpp.o
[ 76%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/masksequence.cpp.o
[ 76%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/maskbygff.cpp.o
[ 77%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/mergeclusters.cpp.o
[ 77%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/mergeresultsbyset.cpp.o
[ 77%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/mergedbs.cpp.o
[ 78%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/msa2profile.cpp.o
[ 78%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/msa2result.cpp.o
[ 79%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/nrtotaxmapping.cpp.o
[ 79%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/countkmer.cpp.o
[ 79%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/pairaln.cpp.o
[ 80%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/prefixid.cpp.o
[ 80%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/profile2pssm.cpp.o
[ 80%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/profile2neff.cpp.o
[ 81%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/profile2seq.cpp.o
[ 81%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/result2dnamsa.cpp.o
[ 82%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/result2flat.cpp.o
[ 82%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/result2msa.cpp.o
[ 82%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/result2rbh.cpp.o
[ 83%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/result2profile.cpp.o
[ 83%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/result2repseq.cpp.o
[ 83%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/result2stats.cpp.o
[ 84%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/reverseseq.cpp.o
[ 84%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/cpmvrmlndb.cpp.o
[ 85%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/extractframes.cpp.o
[ 85%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/sequence2profile.cpp.o
[ 85%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/sortresult.cpp.o
[ 86%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/splitdb.cpp.o
[ 86%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/splitsequence.cpp.o
[ 87%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/subtractdbs.cpp.o
[ 87%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/summarizealis.cpp.o
[ 87%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/summarizeheaders.cpp.o
[ 88%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/summarizeresult.cpp.o
[ 88%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/summarizetabs.cpp.o
[ 88%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/swapresults.cpp.o
[ 89%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/transitivealign.cpp.o
[ 89%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/translatenucs.cpp.o
[ 90%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/translateaa.cpp.o
[ 90%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/tsv2db.cpp.o
[ 90%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/tsv2exprofiledb.cpp.o
[ 91%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/tar2db.cpp.o
[ 91%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/unpackdb.cpp.o
[ 91%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/proteinaln2nucl.cpp.o
[ 92%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/versionstring.cpp.o
[ 92%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/util/diskspaceavail.cpp.o
[ 93%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/workflow/Cluster.cpp.o
[ 93%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/workflow/ClusterUpdate.cpp.o
[ 93%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/workflow/Databases.cpp.o
[ 94%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/workflow/Linclust.cpp.o
[ 94%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/workflow/EasySearch.cpp.o
[ 95%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/workflow/EasyRbh.cpp.o
[ 95%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/workflow/EasyCluster.cpp.o
[ 95%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/workflow/EasyLinclust.cpp.o
[ 96%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/workflow/Enrich.cpp.o
[ 96%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/workflow/Linsearch.cpp.o
[ 96%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/workflow/Map.cpp.o
[ 97%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/workflow/Rbh.cpp.o
[ 97%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/workflow/Search.cpp.o
[ 98%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/workflow/Taxonomy.cpp.o
[ 98%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/workflow/EasyTaxonomy.cpp.o
[ 98%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/workflow/CreateIndex.cpp.o
[100%] Building CXX object src/CMakeFiles/mmseqs-framework.dir/MMseqsBase.cpp.o
Linking CXX static library libmmseqs-framework.a
[100%] Built target mmseqs-framework
Scanning dependencies of target mmseqs
[100%] Building CXX object src/CMakeFiles/mmseqs.dir/mmseqs.cpp.o
Linking CXX executable mmseqs
[100%] Built target mmseqs
Rridley7 commented 1 year ago

Our MPI backend I believe is mvapich2, I'm not the most familiar with this end, let me know if this is what you'd be referring to.

milot-mirdita commented 1 year ago

Yes, exactly. We've never tried running MMseqs2 on mvapich2 and apparently you are the first user facing issues with it. (At least the first who told us about it).

Do you have GDB installed? Can you call the following:

gdb --args ~/data/dir/mmseqs2/build-mpi/bin/mmseqs kmermatcher
# wait for a prompt to appear then type
r 
# wait for it to crash then type
bt
# copy paste the output please

You might have to recompile as a Debug build to get more sensible output here: cmake -DHAVE_MPI=1 -DCMAKE_INSTALL_PREFIX=. -DCMAKE_BUILD_TYPE=Debug ..

milot-mirdita commented 1 year ago

Wait sorry, kmermatcher will need some input so it actually gets to the init function.

You can do:

~/data/dir/mmseqs2/build-mpi/bin/mmseqs createdb examples/QUERY.fasta qdb

Then:


gdb --args ~/data/dir/mmseqs2/build-mpi/bin/mmseqs kmermatcher qdb foo
# wait for a prompt to appear then type
r 
# wait for it to crash then type
bt
# copy paste the output please
Rridley7 commented 1 year ago

Okay, I did some looking and found that our cluster is capable of also running openmpi instead of mvapich2. Unsurpisingly, after loading this and recompiling the program, it works as you said. Thank you!

Fyi if you all are looking to debug for the mvapich2 system, this is the error that comes after running those commands on this system (happy to test any issues/fixes here if it's useful to you):

(gdb) bt
#0  0x00007ffff7186abc in MPID_Abort () from /usr/local/pace-apps/spack/packages/linux-rhel7-x86_64/gcc-10.3.0/mvapich2-2.3.6-ouywalrqjnakjlhjxgunwqssb3iongrc/lib/libmpi.so.12
#1  0x00007ffff71419db in MPIR_Handle_fatal_error () from /usr/local/pace-apps/spack/packages/linux-rhel7-x86_64/gcc-10.3.0/mvapich2-2.3.6-ouywalrqjnakjlhjxgunwqssb3iongrc/lib/libmpi.so.12
#2  0x00007ffff7141b59 in MPIR_Err_return_comm () from /usr/local/pace-apps/spack/packages/linux-rhel7-x86_64/gcc-10.3.0/mvapich2-2.3.6-ouywalrqjnakjlhjxgunwqssb3iongrc/lib/libmpi.so.12
#3  0x00007ffff70fa699 in PMPI_Init () from /usr/local/pace-apps/spack/packages/linux-rhel7-x86_64/gcc-10.3.0/mvapich2-2.3.6-ouywalrqjnakjlhjxgunwqssb3iongrc/lib/libmpi.so.12
#4  0x0000000000479714 in MMseqsMPI::init(int, char const**) ()
#5  0x00000000005d44a1 in kmermatcher(int, char const**, Command const&) ()
#6  0x000000000046f2f9 in runCommand(Command*, int, char const**) ()
#7  0x000000000045af9b in main ()
milot-mirdita commented 1 year ago

Great! I don't think that I'll look closer into this right now. If someone else faces issues with mvapich2 in the future I might revisit this.