Segmentation Fault in result2profile during search

kcibul-aeratx commented 2 years ago

Expected Behavior

Not crashing, with a better error message if this is a user problem

Current Behavior

Seg Fault. 90% of profile searches are fine against this database, but others segfault. example below

Steps to Reproduce (for bugs)

search profiles/CLUSTER_100005_profile /mnt/disks/kc-ssd/GURARB/GURARB_cluster75_reps individual_results/profiles/CLUSTER_100005_profile /tmp --num-iterations 2 --db-load-mode 2

MMseqs Output (for bugs)

search profiles/CLUSTER_100005_profile /mnt/disks/kc-ssd/GURARB/GURARB_cluster75_reps individual_results/profiles/CLUSTER_100005_profile /tmp --num-iterations 2 --db-load-mode 2

MMseqs Version:                         67949d702dbfc6e5d54fdd0f14a9ab6740f11c32
Substitution matrix                     aa:blosum62.out,nucl:nucleotide.out
Add backtrace                           true
Alignment mode                          2
Alignment mode                          0
Allow wrapped scoring                   false
E-value threshold                       0.001
Seq. id. threshold                      0
Min alignment length                    0
Seq. id. mode                           0
Alternative alignments                  0
Coverage threshold                      0
Coverage mode                           0
Max sequence length                     65535
Compositional bias                      1
Compositional bias                      1
Max reject                              2147483647
Max accept                              2147483647
Include identical seq. id.              false
Preload mode                            2
Pseudo count a                          substitution:1.100,context:1.400
Pseudo count b                          substitution:4.100,context:5.800
Score bias                              0
Realign hits                            false
Realign score bias                      -0.2
Realign max seqs                        2147483647
Correlation score weight                0
Gap open cost                           aa:11,nucl:5
Gap extension cost                      aa:1,nucl:2
Zdrop                                   40
Threads                                 32
Compressed                              0
Verbosity                               3
Seed substitution matrix                aa:VTML80.out,nucl:nucleotide.out
Sensitivity                             5.7
k-mer length                            0
k-score                                 seq:2147483647,prof:2147483647
Alphabet size                           aa:21,nucl:5
Max results per query                   300
Split database                          0
Split mode                              2
Split memory limit                      0
Diagonal scoring                        true
Exact k-mer matching                    0
Mask residues                           1
Mask residues probability               0.9
Mask lower case residues                0
Minimum diagonal score                  15
Selected taxa
Spaced k-mers                           1
Spaced k-mer pattern
Local temporary path
Rescore mode                            0
Remove hits by seq. id. and coverage    false
Sort results                            0
Mask profile                            1
Profile E-value threshold               0.1
Global sequence weighting               false
Allow deletions                         false
Filter MSA                              1
Use filter only at N seqs               0
Maximum seq. id. threshold              0.9
Minimum seq. id.                        0.0
Minimum score per column                -20
Minimum coverage                        0
Select N most diverse seqs              1000
Pseudo count mode                       0
Gap pseudo count                        10
Min codons in orf                       30
Max codons in length                    32734
Max orf gaps                            2147483647
Contig start mode                       2
Contig end mode                         2
Orf start mode                          1
Forward frames                          1,2,3
Reverse frames                          1,2,3
Translation table                       1
Translate orf                           0
Use all table starts                    false
Offset of numeric ids                   0
Create lookup                           0
Add orf stop                            false
Overlap between sequences               0
Sequence split mode                     1
Header split mode                       0
Chain overlapping alignments            0
Merge query                             1
Search type                             0
Search iterations                       2
Start sensitivity                       4
Search steps                            1
Exhaustive search mode                  false
Filter results during exhaustive search 0
Strand selection                        1
LCA search mode                         false
Disk space limit                        0
MPI runner
Force restart with latest tmp           false
Remove temporary files                  false

prefilter profiles/CLUSTER_100005_profile /mnt/disks/kc-ssd/GURARB/GURARB_cluster75_reps.idx /tmp/15358673946673749737/pref_0 --sub-mat 'aa:blosum62.out,nucl:nucleotide.out' --seed-sub-mat 'aa:VTML80.out,nucl:nucleotide.out' -s 5.7 -k 0 --k-score seq:2147483647,prof:2147483647 --alph-size aa:21,nucl:5 --max-seq-len 65535 --max-seqs 300 --split 0 --split-mode 2 --split-memory-limit 0 -c 0 --cov-mode 0 --comp-bias-corr 1 --comp-bias-corr-scale 1 --diag-score 1 --exact-kmer-matching 0 --mask 1 --mask-prob 0.9 --mask-lower-case 0 --min-ungapped-score 15 --add-self-matches 0 --spaced-kmer-mode 1 --db-load-mode 2 --pca substitution:1.100,context:1.400 --pcb substitution:4.100,context:5.800 --threads 32 --compressed 0 -v 3

Index version: 16
Generated by:  13.45111
ScoreMatrix:  VTML80.out
Query database size: 1 type: Profile
Estimated memory consumption: 127G
Target database size: 34746192 type: Aminoacid
Process prefiltering step 1 of 1

k-mer similarity threshold: 110
Starting prefiltering scores calculation (step 1 of 1)
Query db start 1 to 1
Target db start 1 to 34746192
[=================================================================] 1 0s 1ms

5111.040161 k-mers per position
5884143 DB matches per sequence
0 overflows
0 queries produce too many hits (truncated result)
300 sequences passed prefiltering per query sequence
300 median result list length
0 sequences with 0 size result lists
Time for merging to pref_0: 0h 0m 0s 0ms
Time for processing: 0h 0m 1s 434ms
align profiles/CLUSTER_100005_profile /mnt/disks/kc-ssd/GURARB/GURARB_cluster75_reps.idx /tmp/15358673946673749737/pref_0 /tmp/15358673946673749737/aln_0 --sub-mat 'aa:blosum62.out,nucl:nucleotide.out' -a 1 --alignment-mode 2 --alignment-output-mode 0 --wrapped-scoring 0 -e 0.001 --min-seq-id 0 --min-aln-len 0 --seq-id-mode 0 --alt-ali 0 -c 0 --cov-mode 0 --max-seq-len 65535 --comp-bias-corr 1 --comp-bias-corr-scale 1 --max-rejected 2147483647 --max-accept 2147483647 --add-self-matches 0 --db-load-mode 2 --pca substitution:1.100,context:1.400 --pcb substitution:4.100,context:5.800 --score-bias 0 --realign 1 --realign-score-bias -0.2 --realign-max-seqs 2147483647 --corr-score-weight 0 --gap-open aa:11,nucl:5 --gap-extend aa:1,nucl:2 --zdrop 40 --threads 32 --compressed 0 -v 3

Index version: 16
Generated by:  13.45111
ScoreMatrix:  VTML80.out
Compute score only
Query database size: 1 type: Profile
Target database size: 34746192 type: Aminoacid
Calculation of alignments
[=================================================================] 1 0s 8ms
Time for merging to aln_0: 0h 0m 0s 0ms
300 alignments calculated
191 sequence pairs passed the thresholds (0.636667 of overall calculated)
191.000000 hits per query sequence
Time for processing: 0h 0m 0s 69ms
result2profile profiles/CLUSTER_100005_profile /mnt/disks/kc-ssd/GURARB/GURARB_cluster75_reps.idx /tmp/15358673946673749737/aln_0 /tmp/15358673946673749737/profile_0 --sub-mat 'aa:blosum62.out,nucl:nucleotide.out' -e 0.001 --mask-profile 1 --e-profile 0.1 --comp-bias-corr 1 --comp-bias-corr-scale 1 --wg 0 --allow-deletion 0 --filter-msa 1 --filter-min-enable 0 --max-seq-id 0.9 --qid '0.0' --qsc -20 --cov 0 --diff 1000 --pseudo-cnt-mode 0 --pca substitution:1.100,context:1.400 --pcb substitution:4.100,context:5.800 --db-load-mode 2 --gap-open aa:11,nucl:5 --gap-extend aa:1,nucl:2 --gap-pc 10 --threads 32 --compressed 0 -v 3

Index version: 16
Generated by:  13.45111
ScoreMatrix:  VTML80.out
Query database size: 1 type: Profile
Target database size: 34746192 type: Aminoacid
[=================================================================Segmentation fault
Error: Create profile died

Context

Your Environment

Include as many relevant details about the environment you experienced the bug in.

Git commit used: 67949d702dbfc6e5d54fdd0f14a9ab6740f11c32
Which MMseqs version was used:Statically-compiled
For self-compiled and Homebrew: Compiler and Cmake versions used and their invocation:
Server specifications (especially CPU support for AVX2/SSE and amount of system memory): 32-core, 256 GB ram, vmtouch'ed target into memory (crash happens either way though)
Operating system and version: Debian GNU/Linux 11 (bullseye)

kcibul-aeratx commented 2 years ago

I ran the final command (result2profile) in gdb with

gdb --args /mnt/disks/kc-ssd/MMseqs2/build/bin/mmseqs result2profile profiles/CLUSTER_100005_profile /mnt/disks/kc-ssd/GURARB/GURARB_cluster75_reps.idx /tmp/15358673946673749737/aln_0 /tmp/15358673946673749737/profile_0 --sub-mat 'aa:blosum62.out,nucl:nucleotide.out' -e 0.001 --mask-profile 1 --e-profile 0.1 --comp-bias-corr 1 --comp-bias-corr-scale 1 --wg 0 --allow-deletion 0 --filter-msa 1 --filter-min-enable 0 --max-seq-id 0.9 --qid '0.0' --qsc -20 --cov 0 --diff 1000 --pseudo-cnt-mode 0 --pca substitution:1.100,context:1.400 --pcb substitution:4.100,context:5.800 --db-load-mode 2 --gap-open aa:11,nucl:5 --gap-extend aa:1,nucl:2 --gap-pc 10 --threads 32 --compressed 0 -v 3

and got the following backtrace via bt

(gdb) #0  0x0000555555a8d42b in tantan::Tantan::calcForwardTransitionAndEmissionProbs() ()
#1  0x0000555555a8d865 in tantan::Tantan::calcRepeatProbs(float*) ()
#2  0x0000555555a8bf8b in tantan::getProbabilities(char const*, char const*, int, double const* const*, double, double, double, double, double, float*) ()
#3  0x0000555555a8be7f in tantan::maskSequences(char*, char*, int, double const* const*, double, double, double, double, double, double, char const*) ()
#4  0x000055555597de9b in PSSMMasker::mask(Sequence&, float, PSSMCalculator::Profile&) ()
#5  0x00005555559f918b in result2profile(int, char const**, Command const&, bool) [clone ._omp_fn.0] ()
#6  0x00007ffff7c7c4c2 in GOMP_parallel () from /lib/x86_64-linux-gnu/libgomp.so.1
#7  0x00005555559f86d7 in result2profile(int, char const**, Command const&, bool) ()
#8  0x00005555559f887c in result2profile(int, char const**, Command const&) ()
#9  0x00005555557ea0d5 in runCommand(Command*, int, char const**) ()
#10 0x00005555557eb00a in main ()

kcibul-aeratx commented 2 years ago

Recompiled in Debug mode, more details

Thread 1 "mmseqs" received signal SIGSEGV, Segmentation fault.
0x0000555555a8d40b in tantan::Tantan::calcForwardTransitionAndEmissionProbs (this=0x7ffffffbaaa0)
    at /mnt/disks/kc-ssd/MMseqs2/src/commons/tantan.cpp:319
319                 foregroundBeg[i] = (b * b2fProbs[i] + f * f2f0) * lrRow[(int)seqPtr[-i-1]];
(gdb) bt
#0  0x0000555555a8d40b in tantan::Tantan::calcForwardTransitionAndEmissionProbs (this=0x7ffffffbaaa0)
    at /mnt/disks/kc-ssd/MMseqs2/src/commons/tantan.cpp:319
#1  0x0000555555a8d845 in tantan::Tantan::calcRepeatProbs (this=0x7ffffffbaaa0, letterProbs=0x555556582704)
    at /mnt/disks/kc-ssd/MMseqs2/src/commons/tantan.cpp:373
#2  0x0000555555a8bf6b in tantan::getProbabilities (
    seqBeg=0x5555564e06c0 "\025\025\025\025\025\025\025\n\f\003\v\020\f\016\020\020\016\017", seqEnd=0x5555564e07b9 "",
    maxRepeatOffset=50, likelihoodRatioMatrix=0x55555643d8f0, repeatProb=0.0050000000000000001, repeatEndProb=0.050000000000000003,
    repeatOffsetProbDecay=0.90000000000000002, firstGapProb=0, otherGapProb=0, probabilities=0x555556582700)
    at /mnt/disks/kc-ssd/MMseqs2/src/commons/tantan.cpp:470
#3  0x0000555555a8be5f in tantan::maskSequences (
    seqBeg=0x5555564e06c0 "\025\025\025\025\025\025\025\n\f\003\v\020\f\016\020\020\016\017", seqEnd=0x5555564e07b9 "",
    maxRepeatOffset=50, likelihoodRatioMatrix=0x55555643d8f0, repeatProb=0.0050000000000000001, repeatEndProb=0.050000000000000003,
    repeatOffsetProbDecay=0.90000000000000002, firstGapProb=0, otherGapProb=0, minMaskProb=0.89999997615814209,
    maskTable=0x7ffffffdc530 '\024' <repeats 200 times>...) at /mnt/disks/kc-ssd/MMseqs2/src/commons/tantan.cpp:449
#4  0x000055555597de7b in PSSMMasker::mask (this=0x7ffffffbaca0, centerSequence=..., maskProb=0.899999976, pssmRes=...)
    at /mnt/disks/kc-ssd/MMseqs2/src/alignment/PSSMMasker.h:25
#5  0x00005555559f916b in result2profile () at /mnt/disks/kc-ssd/MMseqs2/src/util/result2profile.cpp:262
#6  0x00007ffff7c7c4c2 in GOMP_parallel () from /lib/x86_64-linux-gnu/libgomp.so.1
#7  0x00005555559f86b7 in result2profile (argc=54, argv=0x7fffffffe138, command=..., returnAlnRes=false)
    at /mnt/disks/kc-ssd/MMseqs2/src/util/result2profile.cpp:140
#8  0x00005555559f885c in result2profile (argc=54, argv=0x7fffffffe138, command=...)
    at /mnt/disks/kc-ssd/MMseqs2/src/util/result2profile.cpp:311
#9  0x00005555557ea0b5 in runCommand (p=0x55555641eac0, argc=54, argv=0x7fffffffe138)
    at /mnt/disks/kc-ssd/MMseqs2/src/commons/Application.cpp:40
#10 0x00005555557eafea in main (argc=56, argv=0x7fffffffe128) at /mnt/disks/kc-ssd/MMseqs2/src/commons/Application.cpp:203

milot-mirdita commented 2 years ago

Can you try to remove the precomputed index? It seems to have been generated by a different version than the one you use to run.

You can also set the MMSEQS_IGNORE_INDEX environment variable to ignore the precomputed index during the run.

Would it be possible to upload the query/target set that is crashing?

soedinglab / MMseqs2