steineggerlab / foldseek

Foldseek enables fast and sensitive comparisons of large structure sets.
https://foldseek.com
GNU General Public License v3.0
775 stars 99 forks source link

createdb with ProstT5 on gpu fails: Error loading ProstT5: DriverError(CUDA_ERROR_INVALID_PTX, "a PTX JIT compilation failed") when loading cast_f32_f16 #313

Closed HaimAshk closed 1 month ago

HaimAshk commented 1 month ago

Expected Behavior

Current Behavior

When using the ProstT5 model and --gpu 1 I get the error: Error loading ProstT5: DriverError(CUDA_ERROR_INVALID_PTX, "a PTX JIT compilation failed") when loading cast_f32_f16 The same command runs OK on CPU

Foldssek Output (for bugs)

foldseek easy-search test_arabidopsis_alphafold.fas Arabidopsis_thaliana_taxid_3702_AlphaFold_v3.AlphaFold.foldseekdb test_arabidopsis_alphafold.result.m8 /tmp/hashkenazy/foldseek_Prost5/tmp --prostt5-model /ebio/scratch/hashkenazy/foldseek_9_Prost5_model/weights --format-output query,target,evalue,fident,nident,qstart,qend,qlen,tstart,tend,tlen,alnlen,bits,qheader,theader,mismatch,qcov,tcov --max-seqs 50000 --gpu 1 easy-search test_arabidopsis_alphafold.fas Arabidopsis_thaliana_taxid_3702_AlphaFold_v3.AlphaFold.foldseekdb test_arabidopsis_alphafold.result.m8 /tmp/hashkenazy/foldseek_Prost5/tmp --prostt5-model /ebio/scratch/hashkenazy/foldseek_9_Prost5_model/weights --format-output query,target,evalue,fident,nident,qstart,qend,qlen,tstart,tend,tlen,alnlen,bits,qheader,theader,mismatch,qcov,tcov --max-seqs 50000 --gpu 1

MMseqs Version: bc212bc8602ef426c7b58368c65dd744443f802c Seq. id. threshold 0 Coverage threshold 0 Coverage mode 0 Max reject 2147483647 Max accept 2147483647 Add backtrace false TMscore threshold 0 TMalign hit order 0 TMalign fast 1 Preload mode 0 Threads 128 Verbosity 3 LDDT threshold 0 Sort by structure bit score 1 Alignment type 2 Exact TMscore 0 Substitution matrix aa:3di.out,nucl:3di.out Alignment mode 3 Alignment mode 0 E-value threshold 10 Min alignment length 0 Seq. id. mode 0 Alternative alignments 0 Max sequence length 65535 Compositional bias 1 Compositional bias 1 Gap open cost aa:10,nucl:10 Gap extension cost aa:1,nucl:1 Compressed 0 Seed substitution matrix aa:3di.out,nucl:3di.out Sensitivity 9.5 k-mer length 6 Target search mode 0 k-score seq:2147483647,prof:2147483647 Max results per query 50000 Split database 0 Split mode 2 Split memory limit 0 Diagonal scoring true Exact k-mer matching 0 Mask residues 0 Mask residues probability 0.99995 Mask lower case residues 1 Minimum diagonal score 30 Selected taxa
Spaced k-mers 1 Spaced k-mer pattern
Local temporary path
Exhaustive search mode false Prefilter mode 0 Search iterations 1 Remove temporary files true MPI runner
Force restart with latest tmp false Cluster search 0 Use GPU 1 Path to ProstT5 /ebio/scratch/hashkenazy/foldseek_9_Prost5_model/weights Chain name mode 0 Write mapping file 0 Mask b-factor threshold 0 Coord store mode 2 Write lookup file 1 Input format 0 File Inclusion Regex .* File Exclusion Regex ^$ Alignment format 0 Format alignment output query,target,evalue,fident,nident,qstart,qend,qlen,tstart,tend,tlen,alnlen,bits,qheader,theader,mismatch,qcov,tcov Database output false Greedy best hits false

createdb test_arabidopsis_alphafold.fas /tmp/hashkenazy/foldseek_Prost5/tmp/4959022769211143606/query --gpu 1 --prostt5-model /ebio/scratch/hashkenazy/foldseek_9_Prost5_model/weights --chain-name-mode 0 --write-mapping 0 --mask-bfactor-threshold 0 --coord-store-mode 2 --write-lookup 1 --input-format 0 --file-include '.*' --file-exclude '^$' --threads 128 -v 3

Converting sequences

Time for merging to query_h: 0h 0m 0s 1ms Time for merging to query: 0h 0m 0s 1ms Database type: Aminoacid Error loading ProstT5: DriverError(CUDA_ERROR_INVALID_PTX, "a PTX JIT compilation failed") when loading cast_f32_f16 Time for processing: 0h 0m 0s 992ms Error: query createdb died

Context

Providing context helps us come up with a solution and improve our documentation for the future.

Your Environment

Include as many relevant details about the environment you experienced the bug in.

Thanks in advance for any advice!

HaimAshk commented 1 month ago

Was a compilation problem I missed. Solved by recompiling.