Open charlesfoster opened 6 months ago
Hello again,
I've been revisiting mmseqs
again for taxonomic assignment, and unwittingly ran into this problem again before finding my own Github issue (the circle of life!). I was just wondering whether by now there is any advice on creating a taxdb when RAM is limited? I;m working with a pre-clustered version of the NR database that is currently not available directly through mmseqs databases
.
After the standard createdb
command, I ran the following:
mmseqs createtaxdb nr_clustered_mmseqs ~/TMP --ncbi-tax-dump ~/.taxonkit/ --tax-mapping-file /data/clustered_nr/clustered_nr_taxmapping.tsv
I get output as per the OP in this issue, until the process dies with:
[truncated]
Loading names file ... Done
Init RMQ ...Done
Killed
I can see that the problem was most likely the RAM being exhausted (I received exit status 137). My workstation has 64GB of RAM, and accessing a server with more RAM for the creation of this database is not likely to be feasible.
Thanks
p.s. in case you've missed it for any reason, I would also like to point out that the current automated download of the NR/NT fasta files from NCBI using mmseqs databases
might not work as desired moving forwards. As noted at https://ftp.ncbi.nlm.nih.gov/blast/db/FASTA/README.txt:
In April 2024, the BLAST FASTA files in this directory will no longer be
available. You can easily generate FASTA files yourself from the formatted
BLAST databases by using the BLAST utility blastdbcmd that comes with the
standalone BLAST programs. See NCBI Insights for more details
https://ncbiinsights.ncbi.nlm.nih.gov/2024/01/25/blast-fasta-unavailable-on-ftp/
I'm trying to use a clustered version of the NR database for taxonomy assignment but am running into some issues. Any assistance would be appreciated.
Expected Behavior
When running
mmseqs createtaxdb db_name tmp --tax-mapping-file taxid.map
, I would expect to successfully create a seqTaxDB as per here.Current Behavior
The job begins but is unexpectedly killed (see mmseqs output section below).
MMseqs Output (for bugs)
Context
I want to search some query sequences locally against a clustered version of the NR database. I downloaded the clustered database and taxonomy files (nr_cluster_taxid_formatted_final.tsv.gz, nr_rep_seq.fasta.gz) from here, which was created as per: https://research.arcadiascience.com/pub/resource-nr-clustering/release/3. I then processed the files like so:
After these completed successfully, I tried to create the taxdb as per the above:
But the job was killed.
Questions:
--split-memory-limit 50G
but gotUnrecognized parameter "--split-memory-limit"
mmseqs createdb
) is 88.7GB. I haven't tried freeing up more space yet in case this is not the issue.Your Environment
wget https://mmseqs.com/latest/mmseqs-linux-avx2.tar.gz; tar xvfz mmseqs-linux-avx2.tar.gz
)