Failed building databases using GTDB-taxdump taxonomy files

[emilhaegglund]

I was trying to build a database using the taxonomy files from gtdb-taxdump, however it failed when reading the names.dmp with the following message:

zcat  gtdb_proteomes/* | diamond makedb --db gtdb --taxonnames gtdb-taxdump/R207/names.dmp --taxonnodes gtdb_data/gtdb-taxdump/R207/nodes.dmp --taxonmap gtdb.protein.taxid.map

diamond v2.0.15.153 (C) Max Planck Society for the Advancement of Science
Documentation, support and updates available at http://www.diamondsearch.org
Please cite: http://dx.doi.org/10.1038/s41592-021-01101-x Nature Methods (2021)

#CPU threads: 32
Scoring parameters: (Matrix=BLOSUM62 Lambda=0.267 K=0.041 Penalties=11/1)
Input file parameter (--in) is missing. Input will be read from stdin.
Opening the database file...  [0s]
Loading sequences...  [0.772s]
Masking sequences...  [0.157s]
Writing sequences...  [0.035s]
Writing accessions...  [0.072s]
Hashing sequences...  [0.013s]
Loading sequences...  [0s]
Writing trailer...  [0.003s]
Loading taxonomy nodes...  [28.213s]
Loading taxonomy names...  [78.105s]
Failed to allocate sufficient memory. Please refer to the manual for instructions on memory usage.

Here is an example of the names.dmp from gtdb-taxdump

head -20 gtdb-taxdump/R207/names.dmp
1   |   root    |       |   scientific name |
13926   |   001393675   |       |   scientific name |
14375   |   RUG14239 sp902797145    |       |   scientific name |
17689   |   001423155   |       |   scientific name |
20514   |   018334475   |       |   scientific name |
23859   |   013185635   |       |   scientific name |
34402   |   002214285   |       |   scientific name |
38289   |   001509495   |       |   scientific name |
66445   |   009903045   |       |   scientific name |
74747   |   000419015   |       |   scientific name |
78978   |   014222245   |       |   scientific name |
85313   |   001742655   |       |   scientific name |
88808   |   E44-bin52 sp004375875   |       |   scientific name |
121310  |   001585965   |       |   scientific name |
138721  |   VXYK01  |       |   scientific name |
147972  |   007121265   |       |   scientific name |
151528  |   007830495   |       |   scientific name |
157756  |   003411905   |       |   scientific name |
160336  |   002878095   |       |   scientific name |
173955  |   001247185   |       |   scientific name |

Do you have any idea why this could be happening? I haven't had any problems building databases with the NCBI-taxdumps.

Best regards, Emil Hägglund

[bbuchfink]

The taxids used in these files are > 2^31, that is not supported at the moment. I'll see what I can do about this.

[emilhaegglund]

Ah, suspected it was something like this. Then I know the cause of the error. Thanks for the quick reply!