search

NVIDIA / OpenSeq2Seq

Toolkit for efficient experimentation with Speech Recognition, Text2Speech and NLP
https://nvidia.github.io/OpenSeq2Seq
Apache License 2.0
1.54k stars 369 forks source link

generate_trie eat whole system memory! #326

Closed dingguijin closed 5 years ago

dingguijin commented 5 years ago

ctc_decoder_with_lm/generate_trie eat all memory of system (32G) and killed by system.

Since my 4-gram binary file is almost 8G. Suggest read a block and write back to file not read everything into memory.

borisgin commented 5 years ago

Can you check, please, what happens if you change POPULATE_OR_READ to LAZY or to POPULATE_OR_LAZY in generate_trie.cpp config.load_method = util::LAZY;