kpu / kenlm

KenLM: Faster and Smaller Language Model Queries
http://kheafield.com/code/kenlm/
Other
2.46k stars 514 forks source link

Cannot allocate memory Failed to allocate when using build_binary #425

Open amitliron opened 1 year ago

amitliron commented 1 year ago

I have a arpa file which I created by the following command:

./lmplz -o 4 -S 1G <tmp_100M.txt >100m.arpa

Now I want to convert this arpa file to binary file: ./build_binary 100m.arpa 100m.bin

And I'm getting error:

mmap.cc:225 in void util::HugeMalloc(std::size_t, bool, util::scoped_memory&) threw ErrnoException because `!to.get()'.
Cannot allocate memory Failed to allocate 106122412848 bytes Byte: 80
ERROR

I tried to add -S parameter: ./build_binary -S 1G 100m.arpa 100m.bin and I got the same error.

  1. How can I convert to binary file ?
  2. Why I'm getting this error ?