kpu / kenlm

KenLM: Faster and Smaller Language Model Queries
http://kheafield.com/code/kenlm/
Other
2.46k stars 514 forks source link

Running lmplz getting killed #403

Closed felicitywang1 closed 1 year ago

felicitywang1 commented 1 year ago

Hi I'm running https://github.com/NVIDIA/NeMo/blob/main/scripts/asr_language_modeling/ngram_lm/train_kenlm.py and got (part of the log)

[NeMo I 2022-11-22 01:06:15 train_kenlm:105] Encoding the train file '/home/user/tmp.txt' ...
Read 0 lines: 1000 lines [00:00, 389334.82 lines/s]
Chunking 999 rows into 0.1219 tasks (each chunk contains 8192 elements)
Deleting previous file : /home/user/tmp.nemo.tmp.txt
[Parallel(n_jobs=-2)]: Using backend LokyBackend with 7 concurrent workers.
[NeMo W 2022-11-22 01:06:21 optimizers:55] Apex was not found. Using the lamb or fused_adam optimizer will error out.
[Parallel(n_jobs=-2)]: Done   1 tasks      | elapsed:    6.0s
[Parallel(n_jobs=-2)]: Done   1 out of   1 | elapsed:    6.0s finished
Chunk : 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 478.80 chunks/s]
Finished writing 1 chunks to /home/user/tmp.nemo.tmp.txt. Current chunk index = 1
=== 1/5 Counting and sorting n-grams ===
Reading /home/user/tmp.nemo.tmp.txt
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
-9
Traceback (most recent call last):
  File "train_kenlm.py", line 165, in <module>
    main()
  File "train_kenlm.py", line 144, in main
    raise RuntimeError("Training KenLM was not successful!")
RuntimeError: Training KenLM was not successful!

When I run lmplz directly I got

=== 1/5 Counting and sorting n-grams ===
Reading /home/user/tmp.nemo.tmp.txt
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
Killed
f.wang1@nemo:/group-volume/Compression/sharing/felicity.wang/NeMo/scripts/asr_language_modeling/ngram_lm/data$ /group-volume/Compression/sharing/felicity.wang/NeMo/decoders/kenlm/build/bin/lmplz -o 3 --text /home/user/tmp.nemo.tmp.txt --arpa /home/user/tmp.nemo.tmp.arpa
=== 1/5 Counting and sorting n-grams ===
Reading /home/user/tmp.nemo.tmp.txt
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
Killed

Any reason why this is happening and how could I solve it?

kpu commented 1 year ago

Use a smaller -S setting on the command line. The kernel OOM killer is coming by.