Converting to lowercase and counting word occurrences ...
| |# | 407 Elapsed Time: 0:00:00
Saving top 1000 words ...
Calculating word statistics ...
Your text file has 3134 words in total
It has 39 unique words
Your top-1000 words are 100.0000 percent of all words
Your most common word "the" occurred 406 times
The least common word in your top-k is "off" with 2 times
The first word with 4 occurrences is "device" at place 36
Creating ARPA file ...
/code/kenlm/build/bin/lmplz: error while loading shared libraries: libboost_program_options.so.1.71.0: cannot open shared object file: No such file or directory
Traceback (most recent call last):
File "/code/data/lm/generate_lm.py", line 231, in <module>
main()
File "/code/data/lm/generate_lm.py", line 215, in main
build_lm(args, data_lower, vocab_str)
File "/code/data/lm/generate_lm.py", line 98, in build_lm
subprocess.check_call(subargs)
File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/code/kenlm/build/bin/lmplz', '--order', '4', '--temp_prefix', '/app/assets/scorer', '--memory', '85%', '--text', '/app/assets/scorer/lower.txt.gz', '--arpa', '/app/assets/scorer/lm.arpa', '--prune', '0', '--discount_fallback']' returned non-zero exit status 127.
Installing the boost libraries and running the command again leads to a segmentation fault.
Converting to lowercase and counting word occurrences ...
| |# | 407 Elapsed Time: 0:00:00
Saving top 1000 words ...
Calculating word statistics ...
Your text file has 3134 words in total
It has 39 unique words
Your top-1000 words are 100.0000 percent of all words
Your most common word "the" occurred 406 times
The least common word in your top-k is "off" with 2 times
The first word with 4 occurrences is "device" at place 36
Creating ARPA file ...
=== 1/5 Counting and sorting n-grams ===
Reading /app/assets/scorer/lower.txt.gz
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
Traceback (most recent call last):
File "/code/data/lm/generate_lm.py", line 231, in <module>
main()
File "/code/data/lm/generate_lm.py", line 215, in main
build_lm(args, data_lower, vocab_str)
File "/code/data/lm/generate_lm.py", line 98, in build_lm
subprocess.check_call(subargs)
File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/code/kenlm/build/bin/lmplz', '--order', '4', '--temp_prefix', '/app/assets/scorer', '--memory', '85%', '--text', '/app/assets/scorer/lower.txt.gz', '--arpa', '/app/assets/scorer/lm.arpa', '--prune', '0', '--discount_fallback']' died with <Signals.SIGSEGV: 11>.
set value of property to zero
set value of property to one
set value of property to two
set value of property to three
...
set the value of the property to one hundred
switch device on
switch device off
switch the device on
switch the device off
Describe the bug
Using the
generate_lm.py
script returns status 127. The boost libraries cannot be found.Installing the boost libraries and running the command again leads to a segmentation fault.
Executing
lmplz
directly.To Reproduce
Dockerfile
Corpus