search

jbaiter / kenlm-java

Java SWIG Wrapper around kenlm
GNU Lesser General Public License v3.0
0 stars 2 forks source link

KenLM #2

Open jonashartwig opened 6 years ago

jonashartwig commented 6 years ago

Hi,

I am looking at your solution to use the JNI bridge to use Kenlm. I have problems with it not finding the kenlm libraries. I followed the example on the webpage on how to build it. The build gives .a files and I put them into java.library.path. However the java runtime requires dylib files. I have been looking forever through cmake to try to find a solution so it builds dylibs or so for that matter. Any suggestion?

regards

jonashartwig commented 6 years ago

Ok, so seems I added the parameter on the wrong place for cmake. Should be like this:

cmake -DBUILD_SHARED_LIBS=ON ~/Downloads/kenlm

However running the program gives me the following error:

java.lang.UnsatisfiedLinkError: com.github.jbaiter.kenlm.jni.KenLMJNI.new_Config()J at com.github.jbaiter.kenlm.jni.KenLMJNI.new_Config(Native Method) ~[na:na] at com.github.jbaiter.kenlm.jni.Config.(Config.java:96) ~[na:na] at com.github.jbaiter.kenlm.Config.(Config.java:9) ~[kenlm-0.0.2.jar:na] at com.github.jbaiter.kenlm.Model.(Model.java:27) ~[kenlm-0.0.2.jar:na]

I use 0.0.2 from github.

regards

jonashartwig commented 6 years ago

Hi,

I figured out I need to put the kenlm lib that was provided into my java.library.path/LD_LIBRARY_PATH. Starting the program works and I get scoring results. However if I have a sentence "I like science fiction" then all scores are index: 0, ngram: 2, score: -2.7. I attached a debugger and within the model if I evaluate this.vocabulary.Index("word") it always returns 0. Something is wrong but I have a hard time understanding what. My model is in binary format trained on english and i 3gb big. I load it like this: new Model("/model") where /model is the file. Any hints?

Regards

jonashartwig commented 6 years ago

So turns out my model was "bad" and the vocabulary was all upper case. I added the possibility to check the vocabulary in my fork if you are interested. It is very useful if you want to build a spell checker with it like I did. Also I fixed a critical bug where the state was not handled correctly.

Feel free to have a look and merge it, otherwise you can close this issue.

Regards