search

dalab / pboh-entity-linking

Source code for the paper "Probabilistic Bag-Of-Hyperlinks Model for Entity Linking" , http://dl.acm.org/citation.cfm?id=2882988
58 stars 15 forks source link

Error while running in console mode. #4

Closed gvishal closed 6 years ago

gvishal commented 7 years ago

I ran the code in console mode using this command: scala -J-Xmx90g target/PBoH-1.0-SNAPSHOT-jar-with-dependencies.jar consoleEntityLinking max-product jkgh

Has anyone come across this error: java.lang.UnsatisfiedLinkError: org.fusesource.leveldbjni.internal.NativeOptions.init()V

Here's the log for it: Processed 20400000 lines. Processed 20700000 lines. Processed 21000000 lines. Processed 21300000 lines. Processed 21600000 lines. Now sorting in memory all strings Done sorting. time = 102.992 Loaded mention index. Size = 21540764 Loading word freq index p(w) from /data/pboh-data/pboh_data/textFromAllWikipedia2014Feb.txt_word_frequencies... Done loading word freq index p(w). Size = 582355 java.lang.UnsatisfiedLinkError: org.fusesource.leveldbjni.internal.NativeOptions.init()V at org.fusesource.leveldbjni.internal.NativeOptions.init(Native Method) at org.fusesource.leveldbjni.internal.NativeOptions.(NativeOptions.java:54) at org.fusesource.leveldbjni.JniDBFactory$OptionsResourceHolder.init(JniDBFactory.java:98) at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:167) at index.WordEntityProbsIndex.(WordEntityProbsIndex.scala:27) at index.AllIndexesBox.(AllIndexesBox.scala:63) at console_el.ConsoleEntityLinking.(ConsoleEntityLinking.scala:42) at el.EL_LBP_Spark$.main(EL_LBP_Spark.scala:98) at el.EL_LBP_Spark.main(EL_LBP_Spark.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at scala.reflect.internal.util.ScalaClassLoader$$anonfun$run$1.apply(ScalaClassLoader.scala:70) at scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31) at scala.reflect.internal.util.ScalaClassLoader$URLClassLoader.asContext(ScalaClassLoader.scala:101) at scala.reflect.internal.util.ScalaClassLoader$class.run(ScalaClassLoader.scala:70) at scala.reflect.internal.util.ScalaClassLoader$URLClassLoader.run(ScalaClassLoader.scala:101) at scala.tools.nsc.CommonRunner$class.run(ObjectRunner.scala:22) at scala.tools.nsc.JarRunner$.run(MainGenericRunner.scala:13) at scala.tools.nsc.CommonRunner$class.runAndCatch(ObjectRunner.scala:29) at scala.tools.nsc.JarRunner$.runJar(MainGenericRunner.scala:25) at scala.tools.nsc.MainGenericRunner.runTarget$1(MainGenericRunner.scala:69) at scala.tools.nsc.MainGenericRunner.run$1(MainGenericRunner.scala:87) at scala.tools.nsc.MainGenericRunner.process(MainGenericRunner.scala:98) at scala.tools.nsc.MainGenericRunner$.main(MainGenericRunner.scala:103) at scala.tools.nsc.MainGenericRunner.main(MainGenericRunner.scala)

octavian-ganea commented 7 years ago

Did you use maven or sbt ?

gvishal commented 7 years ago

maven

Followed instructions mentioned in the readme.

octavian-ganea commented 7 years ago

This is a leveldb error. Looks like something is wrong already from the creation of your db. Please write some simple script to check if leveldb is working properly for you . See simple examples here: https://github.com/fusesource/leveldbjni

gvishal commented 7 years ago

Alright! Thanks! I'll take a look.

gvishal commented 7 years ago

I see that in https://github.com/dalab/pboh-entity-linking/blob/master/src/main/scala/index/WordEntityProbsIndex.scala#L23 you have used an in-memory db instead of leveldb. I'll try running with that.

octavian-ganea commented 7 years ago

Yes, that's a simple in-memory hashmap, but be aware that it will eat a bunch of GB of RAM more compared to using an in=memory db.

gvishal commented 7 years ago

Yes, in fact it exceeds my memory and fails in between loading: Loading word entity prob index p(w|e). I had 60gigs of RAM.

I found an alternative (RocksDb, based on leveldb) that works without any errors. See links below.

  1. https://github.com/facebook/rocksdb
  2. https://gist.github.com/pauca/c771ccc5ed869dc42b3127e77d73fca5