search

amplab / SparkNet

Distributed Neural Networks for Spark
MIT License
603 stars 172 forks source link

UnsatisfiedLinkError when running Caffe Test #119

Open tutucute5678 opened 8 years ago

tutucute5678 commented 8 years ago

Hi,

I try to build SparkNet in CentOS 6.6 server with CPU only and install it in my spark cluster. I follow https://github.com/amplab/SparkNet/issues/112 and comment out the @Ignore line in src/test/scala/libs/CaffeNetSpec.scala. After building successfully, i run sbt "test-only CaffeNetSpec", but it failed.

java.lang.UnsatisfiedLinkError: no jnicaffe in java.library.path
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1865)
    at java.lang.Runtime.loadLibrary0(Runtime.java:870)
    at java.lang.System.loadLibrary(System.java:1122)
    at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:632)
    at org.bytedeco.javacpp.Loader.load(Loader.java:470)
    at org.bytedeco.javacpp.Loader.load(Loader.java:407)
    at org.bytedeco.javacpp.caffe.<clinit>(caffe.java:16)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at org.bytedeco.javacpp.Loader.load(Loader.java:442)
    at org.bytedeco.javacpp.Loader.load(Loader.java:407)
    at org.bytedeco.javacpp.caffe$NetParameter.<clinit>(caffe.java:1946)
    at CaffeNetSpec$$anonfun$1.apply$mcV$sp(CaffeNetSpec.scala:16)
    at CaffeNetSpec$$anonfun$1.apply(CaffeNetSpec.scala:15)
    at CaffeNetSpec$$anonfun$1.apply(CaffeNetSpec.scala:15)
.
......
Caused by: java.lang.UnsatisfiedLinkError: /tmp/javacpp21624919454281404/libjnicaffe.so: /tmp/javacpp21624919454281404/libjnicaffe.so: undefined symbol: _ZN5caffe15WindowDataLayerIdED1Ev
    at java.lang.ClassLoader$NativeLibrary.load(Native Method)
    at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1937)
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1822)
    at java.lang.Runtime.load0(Runtime.java:809)
    at java.lang.System.load(System.java:1086)
    at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:615)
    at org.bytedeco.javacpp.Loader.load(Loader.java:470)
    at org.bytedeco.javacpp.Loader.load(Loader.java:407)
    at org.bytedeco.javacpp.caffe.<clinit>(caffe.java:16)

Could anyone give me some suggestions of what's wrong in my building process? Many thanks!

robertnishihara commented 8 years ago

Some questions/thoughts:

  1. Are you using the binaries that we built for CentOS? You have to modify build.sbt. The instructions are in the same issue #112.
  2. Since it's for CPUs, the Caffe tests won't work. You'll need to comment out all the calls to Caffe.set_mode(Caffe.GPU) in SparkNet/src/test/scala/libs/CaffeNetSpec.scala. Similarly, if you want to run the apps (e.g., the CifarApp), you'll need to comment out the calls to Caffe.set_mode(Caffe.GPU).
  3. Does sbt assembly work without the tests? That is, if you comment in test in assembly := {} in build.sbt, does sbt assembly work?
  4. Does CifarApp run (after commenting out the call to Caffe.set_mode(Caffe.GPU))?.

If you encounter problems, please let us know and post the full error messages!

tutucute5678 commented 8 years ago

@robertnishihara Thanks for your response!

Yes, i have read all existed issues before and just did as instructions from issue #122. I modified libraries dependencies and uncomment test in assembly := {} line in build.sbt . I removed @ignore and commented Caffe.set_mode(Caffe.GPU) in SparkNet/src/test/scala/libs/CaffeNetSpec.scala. After doing these, i can run sbt assembly successfully. But once i run sbt "test-only CaffeNetSpec", i received below error messages, seems error occurs when calling val netParam = new NetParameter()

[info] Loading project definition from /usr/SparkNet/project
[info] Set current project to sparknet (in build file:/usr/SparkNet/)
[info] Compiling 1 Scala source to /usr/SparkNet/target/scala-2.10/test-classes...
[info] CaffeNetSpec:
[info] NetParam
java.lang.UnsatisfiedLinkError: no jnicaffe in java.library.path
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1865)
    at java.lang.Runtime.loadLibrary0(Runtime.java:870)
    at java.lang.System.loadLibrary(System.java:1122)
    at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:632)
    at org.bytedeco.javacpp.Loader.load(Loader.java:470)
    at org.bytedeco.javacpp.Loader.load(Loader.java:407)
    at org.bytedeco.javacpp.caffe.<clinit>(caffe.java:16)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at org.bytedeco.javacpp.Loader.load(Loader.java:442)
    at org.bytedeco.javacpp.Loader.load(Loader.java:407)
    at org.bytedeco.javacpp.caffe$NetParameter.<clinit>(caffe.java:1946)
    at CaffeNetSpec$$anonfun$1.apply$mcV$sp(CaffeNetSpec.scala:16)
    at CaffeNetSpec$$anonfun$1.apply(CaffeNetSpec.scala:15)
    at CaffeNetSpec$$anonfun$1.apply(CaffeNetSpec.scala:15)
    at org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)
    at org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)
    at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
    at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
    at org.scalatest.Transformer.apply(Transformer.scala:22)
    at org.scalatest.Transformer.apply(Transformer.scala:20)
    at org.scalatest.FlatSpecLike$$anon$1.apply(FlatSpecLike.scala:1636)
    at org.scalatest.Suite$class.withFixture(Suite.scala:1121)
    at org.scalatest.FlatSpec.withFixture(FlatSpec.scala:1683)
    at org.scalatest.FlatSpecLike$class.invokeWithFixture$1(FlatSpecLike.scala:1633)
    at org.scalatest.FlatSpecLike$$anonfun$runTest$1.apply(FlatSpecLike.scala:1645)
    at org.scalatest.FlatSpecLike$$anonfun$runTest$1.apply(FlatSpecLike.scala:1645)
    at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
    at org.scalatest.FlatSpecLike$class.runTest(FlatSpecLike.scala:1645)
    at org.scalatest.FlatSpec.runTest(FlatSpec.scala:1683)
    at org.scalatest.FlatSpecLike$$anonfun$runTests$1.apply(FlatSpecLike.scala:1703)
    at org.scalatest.FlatSpecLike$$anonfun$runTests$1.apply(FlatSpecLike.scala:1703)
    at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413)
    at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)
    at scala.collection.immutable.List.foreach(List.scala:318)
    at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
    at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:390)
    at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:427)
    at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)
    at scala.collection.immutable.List.foreach(List.scala:318)
    at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
    at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:396)
    at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:483)
    at org.scalatest.FlatSpecLike$class.runTests(FlatSpecLike.scala:1703)
    at org.scalatest.FlatSpec.runTests(FlatSpec.scala:1683)
    at org.scalatest.Suite$class.run(Suite.scala:1423)
    at org.scalatest.FlatSpec.org$scalatest$FlatSpecLike$$super$run(FlatSpec.scala:1683)
    at org.scalatest.FlatSpecLike$$anonfun$run$1.apply(FlatSpecLike.scala:1749)
    at org.scalatest.FlatSpecLike$$anonfun$run$1.apply(FlatSpecLike.scala:1749)
    at org.scalatest.SuperEngine.runImpl(Engine.scala:545)
    at org.scalatest.FlatSpecLike$class.run(FlatSpecLike.scala:1749)
    at org.scalatest.FlatSpec.run(FlatSpec.scala:1683)
    at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:444)
    at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:651)
    at sbt.TestRunner.runTest$1(TestFramework.scala:76)
    at sbt.TestRunner.run(TestFramework.scala:85)
    at sbt.TestFramework$$anon$2$$anonfun$$init$$1$$anonfun$apply$8.apply(TestFramework.scala:202)
    at sbt.TestFramework$$anon$2$$anonfun$$init$$1$$anonfun$apply$8.apply(TestFramework.scala:202)
    at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:185)
    at sbt.TestFramework$$anon$2$$anonfun$$init$$1.apply(TestFramework.scala:202)
    at sbt.TestFramework$$anon$2$$anonfun$$init$$1.apply(TestFramework.scala:202)
    at sbt.TestFunction.apply(TestFramework.scala:207)
    at sbt.Tests$$anonfun$9.apply(Tests.scala:216)
    at sbt.Tests$$anonfun$9.apply(Tests.scala:216)
    at sbt.std.Transform$$anon$3$$anonfun$apply$2.apply(System.scala:44)
    at sbt.std.Transform$$anon$3$$anonfun$apply$2.apply(System.scala:44)
    at sbt.std.Transform$$anon$4.work(System.scala:63)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
    at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
    at sbt.Execute.work(Execute.scala:235)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
    at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
    at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.UnsatisfiedLinkError: /tmp/javacpp21670593001062948/libjnicaffe.so: /tmp/javacpp21670593001062948/libjnicaffe.so: undefined symbol: _ZN5caffe15WindowDataLayerIdED1Ev
    at java.lang.ClassLoader$NativeLibrary.load(Native Method)
    at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1937)
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1822)
    at java.lang.Runtime.load0(Runtime.java:809)
    at java.lang.System.load(System.java:1086)
    at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:615)
    at org.bytedeco.javacpp.Loader.load(Loader.java:470)
    at org.bytedeco.javacpp.Loader.load(Loader.java:407)
    at org.bytedeco.javacpp.caffe.<clinit>(caffe.java:16)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at org.bytedeco.javacpp.Loader.load(Loader.java:442)
    at org.bytedeco.javacpp.Loader.load(Loader.java:407)
    at org.bytedeco.javacpp.caffe$NetParameter.<clinit>(caffe.java:1946)
    at CaffeNetSpec$$anonfun$1.apply$mcV$sp(CaffeNetSpec.scala:16)
    at CaffeNetSpec$$anonfun$1.apply(CaffeNetSpec.scala:15)
    at CaffeNetSpec$$anonfun$1.apply(CaffeNetSpec.scala:15)
    at org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)
    at org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)
    at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
    at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
    at org.scalatest.Transformer.apply(Transformer.scala:22)
    at org.scalatest.Transformer.apply(Transformer.scala:20)
    at org.scalatest.FlatSpecLike$$anon$1.apply(FlatSpecLike.scala:1636)
    at org.scalatest.Suite$class.withFixture(Suite.scala:1121)
    at org.scalatest.FlatSpec.withFixture(FlatSpec.scala:1683)
    at org.scalatest.FlatSpecLike$class.invokeWithFixture$1(FlatSpecLike.scala:1633)
    at org.scalatest.FlatSpecLike$$anonfun$runTest$1.apply(FlatSpecLike.scala:1645)
    at org.scalatest.FlatSpecLike$$anonfun$runTest$1.apply(FlatSpecLike.scala:1645)
    at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
    at org.scalatest.FlatSpecLike$class.runTest(FlatSpecLike.scala:1645)
    at org.scalatest.FlatSpec.runTest(FlatSpec.scala:1683)
    at org.scalatest.FlatSpecLike$$anonfun$runTests$1.apply(FlatSpecLike.scala:1703)
    at org.scalatest.FlatSpecLike$$anonfun$runTests$1.apply(FlatSpecLike.scala:1703)
    at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413)
    at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)
    at scala.collection.immutable.List.foreach(List.scala:318)
    at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
    at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:390)
    at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:427)
    at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)
    at scala.collection.immutable.List.foreach(List.scala:318)
    at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
    at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:396)
    at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:483)
    at org.scalatest.FlatSpecLike$class.runTests(FlatSpecLike.scala:1703)
    at org.scalatest.FlatSpec.runTests(FlatSpec.scala:1683)
    at org.scalatest.Suite$class.run(Suite.scala:1423)
    at org.scalatest.FlatSpec.org$scalatest$FlatSpecLike$$super$run(FlatSpec.scala:1683)
    at org.scalatest.FlatSpecLike$$anonfun$run$1.apply(FlatSpecLike.scala:1749)
    at org.scalatest.FlatSpecLike$$anonfun$run$1.apply(FlatSpecLike.scala:1749)
    at org.scalatest.SuperEngine.runImpl(Engine.scala:545)
    at org.scalatest.FlatSpecLike$class.run(FlatSpecLike.scala:1749)
    at org.scalatest.FlatSpec.run(FlatSpec.scala:1683)
    at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:444)
    at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:651)
    at sbt.TestRunner.runTest$1(TestFramework.scala:76)
    at sbt.TestRunner.run(TestFramework.scala:85)
    at sbt.TestFramework$$anon$2$$anonfun$$init$$1$$anonfun$apply$8.apply(TestFramework.scala:202)
    at sbt.TestFramework$$anon$2$$anonfun$$init$$1$$anonfun$apply$8.apply(TestFramework.scala:202)
    at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:185)
    at sbt.TestFramework$$anon$2$$anonfun$$init$$1.apply(TestFramework.scala:202)
    at sbt.TestFramework$$anon$2$$anonfun$$init$$1.apply(TestFramework.scala:202)
    at sbt.TestFunction.apply(TestFramework.scala:207)
    at sbt.Tests$$anonfun$9.apply(Tests.scala:216)
    at sbt.Tests$$anonfun$9.apply(Tests.scala:216)
    at sbt.std.Transform$$anon$3$$anonfun$apply$2.apply(System.scala:44)
    at sbt.std.Transform$$anon$3$$anonfun$apply$2.apply(System.scala:44)
    at sbt.std.Transform$$anon$4.work(System.scala:63)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
    at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
    at sbt.Execute.work(Execute.scala:235)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
    at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
    at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
[error] Could not run test CaffeNetSpec: java.lang.UnsatisfiedLinkError: no jnicaffe in java.library.path
[info] Run completed in 2 seconds, 16 milliseconds.
[info] Total number of tests run: 0
[info] Suites: completed 0, aborted 0
[info] Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[error] Error during tests:
[error]     CaffeNetSpec
[error] (test:testOnly) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 11 s, completed Apr 5, 2016 10:59:09 AM

CifarApp run failed with the same error messges.

pcmoritz commented 8 years ago

Hey,

I managed to reproduce the problem you are running into, most likely there is a problem in the way the CentOS 6 binaries are built. We are looking into it!

-- Philipp.