search

kurator-org / kurator-akka

Provenance-enabled workflow platform and toolkit to curate biodiversity data https://doi.org/10.5281/zenodo.1068311
http://wiki.datakurator.org/wiki/Kurator
7 stars 2 forks source link

NoClassDefFoundError for PythonInterpreter thrown by native python actor #9

Open lowery opened 6 years ago

lowery commented 6 years ago

Issues when trying to run native workflows using the latest version of "kurator-akka-1.0.1-jar-with-dependencies-with-native-python.jar" on MacOS. Likely to be a problem with the library and might be resolved in 1.0.2-SNAPSHOT.

alias ka='java -jar ~/bin/kurator-akka.jar' kurator-validation/packages/kurator_dwca/workflows

Stack trace:

java.lang.NoClassDefFoundError: Could not initialize class org.kurator.akka.interpreters.PythonInterpreter at org.kurator.akka.actors.NativePythonActor.onInitialize(NativePythonActor.java:20) at org.kurator.akka.KuratorActor.onReceive(KuratorActor.java:308) at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167) at akka.actor.Actor$class.aroundReceive(Actor.scala:480) at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526) at akka.actor.ActorCell.invoke(ActorCell.scala:495) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) at akka.dispatch.Mailbox.run(Mailbox.scala:224) at akka.dispatch.Mailbox.exec(Mailbox.scala:234) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

tucotuco commented 6 years ago

Where can 1.0.2-SNAPSHOT be found?

chicoreus commented 6 years ago

On Tue, 13 Feb 2018 13:25:11 +0000 (UTC) John Wieczorek notifications@github.com wrote:

Where can 1.0.2-SNAPSHOT be found?

That's the current version, checkout kurator-akka and do a mvn install.

It should be in the sontatype snapshot respository as well, I'm doing a mvn deploy right now to make sure what is there is current.

-Paul

chicoreus commented 6 years ago

Deploy to sonatype snapshot repository seems to be failing right now. Need to build locally.

tucotuco commented 6 years ago

I do not see that anything has been committed since 30 Nov 2017.

On Tue, Feb 13, 2018 at 11:08 AM, Paul J. Morris notifications@github.com wrote:

Deploy to sonatype snapshot repository seems to be failing right now. Need to build locally.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kurator-org/kurator-akka/issues/9#issuecomment-365276764, or mute the thread https://github.com/notifications/unsubscribe-auth/AAcP60jtV3em7DjUWqaE4uBb6g4Ngjp0ks5tUZdfgaJpZM4R74uL .

chicoreus commented 6 years ago

Available now in the sonatype snapshot repository. https://oss.sonatype.org/content/repositories/snapshots/org/datakurator/kurator-akka/1.0.2-SNAPSHOT/maven-metadata.xml

chicoreus commented 6 years ago

Last commit in kurator-akka master is indeed commit cdc8356a9208c6a83f4784065dc3413bf698233a Date: Thu Nov 30 15:48:14 2017 -0500

tucotuco commented 6 years ago

Ran mvn install on that most recent commit.

The snapshot kurator-akka-1.0.2-SNAPSHOT was created successfully in ./target.

I made a copy of kurator-akka-1.0.2-SNAPSHOT-jar-with-dependencies.jar to my bin directory, renamed it to kurator-akka.jar and have an alias

alias ka='java -jar ~/bin/kurator-akka.jar'

When I run workflows, I still get the following exception:

java.lang.UnsatisfiedLinkError: /private/var/folders/72/10kzx9697h992d7rb6566z9m0000gn/T/native405151464149797879libkurator.dylib: dlopen(/private/var/folders/72/10kzx9697h992d7rb6566z9m0000gn/T/native405151464149797879libkurator.dylib, 1): Library not loaded: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libR.dylib Referenced from: /private/var/folders/72/10kzx9697h992d7rb6566z9m0000gn/T/native405151464149797879libkurator.dylib Reason: image not found at java.lang.ClassLoader$NativeLibrary.load(Native Method) at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941) at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824) at java.lang.Runtime.load0(Runtime.java:809) at java.lang.System.load(System.java:1086) at org.kurator.util.NativeUtil.loadDarwin64(NativeUtil.java:53) at org.kurator.util.NativeUtil.loadLibrary(NativeUtil.java:18) at org.kurator.akka.interpreters.PythonInterpreter.(PythonInterpreter.java:15) at org.kurator.akka.actors.NativePythonActor.onInitialize(NativePythonActor.java:20) at org.kurator.akka.KuratorActor.onReceive(KuratorActor.java:308) at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167) at akka.actor.Actor$class.aroundReceive(Actor.scala:480) at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526) at akka.actor.ActorCell.invoke(ActorCell.scala:495) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) at akka.dispatch.Mailbox.run(Mailbox.scala:224) at akka.dispatch.Mailbox.exec(Mailbox.scala:234) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Workflow] java.lang.NoClassDefFoundError: Could not initialize class org.kurator.akka.interpreters.PythonInterpreter at org.kurator.akka.actors.NativePythonActor.onInitialize(NativePythonActor.java:20) at org.kurator.akka.KuratorActor.onReceive(KuratorActor.java:308) at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167) at akka.actor.Actor$class.aroundReceive(Actor.scala:480) at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97)

chicoreus commented 6 years ago

mvn install on kurator-akka

kurator-akka working on hello world workflow:

$ java -jar target/kurator-akka-1.0.2-SNAPSHOT-jar-with-dependencies.jar src/main/resources/org/kurator/akka/samples/hello.yaml Hello World!

cd into kurator-validation mvn package on kurator-validation

Seeing an error, potentially from a different native library issue when running the darwinize_file workflow (the file /tmp/libkurator3164968784711923780.so does exist after the workflow runs):

java -jar target/kurator-validation-1.0.2-SNAPSHOT-jar-with-dependencies.jar -f packages/kurator_dwca/workflows/darwinize_file.yaml -p "inputfile=packages/kurator_dwca_data/tests/test_barcelona2.txt" Loading native lib: /tmp/libkurator3164968784711923780.so Failed to load "�\��" Actor[akka://Workflow/user/DownloadDarwinCloudLookup#-445428231] threw an uncaught exception: java.lang.NullPointerException at org.kurator.akka.interpreters.PythonInterpreter.run(Native Method) at org.kurator.akka.actors.NativePythonActor.onData(NativePythonActor.java:91) at org.kurator.akka.KuratorActor.onReceive(KuratorActor.java:372) at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167) at akka.actor.Actor$class.aroundReceive(Actor.scala:480) at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526) at akka.actor.ActorCell.invoke(ActorCell.scala:495) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) at akka.dispatch.Mailbox.run(Mailbox.scala:224) at akka.dispatch.Mailbox.exec(Mailbox.scala:234) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) null

Started MakeWorkspace

java.lang.NullPointerException at org.kurator.akka.interpreters.PythonInterpreter.run(Native Method) at org.kurator.akka.actors.NativePythonActor.onData(NativePythonActor.java:91) at org.kurator.akka.KuratorActor.onReceive(KuratorActor.java:372) at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167) at akka.actor.Actor$class.aroundReceive(Actor.scala:480) at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526) at akka.actor.ActorCell.invoke(ActorCell.scala:495) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) at akka.dispatch.Mailbox.run(Mailbox.scala:224) at akka.dispatch.Mailbox.exec(Mailbox.scala:234) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

lowery commented 6 years ago

From the stacktrace looks like the native library loaded but ran into issues initializing the Python script. Usually this happens if Python dependencies are missing and an import statement throws a Python exception during initialization.

If that is the case, running the following command should install any missing Python dependencies:

pip install -r kurator-validation/packages/kurator_dwca/requirements.txt

At the moment the native library C code will display a generic "Failed to load" error if the interpreter runs into trouble loading the Python script so it might help to try and catch the import exceptions separately and to also add some more logging to the C code to indicate that the interpreter was initialized successfully.

lowery commented 6 years ago

The previous stack trace in the issue looks like an error satisfying the linked dependencies for the R library:

java.lang.UnsatisfiedLinkError: /private/var/folders/72/10kzx9697h992d7rb6566z9m0000gn/T/native405151464149797879libkurator.dylib: dlopen(/private/var/folders/72/10kzx9697h992d7rb6566z9m0000gn/T/native405151464149797879libkurator.dylib, 1): Library not loaded: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libR.dylib

Will try the following command on the Mac and update the documentation if that doesn't satisfy the dependency:

brew install python r

See also for linux: https://github.com/kurator-org/kurator-akka#install-dependencies-for-native-actor-support

chicoreus commented 6 years ago

pip install of requirements and export of PYTHONPATH resolved the issue I was seeing (on a machine with a fresh install). We should make this environment not set up error condition handling more robust. I've added comments to the kurator-validation readme file.

lowery commented 6 years ago

Added error condition for PYTHONPATH environment variable not set in: eb5272b