Cannot load binaries pulled in by transient dependencies

[crockpotveggies]

Attempting to run Deeplearning4j but it appears that required binaries that are pulled in by classifiers by transient dependencies are either not downloaded or made available on the classpath.

Sometimes, maven dependencies with classifiers or platform-specific binaries are not loaded properly when using a dependency manager other than the one included with maven. Happy to help here.

To reproduce, try the following:

#@dependency(group="org.nd4j", module="nd4j-cuda-8.0-platform", version="0.9.1")
#@dependency(group="org.deeplearning4j", module="deeplearning4j-zoo", version="0.9.1")

then

#!scala
import org.deeplearning4j.zoo.model.VGG16
import org.deeplearning4j.zoo._

val zooModel = new VGG16()
val net = zooModel.initPretrained(PretrainedType.IMAGENET)

You will get the error:

Caused by: org.nd4j.linalg.factory.Nd4jBackend$NoAvailableBackendException: Please ensure that you have an nd4j backend on your classpath. Please see: http://nd4j.org/getstarted.html
    at org.nd4j.linalg.factory.Nd4jBackend.load(Nd4jBackend.java:258)
    at org.nd4j.linalg.factory.Nd4j.initContext(Nd4j.java:6086)
    ... 44 more

[crockpotveggies]

Note: if I try a direct variation of the above it also doesn't work

#@dependency(group="org.nd4j", module="nd4j-cuda-8.0", version="0.9.1")
#@dependency(group="org.deeplearning4j", module="deeplearning4j-zoo", version="0.9.1")
#@dependency(group="org.nd4j", module="nd4j-cuda-8.0", version="0.9.1", classifier="linux-x86_64")

the following error is produced

Caused by: java.lang.UnsatisfiedLinkError: /home/justin/.javacpp/cache/nd4j-cuda-8.0-0.9.1-linux-x86_64.jar/org/nd4j/nativeblas/linux-x86_64/libjnind4jcuda.so: /home/justin/anaconda2/bin/../lib/libgomp.so.1: version `GOMP_4.0' not found (required by /home/justin/.javacpp/cache/nd4j-cuda-8.0-0.9.1-linux-x86_64.jar/org/nd4j/nativeblas/linux-x86_64/libnd4jcuda.so)
    at java.lang.ClassLoader$NativeLibrary.load(Native Method)
    at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
    at java.lang.Runtime.load0(Runtime.java:809)
    at java.lang.System.load(System.java:1086)
    at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:956)
    ... 66 more

although OMP is installed on the local machine.

[ctrueden]

Just wanted to chime in that I did see this bug report, and plan on investigating, but I am super backlogged right now and not sure when I'll get the chance. This may be an issue with scijava-grab rather than SJJK directly; further investigation will shed more light.

I briefly tried using the #@dependency directive from within ImageJ's Script Editor to see whether that also has similar problems. I did this by running mvn -Dimagej.app.directory=/Applications/Fiji.app from a clone of scijava/scijava-grab and then launching Fiji and running the following script:

#@dependency(group="org.nd4j", module="nd4j-api", version="0.9.1")
println(org.nd4j.linalg.api.environment.Nd4jEnvironment.class)

And I received the following exception:

Started New_.groovy at Mon Aug 28 15:00:43 CDT 2017
java.lang.RuntimeException: Error grabbing Grapes -- [download failed: org.slf4j#slf4j-api;1.7.10!slf4j-api.jar, download failed: org.apache.commons#commons-lang3;3.3.1!commons-lang3.jar]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:83)
    at org.codehaus.groovy.reflection.CachedConstructor.doConstructorInvoke(CachedConstructor.java:77)
    at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrap.callConstructor(ConstructorSite.java:84)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:60)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:235)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:247)
    at groovy.grape.GrapeIvy.getDependencies(GrapeIvy.groovy:424)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSite.invoke(PogoMetaMethodSite.java:169)
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:59)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:52)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:154)
    at groovy.grape.GrapeIvy.resolve(GrapeIvy.groovy:571)
    at groovy.grape.GrapeIvy$resolve$0.callCurrent(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:52)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:154)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:190)
    at groovy.grape.GrapeIvy.resolve(GrapeIvy.groovy:538)
    at groovy.grape.GrapeIvy$resolve.callCurrent(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:52)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:154)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:182)
    at groovy.grape.GrapeIvy.grab(GrapeIvy.groovy:256)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSite.invoke(PogoMetaMethodSite.java:169)
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:59)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:52)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:154)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:174)
    at groovy.grape.GrapeIvy.grab(GrapeIvy.groovy:237)
    at org.scijava.grab.DefaultGrabService.grab(DefaultGrabService.java:146)
    at org.scijava.grab.script.process.GrabScriptProcessor$1.invoke(GrabScriptProcessor.java:115)
    at org.scijava.script.ScriptModule.run(ScriptModule.java:155)
    at org.scijava.module.ModuleRunner.run(ModuleRunner.java:167)
    at org.scijava.module.ModuleRunner.call(ModuleRunner.java:126)
    at org.scijava.module.ModuleRunner.call(ModuleRunner.java:65)
    at org.scijava.thread.DefaultThreadService$3.call(DefaultThreadService.java:237)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)

It is still very early days for scijava-grab and in particular we need to make it smarter about grabbing dependencies which are already present on the system classpath. Until we do that, there are likely to be persistent issues like this, unfortunately.

[ctrueden]

The scijava-jupyter-kernel project is being retired in favor of other solutions like IJava and BeakerX.