How to add dependencies?

[alexklibisz]

First, thanks for the great project.. It's been very helpful in starting to build an Elasticsearch plugin.

Now I'm trying to add ND4J as a dependency, but I'm having problems getting it to build.

I add this to build.gradle:

// In this section you declare the dependencies for your production and test code
// Note, the two dependencies are not really needed as the buildscript dependency gets them in already
// they are just here as an example
dependencies {
  compile 'org.elasticsearch:elasticsearch:6.2.4'
  compile 'org.nd4j:nd4j-native-platform:0.9.1'

  testCompile 'org.elasticsearch.test:framework:6.2.4'
}

And then I try to import nd4j in IngestAwesomePlugin.java:

...
import org.nd4j.*;

public class IngestAwesomePlugin extends Plugin implements IngestPlugin {
...

When I run gradle build, I get the following error:

=======================================
Elasticsearch Build Hamster says Hello!
=======================================
  Gradle Version        : 4.7
  OS Info               : Linux 4.5.2-040502-generic (amd64)
  JDK Version           : Oracle Corporation 10.0.1 [Java HotSpot(TM) 64-Bit Server VM 10.0.1+10]
  JAVA_HOME             : /home/alex/.installations/jdk-10.0.1
  Random Testing Seed   : 8884BF25AD0D20C1

> Task :compileJava FAILED
/home/alex/Documents/dev/approximate-vector-search/scratch/elasticsearch-plugin/ingest-awesome/src/main/java/org/elasticsearch/plugin/ingest/awesome/IngestAwesomePlugin.java:30: error: package org.nd4j does not exist
import org.nd4j.*;
^
1 error

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':compileJava'.
> Compilation failed with exit code 1; see the compiler error output for details.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
See https://docs.gradle.org/4.7/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 2s
2 actionable tasks: 1 executed, 1 up-to-date

I'm not very familiar with Gradle, and I don't know what the various plugins are doing under the hood. Could you maybe tell me what needs to be added to the build.gradle file to get this compilation working?

[alexklibisz]

I was able to get a simple dependency working: org.json:json:20180130.

I added it to the dependencies in build.gradle:

// Note, the two dependencies are not really needed as the buildscript dependency gets them in already
// they are just here as an example
dependencies {
    compile 'org.json:json:20180130'

And added a simple example in IngestAwesomePlugin.java:

import org.json.JSONObject;
...
@Override
 public Map<String, Processor.Factory> getProcessors(Processor.Parameters parameters) {

     JSONObject jo = new JSONObject();
     jo.put("firstName", "John");
     jo.put("lastName", "Smith");
     jo.put("age", 25);
     System.out.println(jo.toString());
...

This compiles and run fine. However, ND4J still does not completely work.

I was able to get it to compile by adding the dependencies like this:

dependencies {
  compile 'org.json:json:20180130'
  compile 'org.nd4j:nd4j-native:1.0.0-alpha:linux-x86_64'
  compile 'org.nd4j:nd4j-api:1.0.0-alpha'
}

And adding a simple usage, also in IngestAwesomePlugin.java:

INDArray arrWrite = Nd4j.linspace(1,10,10);

This compiles, however when I the integration tests run, I get the following error:

|    [2018-04-30T22:46:24,754][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-0] fatal error in thread [main
], exiting
|    java.lang.NoClassDefFoundError: org/nd4j/linalg/api/complex/IComplexNumber
|       at org.elasticsearch.plugin.ingest.awesome.IngestAwesomePlugin.getProcessors(IngestAwesomePlugin.java:58) ~[?:?
]
|       at org.elasticsearch.ingest.IngestService.<init>(IngestService.java:49) ~[elasticsearch-6.2.4.jar:6.2.4]
|       at org.elasticsearch.node.Node.<init>(Node.java:343) ~[elasticsearch-6.2.4.jar:6.2.4]
|       at org.elasticsearch.node.Node.<init>(Node.java:246) ~[elasticsearch-6.2.4.jar:6.2.4]
|       at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.2.4.jar:6.2.4]
|       at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.2.4.jar:6.2.4]
|       at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) ~[elasticsearch-6.2.4.jar:6.2.4]
|       at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-6.2.4.jar:6.2.4]
|       at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-6.2.4.jar:6.2.4]
|       at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.2.4
.jar:6.2.4]
|       at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.2.4.jar:6.2.4
]
|       at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.2.4.jar:6.2.4]
|       at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-6.2.4.jar:6.2.4]
|       at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) ~[elasticsearch-6.2.4.jar:6.2.4]
|    Caused by: java.lang.ClassNotFoundException: org.nd4j.linalg.api.complex.IComplexNumber
|       at java.net.URLClassLoader.findClass(URLClassLoader.java:466) ~[?:?]
|       at java.lang.ClassLoader.loadClass(ClassLoader.java:566) ~[?:?]
|       at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:890) ~[?:?]
|       at java.lang.ClassLoader.loadClass(ClassLoader.java:499) ~[?:?]
|       ... 14 more

This makes it seem like there is some part of ND4J that needs to be included at runtime as well, but I'm not sure if it's an issue with ND4J, Gradle, or both?

I can provide a tarred copy of the code that produces the error above if that helps.

[spinscale]

you can take a look at other plugins like https://github.com/spinscale/elasticsearch-ingest-langdetect/blob/master/build.gradle or https://github.com/spinscale/elasticsearch-ingest-opennlp/blob/master/build.gradle and see how this is working and check for differences.

The gradle code looks ok so I assume this must be something else. Putting the plugin code somewhere would definately help.

[spinscale]

Closing due to lack of feedback. Please reopen if there is a way to reproduce. Thanks!