Closed rasoolims closed 8 years ago
I suspect that you have build an uberjar on your mac?
As maven doesn't know where you are going to deploy, it only packages stuff for mac.
In order to use your uberjar on a linux system, you will have to add the following dependency:
<dependency>
<groupId>org.nd4j</groupId>
<artifactId>nd4j-native</artifactId>
<version>0.4-rc3.10</version>
<classifier>linux-x86_64</classifier>
</dependency>
After adding the dependency, I see a slightly different error (Caused by: java.lang.UnsatisfiedLinkError: /tmp/javacpp12515585121209620/libjnind4j.so: /tmp/javacpp12515585121209620/libgomp.so.1: version `GOMP_4.0' not found (required by /tmp/javacpp12515585121209620/libnd4j.so)):
Exception in thread "main" java.lang.UnsatisfiedLinkError: no jnind4j in java.library.path at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1889) at java.lang.Runtime.loadLibrary0(Runtime.java:849) at java.lang.System.loadLibrary(System.java:1088) at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:654) at org.bytedeco.javacpp.Loader.load(Loader.java:492) at org.nd4j.nativeblas.NativeOps.
(NativeOps.java:29) at org.nd4j.linalg.cpu.nativecpu.ops.NativeOpExecutioner. (NativeOpExecutioner.java:27) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at java.lang.Class.newInstance(Class.java:383) at org.nd4j.linalg.factory.Nd4j.initWithBackend(Nd4j.java:4770) at org.nd4j.linalg.factory.Nd4j.initContext(Nd4j.java:4716) at org.nd4j.linalg.factory.Nd4j. (Nd4j.java:148) at YaraParser.TransitionBasedSystem.Trainer.StaticNeuralTrainer.trainStaticNeural(StaticNeuralTrainer.java:56) at YaraParser.Parser.YaraParser.createTrainData(YaraParser.java:176) at YaraParser.Parser.YaraParser.main(YaraParser.java:36) Caused by: java.lang.UnsatisfiedLinkError: /tmp/javacpp12515585121209620/libjnind4j.so: /tmp/javacpp12515585121209620/libgomp.so.1: version `GOMP_4.0' not found (required by /tmp/javacpp12515585121209620/libnd4j.so) at java.lang.ClassLoader$NativeLibrary.load(Native Method) at java.lang.ClassLoader.loadLibrary1(ClassLoader.java:1968) at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1893) at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1854) at java.lang.Runtime.load0(Runtime.java:795) at java.lang.System.load(System.java:1062) at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:637) ... 14 more
Here is my pom.xml file:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<prerequisites>
<maven>3.0.5</maven>
</prerequisites>
<groupId>edu.columbia.cs</groupId>
<artifactId>neual-yara</artifactId>
<version>0.4-rc0-SNAPSHOT</version>
<name>DeepLearning4j Examples</name>
<description>Examples of training different data sets</description>
<properties>
<nd4j.backend>nd4j-native</nd4j.backend>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<shadedClassifier>bin</shadedClassifier>
<java.version>1.7</java.version>
<nd4j.version>0.4-rc3.9</nd4j.version>
<dl4j.version>0.4-rc3.9</dl4j.version>
<canova.version>0.0.0.15</canova.version>
<guava.version>19.0</guava.version>
<jfreechart.version>1.0.13</jfreechart.version>
<maven-shade-plugin.version>2.4.3</maven-shade-plugin.version>
<exec-maven-plugin.version>1.4.0</exec-maven-plugin.version>
</properties>
<repositories>
<repository>
<id>snapshots-repo</id>
<url>https://oss.sonatype.org/content/repositories/snapshots</url>
<releases>
<enabled>false</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
</repositories>
<distributionManagement>
<snapshotRepository>
<id>sonatype-nexus-snapshots</id>
<name>Sonatype Nexus snapshot repository</name>
<url>https://oss.sonatype.org/content/repositories/snapshots</url>
</snapshotRepository>
<repository>
<id>nexus-releases</id>
<name>Nexus Release Repository</name>
<url>http://oss.sonatype.org/service/local/staging/deploy/maven2/</url>
</repository>
</distributionManagement>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.nd4j</groupId>
<artifactId>nd4j-native</artifactId>
<version>0.4-rc3.10</version>
<classifier>linux-x86_64</classifier>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>org.deeplearning4j</groupId>
<artifactId>deeplearning4j-nlp</artifactId>
<version>${dl4j.version}</version>
</dependency>
<!-- http://mvnrepository.com/artifact/junit/junit -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
</dependency>
<dependency>
<groupId>org.deeplearning4j</groupId>
<artifactId>deeplearning4j-core</artifactId>
<version>${dl4j.version}</version>
</dependency>
<dependency>
<groupId>org.deeplearning4j</groupId>
<artifactId>deeplearning4j-ui</artifactId>
<version>${dl4j.version}</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>${guava.version}</version>
</dependency>
<dependency>
<groupId>org.nd4j</groupId>
<artifactId>${nd4j.backend}</artifactId>
</dependency>
<dependency>
<artifactId>canova-nd4j-image</artifactId>
<groupId>org.nd4j</groupId>
<version>${canova.version}</version>
</dependency>
<dependency>
<artifactId>canova-nd4j-codec</artifactId>
<groupId>org.nd4j</groupId>
<version>${canova.version}</version>
</dependency>
<!-- Used in the RegressionMathFunctions example -->
<dependency>
<groupId>jfree</groupId>
<artifactId>jfreechart</artifactId>
<version>${jfreechart.version}</version>
</dependency>
<dependency>
<groupId>org.nd4j</groupId>
<artifactId>nd4j-native</artifactId>
<version>0.4-rc3.10</version>
<classifier>linux-x86_64</classifier>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>${exec-maven-plugin.version}</version>
<executions>
<execution>
<goals>
<goal>exec</goal>
</goals>
</execution>
</executions>
<configuration>
<executable>java</executable>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>${maven-shade-plugin.version}</version>
<configuration>
<shadedArtifactAttached>true</shadedArtifactAttached>
<shadedClassifierName>${shadedClassifier}</shadedClassifierName>
<createDependencyReducedPom>true</createDependencyReducedPom>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>org/datanucleus/**</exclude>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>reference.conf</resource>
</transformer>
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.5.1</version>
<configuration>
<source>${java.version}</source>
<target>${java.version}</target>
</configuration>
</plugin>
</plugins>
</build>
</project>
What version of Linux are you using?
And please try to upgrade nd4j.version to 0.4-rc3.10
Still having the problem after upgrading the nd4j.version
DISTRIB_ID=Ubuntu DISTRIB_RELEASE=14.04 DISTRIB_CODENAME=trusty DISTRIB_DESCRIPTION="Ubuntu 14.04.4 LTS" NAME="Ubuntu" VERSION="14.04.4 LTS, Trusty Tahr" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 14.04.4 LTS" VERSION_ID="14.04" HOME_URL="http://www.ubuntu.com/" SUPPORT_URL="http://help.ubuntu.com/" BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
I've just tried it in a fresh Docker container, and dl4j-0.4-examples works fine on Ubuntu 14.04. Does dl4j-0.4-examples also fails on your machine?
I tried the examples in MAC (both from cmd and from IntelliJ) and the examples ran correctly.
BUT in linux:
git clone https://github.com/deeplearning4j/dl4j-0.4-examples
cd dl4j-0.4-examples/
mvn -version
Apache Maven 3.0.5 Maven home: /usr/share/maven Java version: 1.7.0_101, vendor: Oracle Corporation Java home: /usr/lib/jvm/java-7-openjdk-amd64/jre Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "3.13.0-74-generic", arch: "amd64", family: "unix"
mvn -e clean install
[INFO] Error stacktraces are turned on.
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building DeepLearning4j Examples 0.4-rc0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[WARNING] The POM for org.apache.uima:uimaj-core:jar:2.5.0 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
[WARNING] The POM for org.apache.uima:uimafit-core:jar:2.0.0 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
[WARNING] The POM for com.google.guava:guava:jar:17.0 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
[WARNING] The POM for org.cleartk:cleartk-type-system:jar:2.0.0 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
[WARNING] The POM for org.cleartk:cleartk-opennlp-tools:jar:2.0.0 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
[WARNING] The POM for io.dropwizard:dropwizard-assets:jar:0.8.0 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
[WARNING] The POM for io.dropwizard:dropwizard-views-mustache:jar:0.8.0 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
[WARNING] The POM for io.dropwizard:dropwizard-views-freemarker:jar:0.8.0 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
[WARNING] The POM for org.nd4j:nd4j-jackson:jar:0.4-rc3.10 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
[WARNING] The POM for org.deeplearning4j:deeplearning4j-ui:jar:0.4-rc3.10 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
[WARNING] The POM for com.google.guava:guava:jar:19.0 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
[WARNING] The POM for org.nd4j:nd4j-native:jar:0.4-rc3.10 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
[WARNING] The POM for org.nd4j:canova-nd4j-codec:jar:0.0.0.16 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
[WARNING] The POM for jfree:jfreechart:jar:1.0.13 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
[WARNING] The POM for org.deeplearning4j:arbiter-deeplearning4j:jar:0.0.0.7 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ deeplearning4j-examples ---
[WARNING] The POM for org.apache.maven:maven-plugin-api:jar:2.0.6 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.601s
[INFO] Finished at: Wed Jun 08 13:21:38 EDT 2016
[INFO] Final Memory: 22M/1447M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-clean-plugin:2.5:clean (default-clean) on project deeplearning4j-examples: Execution default-clean of goal org.apache.maven.plugins:maven-clean-plugin:2.5:clean failed: A required class was missing while executing org.apache.maven.plugins:maven-clean-plugin:2.5:clean: org/codehaus/plexus/util/Os
[ERROR] -----------------------------------------------------
[ERROR] realm = plugin>org.apache.maven.plugins:maven-clean-plugin:2.5
[ERROR] strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy
[ERROR] urls[0] = file:/home/rasooli/.m2/repository/org/apache/maven/plugins/maven-clean-plugin/2.5/maven-clean-plugin-2.5.jar
[ERROR] urls[1] = file:/home/rasooli/.m2/repository/org/codehaus/plexus/plexus-utils/3.0/plexus-utils-3.0.jar
[ERROR] Number of foreign imports: 1
[ERROR] import: Entry[import from realm ClassRealm[maven.api, parent: null]]
[ERROR]
[ERROR] -----------------------------------------------------: org.codehaus.plexus.util.Os
[ERROR] -> [Help 1]
realm = plugin>org.apache.maven.plugins:maven-clean-plugin:2.5 strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy urls[0] = file:/home/rasooli/.m2/repository/org/apache/maven/plugins/maven-clean-plugin/2.5/maven-clean-plugin-2.5.jar urls[1] = file:/home/rasooli/.m2/repository/org/codehaus/plexus/plexus-utils/3.0/plexus-utils-3.0.jar Number of foreign imports: 1 import: Entry[import from realm ClassRealm[maven.api, parent: null]]
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:225)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
realm = plugin>org.apache.maven.plugins:maven-clean-plugin:2.5 strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy urls[0] = file:/home/rasooli/.m2/repository/org/apache/maven/plugins/maven-clean-plugin/2.5/maven-clean-plugin-2.5.jar urls[1] = file:/home/rasooli/.m2/repository/org/codehaus/plexus/plexus-utils/3.0/plexus-utils-3.0.jar Number of foreign imports: 1 import: Entry[import from realm ClassRealm[maven.api, parent: null]]
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:127)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
... 19 more
realm = plugin>org.apache.maven.plugins:maven-clean-plugin:2.5 strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy urls[0] = file:/home/rasooli/.m2/repository/org/apache/maven/plugins/maven-clean-plugin/2.5/maven-clean-plugin-2.5.jar urls[1] = file:/home/rasooli/.m2/repository/org/codehaus/plexus/plexus-utils/3.0/plexus-utils-3.0.jar Number of foreign imports: 1 import: Entry[import from realm ClassRealm[maven.api, parent: null]]
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:125)
... 20 more
Caused by: java.lang.NoClassDefFoundError: org/codehaus/plexus/util/Os
at org.apache.maven.plugin.clean.Cleaner.
Look at that last part...this looks REALLY suspicious: Caused by: java.lang.NoClassDefFoundError: org/codehaus/plexus/util/Os
Could you try upgrading your maven maybe?
Ok, now after updating Maven, I could successfully run the examples but I still have the same problem when running my code (i.e. Caused by: java.lang.UnsatisfiedLinkError: /tmp/javacpp12671829276729217/libjnind4j.so: /tmp/javacpp12671829276729217/libgomp.so.1: version `GOMP_4.0' not found (required by /tmp/javacpp12671829276729217/libnd4j.so))
I also ran in a friend's machine (it ran ok on his laptop Linux but could faced the same problem when he ran on a server linux machine):
Linux version 3.13.0-85-generic (buildd@lgw01-32) (gcc version 4.8.2 (Ubuntu 4.8.2-19ubuntu1) ) #129-Ubuntu SMP Thu Mar 17 20:50:15 UTC 2016 No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 14.04.4 LTS Release: 14.04 Codename: trusty
My only guess is that both of our servers don't have any GPU drivers and I am adding some GPU-specific maven dependencies in the POM.xml file. Is that right?
Upgrade your maven please.
Finally solved it: I had to specify
Thanks
@rasoolims It's probably not related to GPU, Maven version, or source code directory. If the examples are working fine, but your own pom.xml isn't, it's mostly likely because your pom.xml file makes Maven use an old version of DL4J. Just base your pom.xml file on the examples, and everything will work fine!
@rasoolims hello i am meeting the same problem and can you show ur final pom.xml,thank u very much
@cqiang5882 could you please come in to gitter? https://gitter.im/deeplearning4j/deeplearning4j
I created a simple code with DeepLearning4J that works in my Mac OS but when I go to my linux machine (to run real experiments), I see the following error: