[jvm-packages] issue jni.h file cmake: subprocess.CalledProcessError: Command 'cmake --build . --config Release' returned non-zero exit status 2

[blomqvistkj]

Hi, I'm trying to download XGBoost and the Java package. I think I've successfully installed XGBoost, but run into problems when trying to install the jvm. I've googled like crazy and read so many issues here, but none seem to help me. Would be super thankful to get some help!

Environment info

Operating System: OS Sierra 10.12.6 Compiler: gcc7 other: maven 3.5.2, CMake 3.10.2, java: JDK 1.8.0_161 Package used (python/R/jvm/C++): jvm xgboost version used: the latest?

Steps to reproduce

1. Downloading XGBoost and all other required software
2. Set JAVA_HOME environment variable
3. run 'mvn package'

What have you tried?

1. this issue: https://github.com/dmlc/xgboost/issues/2596 (these problems seems to be fixed)
2. this issue: https://github.com/dmlc/xgboost/issues/2603 (did not work)

I get the following error when running "mvn package": What could be the problem?

[INFO] --- exec-maven-plugin:1.6.0:exec (native) @ xgboost4j --- -- Could NOT find OpenMP_C (missing: OpenMP_C_FLAGS OpenMP_C_LIB_NAMES) -- Could NOT find OpenMP_CXX (missing: OpenMP_CXX_FLAGS OpenMP_CXX_LIB_NAMES) -- Could NOT find OpenMP (missing: OpenMP_C_FOUND OpenMP_CXX_FOUND) -- Configuring done -- Generating done -- Build files have been written to: /Users/johannablomqvist/xgboost/build [ 96%] Built target dmlccore [ 96%] Built target objxgboost [ 96%] Built target rabit [ 96%] Built target runxgboost [ 96%] Building CXX object CMakeFiles/xgboost4j.dir/jvm-packages/xgboost4j/src/native/xgboost4j.cpp.o In file included from /Users/johannablomqvist/xgboost/jvm-packages/xgboost4j/src/native/xgboost4j.cpp:17: In file included from /Users/johannablomqvist/xgboost/include/xgboost/base.h:10: /Users/johannablomqvist/xgboost/dmlc-core/include/dmlc/omp.h:13:9: warning: Warning: OpenMP is not available, project will be compiled into single-thread code. Use OpenMP-enabled compiler to get benefit of multi-threading. [-W#pragma-messages]

pragma message("Warning: OpenMP is not available, " \

    ^

In file included from /Users/johannablomqvist/xgboost/jvm-packages/xgboost4j/src/native/xgboost4j.cpp:19: /Users/johannablomqvist/xgboost/jvm-packages/xgboost4j/src/native/./xgboost4j.h:2:10: fatal error: 'jni.h' file not found

include

     ^~~~~~~

1 warning and 1 error generated. make[2]: [CMakeFiles/xgboost4j.dir/jvm-packages/xgboost4j/src/native/xgboost4j.cpp.o] Error 1 make[1]: [CMakeFiles/xgboost4j.dir/all] Error 2 make: * [all] Error 2 building Java wrapper Traceback (most recent call last): cd .. mkdir -p build File "create_jni.py", line 89, in cd build cmake .. -DUSE_S3:BOOL=OFF -DUSE_OPENMP:BOOL=OFF -DPLUGIN_UPDATER_GPU:BOOL=OFF -DUSE_AZURE:BOOL=OFF -DUSE_HDFS:BOOL=OFF -DJVM_BINDINGS:BOOL=ON cmake --build . --config Release run("cmake --build . --config Release") File "create_jni.py", line 51, in run subprocess.check_call(command, shell=True, kwargs) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 540, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'cmake --build . --config Release' returned non-zero exit status 2 [ERROR] Command execution failed. org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1) at org.apache.commons.exec.DefaultExecutor.executeInternal (DefaultExecutor.java:404) at org.apache.commons.exec.DefaultExecutor.execute (DefaultExecutor.java:166) at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:804) at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:751) at org.codehaus.mojo.exec.ExecMojo.execute (ExecMojo.java:313) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:134) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:208) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:51) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:309) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:194) at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:107) at org.apache.maven.cli.MavenCli.execute (MavenCli.java:955) at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:290) at org.apache.maven.cli.MavenCli.main (MavenCli.java:194) at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke (Method.java:498) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289) at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415) at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356) [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] xgboost-jvm ........................................ SUCCESS [ 9.660 s] [INFO] xgboost4j .......................................... FAILURE [ 3.887 s] [INFO] xgboost4j-spark .................................... SKIPPED [INFO] xgboost4j-flink .................................... SKIPPED [INFO] xgboost4j-example .................................. SKIPPED [INFO] keystroke-dynamics ................................. SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 13.715 s [INFO] Finished at: 2018-02-12T15:27:02+01:00 [INFO] Final Memory: 28M/438M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.6.0:exec (native) on project xgboost4j: Command execution failed.: Process exited with an error: 1 (Exit value: 1) -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :xgboost4j

[Craigacp]

Is JAVA_HOME set in the terminal? It's failing to find the jni header file which should be in $JAVA_HOME/include/ on your JDK.

[superbobry]

Try this before compiling (apparently macOS does not define $JAVA_HOME):

export JAVA_HOME=$(/usr/libexec/java_home)

[blomqvistkj]

Thank you for replying!

I think the JAVA_HOME is set correct, or maybe it should be "further down"? (like Contents/Home/include/)

echo $JAVA_HOME returns /Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home

[Craigacp]

Looks like create_jni.py checks for Java home and instantiates it for you, probably overriding anything you've set.

I'd try running "make jvm" in the xgboost root dir and seeing if that completes correctly (with JAVA_HOME set). That will make the JNI binding, but it won't put it in the right place, so it'll need renaming and moving to xgboost/jvm-packages/xgboost4j/src/main/resources/lib.

[blomqvistkj]

Ah, I see!

I tried "make jvm" and this was the output:

Makefile:31: MAKE [/Applications/Xcode.app/Contents/Developer/usr/bin/make] - checked OK g++-7 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -msse2 -fPIC -fopenmp -I/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/include -I./java -I/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/include/darwin -shared -o jvm-packages/lib/libxgboost4j.so jvm-packages/xgboost4j/src/native/xgboost4j.cpp build/learner.o build/logging.o build/c_api/c_api.o build/c_api/c_api_error.o build/common/common.o build/common/hist_util.o build/common/host_device_vector.o build/data/data.o build/data/simple_csr_source.o build/data/simple_dmatrix.o build/data/sparse_page_dmatrix.o build/data/sparse_page_raw_format.o build/data/sparse_page_source.o build/data/sparse_page_writer.o build/gbm/gblinear.o build/gbm/gbm.o build/gbm/gbtree.o build/metric/elementwise_metric.o build/metric/metric.o build/metric/multiclass_metric.o build/metric/rank_metric.o build/objective/multiclass_obj.o build/objective/objective.o build/objective/rank_obj.o build/objective/regression_obj.o build/predictor/cpu_predictor.o build/predictor/predictor.o build/tree/tree_model.o build/tree/tree_updater.o build/tree/updater_colmaker.o build/tree/updater_fast_hist.o build/tree/updater_histmaker.o build/tree/updater_prune.o build/tree/updater_refresh.o build/tree/updater_skmaker.o build/tree/updater_sync.o dmlc-core/libdmlc.a rabit/lib/librabit.a -pthread -lm -fopenmp

I can't really decipher it, did it compile as expected? If so, how do I move the "JNI binding"?

[Craigacp]

It compiled as expected, and will have made a libxgboost4j.so in jvm-packages/lib/, which you should rename to libxgboost4j.dylib. It looks like the python script doesn't check to see if the file has been updated, so it might trigger CMake again, which hopefully should notice the library is already there and not do anything. If it fails to compile again then comment out line 67 in xgboost/jvm-packages/xgboost4j/pom.xml, and rerun mvn package.

[blomqvistkj]

Thank you!! I'm running right now, will update when it has finished!

[blomqvistkj]

I tried you approach, and it seems like the installation is getting a bit further, but it gets stuck. I ran it for approx 6h yesterday, with no luck.

This is the output/where it gets stuck:

[INFO] --- scalatest-maven-plugin:1.0:test (test) @ xgboost-jvm --- java.io.FileNotFoundException: /Users/johannablomqvist/xgboost/jvm-packages/target/jacoco.exec (Permission denied) at java.io.FileOutputStream.open0(Native Method) at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.(FileOutputStream.java:213) at org.jacoco.agent.rt.internal_8ff85ea.output.FileOutput.openFile(FileOutput.java:67) at org.jacoco.agent.rt.internal_8ff85ea.output.FileOutput.startup(FileOutput.java:49) at org.jacoco.agent.rt.internal_8ff85ea.Agent.startup(Agent.java:122) at org.jacoco.agent.rt.internal_8ff85ea.Agent.getInstance(Agent.java:50) at org.jacoco.agent.rt.internal_8ff85ea.PreMain.premain(PreMain.java:45) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386) at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401) Discovery starting. Discovery completed in 82 milliseconds. Run starting. Expected test count is: 0 DiscoverySuite: Run completed in 172 milliseconds. Total number of tests run: 0 Suites: completed 1, aborted 0 Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0 No tests were executed. java.io.FileNotFoundException: /Users/johannablomqvist/xgboost/jvm-packages/target/jacoco.exec (Permission denied) at java.io.FileOutputStream.open0(Native Method) at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.(FileOutputStream.java:213) at org.jacoco.agent.rt.internal_8ff85ea.output.FileOutput.openFile(FileOutput.java:67) at org.jacoco.agent.rt.internal_8ff85ea.output.FileOutput.writeExecutionData(FileOutput.java:53) at org.jacoco.agent.rt.internal_8ff85ea.Agent.shutdown(Agent.java:137) at org.jacoco.agent.rt.internal_8ff85ea.Agent$1.run(Agent.java:54) [INFO] [INFO] --- jacoco-maven-plugin:0.7.9:report (report) @ xgboost-jvm --- [INFO] Skipping JaCoCo execution due to missing classes directory. [INFO] [INFO] --- maven-assembly-plugin:2.6:single (make-assembly) @ xgboost-jvm --- [INFO] Assemblies have been skipped per configuration of the skipAssembly parameter. [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building xgboost4j 0.8-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- scalastyle-maven-plugin:0.8.0:check (checkstyle) @ xgboost4j --- Processed 14 file(s) Found 0 errors Found 0 warnings Found 0 infos Finished in 1756 ms [INFO] [INFO] --- maven-checkstyle-plugin:2.17:check (checkstyle) @ xgboost4j --- [INFO] [INFO] --- jacoco-maven-plugin:0.7.9:prepare-agent (default) @ xgboost4j --- [INFO] argLine set to -javaagent:/Users/johannablomqvist/.m2/repository/org/jacoco/org.jacoco.agent/0.7.9/org.jacoco.agent-0.7.9-runtime.jar=destfile=/Users/johannablomqvist/xgboost/jvm-packages/xgboost4j/target/jacoco.exec [INFO] [INFO] --- exec-maven-plugin:1.6.0:exec (native) @ xgboost4j ---

[Craigacp]

That error looks unrelated (you might have some permissions error, but jacoco is doing code coverage, it's unrelated to the compilation). What's the error from the bottom of the output?

[blomqvistkj]

This is the last output, it just gets stuck after the last line (ran it for approx 10 hours). I have now moved on to writing in python instead (which worked fine to install). Should I close the issue or keep it open if someone wants to add/help others in the future?

[Craigacp]

I think this just indicates that the JVM build path needs fixing on macOS, so I'd leave it open until someone gets around to doing that.

[tqchen]

installation questions are now moved to https://discuss.xgboost.ai/