Closed suming closed 1 year ago
Those makefiles are just think wrappers around some cmake commands and aren't used much so might not be up to date (as is the case here).
Please see Java build instructions here: https://github.com/VowpalWabbit/vowpal_wabbit/wiki/Java#build-from-source
Thanks for the tip, following those instructions exactly, before I run
make -j$(nproc)
I assume I need to be in the root directory right? Since running from the 'build/' directory gives me
make: *** No targets specified and no makefile found. Stop.
Nvm, re-cloned and ran from a fresh directory. It's building in that directory now, but now the issue is when it gets to 100%, I see
[100%] Linking CXX executable vw-unit-test.out
ld: warning: ignoring file /opt/homebrew/lib/libboost_unit_test_framework-mt.dylib, building for macOS-x86_64 but attempting to link with file built for macOS-arm64
Undefined symbols for architecture x86_64:
I am on arm64, so not sure where I can change the instructinos to build for arm64 instead of x86_64
@jackgerrits when I run the final command from the "Build from Source" section on the link you shared, I see build failures still:
[ 1%] Built target gtest
[ 2%] Built target gmock
[ 2%] Built target gmock_main
[ 2%] Built target gtest_main
[ 2%] Built target fmt
[ 4%] Built target spdlog
[ 4%] Built target vw_io
[ 5%] Built target vw_allreduce
[ 8%] Built target vw_config
[ 56%] Built target vw_core
[ 57%] Built target library_example
[ 57%] Built target test_search
[ 58%] Built target search_generate
[ 58%] Built target recommend
[ 59%] Built target gd_mf_weights
[ 59%] Built target vw_active_interactor_bin
[ 60%] Built target vw_c_wrapper
[ 61%] Built target vw_c_wrapper_test
[ 61%] Built target vw_cli_bin
[ 62%] Built target vw_config_test
[ 65%] Built target vw_core_test
[ 66%] Built target vw_explore_test
[ 68%] Built target vw_io_test
[ 69%] Built target vw_model_merger_bin
[ 72%] Built target vw_slim
[ 73%] Built target vw_slim_test
[ 74%] Built target vw_spanning_tree
[ 75%] Built target vw_spanning_tree_bin
[ 76%] Built target vw-dump-options
[ 77%] Built target vw_jni_generate_native_headers_do_not_use_jar
[ 77%] Linking CXX shared library libvw_jni.dylib
Copying shared libary dependencies to output directory
[INFO] Scanning for projects...
[INFO] Inspecting build with total of 1 modules...
[INFO] Installing Nexus Staging features:
[INFO] ... total of 1 executions of maven-deploy-plugin replaced with nexus-staging-maven-plugin
[INFO]
[INFO] -------------------< com.github.vowpalwabbit:vw-jni >-------------------
[INFO] Building Vowpal Wabbit JNI Layer 9.6.0
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-enforcer-plugin:1.1:enforce (enforce-ban-duplicate-classes) @ vw-jni ---
[INFO]
[INFO] --- maven-enforcer-plugin:1.1:enforce (enforce-ban-version-downgrades) @ vw-jni ---
[INFO]
[INFO] --- maven-resources-plugin:2.7:resources (default-resources) @ vw-jni ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO]
[INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ vw-jni ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 35 source files to /Users/such/code/vw3/vowpal_wabbit/java/target/classes
[INFO]
[INFO] --- maven-resources-plugin:2.7:copy-resources (copy-resources) @ vw-jni ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 0 resource
[INFO]
[INFO] --- maven-resources-plugin:2.7:testResources (default-testResources) @ vw-jni ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 0 resource
[INFO]
[INFO] --- maven-compiler-plugin:3.8.0:testCompile (default-testCompile) @ vw-jni ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-surefire-plugin:2.17:test (default-test) @ vw-jni ---
[INFO] Surefire report directory: /Users/such/code/vw3/vowpal_wabbit/java/target/surefire-reports
-------------------------------------------------------
T E S T S
-------------------------------------------------------
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running vowpalWabbit.ClosingTest
Tests run: 8, Failures: 0, Errors: 8, Skipped: 0, Time elapsed: 0.046 sec <<< FAILURE! - in vowpalWabbit.ClosingTest
testCloseAsyncIsNearImmediate(vowpalWabbit.ClosingTest) Time elapsed: 0.018 sec <<< ERROR!
java.lang.ExceptionInInitializerError: null
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.<init>(ZipFile.java:230)
at java.util.zip.ZipFile.<init>(ZipFile.java:160)
at java.util.jar.JarFile.<init>(JarFile.java:168)
at java.util.jar.JarFile.<init>(JarFile.java:132)
at common.Native.try_load_from_jar(Native.java:26)
at common.Native.<clinit>(Native.java:60)
at vowpalWabbit.learner.VWLearners.<clinit>(VWLearners.java:22)
at vowpalWabbit.ClosingTest.model(ClosingTest.java:185)
at vowpalWabbit.ClosingTest.model(ClosingTest.java:171)
at vowpalWabbit.ClosingTest.testCloseAsyncIsNearImmediate(ClosingTest.java:40)
multipleAsynchronousCloserCallsReturnTrueOnlyOnce(vowpalWabbit.ClosingTest) Time elapsed: 0 sec <<< ERROR!
java.lang.NoClassDefFoundError: Could not initialize class vowpalWabbit.learner.VWLearners
at vowpalWabbit.ClosingTest.model(ClosingTest.java:185)
at vowpalWabbit.ClosingTest.model(ClosingTest.java:171)
at vowpalWabbit.ClosingTest.multipleAsynchronousCloserCallsReturnTrueOnlyOnce(ClosingTest.java:97)
closerThenClose(vowpalWabbit.ClosingTest) Time elapsed: 0 sec <<< ERROR!
java.lang.NoClassDefFoundError: Could not initialize class vowpalWabbit.learner.VWLearners
at vowpalWabbit.ClosingTest.model(ClosingTest.java:185)
at vowpalWabbit.ClosingTest.model(ClosingTest.java:171)
at vowpalWabbit.ClosingTest.closerThenClose(ClosingTest.java:69)
testRegisteringClosingCallbackWithLongDelay(vowpalWabbit.ClosingTest) Time elapsed: 0.001 sec <<< ERROR!
java.lang.NoClassDefFoundError: Could not initialize class vowpalWabbit.learner.VWLearners
at vowpalWabbit.ClosingTest.model(ClosingTest.java:185)
at vowpalWabbit.ClosingTest.model(ClosingTest.java:171)
at vowpalWabbit.ClosingTest.testClosingCallback(ClosingTest.java:143)
at vowpalWabbit.ClosingTest.testRegisteringClosingCallbackWithLongDelay(ClosingTest.java:133)
...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.732 s
[INFO] Finished at: 2023-02-28T16:01:23-08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on project vw-jni: There are test failures.
[ERROR]
[ERROR] Please refer to /Users/such/code/vw3/vowpal_wabbit/java/target/surefire-reports for the individual test results.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
make[2]: *** [java/CMakeFiles/vw_jni.dir/build.make:318: java/libvw_jni.dylib] Error 1
make[2]: *** Deleting file 'java/libvw_jni.dylib'
make[1]: *** [CMakeFiles/Makefile2:2283: java/CMakeFiles/vw_jni.dir/all] Error 2
make: *** [Makefile:121: all] Error 2
any idea?
Some more context -- I have maven and jdk both installed: Apache Maven 3.9.0 (9b58d2bad23a66be161c4664ef21ce219c2c8584) Maven home: /opt/homebrew/Cellar/maven/3.9.0/libexec Java version: 1.8.0_275, vendor: Azul Systems, Inc., runtime: /usr/local/lib/java/Contents/Home/jre Default locale: en_US, platform encoding: UTF-8 OS name: "mac os x", version: "12.5.1", arch: "aarch64", family: "mac"
I ran exactly
git clone --recursive https://github.com/VowpalWabbit/vowpal_wabbit.git
cd vowpal_wabbit/
mkdir build
cd build
cmake -DBUILD_JAVA=ON ..
make
after running cmake
command, I see
cmake -DBUILD_JAVA=ON .. ✘ 130 master
-- VowpalWabbit Version: 9.7.0
-- No build type selected, defaulting to Release
-- The C compiler identification is AppleClang 13.1.6.13160021
-- The CXX compiler identification is AppleClang 13.1.6.13160021
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Using C++ standard: 11
CMake Warning (dev) at /opt/homebrew/Cellar/cmake/3.25.2/share/cmake/Modules/FetchContent.cmake:1284 (message):
The DOWNLOAD_EXTRACT_TIMESTAMP option was not given and policy CMP0135 is
not set. The policy's OLD behavior will be used. When using a URL
download, the timestamps of extracted files should preferably be that of
the time of extraction, otherwise code that depends on the extracted
contents might not be rebuilt if the URL changes. The OLD behavior
preserves the timestamps from the archive instead, but this is usually not
what you want. Update your project to the NEW behavior or specify the
DOWNLOAD_EXTRACT_TIMESTAMP option with a value of true to avoid this
robustness issue.
Call Stack (most recent call first):
cmake/VowpalWabbitUtils.cmake:23 (FetchContent_Declare)
CMakeLists.txt:85 (include)
This warning is for project developers. Use -Wno-dev to suppress it.
-- Found Python: /opt/homebrew/Frameworks/Python.framework/Versions/3.11/bin/python3.11 (found version "3.11.2") found components: Interpreter
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Found Git: /opt/homebrew/bin/git (found version "2.39.0")
-- Git Version: c0ba1804b
-- Module support is disabled.
-- Version: 9.1.0
-- Build type: Release
-- CXX_STANDARD: 11
-- Performing Test has_std_11_flag
-- Performing Test has_std_11_flag - Success
-- Performing Test has_std_0x_flag
-- Performing Test has_std_0x_flag - Success
-- Required features: cxx_variadic_templates
-- Performing Test HAS_NULLPTR_WARNING
-- Performing Test HAS_NULLPTR_WARNING - Success
-- Build spdlog: 1.11.0
-- Build type: Release
-- Boost.Math: standalone mode ON
-- Found ZLIB: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk/usr/lib/libz.tbd (found version "1.2.11")
-- help2man not found, please install it to generate manpages
-- Found JNI: /usr/local/lib/java/Contents/Home/include found components: AWT JVM
-- Found Java: /usr/local/lib/java/Contents/Home/bin/java (found version "1.8.0.275")
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/such/code/vw4/vowpal_wabbit/build
Then when running make
I see the same error as above,
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 6.154 s
[INFO] Finished at: 2023-02-28T16:19:55-08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on project vw-jni: There are test failures.
[ERROR]
[ERROR] Please refer to /Users/such/code/vw4/vowpal_wabbit/java/target/surefire-reports for the individual test results.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
where (maybe?) due to these errors, compiled JNI library is not found at anywhere in the directory.
I know for the released jar the library loading logic is only implemented for Linux x86 (it searches in the jar). It's possible the loading logic is not implemented for Mac or Mac arm. I'm not sure as it's been a while since I've looked at this.
See related #2521, #1579
Thanks for the response. Is there any version in the past, e.g. 8.x.x where Mac arm is supported?
Any other workarounds that may be possible?
Thanks
On Tue, Feb 28, 2023, 5:19 PM Jack Gerrits @.***> wrote:
I know for the released jar the library loading logic is only implemented for Linux x86 (it searches in the jar). It's possible the loading logic is not implemented for Mac or Mac arm. I'm not sure as it's been a while since I've looked at this.
— Reply to this email directly, view it on GitHub https://github.com/VowpalWabbit/vowpal_wabbit/issues/4515#issuecomment-1449172729, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAF4TV63KCRNXJ23UGXS5DDWZ2P3JANCNFSM6AAAAAAVLKBBV4 . You are receiving this because you authored the thread.Message ID: @.***>
Mac arm is not explicitly supported or unsupported. The Jar has only ever had Linux x86 in it though. For usage without the jar this code tries to load the lib, so you may need to move the built vw_jni shared object or update the path Java is using to search for the shared object
thanks for the response!
The issue is I can't build vw_jni shared object with ARM. Does anyone have a built vw_jni version they can share?
On Wed, Mar 1, 2023, 9:35 AM Jack Gerrits @.***> wrote:
Mac arm is not explicitly supported or unsupported. The Jar has only ever had Linux x86 in it though. For usage without the jar this code https://github.com/VowpalWabbit/vowpal_wabbit/blob/c0ba1804be1b94a12db1b7ef5d891af24ff37748/java/src/main/java/common/Native.java#L14 tries to load the lib, so you may need to move the built vw_jni shared object or update the path Java is using to search for the shared object
— Reply to this email directly, view it on GitHub https://github.com/VowpalWabbit/vowpal_wabbit/issues/4515#issuecomment-1450561153, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAF4TVZGG2IHXX5KYX7J2UDWZ6CGFANCNFSM6AAAAAAVLKBBV4 . You are receiving this because you authored the thread.Message ID: @.***>
The build error you posted was related to unit tests which aren't needed to get vw_jni
built, did vw_jni
itself build successfully?
Use cmake --build build -t vw_jni
just just build vw_jni
Thanks for the response, sorry I wasn't clear before -- vw_jni
doesn't look like it is getting built. The only vw_jni
that I see is built for linux_64
. Is that a bug and that is actually OK to use for mac?
classes/natives/linux_64/libvw_jni.dylib
Btw, even when I run cmake --build build -t vw_jni
, I see a variety of test errors:
testConcurrency(vowpalWabbit.learner.VWLearnersTest) Time elapsed: 0.001 sec <<< ERROR!
java.lang.NoClassDefFoundError: Could not initialize class vowpalWabbit.learner.VWLearners
at vowpalWabbit.learner.VWLearnersTest.testConcurrency(VWLearnersTest.java:99)
testSaveModel(vowpalWabbit.learner.VWLearnersTest) Time elapsed: 0.001 sec <<< ERROR!
java.lang.NoClassDefFoundError: Could not initialize class vowpalWabbit.learner.VWLearners
at vowpalWabbit.learner.VWLearnersTest.testSaveModel(VWLearnersTest.java:78)
testOldModel(vowpalWabbit.learner.VWLearnersTest) Time elapsed: 0.002 sec <<< FAILURE!
That path is hardcoded, it doesn't mean it is a linux x64 binary. The test failures show it fails to load the vw_jni
lib, so you may need to move the built vw_jni shared object or update the path Java is using to search for the shared object
Is there any other way besides running the tests that I can verify that vw_jni
is being built successfully? From the build/
directory if I am able to run vowpalwabbit/cli/vw
and see :
using no cache
Reading datafile = stdin
num sources = 1
Num weight bits = 18
learning rate = 0.5
initial_t = 0
power_t = 0.5
Enabled learners: gd, scorer-identity, count_label
Does this mean that vw_jni
works?
I did what you suggested and moved classes/natives/linux_64/libvw_jni.dylib
to /Library/Java/Extensions
(which is what shows up under java -XshowSettings:properties
), but when I run cmake --build build -t vw_jni
I still see test errors, so it's unclear if vw_jni
was built correctly.
Does this mean that vw_jni works?
It's a separate binary so its positive but doesn't necessary mean the Java is all good.
Getting the shared object loaded can be a bit tricky and can be environment dependent so I am not sure how to resolve your loading issue.
I'm going to go ahead and close this one, please feel free to reopen if you are still facing issues or open a new issue
Describe the bug
I cloned via
git clone --recursive https://github.com/VowpalWabbit/vowpal_wabbit.git
make java
results in
~/c/v/vowpal_wabbit ❯❯❯ java -version ✘ 1 newmaster ✱ ◼ openjdk version "1.8.0_275" OpenJDK Runtime Environment (Zulu 8.50.0.1017-CA-macos-aarch64) (build 1.8.0_275-b01) OpenJDK 64-Bit Server VM (Zulu 8.50.0.1017-CA-macos-aarch64) (build 25.275-b01, mixed mode)