Missing onnxruntime-genai-jni library

[pekzeki]

Describe the bug Building Java API and use the generated artifacts in another application however I got the below error while using the sample SimpleGenAI class.

Exception in thread "main" java.lang.UnsatisfiedLinkError: no onnxruntime-genai-jni in java.library.path: .../onnxruntime-genai/build/macOS/Release
    at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2434)
    at java.base/java.lang.Runtime.loadLibrary0(Runtime.java:818)
    at java.base/java.lang.System.loadLibrary(System.java:1993)
    at ai.onnxruntime.genai.GenAI.load(GenAI.java:214)
    at ai.onnxruntime.genai.GenAI.init(GenAI.java:61)
    at ai.onnxruntime.genai.Model.<clinit>(Model.java:81)
    at SimpleGenAI.<init>(SimpleGenAI.java:34)
    at Main.main(Main.java:7)

To Reproduce Steps to reproduce the behavior:

1. Build Java API from source
2. Add onnxruntime-genai/src/java/build/libs/onnxruntime-genai-0.4.0-dev as a dependency to your project
3. Add onnxruntime-genai/build/macOS/Release as native library location to your project
4. Run below script

import ai.onnxruntime.genai.GenAIException;
import ai.onnxruntime.genai.GeneratorParams;

public class Main {

    public static void main(String[] args) throws GenAIException {
        SimpleGenAI generator = new SimpleGenAI(".../gemma_2_2b_it");
        GeneratorParams params = generator.createGeneratorParams("What's 6 times 7?");
        String result = generator.generate(params, null);
        System.out.println(result);
    }
}

Expected behavior I expect no missing JNI library errors

Screenshots

Desktop (please complete the following information):

• OS: MacOS (m2)

Additional context

• Python build is successful ✅
• Java build results like below ⬇️

  ...
  BUILD SUCCESSFUL in 1s
  5 actionable tasks: 5 executed
  [ 96%] Built target onnxruntime-genai4j
  [ 96%] Building CXX object src/java/CMakeFiles/onnxruntime-genai-jni.dir/src/main/native/ai_onnxruntime_genai_Tensor.cpp.o
  /onnxruntime-genai/src/java/src/main/native/ai_onnxruntime_genai_Tensor.cpp:24:18: error: cannot initialize a variable of type 'const int64_t *' (aka 'const long long *') with an rvalue of type 'jlong *' (aka 'long *')
  const int64_t* shape_dims = env->GetLongArrayElements(shape_dims_in, /*isCopy*/ 0);
               ^            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  1 error generated.
  make[2]: *** [src/java/CMakeFiles/onnxruntime-genai-jni.dir/src/main/native/ai_onnxruntime_genai_Tensor.cpp.o] Error 1
  make[1]: *** [src/java/CMakeFiles/onnxruntime-genai-jni.dir/all] Error 2
  make: *** [all] Error 2
  …
  'Release']' returned non-zero exit status 2.

[skyline75489]

Hi @pekzeki thanks for the report. I'll see what I can do after I get back my mac device.

[skyline75489]

It seems that AppleClang has more strict type requirement, @pekzeki , could you please try the following change:

-  const int64_t* shape_dims = env->GetLongArrayElements(shape_dims_in, /*isCopy*/ 0);
+  const int64_t* shape_dims = reinterpret_cast<int64_t *>(env->GetLongArrayElements(shape_dims_in, /*isCopy*/ 0));

[pekzeki]

Error has gone. Thanks for the help! @skyline75489

[skyline75489]

@pekzeki Could you please share you experience with GenAI? Did you get the expected output? We have limited resource to fully test GenAI across different devices, especially Apple devices. Any feedback would be helpful. Thanks.

[pekzeki]

Sure!

Currently, I am exploring the capabilities of onnxruntime-genai. Previously, my team built an ML toolkit on onnxruntime, and now I am searching for a component that can help run SLMs (e.g., gemma-2b, phi-4b, etc.) on a CPU. I am specifically interested in running those models on the JVM.

I had to spend some time building the library and following related GitHub issues to get it running. It would have been easier if I could directly download artifacts from somewhere. The library works as expected on my macOS (M3) device. Although I haven't run extensive tests, it seems that running these SLMs on the JVM generates content approximately 20% faster.

One thing I haven't been able to do is compile the library for Windows properly. As I don't have a native Windows machine, I tried doing it on a virtual Windows 11 machine, and while I was able to build the library successfully with the related ARM64 packages, I kept getting the following error when loading:

Can't load ARM 64-bit .dll on an AMD 64-bit platform

Hope this helps 🙏

[pekzeki]

I managed to resolve the ARM <-> AMD compatibility issue after switching to Microsoft ARM64 JDK however I cannot produce the onnxruntime-genai-jni library due to below error.

  BUILD SUCCESSFUL in 15s
  5 actionable tasks: 5 executed
C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(254,5): warning MSB8065: Custom build for item "C:\Users\u1\x64-jdk\onnxruntime-genai\build\Windows\Release\CMakeFiles\4d3ff2e7ea1390088edab65ca44f29c6\onnxruntime-genai.jar.rule" succeeded, but specified output "c:\users\u1\x64-jdk\onnxruntime-genai\src\java\build\libs\onnxruntime-genai.jar" has not been created. This may cause incremental build to work incorrectly. [C:\Users\u1\x64-jdk\onnxruntime-genai\build\Windows\Release\src\java\onnxruntime-genai4j.vcxproj]
  Building Custom Rule C:/Users/u1/x64-jdk/onnxruntime-genai/src/java/CMakeLists.txt
  ai_onnxruntime_genai_Generator.cpp
  ai_onnxruntime_genai_GeneratorParams.cpp
C:\Users\u1\x64-jdk\onnxruntime-genai\src\java\src\main\native\ai_onnxruntime_genai_Generator.cpp(60,47): error C2220: the following warning is treated as an error [C:\Users\u1\x64-jdk\onnxruntime-genai\build\Windows\Release\src\java\onnxruntime-genai-jni.vcxproj]
C:\Users\u1\x64-jdk\onnxruntime-genai\src\java\src\main\native\ai_onnxruntime_genai_Generator.cpp(60,47): warning C4267: 'argument': conversion from 'size_t' to 'jsize', possible loss of data [C:\Users\u1\x64-jdk\onnxruntime-genai\build\Windows\Release\src\java\onnxruntime-genai-jni.vcxproj]
C:\Users\u1\x64-jdk\onnxruntime-genai\src\java\src\main\native\ai_onnxruntime_genai_Generator.cpp(62,45): warning C4267: 'argument': conversion from 'size_t' to 'jsize', possible loss of data [C:\Users\u1\x64-jdk\onnxruntime-genai\build\Windows\Release\src\java\onnxruntime-genai-jni.vcxproj]
  ai_onnxruntime_genai_Images.cpp
  ai_onnxruntime_genai_Model.cpp
  ai_onnxruntime_genai_MultiModalProcessor.cpp
  ai_onnxruntime_genai_NamedTensors.cpp
  ai_onnxruntime_genai_Sequences.cpp
C:\Users\u1\x64-jdk\onnxruntime-genai\src\java\src\main\native\ai_onnxruntime_genai_Sequences.cpp(37,47): error C2220: the following warning is treated as an error [C:\Users\u1\x64-jdk\onnxruntime-genai\build\Windows\Release\src\java\onnxruntime-genai-jni.vcxproj]
C:\Users\u1\x64-jdk\onnxruntime-genai\src\java\src\main\native\ai_onnxruntime_genai_Sequences.cpp(37,47): warning C4267: 'argument': conversion from 'size_t' to 'jsize', possible loss of data [C:\Users\u1\x64-jdk\onnxruntime-genai\build\Windows\Release\src\java\onnxruntime-genai-jni.vcxproj]
C:\Users\u1\x64-jdk\onnxruntime-genai\src\java\src\main\native\ai_onnxruntime_genai_Sequences.cpp(38,45): warning C4267: 'argument': conversion from 'size_t' to 'jsize', possible loss of data [C:\Users\u1\x64-jdk\onnxruntime-genai\build\Windows\Release\src\java\onnxruntime-genai-jni.vcxproj]
  ai_onnxruntime_genai_Tensor.cpp
  ai_onnxruntime_genai_Tokenizer.cpp
  ai_onnxruntime_genai_TokenizerStream.cpp
  utils.cpp
  Generating Code...
  Building Custom Rule C:/Users/u1/x64-jdk/onnxruntime-genai/build/Windows/Release/_deps/googletest-src/googletest/CMakeLists.txt
  gtest-all.cc
  gtest.vcxproj -> C:\Users\u1\x64-jdk\onnxruntime-genai\build\Windows\Release\lib\Release\gtest.lib
  Building Custom Rule C:/Users/u1/x64-jdk/onnxruntime-genai/build/Windows/Release/_deps/googletest-src/googletest/CMakeLists.txt
  gtest_main.cc
  gtest_main.vcxproj -> C:\Users\u1\x64-jdk\onnxruntime-genai\build\Windows\Release\lib\Release\gtest_main.lib
  Building Custom Rule C:/Users/u1/x64-jdk/onnxruntime-genai/test/CMakeLists.txt
  main.cpp
  c_api_tests.cpp
  model_tests.cpp
  sampling_tests.cpp
  sampling_benchmark.cpp
  Generating Code...
     Creating library C:/Users/u1/x64-jdk/onnxruntime-genai/build/Windows/Release/test/Release/unit_tests.lib and object C:/Users/u1/x64-jdk/onnxruntime-genai/build/Windows/Release/test/Release/unit_tests.exp
  unit_tests.vcxproj -> C:\Users\u1\x64-jdk\onnxruntime-genai\build\Windows\Release\test\Release\unit_tests.exe
Traceback (most recent call last):
  File "C:\Users\u1\x64-jdk\onnxruntime-genai\build.py", line 571, in <module>
    build(arguments, environment)
  File "C:\Users\u1\x64-jdk\onnxruntime-genai\build.py", line 520, in build
    util.run(make_command, env=env)
  File "C:\Users\u1\x64-jdk\onnxruntime-genai\tools\python\util\run.py", line 56, in run
    completed_process = subprocess.run(
                        ^^^^^^^^^^^^^^^
  File "C:\Users\u1\AppData\Local\Programs\Python\Python312-arm64\Lib\subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['C:\\Program Files\\CMake\\bin\\cmake.EXE', '--build', 'C:\\Users\\u1\\x64-jdk\\onnxruntime-genai\\build\\Windows\\Release', '--config', 'Release']' returned non-zero exit status 1.

Do you have any suggestions?

[skyline75489]

We're treating all warnings as errors on Windows. I think a cast would work. Could you please try:

- jintArray java_int_array = env->NewIntArray(num_tokens);
+ jintArray java_int_array = env->NewIntArray((jsize)num_tokens);

[natke]

@pekzeki Did you manage to try the above suggestion?