native-image: support statically linking JNI libraries

[smasher164]

native-image can currently produce static or mostly static executables, except for object files loaded with System/loadLibrary when using JNI. This makes it inconvenient to distribute binaries for applications where the user doesn't have the JNI library present on their system.

The only alternative appears to be to build GraalVM from source with library linked, but I would prefer a solution where I could tell native-image a list of object files to statically link into the resulting binary.

[kristofdho]

You can do this using -H:AdditionalLinkerOptions=<path-to-your-lib>, not sure if that's strictly the right way to do it. from the output of native-image --expert-options-all: -H:AdditionalLinkerOptions="" String which would be appended to the linker call.

Alternatively you can write your own feature for your lib and tell native-image exactly what to do with it: https://github.com/oracle/graal/blob/fbab70f9d788f997c862bdee186ef1d8e6c435f1/substratevm/src/com.oracle.svm.hosted/src/com/oracle/svm/hosted/SecurityServicesFeature.java#L245-L258

[smasher164]

Thanks for following up @kristofdho! I made a repo for a minimal proof-of-concept here: https://github.com/smasher164/staticjni. I passed in AdditionalLinkerOptions with the absolute path to the library, and I no longer compile the library with -shared, however I still see the following error:

root@8b3118f6d2a6:~/staticjni# ./helloworld
Exception in thread "main" java.lang.UnsatisfiedLinkError: no HelloWorld in java.library.path
    at com.oracle.svm.core.jdk.NativeLibrarySupport.loadLibraryRelative(NativeLibrarySupport.java:132)
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:275)
    at java.lang.Runtime.loadLibrary0(Runtime.java:830)
    at java.lang.System.loadLibrary(System.java:1871)
    at HelloWorld.main(HelloWorld.java:5)

(Note, I ran this example inside an ubuntu docker container)

Should I not be using loadLibrary? Or is there something I'm missing here?

[kristofdho]

I'm having troubles getting it to work with -H:AdditionalLinkerOptions to the point that I'm doubting if I ever got it working that way..

The second example does work however. I am on windows btw, so I had to figure out some stuff before I could run your example.

In the end, to get it statically linked into the image, I had to add this custom feature to my classpath:

import com.oracle.svm.core.annotate.AutomaticFeature;
import com.oracle.svm.core.jdk.NativeLibrarySupport;
import com.oracle.svm.core.jdk.PlatformNativeLibrarySupport;
import com.oracle.svm.hosted.FeatureImpl;
import com.oracle.svm.hosted.c.NativeLibraries;
import org.graalvm.nativeimage.hosted.Feature;

@AutomaticFeature
public class HelloWorldFeature implements Feature {

    @Override
    public void beforeAnalysis(BeforeAnalysisAccess access) {
        NativeLibrarySupport.singleton().preregisterUninitializedBuiltinLibrary("HelloWorld");
        PlatformNativeLibrarySupport.singleton().addBuiltinPkgNativePrefix("HelloWorld");
        NativeLibraries nativeLibraries = ((FeatureImpl.BeforeAnalysisAccessImpl) access).getNativeLibraries();
        nativeLibraries.addStaticJniLibrary("HelloWorld");
    }
}

For this to compile, you need this dependency:

<dependency>
    <groupId>org.graalvm.nativeimage</groupId>
    <artifactId>svm</artifactId>
    <version>21.0.0</version>
    <scope>provided</scope>
</dependency>

After this, native-image will itself put HelloWorld.lib (since I'm on windows) in the linker command, so all we have to do is make sure the linker can find it, so I had to add -H:CLibraryPath="<path to folder containing library>" to the native-image command.

In the end, my native-image command looked like this:

native-image --no-fallback --no-server --initialize-at-build-time -H:Name=hello -cp HelloWorld.jar -H:CLibraryPath="<path to folder containing library>"

The part that is missing when using AdditionalLinkerOptions is probably the part where you tell native-image that the Java_HelloWorld* symbols are to be resolved internally, which the addBuiltinPkgNativePrefix call does. But don't quote me on that, it's merely a slightly educated guess.

[smasher164]

Thanks @kristofdho, that works! Did you mean -jar HelloWorld.jar instead of -cp HelloWorld.jar in the native-image command? I've updated my repo accordingly.

[kristofdho]

Ah yes, I didn't have a manifest, so I actually used -cp HelloWorld.jar HelloWorld, but I only partially changed it back to your use case. Should indeed be -jar HelloWorld.jar. Glad I could help you!

[smasher164]

Great. I'll go ahead and close this issue.

[smasher164]

Actually, is it possible to put this information in the GraalVM docs somewhere? I understand the user must implement a Feature, but I feel that this is a common enough use-case to warrant its own place in the docs.

[kristofdho]

Unfortunately that's not something I can help you with. And strictly speaking, I think those calls are implementation details, and not part of the API. So for an actual API conform solution some code changes and indeed, definitely documentation, would be required.

[smasher164]

Gotcha. In that case, would the original purpose of this issue still be valid? Since there is no stable (or at least a way we would want to document) way to currently do this. I could imagine a flag like -H:RegisterBuiltin=<comma-delimited list of libraries to resolve internally>

[munishchouhan]

@olyagpl please check it out

[smasher164]

I published a blog post about this method today, and mentioned the caveat that it's undocumented and unstable. See https://www.blog.akhil.cc/static-jni.

[pquiring]

An alternative is to build a --shared library and then build your own loader with JNI libraries baked in. That's what I do now. The loader just has to registerNatives()

[chanseokoh]

@pquiring very interesting. Could you elaborate on how exactly you made it work in your way, step by step? Actually I don't have much experience with JNI, so I don't know what's a loader in this context, how you can build their own loader, how you make .so files baked in where (e.g., as "resources" for native-image like -H:IncludeResources=...)? And how did you confirm that the System.loadLibrary() loads JNI libraries from the baked-in .so files and not from the runtime env?

EDIT: and does it require having source code for .so files so that you can modify and build from source?

[pquiring]

My loaders are here: https://github.com/pquiring/javaforce/tree/master/stubs And JNI libraries are here: https://github.com/pquiring/javaforce/tree/master/native

[Noisyfox]

Should I not be using loadLibrary? Or is there something I'm missing here?

For statically linked lib to make loadLibrary work there have to be an exported function called JNI_OnLoad_<library name>() according to JEP 178, or you can simply don't call loadLibrary at all, and the native method should simply work as long as the native function name matches.

This is what I've done in one of my project that using native-image: https://github.com/multi-os-engine/moe-core/blob/7199da6723cc42c0f6ce4a9afc1fb91f03c1f6d1/moe.apple/moe.core.native/moe.sdk/src/MOE.mm#L313 https://github.com/multi-os-engine/moe-core/blob/7199da6723cc42c0f6ce4a9afc1fb91f03c1f6d1/moe.apple/moe.core.java/src/main/java/org/moe/MOE.java#L27

[iseki0]

Does it easier to be used in Panama?