Documentation for shipping native libraries in JARs

[lvh]

Is there any documentation for how to structure a JAR so that native libraries can get shipped as part of a distribution, and LibraryLoader will be able to load them? Ostensibly, LibraryLoader just takes filesystem paths.

[ghost]

Would be good if there's also mention on how this works to support different operating systems.

[headius]

There's unfortunately no standard way to do this. Native libraries (on every platform I know) must be loose on the filesystem to be loaded, and certainly not within a compressed jar file. That's why jnr-ffi attempts to unpack the libraries somewhere, and why JRuby ships with them already unpacked in a lib directory.

The logic we use to unpack those jars could be made generally usable, of course. I don't expect it would be too much work. Someone want to put together a PR?

[lvh]

I could probably do it if I could figure out how. Is there some sample project or something that users jnr-ffi and does this?

[m4dc4p]

Does jnr-ffi support having libraries in a JAR? Where is that code? I would love to use this feature. Thank you!

[danielcompton]

@lvh this is how sqlite-jdbc does it (using JNI, not JNR). They extract the platform specific library into a temp directory, and work with it from there.

[lvh]

@danielcompton Yeah; I imagine JNR has to do something similar. Thanks!

[lvh]

@headius You mentioned jnr-ffi unpacks libraries somewhere, but I was unable to find any code that does that. Do you have a blessed sample, or should I try to hack this together myself?

[goto1134]

@lvh There is no code for unpacking, as I see. Need PR.

[headius]

It's not a bad idea to add an API for unpacking your FFI-accessed native libraries, but it's not something I have time to work on right now.

The logic for this in JNR lives in the base jffi library here: https://github.com/jnr/jffi/blob/master/src/main/java/com/kenai/jffi/internal/StubLoader.java

[Kaiser1989]

I pack all libraries in jar's root. At runtime they're copied into temp folder to load them as Library:

public class NativeLibraryLoader {

    /**
     * Unpack library from jar and copy it to temp file
     * @param library Name of the library file inside the jar
     * @return Return temp file with copy of library
     * @throws IOException
     */
    public static File createTempFileFromJar(final String library) throws IOException {
        // create temp folder
        final Path tempFolder = Files.createDirectory(new File(System.getProperty("java.io.tmpdir"), "lib" + System.nanoTime()).toPath());
        // copy library to temp file in temp folder
        final File tempFile = new File(tempFolder.toFile(), library);
        try (InputStream is = NativeLibraryLoader.class.getResourceAsStream("/" + library)) {
            Files.copy(is, tempFile.toPath(), StandardCopyOption.REPLACE_EXISTING);
        } finally {
            tempFolder.toFile().deleteOnExit();
        }
        // set folder to be deleted after VM shuts down
        return tempFile;
    }
}

[headius]

Since this issue woke up again I'd like to point out the maven native archive plugin which handles shipping and building sources and shipping and unpacking pre-built native binaries. if I were to make a recommendation today, I'd say to look into this, and I hope to have time to explore using this in JNR itself (e.g. for the jffi native stub).

[PumpkinXD]

It's not a bad idea to add an API for unpacking your FFI-accessed native libraries, but it's not something I have time to work on right now.

The logic for this in JNR lives in the base jffi library here: https://github.com/jnr/jffi/blob/master/src/main/java/com/kenai/jffi/internal/StubLoader.java

@headius

uh... if I want use jnr-ffi in a minecraft mod(forge1.8.9, it's a single jar and it uses gradle not maven) should I write the extract part? you mentioned the logic exist in the base lib jffi but I didn't find any reference or calling in jnr-ffi yet :confused: (currently, I extracting natives from the jar myself, because jna3.4(mc1.8.9 is using it so I can't change version or shading newer version) don't have such feature)

[PumpkinXD]

well... I guess I messed up something...

package io.github.pumpkinxd.examples;

import com.kenai.jffi.internal.StubLoader;
import jnr.ffi.LibraryLoader;

public class Main {
    public interface testLib{
        void sayHiJavaFromC();
    }

//    private static final testLib testLib = LibraryLoader.create(testLib.class).load("test");

    public static void main(String[] args) throws InterruptedException {
        try {
            System.out.println("hello from java\n");

            final testLib testLib = LibraryLoader.create(testLib.class).load("test");
            testLib.sayHiJavaFromC();
        }catch(Exception e) {
        String emsg = e.getMessage().toString();
            System.out.println(emsg+"\n");
            Thread.sleep(2000);
        }finally {
            System.out.println("\nbye\n");
//            System.exit(0);
        }

    }
}

build.gradle.kt

``` kotlin import org.gradle.nativeplatform.platform.internal.Architectures import org.gradle.nativeplatform.platform.internal.OperatingSystemInternal import org.gradle.nativeplatform.platform.internal.DefaultNativePlatform plugins { id("java") id("com.github.johnrengelman.shadow") version "8.1.1" id("org.tboox.gradle-xmake-plugin") version "1.1.5" } group = "io.github.pumpkinxd.examples" version = "1.0-SNAPSHOT" repositories { maven("https://maven.aliyun.com/repository/public/") mavenCentral() } val shadowImpl: Configuration by configurations.creating { configurations.implementation.get().extendsFrom(this) } dependencies { shadowImpl("com.github.jnr:jnr-ffi:2.2.16") shadowImpl("com.github.jnr:jffi:1.3.13") // implementation("com.github.jnr:jnr-ffi:2.2.16") // implementation("com.github.jnr:jffi:1.3.13") testImplementation(platform("org.junit:junit-bom:5.9.1")) testImplementation("org.junit.jupiter:junit-jupiter") } tasks.test { useJUnitPlatform() } tasks.register("buildNative"){ fun String.execute() = org.codehaus.groovy.runtime.ProcessGroovyMethods.execute(this) fun Process.waitForProcessOutput(out: java.io.OutputStream,err: java.io.OutputStream)= org.codehaus.groovy.runtime.ProcessGroovyMethods.waitForProcessOutput(this,out,err) // Runtime.getRuntime().exec("CHDIR /jni").waitForProcessOutput(System.out,System.err) Runtime.getRuntime().exec("./jni/build.bat").waitForProcessOutput(System.out,System.err) } //tasks.register("copyNative") { // from("build/lib/test")//? // to("build/jni")//https://github.com/jnr/jffi/blob/f2ccd60d8ab52387d9bfcd46cc15349df7b77dbf/src/main/java/com/kenai/jffi/internal/StubLoader.java#L295 //} tasks.jar{ println("hello from jar ") dependsOn("buildNative") archiveBaseName.set("NOshadow") archiveClassifier.set("") archiveVersion.set("") enabled=false } tasks.shadowJar { enabled=true // dependsOn("jar") println("hello from shadow") dependsOn("buildNative") configurations = listOf(shadowImpl) duplicatesStrategy = DuplicatesStrategy.EXCLUDE archiveBaseName.set("test") archiveClassifier.set("shadow") archiveVersion.set("") manifest { attributes["Main-Class"] = "io.github.pumpkinxd.examples.Main" } if(DefaultNativePlatform.getCurrentOperatingSystem().isWindows&&DefaultNativePlatform.getCurrentArchitecture().isAmd64){ from( "./jni/build/" + DefaultNativePlatform.getCurrentOperatingSystem().internalOs.familyName + "/" + "x64/release" ) { include(System.mapLibraryName("test")) into( "jni/" +(if(DefaultNativePlatform.getCurrentArchitecture().name.equals("x86-64"))"x86_64" else DefaultNativePlatform.getCurrentArchitecture().name) + "-" + DefaultNativePlatform.getCurrentOperatingSystem().internalOs.familyName ) } } else{ from( "./jni/build/" + DefaultNativePlatform.getCurrentOperatingSystem().internalOs.familyName + "/" + DefaultNativePlatform.getCurrentArchitecture().name + "/release" ) { include(System.mapLibraryName("test")) into( "jni/" +(if(DefaultNativePlatform.getCurrentArchitecture().name.equals("x86-64"))"x86_64" else DefaultNativePlatform.getCurrentArchitecture().name) + "_" + DefaultNativePlatform.getCurrentOperatingSystem().internalOs.familyName ) } } } tasks.assemble.get().dependsOn(tasks.shadowJar) ```

[headius]

I'd love to revisit this and see what we can come up with, because with the rise of Project Panama FFI for JVM, lots of folks are going to want to have this functionality.

First off, has anyone explored the Maven plugins for native libraries that I mentioned above? I never circled back to it, but it seems like a good place to start (even if we end up having to contribute to it to make it do what we need).

@PumpkinXD Your example code is probably the same sort of thing that I would hack together myself. I'm just really hoping we can find someone else's project that does all of this better than what we have today!