loading native library from a (non-first) jar on the classpath doesn't seem to work

[slyrus]

Using the CDK (Chemistry Development Kit), which uses a JNA library to the InChI native code library, ABCL fails to find the native code library when it is contained in a jar. If I put the dylib in /usr/local/lib/ things work, but this shouldn't be necessary.

The problem and some further information on it are described here: https://github.com/cdk/cdk/issues/894

[johnmay]

Here is a more minimal example summarised example. The issue is JNA's NativeLibrary.getInstance("nativelibname") does not seem to be able to load from the classpath like it does from Java. Note there is an optional second argument for a different classloader.

edit: added jna-inchi-linux-x86-64 so example works on linux.

:

; jna-test/jna-test.asd

(eval-when (:compile-toplevel :load-toplevel :execute)
  (cl:require 'abcl-contrib)
  (cl:require 'abcl-asdf)
  (cl:require 'jss)
  (cl:require 'extensible-sequences)
  (cl:require 'java-collections))

(asdf:defsystem :jna-test
  :name "jna-test"
  :serial t
  :defsystem-depends-on (asdf-mvn-module)
  :components
  ((:mvn-module jna-test-mvn
                :dependencies
                ("io.github.dan2097/jna-inchi-darwin-aarch64/1.1"
                 "io.github.dan2097/jna-inchi-linux-x86-64/1.1"
                 "net.java.dev.jna/jna/5.10.0"))   
   (:file "package")
   (:file "jna-test")))

; jna-test/jna-test.lisp

(cl:in-package #:jna-test)

(eval-when (:compile-toplevel :load-toplevel :execute)
  (set-dispatch-macro-character #\# #\" 'jss::read-invoke))

(java:jstatic "getInstance" "com.sun.jna.NativeLibrary" "jnainchi")

; jna-test/package.asd
(cl:defpackage #:jna-test
  (:use :cl))

(cl:in-package #:jna-test)

CL-USER(1): (require :asdf)
("uiop" "UIOP" "asdf" "ASDF"
CL-USER(2): (pushnew "/tmp/jna-test/" asdf:*central-registry*)
("/tmp/jna-test/")
CL-USER(3): (asdf:load-system "jna-test")

; Compiling /private/tmp/jna-test/jna-test.lisp ...
; (IN-PACKAGE #:JNA-TEST)
; (SET-DISPATCH-MACRO-CHARACTER #\# ...)
; (JAVA:JSTATIC "getInstance" ...)
; Wrote /Users/john/.cache/common-lisp/abcl-1.9.0-fasl43-macosx-arm64/private/tmp/jna-test/jna-test-tmpXNEWPR2O.abcl (0.008 seconds)
Error loading /Users/john/.cache/common-lisp/abcl-1.9.0-fasl43-macosx-arm64/private/tmp/jna-test/jna-test.abcl at line 10 (offset 332)
#<THREAD "interpreter" {4EBD8C3C}>: Debugger invoked on condition of type JAVA-EXCEPTION
  Java exception 'java.lang.UnsatisfiedLinkError: Unable to load library 'jnainchi':
dlopen(libjnainchi.dylib, 0x0009): tried: '/opt/jdk-17.0.1.jdk/Contents/Home/bin/./libjnainchi.dylib' (no such file), '/opt/jdk-17.0.1.jdk/Contents/Home/bin/../lib/libjnainchi.dylib' (no such file), 'libjnainchi.dylib' (no such file), '/usr/lib/libjnainchi.dylib' (no such file), '/private/tmp/libjnainchi.dylib' (no such file)
dlopen(libjnainchi.dylib, 0x0009): tried: '/opt/jdk-17.0.1.jdk/Contents/Home/bin/./libjnainchi.dylib' (no such file), '/opt/jdk-17.0.1.jdk/Contents/Home/bin/../lib/libjnainchi.dylib' (no such file), 'libjnainchi.dylib' (no such file), '/usr/lib/libjnainchi.dylib' (no such file), '/private/tmp/libjnainchi.dylib' (no such file)
dlopen(/Users/john/Library/Frameworks/jnainchi.framework/jnainchi, 0x0009): tried: '/Users/john/Library/Frameworks/jnainchi.framework/jnainchi' (no such file), '/System/Library/Frameworks/jnainchi.framework/jnainchi' (no such file)
dlopen(/Library/Frameworks/jnainchi.framework/jnainchi, 0x0009): tried: '/Library/Frameworks/jnainchi.framework/jnainchi' (no such file), '/System/Library/Frameworks/jnainchi.framework/jnainchi' (no such file)
dlopen(/System/Library/Frameworks/jnainchi.framework/jnainchi, 0x0009): tried: '/System/Library/Frameworks/jnainchi.framework/jnainchi' (no such file)
Native library (darwin-aarch64/libjnainchi.dylib) not found in resource path (/opt/homebrew/Cellar/abcl/1.9.0/libexec/abcl.jar:)'.

The key issue being it only looks just in abcl.jar for the lib:

Native library (darwin-aarch64/libjnainchi.dylib) not found in resource path (/opt/homebrew/Cellar/abcl/1.9.0/libexec/abcl.jar:)

If we dump the classpath we can see the JAR we expect to find it in is found:

(java:dump-classpath)
; ...
#P"file:///Users/john/.m2/repository/io/github/dan2097/jna-inchi-darwin-aarch64/1.1/jna-inchi-darwin-aarch64-1.1.jar"
; ...

[john@sentinel:/tmp]% jar tvf /Users/john/.m2/repository/io/github/dan2097/jna-inchi-darwin-aarch64/1.1/jna-inchi-darwin-aarch64-1.1.jar
     0 Sun Jan 23 15:22:32 GMT 2022 META-INF/
   132 Sun Jan 23 15:22:30 GMT 2022 META-INF/MANIFEST.MF
     0 Sun Jan 23 15:22:32 GMT 2022 darwin-aarch64/
1237264 Sun Jan 23 15:22:32 GMT 2022 darwin-aarch64/libjnainchi.dylib
     0 Sun Jan 23 15:22:32 GMT 2022 META-INF/maven/
     0 Sun Jan 23 15:22:32 GMT 2022 META-INF/maven/io.github.dan2097/
     0 Sun Jan 23 15:22:32 GMT 2022 META-INF/maven/io.github.dan2097/jna-inchi-darwin-aarch64/
   543 Sun Jan 23 15:22:10 GMT 2022 META-INF/maven/io.github.dan2097/jna-inchi-darwin-aarch64/pom.xml
   129 Sun Jan 23 15:22:32 GMT 2022 META-INF/maven/io.github.dan2097/jna-inchi-darwin-aarch64/pom.properties

[alanruttenberg]

Does this work for you on other systems? I've not had it work on mine, which is why I landed up using OSGI for the case where I need it, since that's defined to work. https://stackoverflow.com/questions/38248983/load-jna-library-which-is-shipped-with-jar

[johnmay]

Yes this is a standard mechanism in JNA. You can see on the linked project (CDK) the git workflow runs the test suite on Linux which needs to load the native library to pass. Likewise we have no issues on Windows or OS X.

As @slyrus has tested a work around is to manually unpack the library and put it somewhere on the file system that is expected. This is inconvenient though