Closed artur-kovalchuk closed 2 years ago
Yes it could be that bug, but we might be able to work around it.
Can you please explain the situation in a little more detail:
For all 3 of the libraries, tell me each one's location and dependency to other ones.
Which of the libraries are you trying to load? Just 1 or more?
You may have to load them in their dependency order if they are in a custom path (non-system path)
@basshelal I created a sample repository for testing.
I have such a sequence: 1) libuapkic.2.0.0.dylib <- 2) libuapkif.2.0.0.dylib <- 3) libuapki.dylib I tried load by libName and with absolute path.
In result I had:
Exception in thread "main" java.lang.UnsatisfiedLinkError: dlopen(/Users/arkovalchuk/Library/Java/Extensions/libuapkif.2.0.0.dylib, 9): Library not loaded: libuapkic.2.0.0.dylib
Referenced from: /Users/arkovalchuk/Library/Java/Extensions/libuapkif.2.0.0.dylib
Reason: image not found
I try to load the libraries in a logical order, but it is useless
I tried everything but to no avail, it seems this is a MacOS specific problem relating to dlopen
which returns the error Reason: image not found
.
Running otool -L libuapki.dylib
yields:
@rpath/libuapki.dylib (compatibility version 0.0.0, current version 0.0.0)
libuapkif.2.0.0.dylib (compatibility version 0.0.0, current version 2.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)
/usr/lib/libcurl.4.dylib (compatibility version 7.0.0, current version 9.0.0)
/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
libuapkic.2.0.0.dylib (compatibility version 0.0.0, current version 2.0.0)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1200.3.0)
But if you try and load the libraries yourself from bash using this script I found here:
#!/usr/bin/env bash
while getopts "r" OPTION; do
case $OPTION in
r) export DYLD_PRINT_RPATHS=1;;
esac
done
shift $((OPTIND-1))
cp `which true` .
DYLD_PRINT_LIBRARIES=1 \
DYLD_PRINT_LIBRARIES_POST_LAUNCH=1 \
DYLD_INSERT_LIBRARIES=$1 \
./true
rm ./true
By doing in resources/
:
./my-ldd libuapkic.2.0.0.dylib
./my-ldd libuapkif.2.0.0.dylib
./my-ldd libcm-pkcs12.dylib
./my-ldd libuapki.dylib
None of them show failures!
Now run your JVM with the following environment (taken from here):
DYLD_PRINT_LIBRARIES=1;DYLD_PRINT_LIBRARIES_POST_LAUNCH=1;DYLD_PRINT_RPATHS=1
I used IntelliJ's environment variables feature in Run Configurations to do this.
I successfully loaded libuapkic.2.0.0.dylib
and ran a function I found from the repository here:
UAPKIC_EXPORT int md5_self_test(void);
But as soon as I try to load libuapkif.2.0.0.dylib
which depends on libuapkic.2.0.0.dylib
I get:
Exception in thread "main" java.lang.UnsatisfiedLinkError: dlopen(/Users/user/IdeaProjects/uapki-mac-os-integration/src/main/resources/libuapkif.2.0.0.dylib, 9): Library not loaded: libuapkic.2.0.0.dylib
Referenced from: /Users/user/IdeaProjects/uapki-mac-os-integration/src/main/resources/libuapkif.2.0.0.dylib
Reason: image not found
It seems libuapkic.2.0.0.dylib
fails to be loaded when libuapkif.2.0.0.dylib
is being loaded.
Even though the environment variables printed to the output:
RPATH failed expanding uapkif.2.0.0 to: /Users/user/Library/Java/JavaVirtualMachines/openjdk-16/Contents/Home/bin/./uapkif.2.0.0
RPATH failed expanding uapkif.2.0.0 to: /Users/user/Library/Java/JavaVirtualMachines/openjdk-16/Contents/Home/bin/../lib/uapkif.2.0.0
dyld: loaded: <42B29783-1F62-31B7-81CB-5576FF6C0ADE> /Users/user/IdeaProjects/uapki-mac-os-integration/src/main/resources/libuapkif.2.0.0.dylib
dyld: unloaded: <42B29783-1F62-31B7-81CB-5576FF6C0ADE> /Users/user/IdeaProjects/uapki-mac-os-integration/src/main/resources/libuapkif.2.0.0.dylib
Which means the library was indeed successfully loaded (after some initial failures because of paths).
I really don't know what to do next, I'll post the code so that you can try out yourself what I did and see if you can get something working from it maybe. Here is my fork with the code I tried. I'm running it on an Intel Macbook running macOS Big Sur 11.5.2.
My instinct is to somehow build all the libraries into one dylib
that way there's no extra loading but that may not be a possible solution (because of shared function names etc).
The other solution would be to move all libraries to /usr/lib/
and see if that works, I think it will but I haven't tried it, it's a pretty bad idea but it's a last resort.
Pinging @headius to see if he knows anything about this or anyone that can help
"image not found" looks like a generic error from jnr-ffi but I have not poked around yet.
Does the platform match for all libraries, i.e. all universal binary or x86 or arm?
I will try to reproduce on my systems today.
I was able to reproduce, and also able to get the libraries to load.
The first thing I tried was simply adding the resources dir with loader.search(...)
. This allowed the libuapkif library to load, but then the libuapkic library fails to find it and the program exits.
diff --git a/src/main/java/ua/koval4uk/Main.java b/src/main/java/ua/koval4uk/Main.java
index d2f0677..05fef08 100644
--- a/src/main/java/ua/koval4uk/Main.java
+++ b/src/main/java/ua/koval4uk/Main.java
@@ -14,6 +14,7 @@ public class Main {
*/
public static void main(String[] args) {
LibraryLoader<LibUapki> loader = LibraryLoader.create(LibUapki.class);
+ loader.search("src/main/resources");
loader.load("uapkic.2.0.0");
loader.load("uapkif.2.0.0");
loader.load("cm-pkcs12");
Output:
java.lang.UnsatisfiedLinkError: dlopen(/Users/headius/projects/tmp/uapki-mac-os-integration/src/main/resources/libuapkif.2.0.0.dylib, 9): Library not loaded: libuapkic.2.0.0.dylib
Referenced from: /Users/headius/projects/tmp/uapki-mac-os-integration/src/main/resources/libuapkif.2.0.0.dylib
Reason: image not found
Library names
[uapkic.2.0.0, uapkif.2.0.0, cm-pkcs12, uapki]
Search paths:
[/Users/headius/projects/tmp/uapki-mac-os-integration/src/main/resources, /Users/headius/Library/Java/Extensions, /Library/Java/Extensions, /Network/Library/Java/Extensions, /System/Library/Java/Extensions, /usr/lib/java, ., /usr/local/lib, /usr/lib, /lib]
at jnr.ffi.provider.jffi.NativeLibrary.loadNativeLibraries (NativeLibrary.java:111)
at jnr.ffi.provider.jffi.NativeLibrary.getNativeLibraries (NativeLibrary.java:85)
at jnr.ffi.provider.jffi.NativeLibrary.getSymbolAddress (NativeLibrary.java:64)
at jnr.ffi.provider.jffi.NativeLibrary.findSymbolAddress (NativeLibrary.java:74)
at jnr.ffi.provider.jffi.AsmLibraryLoader.generateInterfaceImpl (AsmLibraryLoader.java:138)
at jnr.ffi.provider.jffi.AsmLibraryLoader.loadLibrary (AsmLibraryLoader.java:86)
at jnr.ffi.provider.jffi.NativeLibraryLoader.loadLibrary (NativeLibraryLoader.java:44)
at jnr.ffi.LibraryLoader.load (LibraryLoader.java:420)
at jnr.ffi.LibraryLoader.load (LibraryLoader.java:399)
at ua.koval4uk.Main.main (Main.java:22)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:254)
at java.lang.Thread.run (Thread.java:831)
Based on various posts online it seems like this is largely a quirk of library loading on MacOS. Specifying a path to find the libraries does not seem to be working like we expect, and so when it tries to load the c
part of the library it fails to see that it already loaded the f
part. Perhaps this is because we are loading from a relative path, and it registers the image using that path? I am not sure.
However, I found one thing that makes it work: copy the libraries to CWD:
[] ~/projects/tmp/uapki-mac-os-integration $ git st
On branch master
Your branch is up to date with 'origin/master'.
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
modified: pom.xml
modified: src/main/java/ua/koval4uk/Main.java
Untracked files:
(use "git add <file>..." to include in what will be committed)
libcm-pkcs12.dylib
libuapki.dylib
libuapkic.2.0.0.dylib
libuapkif.2.0.0.dylib
uapki-test.iml
With this, the Main program is able to run to completion:
[] ~/projects/tmp/uapki-mac-os-integration $ mvn exec:java -Dexec.mainClass=ua.koval4uk.Main
[INFO] Scanning for projects...
[INFO]
[INFO] -----------------------< org.example:uapki-test >-----------------------
[INFO] Building uapki-test 1.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- exec-maven-plugin:3.0.0:java (default-cli) @ uapki-test ---
Result: {"errorCode":0,"method":"VERSION","result":{"name":"UAPKI","version":"2.0.6"}}
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.497 s
[INFO] Finished at: 2022-01-25T08:33:36-06:00
[INFO] ------------------------------------------------------------------------
I cannot say what is happening here but it does not seem to be a jnr-ffi bug. We can keep this open until we have a satisfactory answer for why loader.search
does not seem to be sufficient for dlopen to see all loaded images.
I confused some filenames so here are my before and after adding library.search
.
Before:
java.lang.UnsatisfiedLinkError: dlopen(libuapkic.2.0.0.dylib, 9): image not found
Library names
[uapkic.2.0.0, uapkif.2.0.0, cm-pkcs12, uapki]
Search paths:
[/Users/headius/Library/Java/Extensions, /Library/Java/Extensions, /Network/Library/Java/Extensions, /System/Library/Java/Extensions, /usr/lib/java, ., /usr/local/lib, /usr/lib, /lib]
at jnr.ffi.provider.jffi.NativeLibrary.loadNativeLibraries (NativeLibrary.java:111)
at jnr.ffi.provider.jffi.NativeLibrary.getNativeLibraries (NativeLibrary.java:85)
at jnr.ffi.provider.jffi.NativeLibrary.getSymbolAddress (NativeLibrary.java:64)
at jnr.ffi.provider.jffi.NativeLibrary.findSymbolAddress (NativeLibrary.java:74)
at jnr.ffi.provider.jffi.AsmLibraryLoader.generateInterfaceImpl (AsmLibraryLoader.java:138)
at jnr.ffi.provider.jffi.AsmLibraryLoader.loadLibrary (AsmLibraryLoader.java:86)
at jnr.ffi.provider.jffi.NativeLibraryLoader.loadLibrary (NativeLibraryLoader.java:44)
at jnr.ffi.LibraryLoader.load (LibraryLoader.java:420)
at jnr.ffi.LibraryLoader.load (LibraryLoader.java:399)
at ua.koval4uk.Main.main (Main.java:21)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:254)
at java.lang.Thread.run (Thread.java:831)
After:
java.lang.UnsatisfiedLinkError: dlopen(/Users/headius/projects/tmp/uapki-mac-os-integration/src/main/resources/libuapkif.2.0.0.dylib, 9): Library not loaded: libuapkic.2.0.0.dylib
Referenced from: /Users/headius/projects/tmp/uapki-mac-os-integration/src/main/resources/libuapkif.2.0.0.dylib
Reason: image not found
Library names
[uapkic.2.0.0, uapkif.2.0.0, cm-pkcs12, uapki]
Search paths:
[/Users/headius/projects/tmp/uapki-mac-os-integration/src/main/resources, /Users/headius/Library/Java/Extensions, /Library/Java/Extensions, /Network/Library/Java/Extensions, /System/Library/Java/Extensions, /usr/lib/java, ., /usr/local/lib, /usr/lib, /lib]
at jnr.ffi.provider.jffi.NativeLibrary.loadNativeLibraries (NativeLibrary.java:111)
at jnr.ffi.provider.jffi.NativeLibrary.getNativeLibraries (NativeLibrary.java:85)
at jnr.ffi.provider.jffi.NativeLibrary.getSymbolAddress (NativeLibrary.java:64)
at jnr.ffi.provider.jffi.NativeLibrary.findSymbolAddress (NativeLibrary.java:74)
at jnr.ffi.provider.jffi.AsmLibraryLoader.generateInterfaceImpl (AsmLibraryLoader.java:138)
at jnr.ffi.provider.jffi.AsmLibraryLoader.loadLibrary (AsmLibraryLoader.java:86)
at jnr.ffi.provider.jffi.NativeLibraryLoader.loadLibrary (NativeLibraryLoader.java:44)
at jnr.ffi.LibraryLoader.load (LibraryLoader.java:420)
at jnr.ffi.LibraryLoader.load (LibraryLoader.java:399)
at ua.koval4uk.Main.main (Main.java:21)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:254)
at java.lang.Thread.run (Thread.java:831)
Clearly library.search
is having an effect, but it is not sufficient to get all libraries to load happily.
A couple posts I read while digging into this:
Also note the improved error output (with listing of library names and search paths) is available in newer jnr-ffi (I don't remember when it was merged but latest release is 2.2.11).
Success! I think I cracked the code!
Due to MacOS security protections (e.g. Security Integrity Protection) certain executables do not inherit LD/DYLD environment variables:
I believe this includes processes that are setuid root, which includes sh/bash/zsh on MacOS.
Funny thing, the mvn
launcher is a shell script, which means running the Main here does not inherit my DYLD env. However if I run java
directly with an appropriate DYLD_FALLBACK_LIBRARY_PATH and classpath, your example works just fine even without the library.search
call:
$ DYLD_FALLBACK_LIBRARY_PATH=`pwd`/src/main/resources java -cp /Users/headius/.m2/repository/com/github/jnr/jnr-ffi/2.2.11/jnr-ffi-2.2.11.jar:/Users/headius/.m2/repository/com/github/jnr/jffi/1.3.9/jffi-1.3.9.jar:/Users/headius/.m2/repository/com/github/jnr/jffi/1.3.9/jffi-1.3.9-native.jar:/Users/headius/.m2/repository/org/ow2/asm/asm/9.2/asm-9.2.jar:/Users/headius/.m2/repository/org/ow2/asm/asm-commons/9.2/asm-commons-9.2.jar:/Users/headius/.m2/repository/org/ow2/asm/asm-analysis/9.2/asm-analysis-9.2.jar:/Users/headius/.m2/repository/org/ow2/asm/asm-tree/9.2/asm-tree-9.2.jar:/Users/headius/.m2/repository/org/ow2/asm/asm-util/9.2/asm-util-9.2.jar:/Users/headius/.m2/repository/com/github/jnr/jnr-a64asm/1.0.0/jnr-a64asm-1.0.0.jar:/Users/headius/.m2/repository/com/github/jnr/jnr-x86asm/1.0.2/jnr-x86asm-1.0.2.jar:target/uapki-test-1.0-SNAPSHOT.jar ua.koval4uk.Main
Result: {"errorCode":0,"method":"VERSION","result":{"name":"UAPKI","version":"2.0.6"}}
I think we have an understanding of the problem now, and it is not a jnr-ffi issue. MacOS is weird sometimes!
Note @basshelal were both able to reproduce even with the direct java
command when running on certain newer builds of the JVM (from 11 up to 17). It seems the DYLD_FALLBACK_LIBRARY_PATH is not propagating properly in some of these cases, or for whatever reason not being honored. Using DYLD_LIBRARY_PATH allowed it to work everywhere for me.
Following up on this DYLD_FALLBACK_LIBRARY_PATH thing...
It seems like this worked at one point and was "broken" by other fixes to OpenJDK. The breakage probably happened in a more recent release, but was eventually backported to at least Java 11:
$ java -version
openjdk version "11.0.4" 2019-07-16
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.4+11)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.4+11, mixed mode)
$ DYLD_FALLBACK_LIBRARY_PATH=`pwd`/src/main/resources java -Djava.library.path=`pwd`/src/main/resources -cp /Users/headius/.m2/repository/com/github/jnr/jnr-ffi/2.2.11/jnr-ffi-2.2.11.jar:/Users/headius/.m2/repository/com/github/jnr/jffi/1.3.9/jffi-1.3.9.jar:/Users/headius/.m2/repository/com/github/jnr/jffi/1.3.9/jffi-1.3.9-native.jar:/Users/headius/.m2/repository/org/ow2/asm/asm/9.2/asm-9.2.jar:/Users/headius/.m2/repository/org/ow2/asm/asm-commons/9.2/asm-commons-9.2.jar:/Users/headius/.m2/repository/org/ow2/asm/asm-analysis/9.2/asm-analysis-9.2.jar:/Users/headius/.m2/repository/org/ow2/asm/asm-tree/9.2/asm-tree-9.2.jar:/Users/headius/.m2/repository/org/ow2/asm/asm-util/9.2/asm-util-9.2.jar:/Users/headius/.m2/repository/com/github/jnr/jnr-a64asm/1.0.0/jnr-a64asm-1.0.0.jar:/Users/headius/.m2/repository/com/github/jnr/jnr-x86asm/1.0.2/jnr-x86asm-1.0.2.jar:target/uapki-test-1.0-SNAPSHOT.jar ua.koval4uk.Main
Result: {"errorCode":0,"method":"VERSION","result":{"name":"UAPKI","version":"2.0.6"}}
vs
$ java -version
openjdk version "11.0.14" 2022-01-18 LTS
OpenJDK Runtime Environment Zulu11.54+23-CA (build 11.0.14+9-LTS)
OpenJDK 64-Bit Server VM Zulu11.54+23-CA (build 11.0.14+9-LTS, mixed mode)
[] ~/projects/tmp/uapki-mac-os-integration $ DYLD_FALLBACK_LIBRARY_PATH=`pwd`/src/main/resources java -Djava.library.path=`pwd`/src/main/resources -cp /Users/headius/.m2/repository/com/github/jnr/jnr-ffi/2.2.11/jnr-ffi-2.2.11.jar:/Users/headius/.m2/repository/com/github/jnr/jffi/1.3.9/jffi-1.3.9.jar:/Users/headius/.m2/repository/com/github/jnr/jffi/1.3.9/jffi-1.3.9-native.jar:/Users/headius/.m2/repository/org/ow2/asm/asm/9.2/asm-9.2.jar:/Users/headius/.m2/repository/org/ow2/asm/asm-commons/9.2/asm-commons-9.2.jar:/Users/headius/.m2/repository/org/ow2/asm/asm-analysis/9.2/asm-analysis-9.2.jar:/Users/headius/.m2/repository/org/ow2/asm/asm-tree/9.2/asm-tree-9.2.jar:/Users/headius/.m2/repository/org/ow2/asm/asm-util/9.2/asm-util-9.2.jar:/Users/headius/.m2/repository/com/github/jnr/jnr-a64asm/1.0.0/jnr-a64asm-1.0.0.jar:/Users/headius/.m2/repository/com/github/jnr/jnr-x86asm/1.0.2/jnr-x86asm-1.0.2.jar:target/uapki-test-1.0-SNAPSHOT.jar ua.koval4uk.Main
Exception in thread "main" java.lang.UnsatisfiedLinkError: dlopen(/Users/headius/projects/tmp/uapki-mac-os-integration/src/main/resources/libuapkif.2.0.0.dylib, 9): Library not loaded: libuapkic.2.0.0.dylib
Referenced from: /Users/headius/projects/tmp/uapki-mac-os-integration/src/main/resources/libuapkif.2.0.0.dylib
Reason: image not found
Library names
[uapkic.2.0.0, uapkif.2.0.0, cm-pkcs12, uapki]
Search paths:
[/Users/headius/projects/tmp/uapki-mac-os-integration/src/main/resources, /usr/local/lib, /usr/lib, /lib]
at jnr.ffi.provider.jffi.NativeLibrary.loadNativeLibraries(NativeLibrary.java:111)
at jnr.ffi.provider.jffi.NativeLibrary.getNativeLibraries(NativeLibrary.java:85)
at jnr.ffi.provider.jffi.NativeLibrary.getSymbolAddress(NativeLibrary.java:64)
at jnr.ffi.provider.jffi.NativeLibrary.findSymbolAddress(NativeLibrary.java:74)
at jnr.ffi.provider.jffi.AsmLibraryLoader.generateInterfaceImpl(AsmLibraryLoader.java:141)
at jnr.ffi.provider.jffi.AsmLibraryLoader.loadLibrary(AsmLibraryLoader.java:87)
at jnr.ffi.provider.jffi.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:44)
at jnr.ffi.LibraryLoader.load(LibraryLoader.java:420)
at jnr.ffi.LibraryLoader.load(LibraryLoader.java:399)
at ua.koval4uk.Main.main(Main.java:21)
$ DYLD_LIBRARY_PATH=`pwd`/src/main/resources java -Djava.library.path=`pwd`/src/main/resources -cp /Users/headius/.m2/repository/com/github/jnr/jnr-ffi/2.2.11/jnr-ffi-2.2.11.jar:/Users/headius/.m2/repository/com/github/jnr/jffi/1.3.9/jffi-1.3.9.jar:/Users/headius/.m2/repository/com/github/jnr/jffi/1.3.9/jffi-1.3.9-native.jar:/Users/headius/.m2/repository/org/ow2/asm/asm/9.2/asm-9.2.jar:/Users/headius/.m2/repository/org/ow2/asm/asm-commons/9.2/asm-commons-9.2.jar:/Users/headius/.m2/repository/org/ow2/asm/asm-analysis/9.2/asm-analysis-9.2.jar:/Users/headius/.m2/repository/org/ow2/asm/asm-tree/9.2/asm-tree-9.2.jar:/Users/headius/.m2/repository/org/ow2/asm/asm-util/9.2/asm-util-9.2.jar:/Users/headius/.m2/repository/com/github/jnr/jnr-a64asm/1.0.0/jnr-a64asm-1.0.0.jar:/Users/headius/.m2/repository/com/github/jnr/jnr-x86asm/1.0.2/jnr-x86asm-1.0.2.jar:target/uapki-test-1.0-SNAPSHOT.jar ua.koval4uk.Main
Result: {"errorCode":0,"method":"VERSION","result":{"name":"UAPKI","version":"2.0.6"}}
I have confirmed this is broken in recent versions of Java 11, 13, 14, 15, 16, and 17. Only this older build of Java 11 works on my system. I have not attempted to test this on 8 since your build requests 11.
I should probably file a bug with OpenJDK, but this additional problem is also not a jnr-ffi issue.
@basshelal @headius Thanks! I appreciate what you did. Saved my time and spend yours.
When I am loading dependent libraries.
Exception is rised
I use Java 11, macOS - 11.6.2.
Is it a bug in JDK - https://bugs.openjdk.java.net/browse/JDK-8213772?