Can't load library on macOS using pyenv

[charlielu15]

Describe the problem I am trying to use jep to call python script in java on macOS m1.

Search for existing solutions At the beginning, it showed me UnsatisfiedLinkError and I have already tracked down both problem #379 and #389. I install the Azul version of java 11 and the UnsatisfiedLinkError is still here but the error message changed from x_86 and arm64 error to Can't load library error.

Environment (please complete the following information): OS Platform, Distribution, and Version: macOS Monterey 12.0.1 with M1 chip Python Distribution and Version: pyenv 3.8.10 Java Distribution and Version: java 11 with ARM (https://www.azul.com/downloads/?version=java-11-lts&os=macos&architecture=arm-64-bit&package=jdk) Jep Version: 4.0.3 Python packages used (e.g. numpy, pandas, tensorflow): Numpy, pandas

Logs the jep's built C library is at: /Users/zhongyuanlu/jep.cpython-38-darwin.so Exception in thread "main" java.lang.UnsatisfiedLinkError: Can't load library: /Users/zhongyuanlu/.pyenv/versions/3.8.10/lib/python3.8/site-packages/jep/jep.cpython-38-darwin.so at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2633) at java.base/java.lang.Runtime.load0(Runtime.java:768) at java.base/java.lang.System.load(System.java:1837) at jep.MainInterpreter.initialize(MainInterpreter.java:126) at jep.MainInterpreter.getMainInterpreter(MainInterpreter.java:101) at jep.Jep.(Jep.java:133) at jep.SharedInterpreter.(SharedInterpreter.java:56) at com.quantum.quantumproject2.repository.RestClient.main(RestClient.java:199) Process finished with exit code 1

Additional context Add any other context about the problem here. Here is the code I tried to run the jep in java:
public static void main(String[] args) throws Exception { String ret = "/Users/zhongyuanlu/.pyenv/versions/3.8.10/lib/python3.8/site-packages/jep/jep.cpython-38-darwin.so"; System.out.println("the jep's built C library is at: "+ret); MainInterpreter.setJepLibraryPath(ret); Interpreter interp = new SharedInterpreter(); interp.exec("print(\"hello world\")"); }

[bsteffensmeier]

If you are not having problems related to the architexture I suspect it is failing to load libpython instead of calling setJepLibraryPath can you use setInitParams instead and set the PythonHome as /Users/zhongyuanlu/.pyenv/versions/3.8.10/

[jenarros-lw]

I have the same problem trying to use jep on a macOS M1 with Java 17.

System.getProperty("os.arch") returns "aarch64".

I have tried both setting the setJepLibraryPath and setInitParams with PythonHome but still doesn't work.

My code (Kotlin):

import jep.MainInterpreter
import jep.PyConfig
import jep.SharedInterpreter
import java.io.File

fun main() {
    println(
        File("/Users/eduardonarros/projects/jep/build/lib.macosx-12.6-arm64-cpython-310/jep/libjep.jnilib").exists())
    MainInterpreter.setJepLibraryPath("/Users/eduardonarros/projects/jep/build/lib.macosx-12.6-arm64-cpython-310/jep/libjep.jnilib");
    MainInterpreter.setInitParams(PyConfig().setPythonHome("/Users/eduardonarros/.pyenv/versions/3.10.7"))
    SharedInterpreter().use { interp ->
        interp.exec("""print("hello world")""");
        val result = interp.getValue("x")
        println(result)
    }
}

/Users/eduardonarros/.jenv/versions/17/bin/java -Xlog:library=info ....MainKt
[0.040s][info][library] Failed to find JNI_OnLoad_instrument in library with handle 0xfffffffffffffffb
[0.041s][info][library] Loaded library /opt/homebrew/Cellar/openjdk@17/17.0.5/libexec/openjdk.jdk/Contents/Home/lib/libinstrument.dylib, handle 0x0000000209f46e80
[0.041s][info][library] Found JNI_OnLoad in library with handle 0x0000000209f46e80
[0.042s][info][library] Failed to find JNI_OnLoad_nio in library with handle 0xfffffffffffffffb
[0.043s][info][library] Loaded library /opt/homebrew/Cellar/openjdk@17/17.0.5/libexec/openjdk.jdk/Contents/Home/lib/libnio.dylib, handle 0x0000000209f45dc0
[0.043s][info][library] Found JNI_OnLoad in library with handle 0x0000000209f45dc0
[0.043s][info][library] Failed to find Java_sun_nio_fs_UnixNativeDispatcher_init in library with handle 0x0000000209f46e80
[0.043s][info][library] Found Java_sun_nio_fs_UnixNativeDispatcher_init in library with handle 0x0000000209f45dc0
[0.044s][info][library] Failed to find Java_sun_nio_fs_UnixNativeDispatcher_getcwd in library with handle 0x0000000209f46e80
[0.044s][info][library] Found Java_sun_nio_fs_UnixNativeDispatcher_getcwd in library with handle 0x0000000209f45dc0
[0.044s][info][library] Failed to find Java_sun_nio_fs_UnixNativeDispatcher_stat0 in library with handle 0x0000000209f46e80
[0.044s][info][library] Found Java_sun_nio_fs_UnixNativeDispatcher_stat0 in library with handle 0x0000000209f45dc0
[0.045s][info][library] Failed to find JNI_OnLoad_zip in library with handle 0xfffffffffffffffb
[0.045s][info][library] Loaded library /opt/homebrew/Cellar/openjdk@17/17.0.5/libexec/openjdk.jdk/Contents/Home/lib/libzip.dylib, handle 0x0000000209f47380
[0.045s][info][library] Found JNI_OnLoad in library with handle 0x0000000209f47380
[0.045s][info][library] Failed to find Java_java_util_zip_Inflater_initIDs in library with handle 0x0000000209f46e80
[0.045s][info][library] Failed to find Java_java_util_zip_Inflater_initIDs in library with handle 0x0000000209f45dc0
[0.045s][info][library] Found Java_java_util_zip_Inflater_initIDs in library with handle 0x0000000209f47380
[0.045s][info][library] Failed to find Java_java_util_zip_Inflater_init in library with handle 0x0000000209f46e80
[0.045s][info][library] Failed to find Java_java_util_zip_Inflater_init in library with handle 0x0000000209f45dc0
[0.045s][info][library] Found Java_java_util_zip_Inflater_init in library with handle 0x0000000209f47380
[0.046s][info][library] Failed to find Java_java_util_zip_Inflater_inflateBytesBytes in library with handle 0x0000000209f46e80
[0.046s][info][library] Failed to find Java_java_util_zip_Inflater_inflateBytesBytes in library with handle 0x0000000209f45dc0
[0.046s][info][library] Found Java_java_util_zip_Inflater_inflateBytesBytes in library with handle 0x0000000209f47380
[0.046s][info][library] Failed to find Java_java_util_zip_Inflater_reset in library with handle 0x0000000209f46e80
[0.046s][info][library] Failed to find Java_java_util_zip_Inflater_reset in library with handle 0x0000000209f45dc0
[0.046s][info][library] Found Java_java_util_zip_Inflater_reset in library with handle 0x0000000209f47380
[0.103s][info][library] Failed to find JNI_OnLoad_zip in library with handle 0xfffffffffffffffb
[0.105s][info][library] Failed to find JNI_OnLoad_net in library with handle 0xfffffffffffffffb
true
[0.105s][info][library] Loaded library /opt/homebrew/Cellar/openjdk@17/17.0.5/libexec/openjdk.jdk/Contents/Home/lib/libnet.dylib, handle 0x0000000209f45fa0
[0.105s][info][library] Found JNI_OnLoad in library with handle 0x0000000209f45fa0
[0.105s][info][library] Found inet_pton in library with handle 0xfffffffffffffffe
[0.106s][info][library] Failed to find Java_java_net_InetAddress_init in library with handle 0x0000000209f46e80
[0.106s][info][library] Found Java_java_net_InetAddress_init in library with handle 0x0000000209f45fa0
[0.106s][info][library] Failed to find Java_java_net_InetAddressImplFactory_isIPv6Supported in library with handle 0x0000000209f46e80
[0.106s][info][library] Found Java_java_net_InetAddressImplFactory_isIPv6Supported in library with handle 0x0000000209f45fa0
[0.106s][info][library] Failed to find Java_java_net_Inet4Address_init in library with handle 0x0000000209f46e80
[0.106s][info][library] Found Java_java_net_Inet4Address_init in library with handle 0x0000000209f45fa0
[0.108s][info][library] Failed to find JNI_OnLoad_net in library with handle 0xfffffffffffffffb
[0.108s][info][library] Failed to find JNI_OnLoad_jep. in library with handle 0xfffffffffffffffb
[0.109s][info][library] Failed to load library /Users/eduardonarros/projects/jep/build/lib.macosx-12.6-arm64-cpython-310/jep/libjep.jnilib
[0.109s][info][library] Failed to find JNI_OnLoad_nio in library with handle 0xfffffffffffffffb
[0.109s][info][library] Failed to find Java_sun_nio_ch_IOUtil_initIDs in library with handle 0x0000000209f46e80
[0.109s][info][library] Failed to find Java_sun_nio_ch_IOUtil_initIDs in library with handle 0x0000000209f45fa0
[0.109s][info][library] Found Java_sun_nio_ch_IOUtil_initIDs in library with handle 0x0000000209f45dc0
[0.109s][info][library] Failed to find Java_sun_nio_ch_IOUtil_iovMax in library with handle 0x0000000209f46e80
[0.110s][info][library] Failed to find Java_sun_nio_ch_IOUtil_iovMax in library with handle 0x0000000209f45fa0
[0.110s][info][library] Found Java_sun_nio_ch_IOUtil_iovMax in library with handle 0x0000000209f45dc0
Exception in thread "main" java.lang.UnsatisfiedLinkError: Can't load library: /Users/eduardonarros/projects/jep/build/lib.macosx-12.6-arm64-cpython-310/jep/libjep.jnilib
    at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2393)
    at java.base/java.lang.Runtime.load0(Runtime.java:755)
    at java.base/java.lang.System.load(System.java:1953)
    at jep.MainInterpreter.initialize(MainInterpreter.java:126)
    at jep.MainInterpreter.getMainInterpreter(MainInterpreter.java:101)
    at jep.Jep.<init>(Jep.java:133)
    at jep.SharedInterpreter.<init>(SharedInterpreter.java:60)
    at fluro.MainKt.main(main.kt:14)
    at fluro.MainKt.main(main.kt)

[bsteffensmeier]

@jenarros-lw if you set both the JepLibraryPath and python home then python home is ignored. Try only setting python home and not JepLibraryPath.

[jenarros-lw]

@bsteffensmeier I have tried only setting the python home and I get the same error.

[bsteffensmeier]

If the library fails to load it is likely caused because a dependent library can not be loaded. Usually this is caused because libpython can not be found. I recommend finding the correct libpython, ensuring the architecture for that library matches the java architecture and then loading that library with System.load(). As an alternative to System.load(), Most OSs let you configure the linker with some env variables(LD_LIBRARY_PATH, LD_PRELOAD, PATH) but I am not sure how that is handled on mac.

[ndjensen]

Does the jep script that starts the interactive interpreter work?

[jenarros-lw]

When I install jep via pip install jep and then I try to run it I get the same problem:

~ ll /Users/eduardonarros/.pyenv/versions/3.10.7/lib/python3.10/site-packages/jep/libjep.jnilib
lrwxr-xr-x  1 eduardonarros  staff    25B 14 Nov 21:21 /Users/eduardonarros/.pyenv/versions/3.10.7/lib/python3.10/site-packages/jep/libjep.jnilib -> jep.cpython-310-darwin.so
➜  ~ jep                                                                                          
java.lang.UnsatisfiedLinkError: Can't load library: /Users/eduardonarros/.pyenv/versions/3.10.7/lib/python3.10/site-packages/jep/libjep.jnilib
    at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2393)
    at java.base/java.lang.Runtime.load0(Runtime.java:755)
    at java.base/java.lang.System.load(System.java:1953)
    at jep.LibraryLocator.searchPackageDir(LibraryLocator.java:243)
    at jep.LibraryLocator.searchSitePackages(LibraryLocator.java:146)
    at jep.LibraryLocator.findJepLibrary(LibraryLocator.java:325)
    at jep.MainInterpreter.initialize(MainInterpreter.java:131)
    at jep.MainInterpreter.getMainInterpreter(MainInterpreter.java:101)
    at jep.Jep.<init>(Jep.java:133)
    at jep.SharedInterpreter.<init>(SharedInterpreter.java:60)
    at jep.Run.run(Run.java:49)
    at jep.Run.main(Run.java:146)

[ndjensen]

So what I think is happening (and I could be wrong), is similar what @bsteffensmeier mentioned, i.e. that libjep is found thanks to -Djava.library.path but libpython is not found (for some unknown reason). On macOS the environment variable to give paths to search for libraries is DYLD_LIBRARY_PATH. However, if I remember correctly, an earlier version of macOS disables that environment variable while System Integrity Protection is on. So as a test, you could try setting the environment variable DYLD_LIBRARY_PATH to the directory containing libpython, temporarily disable SIP, and then see if jep loads correctly. If you do do that, you should re-enable SIP after the test as that is not a good long term solution and we don't want to open your mac to security risks. Reading online, there appears to be an entitlement you add to a library somehow to give it an exception to that security rule from SIP. If that is the problem I'm not sure yet how we proceed but it would be good to know if DYLD_LIBRARY_PATH was a temporary workaround and would be good to confirm it's a library search path issue.

Based on responses on other tickets it looks like some people have gotten jep to work on macOS arm, I'm wondering if the difference here is pyenv vs other installation methods of Python.

[ndjensen]

@bsteffensmeier suggested, "I recommend finding the correct libpython, ensuring the architecture for that library matches the java architecture and then loading that library with System.load()." So try loading libpython with System.load(pathToLibPython) before calling into Jep classes.

[jenarros-lw]

Using the python installation that comes with XCode and DYLD_LIBRARY_PATH, jep loads and works now. Thanks a lot for your help!