lvh / caesium

Modern cryptography (libsodium/NaCl) for Clojure
Eclipse Public License 1.0
180 stars 28 forks source link

Error "unable to load native libsodium" but the system does have it #80

Closed mtmr0x closed 2 years ago

mtmr0x commented 3 years ago

I've trying to run a backend API that uses Caesium (version "0.14.0") in my MacOS with M1 chip and I'm getting the following error:

Execution error (ClassNotFoundException) at caesium.binding/load-sodium (binding.clj:716).
unable to load native libsodium; is it installed?

Full error attached below

I've tried to run it with a diversity of JDKs but I keep getting the exact same error. In non M1 chips the project runs with no issues (with some exceptions on Macs that require to use JDK 11).

I could isolate the problem to Caesium, considering the project is using more cryptographic technologies I wanted to verify if any other item were using it.

Removing Caesium from use the project runs.

Here are the JDK I tried, using lein with brew on openjdk and manually installing lein and using jabba to manage the JDK version on zulu.

release pass?
zulu@1.11.0
zulu@1.16.0
adoptopenjdk@16.0
adoptopenjdk@14.0
adoptopenjdk@11.0
FULL ERROR MESSAGE HERE, click to expand ```clojure {:clojure.main/message "Execution error (ClassNotFoundException) at caesium.binding/load-sodium (binding.clj:716).\nunable to load native libsodium; is it installed?\n", :clojure.main/triage {:clojure.error/class java.lang.ClassNotFoundException, :clojure.error/line 716, :clojure.error/cause "unable to load native libsodium; is it installed?", :clojure.error/symbol caesium.binding/load-sodium, :clojure.error/source "binding.clj", :clojure.error/phase :execution}, :clojure.main/trace {:via [{:type clojure.lang.Compiler$CompilerException, :message "Syntax error macroexpanding at (binding.clj:722:3).", :data {:clojure.error/phase :execution, :clojure.error/line 722, :clojure.error/column 3, :clojure.error/source "binding.clj"}, :at [clojure.lang.Compiler$InvokeExpr eval "Compiler.java" 3707]} {:type java.lang.ClassNotFoundException, :message "unable to load native libsodium; is it installed?", :at [caesium.binding$load_sodium invokeStatic "binding.clj" 716]}], :trace [[caesium.binding$load_sodium invokeStatic "binding.clj" 716] [caesium.binding$load_sodium invoke "binding.clj" 707] [caesium.binding$load_sodium invokeStatic "binding.clj" 710] [caesium.binding$load_sodium invoke "binding.clj" 707] [clojure.lang.AFn applyToHelper "AFn.java" 152] [clojure.lang.AFn applyTo "AFn.java" 144] [clojure.lang.Compiler$InvokeExpr eval "Compiler.java" 3702] [clojure.lang.Compiler$DefExpr eval "Compiler.java" 457] [clojure.lang.Compiler eval "Compiler.java" 7182] [clojure.lang.Compiler load "Compiler.java" 7636] [clojure.lang.RT loadResourceScript "RT.java" 381] [clojure.lang.RT loadResourceScript "RT.java" 372] [clojure.lang.RT load "RT.java" 459] [clojure.lang.RT load "RT.java" 424] [clojure.core$load$fn__6839 invoke "core.clj" 6126] [clojure.core$load invokeStatic "core.clj" 6125] [clojure.core$load doInvoke "core.clj" 6109] [clojure.lang.RestFn invoke "RestFn.java" 408] [clojure.core$load_one invokeStatic "core.clj" 5908] [clojure.core$load_one invoke "core.clj" 5903] [clojure.core$load_lib$fn__6780 invoke "core.clj" 5948] [clojure.core$load_lib invokeStatic "core.clj" 5947] [clojure.core$load_lib doInvoke "core.clj" 5928] [clojure.lang.RestFn applyTo "RestFn.java" 142] [clojure.core$apply invokeStatic "core.clj" 667] [clojure.core$load_libs invokeStatic "core.clj" 5985] [clojure.core$load_libs doInvoke "core.clj" 5969] [clojure.lang.RestFn applyTo "RestFn.java" 137] [clojure.core$apply invokeStatic "core.clj" 667] [clojure.core$require invokeStatic "core.clj" 6007] [clojure.core$require doInvoke "core.clj" 6007] [clojure.lang.RestFn invoke "RestFn.java" 436] [caesium.crypto.box$eval26648$loading__6721__auto____26649 invoke "box.clj" 1] [caesium.crypto.box$eval26648 invokeStatic "box.clj" 1] [caesium.crypto.box$eval26648 invoke "box.clj" 1] [clojure.lang.Compiler eval "Compiler.java" 7177] [clojure.lang.Compiler eval "Compiler.java" 7166] [clojure.lang.Compiler load "Compiler.java" 7636] [clojure.lang.RT loadResourceScript "RT.java" 381] [clojure.lang.RT loadResourceScript "RT.java" 372] [clojure.lang.RT load "RT.java" 459] [clojure.lang.RT load "RT.java" 424] [clojure.core$load$fn__6839 invoke "core.clj" 6126] [clojure.core$load invokeStatic "core.clj" 6125] [clojure.core$load doInvoke "core.clj" 6109] [clojure.lang.RestFn invoke "RestFn.java" 408] [clojure.core$load_one invokeStatic "core.clj" 5908] [clojure.core$load_one invoke "core.clj" 5903] [clojure.core$load_lib$fn__6780 invoke "core.clj" 5948] [clojure.core$load_lib invokeStatic "core.clj" 5947] [clojure.core$load_lib doInvoke "core.clj" 5928] [clojure.lang.RestFn applyTo "RestFn.java" 142] [clojure.core$apply invokeStatic "core.clj" 667] [clojure.core$load_libs invokeStatic "core.clj" 5985] [clojure.core$load_libs doInvoke "core.clj" 5969] [clojure.lang.RestFn applyTo "RestFn.java" 137] [clojure.core$apply invokeStatic "core.clj" 667] [clojure.core$require invokeStatic "core.clj" 6007] [clojure.core$require doInvoke "core.clj" 6007] [clojure.lang.RestFn invoke "RestFn.java" 482] [security.crypto$eval26640$loading__6721__auto____26641 invoke "crypto.clj" 1] [security.crypto$eval26640 invokeStatic "crypto.clj" 1] [security.crypto$eval26640 invoke "crypto.clj" 1] [clojure.lang.Compiler eval "Compiler.java" 7177] [clojure.lang.Compiler eval "Compiler.java" 7166] [clojure.lang.Compiler load "Compiler.java" 7636] [clojure.lang.RT loadResourceScript "RT.java" 381] [clojure.lang.RT loadResourceScript "RT.java" 372] [clojure.lang.RT load "RT.java" 459] [clojure.lang.RT load "RT.java" 424] [clojure.core$load$fn__6839 invoke "core.clj" 6126] [clojure.core$load invokeStatic "core.clj" 6125] [clojure.core$load doInvoke "core.clj" 6109] [clojure.lang.RestFn invoke "RestFn.java" 408] [clojure.core$load_one invokeStatic "core.clj" 5908] [clojure.core$load_one invoke "core.clj" 5903] [clojure.core$load_lib$fn__6780 invoke "core.clj" 5948] [clojure.core$load_lib invokeStatic "core.clj" 5947] [clojure.core$load_lib doInvoke "core.clj" 5928] [clojure.lang.RestFn applyTo "RestFn.java" 142] [clojure.core$apply invokeStatic "core.clj" 667] [clojure.core$load_libs invokeStatic "core.clj" 5985] [clojure.core$load_libs doInvoke "core.clj" 5969] [clojure.lang.RestFn applyTo "RestFn.java" 137] [clojure.core$apply invokeStatic "core.clj" 667] [clojure.core$require invokeStatic "core.clj" 6007] [clojure.core$require doInvoke "core.clj" 6007] [clojure.lang.RestFn invoke "RestFn.java" 703] [security.login.login$eval17253$loading__6721__auto____17254 invoke "login.clj" 1] [security.login.login$eval17253 invokeStatic "login.clj" 1] [security.login.login$eval17253 invoke "login.clj" 1] [clojure.lang.Compiler eval "Compiler.java" 7177] [clojure.lang.Compiler eval "Compiler.java" 7166] [clojure.lang.Compiler load "Compiler.java" 7636] [clojure.lang.RT loadResourceScript "RT.java" 381] [clojure.lang.RT loadResourceScript "RT.java" 372] [clojure.lang.RT load "RT.java" 459] [clojure.lang.RT load "RT.java" 424] [clojure.core$load$fn__6839 invoke "core.clj" 6126] [clojure.core$load invokeStatic "core.clj" 6125] [clojure.core$load doInvoke "core.clj" 6109] [clojure.lang.RestFn invoke "RestFn.java" 408] [clojure.core$load_one invokeStatic "core.clj" 5908] [clojure.core$load_one invoke "core.clj" 5903] [clojure.core$load_lib$fn__6780 invoke "core.clj" 5948] [clojure.core$load_lib invokeStatic "core.clj" 5947] [clojure.core$load_lib doInvoke "core.clj" 5928] [clojure.lang.RestFn applyTo "RestFn.java" 142] [clojure.core$apply invokeStatic "core.clj" 667] [clojure.core$load_libs invokeStatic "core.clj" 5985] [clojure.core$load_libs doInvoke "core.clj" 5969] [clojure.lang.RestFn applyTo "RestFn.java" 137] [clojure.core$apply invokeStatic "core.clj" 667] [clojure.core$require invokeStatic "core.clj" 6007] [clojure.core$require doInvoke "core.clj" 6007] [clojure.lang.RestFn invoke "RestFn.java" 457] [security.login.interceptor$eval17247$loading__6721__auto____17248 invoke "interceptor.clj" 1] [security.login.interceptor$eval17247 invokeStatic "interceptor.clj" 1] [security.login.interceptor$eval17247 invoke "interceptor.clj" 1] [clojure.lang.Compiler eval "Compiler.java" 7177] [clojure.lang.Compiler eval "Compiler.java" 7166] [clojure.lang.Compiler load "Compiler.java" 7636] [clojure.lang.RT loadResourceScript "RT.java" 381] [clojure.lang.RT loadResourceScript "RT.java" 372] [clojure.lang.RT load "RT.java" 459] [clojure.lang.RT load "RT.java" 424] [clojure.core$load$fn__6839 invoke "core.clj" 6126] [clojure.core$load invokeStatic "core.clj" 6125] [clojure.core$load doInvoke "core.clj" 6109] [clojure.lang.RestFn invoke "RestFn.java" 408] [clojure.core$load_one invokeStatic "core.clj" 5908] [clojure.core$load_one invoke "core.clj" 5903] [clojure.core$load_lib$fn__6780 invoke "core.clj" 5948] [clojure.core$load_lib invokeStatic "core.clj" 5947] [clojure.core$load_lib doInvoke "core.clj" 5928] [clojure.lang.RestFn applyTo "RestFn.java" 142] [clojure.core$apply invokeStatic "core.clj" 667] [clojure.core$load_libs invokeStatic "core.clj" 5985] [clojure.core$load_libs doInvoke "core.clj" 5969] [clojure.lang.RestFn applyTo "RestFn.java" 137] [clojure.core$apply invokeStatic "core.clj" 667] [clojure.core$require invokeStatic "core.clj" 6007] [clojure.core$require doInvoke "core.clj" 6007] [clojure.lang.RestFn invoke "RestFn.java" 930] [http.routes$eval16978$loading__6721__auto____16979 invoke "routes.clj" 1] [http.routes$eval16978 invokeStatic "routes.clj" 1] [http.routes$eval16978 invoke "routes.clj" 1] [clojure.lang.Compiler eval "Compiler.java" 7177] [clojure.lang.Compiler eval "Compiler.java" 7166] [clojure.lang.Compiler load "Compiler.java" 7636] [clojure.lang.RT loadResourceScript "RT.java" 381] [clojure.lang.RT loadResourceScript "RT.java" 372] [clojure.lang.RT load "RT.java" 459] [clojure.lang.RT load "RT.java" 424] [clojure.core$load$fn__6839 invoke "core.clj" 6126] [clojure.core$load invokeStatic "core.clj" 6125] [clojure.core$load doInvoke "core.clj" 6109] [clojure.lang.RestFn invoke "RestFn.java" 408] [clojure.core$load_one invokeStatic "core.clj" 5908] [clojure.core$load_one invoke "core.clj" 5903] [clojure.core$load_lib$fn__6780 invoke "core.clj" 5948] [clojure.core$load_lib invokeStatic "core.clj" 5947] [clojure.core$load_lib doInvoke "core.clj" 5928] [clojure.lang.RestFn applyTo "RestFn.java" 142] [clojure.core$apply invokeStatic "core.clj" 667] [clojure.core$load_libs invokeStatic "core.clj" 5985] [clojure.core$load_libs doInvoke "core.clj" 5969] [clojure.lang.RestFn applyTo "RestFn.java" 137] [clojure.core$apply invokeStatic "core.clj" 667] [clojure.core$require invokeStatic "core.clj" 6007] [clojure.core$require doInvoke "core.clj" 6007] [clojure.lang.RestFn invoke "RestFn.java" 482] [http.server$eval157$loading__6721__auto____158 invoke "server.clj" 1] [http.server$eval157 invokeStatic "server.clj" 1] [http.server$eval157 invoke "server.clj" 1] [clojure.lang.Compiler eval "Compiler.java" 7177] [clojure.lang.Compiler eval "Compiler.java" 7166] [clojure.lang.Compiler load "Compiler.java" 7636] [clojure.lang.RT loadResourceScript "RT.java" 381] [clojure.lang.RT loadResourceScript "RT.java" 372] [clojure.lang.RT load "RT.java" 459] [clojure.lang.RT load "RT.java" 424] [clojure.core$load$fn__6839 invoke "core.clj" 6126] [clojure.core$load invokeStatic "core.clj" 6125] [clojure.core$load doInvoke "core.clj" 6109] [clojure.lang.RestFn invoke "RestFn.java" 408] [clojure.core$load_one invokeStatic "core.clj" 5908] [clojure.core$load_one invoke "core.clj" 5903] [clojure.core$load_lib$fn__6780 invoke "core.clj" 5948] [clojure.core$load_lib invokeStatic "core.clj" 5947] [clojure.core$load_lib doInvoke "core.clj" 5928] [clojure.lang.RestFn applyTo "RestFn.java" 142] [clojure.core$apply invokeStatic "core.clj" 667] [clojure.core$load_libs invokeStatic "core.clj" 5985] [clojure.core$load_libs doInvoke "core.clj" 5969] [clojure.lang.RestFn applyTo "RestFn.java" 137] [clojure.core$apply invokeStatic "core.clj" 667] [clojure.core$require invokeStatic "core.clj" 6007] [clojure.core$require doInvoke "core.clj" 6007] [clojure.lang.RestFn invoke "RestFn.java" 408] [user$eval140$fn__144 invoke "form-init10770987251725177174.clj" 1] [user$eval140 invokeStatic "form-init10770987251725177174.clj" 1] [user$eval140 invoke "form-init10770987251725177174.clj" 1] [clojure.lang.Compiler eval "Compiler.java" 7177] [clojure.lang.Compiler eval "Compiler.java" 7167] [clojure.lang.Compiler load "Compiler.java" 7636] [clojure.lang.Compiler loadFile "Compiler.java" 7574] [clojure.main$load_script invokeStatic "main.clj" 475] [clojure.main$init_opt invokeStatic "main.clj" 477] [clojure.main$init_opt invoke "main.clj" 477] [clojure.main$initialize invokeStatic "main.clj" 508] [clojure.main$null_opt invokeStatic "main.clj" 542] [clojure.main$null_opt invoke "main.clj" 539] [clojure.main$main invokeStatic "main.clj" 664] [clojure.main$main doInvoke "main.clj" 616] [clojure.lang.RestFn applyTo "RestFn.java" 137] [clojure.lang.Var applyTo "Var.java" 705] [clojure.main main "main.java" 40]], :cause "unable to load native libsodium; is it installed?", :phase :execution}} ```

If anyone have a solution or workaround to run the project locally in my machine I would really appreciate it. Thanks

dspearson commented 3 years ago

What is the full path to your libsodium library? Is it in your java.library.path? This is probably either an architecture mismatch or a JVM library path issue. Running (System/getProperty "java.library.path") at a REPL should show you where the default search path is for the JVM. I believe libsodium needs to be in one of the listed directories to be picked up by JNR-FFI. I don't have a M1 to try on, but more information would be great for trying to pin down the issue.

sandromello commented 3 years ago

I've followed the tip of @dspearson and linked the libsodium library to a path in java.library.path and it worked:

# this may differ depending of each O.S.
sudo ln -s /path/to/libsodium/shared/library/libsodium.dylib /path/to/java/library/libsodium.dylib

My setup is a mac-mini amd64

mtmr0x commented 3 years ago

I just did (apparently) what @sandromello did, but it didn't solve for me.

Based on your message @dspearson I ran (System/getProperty "java.library.path") from my REPL and got:

"/Users/mtmr0x/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:."

I've run a symlink like this (since the path of /Library/Java/Extensions were the only valid/that existed in my system):

sudo ln -s /opt/homebrew/Cellar/libsodium/1.0.18_1/lib/libsodium.dylib /Library/Java/Extensions/libsodium.dylib

Is there something I'm missing? One thing that's a bit confusing is the : between every possible path from (System/getProperty "java.library.path") that I got in the REPL.

Context: I'm quite new to Java/Clojure. Some basic conventions might be off my knowledge

dspearson commented 3 years ago

The : is conventionally a separator on UNIX, so each of those paths will be searched when trying to load libs. That should have worked. You could try cp /opt/homebrew/Cellar/libsodium/1.0.18_1/lib/libsodium.dylib /Users/mtmr0x/Library/Java/Extensions, in case it is somehow a permission issue, though I am not sure I can be of much further help since I don't have access to macOS. Maybe someone else could further assist.

piotr-yuxuan commented 2 years ago

I can confirm that I was able to get it working in the REPL, but I got issues with lein install. I executed

ln -s /usr/local/Cellar/libsodium/1.0.18_1/lib/libsodium.dylib $HOME/Library/Java/Extensions/libsodium.dylib
lein clean

and now lein install works smoothly.

dspearson commented 2 years ago

Great news! A macOS/brew library path idiosyncrasy then, which would affect any JVM code that calls native code outside of known paths, rather than a bug with caesium. There's not really any good way to resolve this within the project. If anything, it'd be upstream brew and java defaults. I guess the issue here can be closed?

mtmr0x commented 2 years ago

Great news! A macOS/brew library path idiosyncrasy then, which would affect any JVM code that calls native code outside of known paths, rather than a bug with caesium. There's not really any good way to resolve this within the project. If anything, it'd be upstream brew and java defaults. I guess the issue here can be closed?

Done! ✌️