Unable to run on macOS BigSur

[meadch]

Using hello-world

; project.clj
(defproject hello-world "0.40.0"
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
  :dependencies [[org.clojure/clojure "1.10.3"]
                 [uncomplicate/neanderthal "0.40.0"]
                 ;;Optional. If bytedeco is not present, a system-wide MKL is used.
                 [org.bytedeco/mkl-platform-redist "2021.1-1.5.5"]]
  ;; Nvidia doesn't ship CUDA for macOS; you have to add this to your project
  :exclusions [[org.jcuda/jcuda-natives :classifier "apple-x86_64"]
               [org.jcuda/jcublas-natives :classifier "apple-x86_64"]]
  ;; If on Java 9+, you have to uncomment the following JVM option.
  :jvm-opts ^:replace [#_"--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED"])

Using Java 8, macOS Big Sur

$ java -version
openjdk version "1.8.0_292"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_292-b10)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.292-b10, mixed mode)

Can start a REPL

$ lein repl
nREPL server started on port 57826 on host 127.0.0.1 - nrepl://127.0.0.1:57826
REPL-y 0.4.4, nREPL 0.6.0
Clojure 1.10.3
OpenJDK 64-Bit Server VM 1.8.0_292-b10
    Docs: (doc function-name-here)
          (find-doc "part-of-name-here")
  Source: (source function-name-here)
 Javadoc: (javadoc java-object-or-class-here)
    Exit: Control+D or (exit) or (quit)
 Results: Stored in vars *1, *2, *3, an exception in *e

user=> 

But uncomplicate.neanderthal fails on:

example=> (ns example (:use [uncomplicate.neanderthal core native]))
Syntax error compiling at (uncomplicate/neanderthal/native.clj:1:1).
No namespace: uncomplicate.neanderthal.internal.host.mkl

I thought that, per walkthrough, by including [org.bytedeco/mkl-platform-redist "2020.3-1.5.4"] as a dependency I'd be good to go.

[blueberry]

Did you disable SIP?

[viesti]

What happens for me is the following link error (with uncomplicate/neanderthal bumped to 0.41.0):

Execution error (UnsatisfiedLinkError) at java.lang.ClassLoader$NativeLibrary/load0 (ClassLoader.java:-2).
/private/var/folders/xr/bbb7t_k5079gj6xslfqnj8780000gn/T/libneanderthal-mkl-0.33.07213617030678858077.jnilib: dlopen(/private/var/folders/xr/bbb7t_k5079gj6xslfqnj8780000gn/T/libneanderthal-mkl-0.33.07213617030678858077.jnilib, 1): Library not loaded: @rpath/libmkl_intel_lp64.dylib
  Referenced from: /private/var/folders/xr/bbb7t_k5079gj6xslfqnj8780000gn/T/libneanderthal-mkl-0.33.07213617030678858077.jnilib
  Reason: image not found

Might be that this link error was not visible in the original error report.

I didn't yet try with a system-wide MKL installation though. Also I didn't yet try disabling SIP, but was thinking that loading native libraries should probably work, even though writing to SIP protected paths would be prevented when SIP is active.

[blueberry]

Some users reported it working with SIP enabled, but I'm not so sure that they can be sure about that. It seems that SIP has to be disabled... I'd recommend to disable SIP, install system-wide MKL (or just put the proper files in proper directories and edit proper paths), make sure Hello World works for you, and then re-enable SIP and see if it still works.

[viesti]

Right. Was suspecting SIP disabling since I got some stuff working in the past: https://github.com/viesti/lin-alg-app/blob/master/project.clj. Though uncomplicate/neanderthal "0.17.2" is probably ancient, so stuff might be different now :)

Have to go to sleep now, might try MKL installation in near future :)

[viesti]

Urgh, turns out I didn't know enough about SIP, seems that it does limit paths from which native libraries can be loaded and it's even mentioned in the docs, which I didn't read :D https://neanderthal.uncomplicate.org/articles/getting_started.html#the-native-library-used-by-neanderthals-native-engine-optional & https://github.com/uncomplicate/neanderthal/issues/31.

I ended up copying the MKL libraries to /usr/local/lib/, instead of rebooting and disabling SIP. The current MKL installer uses a bit different paths through (branded as "oneAPI"):

ln -s /opt/intel/oneapi/mkl/latest/lib/* /usr/local/lib/
ln -s /opt/intel/oneapi/compiler/latest/mac/compiler/lib/* /usr/local/lib/

[blueberry]

If I understand correctly, this issue has been solved. If not, feel free to reopen.

[meadch]

Even after those linking steps, still unable to get hello-world up and running.

1. Downloaded systemwide MKL libraries
2. Copied libraries
3. Confirm my usr/local/lib has the following "mandatory" libraries, per documentation

/usr/local/lib/libmkl_rt.dylib
/usr/local/lib/libmkl_core.dylib
/usr/local/lib/libmkl_intel_lp64.dylib
/usr/local/lib/libiomp5.dylib
/usr/local/lib/libmkl_intel_thread.dylib

4. Java 8

   > java -version
   openjdk version "1.8.0_292"
   OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_292-b10)
   OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.292-b10, mixed mode)

5. project.clj

(defproject hello-world "0.43.1"
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
  :dependencies [[org.clojure/clojure "1.10.3"]
                 [uncomplicate/neanderthal "0.43.1"]
                 ;;Optional. If bytedeco is not present, a system-wide MKL is used.
                 #_[org.bytedeco/mkl-platform-redist "2021.3-1.5.6"]]
  ;; Nvidia doesn't ship CUDA for macOS; you have to add this to your project
  :exclusions [[org.jcuda/jcuda-natives :classifier "apple-x86_64"]
               [org.jcuda/jcublas-natives :classifier "apple-x86_64"]])

6. macOS Big Sur, 11.2.3

And at the repl:

REPL-y 0.4.4, nREPL 0.6.0
Clojure 1.10.3
user=> (ns example
  #_=>   (:use [uncomplicate.neanderthal core native]))

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Execution error (UnsatisfiedLinkError) at java.lang.ClassLoader$NativeLibrary/load (ClassLoader.java:-2).
/private/var/folders/7n/8j215_hd13l3slmlld6kdlcm0000gn/T/libneanderthal-mkl-0.33.0321665570370381334.jnilib: dlopen(/private/var/folders/7n/8j215_hd13l3slmlld6kdlcm0000gn/T/libneanderthal-mkl-0.33.0321665570370381334.jnilib, 1): Library not loaded: @rpath/libmkl_intel_lp64.dylib
  Referenced from: /private/var/folders/7n/8j215_hd13l3slmlld6kdlcm0000gn/T/libneanderthal-mkl-0.33.0321665570370381334.jnilib
  Reason: image not found

Going to try disabling SIP next.

[meadch]

Got it to work! Had to disable SIP.

[HLevering]

@meadch do you use macOs with Intel our M1 chip. I tried running the example with M1. It hangs when requireing neanderthal dependencies. (ns example (:use [uncomplicate.neanderthal core native])) No errors. No warnings. It just hangs.

[blueberry]

@HLevering M1 is not supported (yet).

[sainadh-d]

@blueberry Any estimate when M1 will be supported?

[blueberry]

When I manage to buy an M1 Mac and fund the development of M1 based engine. I can't know for sure, but I can hope that that would be possible in 2023.

[aaelony]

fyi, https://www.zdnet.com/article/aws-amazon-ec2-m1-mac-instances-have-arrived/On Oct 9, 2022, at 15:38, Dragan Djuric @.***> wrote: When I manage to buy an M1 Mac and fund the development of M1 based engine. I can't know for sure, but I can hope that that would be possible in 2023.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: @.***>

[blueberry]

I'm aware of that, but it's not a matter of just recompiling the existing code. There's a sizable chunk of code that has to be specially tailored to what's available on Apple's platform. Doing that development is simply not practical (for me) on a remote instance billed by the hour (not to mention that there's no way I'm trusting Amazon's billing-on-their-whim-with-no-recourse system with my personal CC).