search

uncomplicate / neanderthal

Fast Clojure Matrix Library
http://neanderthal.uncomplicate.org
Eclipse Public License 1.0
1.06k stars 56 forks source link

UnsatisfiedLinkError, no such file `libmkl_rt.so`, with `org.bytedeco/mkl-platform-redist` included as a dep. #102

Closed eihli closed 3 years ago

eihli commented 3 years ago

After reading the docs at https://neanderthal.uncomplicate.org/articles/getting_started.html#the-native-library-used-by-neanderthals-native-engine-optional I expected I wouldn't see an error about missing mkl-related files if I included the bytedeco dep.

But I'm getting this when I launch cider.

Execution error (UnsatisfiedLinkError) at java.lang.ClassLoader$NativeLibrary/load (ClassLoader.java:-2).
/tmp/libneanderthal-mkl-0.33.0724836737567557959.so: libmkl_rt.so: cannot open shared object file: No such file or directory
...
java.lang.NoClassDefFoundError: Could not initialize class uncomplicate.neanderthal.internal.host.MKL

deps.edn

{:paths ["src" "resources"]
 :deps {org.clojure/clojure {:mvn/version "1.10.1"}
        uncomplicate/neanderthal {:mvn/version "0.38.0"}
        org.bytedeco/mkl-platform-redist {:mvn/version "2020.1-1.5.3"}}
 :aliases {:dev {:extra-paths ["dev"]}}}
blueberry commented 3 years ago

You're probably using a mismatched version of javacpp-mkl. See the example hello world: https://github.com/uncomplicate/neanderthal/blob/master/examples/hello-world/project.clj

eihli commented 3 years ago

Ah yes. That was it. mkl-platform-redist @ 2020.3-1.5.4 and neanderthal @ 0.38.0 works. Thanks!

zendevil commented 3 years ago

This doesn't work for me. I'm still getting the error. Here's my project.clj, the same one in hello-world example. I'm running the hello world example from the source:

(defproject hello-world "0.38.0"
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
  :dependencies [[org.clojure/clojure "1.10.1"]
                 [uncomplicate/neanderthal "0.38.0"]
                 ;;Optional. If bytedeco is not present, a system-wide MKL is used.
                 [org.bytedeco/mkl-platform-redist "2020.3-1.5.4"]]
  ;; Nvidia doesn't ship CUDA for macOS; you have to add this to your project
  :exclusions [[org.jcuda/jcuda-natives :classifier "apple-x86_64"]
               [org.jcuda/jcublas-natives :classifier "apple-x86_64"]]
  ;; If on Java 9+, you have to uncomment the following JVM option.
  :jvm-opts ^:replace [#_"--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED"])

Full stack trace

         More information.Execution error (UnsatisfiedLinkError) at java.lang.ClassLoader$NativeLibrary/load (ClassLoader.java:-2).
/private/var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/libneanderthal-mkl-0.33.07947871945054341194.jnilib: dlopen(/private/var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/libneanderthal-mkl-0.33.07947871945054341194.jnilib, 1): Library not loaded: @rpath/libmkl_intel_lp64.dylib
  Referenced from: /private/var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/libneanderthal-mkl-0.33.07947871945054341194.jnilib
  Reason: image not found
ERROR: Unhandled REPL handler exception processing message {:nrepl.middleware.print/stream? 1, :nrepl.middleware.print/print cider.nrepl.pprint/pprint, :nrepl.middleware.print/quota 1048576, :nrepl.middleware.print/options {:right-margin 80}, :op stacktrace, :session 0f1c7690-c341-4daa-97ce-c8c9d2e37dbc, :id 19}
java.lang.NoClassDefFoundError: Could not initialize class uncomplicate.neanderthal.internal.host.MKL
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at clojure.lang.RT.classForName(RT.java:2211)
    at clojure.lang.RT.classForName(RT.java:2220)
    at clojure.lang.Compiler.maybeResolveIn(Compiler.java:7438)
    at clojure.core$ns_resolve.invokeStatic(core.clj:4370)
    at clojure.core$ns_resolve.invokeStatic(core.clj:4359)
    at clojure.core$ns_resolve.invoke(core.clj:4359)
    at cider.nrepl.inlined_deps.orchard.v0v6v0.orchard.java$resolve_class$fn__42350.invoke(java.clj:310)
    at cider.nrepl.inlined_deps.orchard.v0v6v0.orchard.java$resolve_class.invokeStatic(java.clj:310)
    at cider.nrepl.inlined_deps.orchard.v0v6v0.orchard.java$resolve_class.invoke(java.clj:304)
    at cider.nrepl.inlined_deps.orchard.v0v6v0.orchard.java$resolve_symbol.invokeStatic(java.clj:344)
    at cider.nrepl.inlined_deps.orchard.v0v6v0.orchard.java$resolve_symbol.invoke(java.clj:330)
    at cider.nrepl.middleware.stacktrace$analyze_fn.invokeStatic(stacktrace.clj:84)
    at cider.nrepl.middleware.stacktrace$analyze_fn.invoke(stacktrace.clj:63)
    at clojure.core$comp$fn__5807.invoke(core.clj:2569)
    at clojure.core$comp$fn__5807.invoke(core.clj:2569)
    at clojure.core$comp$fn__5807.invoke(core.clj:2569)
    at cider.nrepl.middleware.stacktrace$analyze_frame.invokeStatic(stacktrace.clj:180)
    at cider.nrepl.middleware.stacktrace$analyze_frame.invoke(stacktrace.clj:177)
    at clojure.core$map$fn__5866.invoke(core.clj:2755)
    at clojure.lang.LazySeq.sval(LazySeq.java:42)
    at clojure.lang.LazySeq.seq(LazySeq.java:51)
    at clojure.lang.RT.seq(RT.java:535)
    at clojure.core$seq__5402.invokeStatic(core.clj:137)
    at clojure.core$map$fn__5870.invoke(core.clj:2757)
    at clojure.lang.LazySeq.sval(LazySeq.java:42)
    at clojure.lang.LazySeq.seq(LazySeq.java:51)
    at clojure.lang.Cons.next(Cons.java:39)
    at clojure.lang.RT.next(RT.java:713)
    at clojure.core$next__5386.invokeStatic(core.clj:64)
    at clojure.core.protocols$fn__8159.invokeStatic(protocols.clj:169)
    at clojure.core.protocols$fn__8159.invoke(protocols.clj:124)
    at clojure.core.protocols$fn__8114$G__8109__8123.invoke(protocols.clj:19)
    at clojure.core.protocols$seq_reduce.invokeStatic(protocols.clj:31)
    at clojure.core.protocols$fn__8146.invokeStatic(protocols.clj:75)
    at clojure.core.protocols$fn__8146.invoke(protocols.clj:75)
    at clojure.core.protocols$fn__8088$G__8083__8101.invoke(protocols.clj:13)
    at clojure.core$reduce.invokeStatic(core.clj:6828)
    at clojure.core$into.invokeStatic(core.clj:6895)
    at clojure.core$mapv.invokeStatic(core.clj:6903)
    at clojure.core$mapv.invoke(core.clj:6903)
    at cider.nrepl.middleware.stacktrace$flag_duplicates.invokeStatic(stacktrace.clj:175)
    at cider.nrepl.middleware.stacktrace$flag_duplicates.invoke(stacktrace.clj:164)
    at cider.nrepl.middleware.stacktrace$analyze_stacktrace.invokeStatic(stacktrace.clj:186)
    at cider.nrepl.middleware.stacktrace$analyze_stacktrace.invoke(stacktrace.clj:182)
    at cider.nrepl.middleware.stacktrace$analyze_cause.invokeStatic(stacktrace.clj:290)
    at cider.nrepl.middleware.stacktrace$analyze_cause.invoke(stacktrace.clj:281)
    at cider.nrepl.middleware.stacktrace$analyze_causes$fn__43515.invoke(stacktrace.clj:315)
    at clojure.core$map$fn__5862$fn__5863.invoke(core.clj:2742)
    at clojure.core$take_while$fn__5913$fn__5914.invoke(core.clj:2901)
    at clojure.lang.Iterate.reduce(Iterate.java:81)
    at clojure.core$transduce.invokeStatic(core.clj:6883)
    at clojure.core$into.invokeStatic(core.clj:6899)
    at clojure.core$into.invoke(core.clj:6887)
    at cider.nrepl.middleware.stacktrace$analyze_causes.invokeStatic(stacktrace.clj:314)
    at cider.nrepl.middleware.stacktrace$analyze_causes.invoke(stacktrace.clj:306)
    at cider.nrepl.middleware.stacktrace$handle_stacktrace.invokeStatic(stacktrace.clj:323)
    at cider.nrepl.middleware.stacktrace$handle_stacktrace.invoke(stacktrace.clj:320)
    at clojure.lang.Var.invoke(Var.java:388)
    at cider.nrepl$wrap_stacktrace$fn__35396.invoke(nrepl.clj:410)
    at nrepl.middleware$wrap_conj_descriptor$fn__34293.invoke(middleware.clj:16)
    at cider.nrepl$wrap_debug$fn__35298.invoke(nrepl.clj:152)
    at nrepl.middleware$wrap_conj_descriptor$fn__34293.invoke(middleware.clj:16)
    at cider.nrepl$wrap_enlighten$fn__35306.invoke(nrepl.clj:178)
    at nrepl.middleware$wrap_conj_descriptor$fn__34293.invoke(middleware.clj:16)
    at cider.nrepl$wrap_format$fn__35314.invoke(nrepl.clj:182)
    at nrepl.middleware$wrap_conj_descriptor$fn__34293.invoke(middleware.clj:16)
    at cider.nrepl$wrap_info$fn__35322.invoke(nrepl.clj:196)
    at nrepl.middleware$wrap_conj_descriptor$fn__34293.invoke(middleware.clj:16)
    at cider.nrepl$wrap_out$fn__35356.invoke(nrepl.clj:312)
    at nrepl.middleware$wrap_conj_descriptor$fn__34293.invoke(middleware.clj:16)
    at cider.nrepl$wrap_resource$fn__35380.invoke(nrepl.clj:384)
    at nrepl.middleware$wrap_conj_descriptor$fn__34293.invoke(middleware.clj:16)
    at cider.nrepl$wrap_clojuredocs$fn__35454.invoke(nrepl.clj:504)
    at nrepl.middleware$wrap_conj_descriptor$fn__34293.invoke(middleware.clj:16)
    at cider.nrepl$wrap_undef$fn__35428.invoke(nrepl.clj:471)
    at nrepl.middleware$wrap_conj_descriptor$fn__34293.invoke(middleware.clj:16)
    at nrepl.middleware.load_file$wrap_load_file$fn__34642.invoke(load_file.clj:81)
    at nrepl.middleware$wrap_conj_descriptor$fn__34293.invoke(middleware.clj:16)
    at cider.nrepl$wrap_content_type$fn__35256.invoke(nrepl.clj:98)
    at nrepl.middleware$wrap_conj_descriptor$fn__34293.invoke(middleware.clj:16)
    at cider.nrepl$wrap_ns$fn__35348.invoke(nrepl.clj:278)
    at nrepl.middleware$wrap_conj_descriptor$fn__34293.invoke(middleware.clj:16)
    at cider.nrepl$wrap_inspect$fn__35332.invoke(nrepl.clj:215)
    at nrepl.middleware$wrap_conj_descriptor$fn__34293.invoke(middleware.clj:16)
    at nrepl.middleware.caught$wrap_caught$fn__34550.invoke(caught.clj:97)
    at nrepl.middleware$wrap_conj_descriptor$fn__34293.invoke(middleware.clj:16)
    at cider.nrepl$wrap_apropos$fn__35272.invoke(nrepl.clj:118)
    at nrepl.middleware$wrap_conj_descriptor$fn__34293.invoke(middleware.clj:16)
    at cider.nrepl$wrap_test$fn__35404.invoke(nrepl.clj:420)
    at nrepl.middleware$wrap_conj_descriptor$fn__34293.invoke(middleware.clj:16)
    at nrepl.middleware.print$wrap_print$fn__34517.invoke(print.clj:234)
    at nrepl.middleware$wrap_conj_descriptor$fn__34293.invoke(middleware.clj:16)
    at cider.nrepl$wrap_classpath$fn__35280.invoke(nrepl.clj:126)
    at nrepl.middleware$wrap_conj_descriptor$fn__34293.invoke(middleware.clj:16)
    at cider.nrepl$wrap_trace$fn__35412.invoke(nrepl.clj:442)
    at nrepl.middleware$wrap_conj_descriptor$fn__34293.invoke(middleware.clj:16)
    at cider.nrepl$wrap_tracker$fn__35420.invoke(nrepl.clj:460)
    at nrepl.middleware$wrap_conj_descriptor$fn__34293.invoke(middleware.clj:16)
    at nrepl.middleware.session$session$fn__34738.invoke(session.clj:272)
    at nrepl.middleware$wrap_conj_descriptor$fn__34293.invoke(middleware.clj:16)
    at cider.nrepl$wrap_profile$fn__35364.invoke(nrepl.clj:321)
    at nrepl.middleware$wrap_conj_descriptor$fn__34293.invoke(middleware.clj:16)
    at cider.nrepl$wrap_slurp$fn__35264.invoke(nrepl.clj:110)
    at nrepl.middleware$wrap_conj_descriptor$fn__34293.invoke(middleware.clj:16)
    at nrepl.server$handle_STAR_.invokeStatic(server.clj:18)
    at nrepl.server$handle_STAR_.invoke(server.clj:15)
    at nrepl.server$handle$fn__34775.invoke(server.clj:27)
    at clojure.core$binding_conveyor_fn$fn__5754.invoke(core.clj:2030)
    at clojure.lang.AFn.call(AFn.java:18)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Syntax error compiling at (uncomplicate/neanderthal/native.clj:1:1).
No namespace: uncomplicate.neanderthal.internal.host.mkl
burbma commented 3 years ago

Alright I've got a lead on this but I'm in way over my head.

So uncomplicate.neanderthal.native requires uncomplicate.neanderthal.internal.host.mkl. The problem is running (require '[uncomplicate.neanderthal.internal.host.mkl]) fails with something like:

Execution error (UnsatisfiedLinkError) at java.lang.ClassLoader$NativeLibrary/load0 (ClassLoader.java:-2).
/private/var/folders/8m/p0_shyw51mq8mf2bb7lg52yc0000gn/T/libneanderthal-mkl-0.33.010233505079158403947.jnilib: dlopen(/private/var/folders/8m/p0_shyw51mq8mf2b
b7lg52yc0000gn/T/libneanderthal-mkl-0.33.010233505079158403947.jnilib, 1): Library not loaded: @rpath/libmkl_intel_lp64.dylib
  Referenced from: /private/var/folders/8m/p0_shyw51mq8mf2bb7lg52yc0000gn/T/libneanderthal-mkl-0.33.010233505079158403947.jnilib
  Reason: image not found

After many searches and being generally out of my league I end up reading this https://blog.krzyzanowskim.com/2018/12/05/rpath-what/.

In that article I find that I can use otool -l to find out what this @rpath thing actually resolves to, it's tricky though because the binary /private/var/folders/8m/p0_shyw51mq8mf2bb7lg52yc0000gn/T/libneanderthal-mkl-0.33.010233505079158403947.jnilib only exists while my REPL is running and after I've tried the failing require. So in another shell I run otool -l /private/var/folders/8m/p0_shyw51mq8mf2bb7lg52yc0000gn/T/libneanderthal-mkl-0.33.010233505079158403947.jnilib | grep -B 1 -A 2 LC_RPATH and I get this:

Load command 15
          cmd LC_RPATH
      cmdsize 72
         path /opt/intel/compilers_and_libraries_2018.3.185/mac/mkl/lib (offset 12)

Which makes me think I wouldn't have issues if I installed MKL 2018.3.185 on my system, but I didn't, I'm using org.bytedeco/mkl-platform-redist {:mvn/version "2020.3-1.5.4"}. Seems strange that this rpath was set to such a specific install location, I'm not sure when this linking happens in Clojure/Java land or who has control over it.

Via previous searches and debug logs etc. I had already discovered that the dylib's are actually in ~/.javacpp/cache/mkl-2020.3-1.5.4-macosx-x86_64-redist.jar/org/bytedeco/mkl/macosx-x86_64/ so I simply tried copying them with:

sudo mkdir -p /opt/intel/compilers_and_libraries_2018.3.185/mac/mkl/lib
sudo cp ~/.javacpp/cache/mkl-2020.3-1.5.4-macosx-x86_64-redist.jar/org/bytedeco/mkl/macosx-x86_64/ /opt/intel/compilers_and_libraries_2018.3.185/mac/mkl/lib

And low and behold I can now (require '[uncomplicate.neanderthal.internal.host.mkl]) as well as the original goal which was (require '[uncomplicate.neanderthal.nativel]). Running (uncomplicate.neanderthal.native/dge 3 2 [1 2 3 4 5 6]) works as well.

I don't know where to go next as to tracking down what a more proper solution is than my copying of the dylib's.

cc people who seemed to have cared about this issue (or a potentially related one) in the past since conversations are spread out in multiple places: @blueberry @zendevil @jasonwaack @lccambiaghi @Hendekagon @SolbiatiAlessandro

blueberry commented 3 years ago

I don't know why you did all this, nor whether you've read the Getting Started guide, but it's great that you solved your issue at the end.

For posterity: I recommend reading Getting Started and proceeding accordingly.

burbma commented 3 years ago

Based on this issue and #31 I gathered that people were having trouble getting it to work in anyway besides installing mkl on the system and disabling SIP and exporting the right DYLD_LIBRARY_PATH. Yes I did read the starting guide, it claims one should be able to just include the bytedeco dependency:

Final note If you prefer zero-install, just include [org.bytedeco/mkl-platform-redist "2020.3-1.5.4"] as a dependencly in your leiningen project and none of these is necessary.

But this statement is not true (at least as I interpreted it), you still need to copy the dylib's to the right place, either the one I used or perhaps /usr/local/lib which I haven't tried yet. The starting guide mentions making a copy to /usr/local/lib but I was under the impression based on the quote above that such would only be necessary for a system install of mkl and not when including the bytedeco dependency.

blueberry commented 3 years ago

Based on this issue and #31 I gathered that people were having trouble getting it to work in anyway besides installing mkl on the system and disabling SIP and exporting the right DYLD_LIBRARY_PATH. Yes I did read the starting guide, it claims one should be able to just include the bytedeco dependency:

Final note If you prefer zero-install, just include [org.bytedeco/mkl-platform-redist "2020.3-1.5.4"] as a dependencly in your leiningen project and none of these is necessary.

But this statement is not true (at least as I interpreted it), you still need to copy the dylib's to the right place, either the one I used or perhaps /usr/local/lib which I haven't tried yet. The starting guide mentions making a copy to /usr/local/lib but I was under the impression based on the quote above that such would only be necessary for a system install of mkl and not when including the bytedeco dependency.

You have to disable SIP, then everything works in the same way as linux, and that's it. Now, I understand that some people might have reasons why they don't want to disable SIP, but that's beyond what I can support. It's literally that there is a big Apple dam blocking the river, and people are complaining why there is no water downstream. There is no water because the dam (SIP) is blocking all water. If you don't want to open the dam, that's ok, but then I don't know how to help.

burbma commented 3 years ago

It's very possible that I'm still just missing something and I apologize if I'm being confusing in how I try to explain this.

I tried disabling SIP and including the bytedeco dependency but I still had the Reason: image not found error. Would you have expected those two steps alone to be sufficient?

blueberry commented 3 years ago

It should be. Please try the Hello World example project first https://github.com/uncomplicate/neanderthal/tree/master/examples/hello-world Please read the comments in project.clj regarding Java 9+ and jvm-opts.

If that doesn't work, then it may be that there's something set up on your OS that messes with this, or Apple introduced yet another roadblock for developers. IDK. I only have an old MacBook Air, and no access to latest macOS. But I've helped quite a few people set up new macs and in each case when they had various issues the source of the problem was always SIP. Maybe that reconfiguration requires a reboot, who knows...

burbma commented 3 years ago

I've tried hello-world, using OpenJDK 11.0.9.1 and thus uncommenting the jvm-opts, I disabled SIP and rebooted, it gives the same error

Execution error (UnsatisfiedLinkError) at java.lang.ClassLoader$NativeLibrary/load0 (ClassLoader.java:-2).
/private/var/folders/8m/p0_shyw51mq8mf2bb7lg52yc0000gn/T/libneanderthal-mkl-0.33.013592419731879182342.jnilib: dlopen(/private/var/folders/8m/p0_shyw51mq8mf2bb7lg52yc0000gn/T/libneanderthal-mkl-0.33.013592419731879182342.jnilib, 1): Library not loaded: @rpath/libmkl_intel_lp64.dylib
  Referenced from: /private/var/folders/8m/p0_shyw51mq8mf2bb7lg52yc0000gn/T/libneanderthal-mkl-0.33.013592419731879182342.jnilib
  Reason: image not found

I appreciate your responses and don't expect you to resolve this for me. If you don't mind, I'm wondering if you know any details about this file and where it comes from /private/var/folders/8m/p0_shyw51mq8mf2bb7lg52yc0000gn/T/libneanderthal-mkl-0.33.013592419731879182342.jnilib. It seems to be created once I attempt to require uncomplicate.neanderthal.internal.host.mkl and it is the library that when loaded by dlopen fails to find it's dependency on the mkl dylib's.

blueberry commented 3 years ago

Then it seems that this deserves its own issue. Please create a new with as many detailed info as you can share. I really don't know what could be the problem, nor I have a machine where I could reproduce this, but someone else might help. In the meantime, I guess some other installation option might be a temporary workaround.

malloc82 commented 3 years ago

@burbma I don't know if you have solved this, but I ran into the same problem and was able to fix it. It seems like the mkl redist library is missing libmkl_intel_lp64.dylib file which is supposed to link to libmkl_intel_lp64.1.dylib. I just manually created the link in the same folder in addition to creating a link of /opt/intel/compilers_and_libraries_2018.3.185/mac/mkl/lib that points to the ~/.javacpp/cache/mkl-2020.3-1.5.4-macosx-x86_64-redist.jar/org/bytedeco/mkl/macosx-x86_64/. That seems to fixed the problem.

@blueberry I think the redist mkl library is supposed to work out of the box, but I think they missed a hard link which requires user to manually set it up.

vxe commented 2 years ago

I also could not get the redist mkl library to work however installing the intel libs worked the first try, I did the following (macos):

  1. install intel mkl via gui (make a note of version) link

  2. set DYLD_LIBRARY_PATH

    export  DYLD_LIBRARY_PATH=/opt/intel/oneapi/mkl/VERSION_FROM_STEP_1/lib

  3. try lein repl

➛ lein repl                                                                                                                                                      
OpenJDK 64-Bit Server VM warning: -XX:+UseLargePages not supported in this VM
nREPL server started on port 55698 on host 127.0.0.1 - nrepl://127.0.0.1:55698
REPL-y 0.5.1, nREPL 0.8.3
Clojure 1.10.3
OpenJDK 64-Bit Server VM 17+0
    Docs: (doc function-name-here)
          (find-doc "part-of-name-here")
  Source: (source function-name-here)
 Javadoc: (javadoc java-object-or-class-here)
    Exit: Control+D or (exit) or (quit)
 Results: Stored in vars *1, *2, *3, an exception in *e

user=> (ns hello-world.native
  #_=> (:require [uncomplicate.neanderthal
  #_=>              [core :refer :all]
  #_=>              [native :refer :all]]))
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
nil
hello-world.native=> (def a (dge 2 3 [1 2 3 4 5 6]))
#'hello-world.native/a
hello-world.native=> (def b (dge 3 2 [1 3 5 7 9 11]))
#'hello-world.native/b
hello-world.native=> (mm a b)
#RealGEMatrix[double, mxn:2x2, layout:column, offset:0]
   ▥       ↓       ↓       ┓    
   →      35.00   89.00         
   →      44.00  116.00         
   ┗                       ┛
vxe commented 1 year ago

2021.1 mkl no longer available on intel site, can download here and set DYLD_LIBRARY_PATH as above