[Newbie issue] Try to follow GPU series but cannot install Neanderthal

[kwccoin]

I tried to followed

https://dragan.rocks/articles/19/Deep-Learning-in-Clojure-From-Scratch-to-GPU-1-Representing-Layers-and-Connections

And as I have some AI and statistics background (Fastai, Leela zero etc.), I enjoy both the GPU part and matrix one. But when I tried to run the code, I can't. There are two cliff between clojure and the GPU part and the Neanderthal. I can do cli -m core as under in the clojure in 60s plus and run lein no issue.

But how one can install the jar under Mac? Clicking the jar links to clojar.org just gave you a message. And I am not sure I follow the base one etc. I have a eGPU for Macbook and another old MacBook has old Nvidia. I would try that on top of Intel MKL. But that the other cliff. The first cliff of Neanderthal cannot be crossed yet.

Any help would be much appreciate ... something like this:

https://medium.com/@swlkr/go-from-zero-to-clojure-in-60-seconds-6550613c0699

[blueberry]

Hi.

1. If you are running this on the Mac, you'll have to wait for JCuda build to bundle Mac binaries in the next release. The current release was shipped without CUDA support, since Macs hadn't shipped with Nvidia GPUs since 2012/13. No one had a hardware to build it, but this problem has been solved recently and is waiting for the release. If you need it on Mac now, the solution is to build JCuda 10.0 yourself, and install your build in your local maven repo (given that you are total newbie, this might require a bit of learning of how lein/maven/clojars work) See https://github.com/jcuda/jcuda-main/issues/29 and https://github.com/jcuda/jcuda-main/issues/23

2. The other, more important part is about the Clojure programming workflow. Unless in some rare cases, you do not run this as a CLI program, in the same way you are not rebooting your OS before each mouse button click. There are many ways how you can structure, develop, or run Java/Clojure program, but the main Clojure feature is REPL (Read Evaluate Print Loop). This is probably different from what you are used to, so you might be surprised or put off at first, but the point is that you start Clojure's JVM as a long-running environment, and issue your commands interactively to this long running process, and see the results immediately. There are many ways how you can apply that, so I'll give you a long youtube video link that might be a bit more on the philosophical side (longer than 60 seconds) but will invite some Clojurians that love to explain this, to answer here with better and more practical advice.

https://youtu.be/Qx0-pViyIDU?t=557

PS. I'm closing this because this is not an issue with Neanderthal per se but a question about Clojure workflow, but feel free to discuss this further here despite it being closed.1

[blueberry]

BTW I've looked at the 60 seconds tutorial link that you posted, and although it might help with starting up with Clojure as fast as promised, I couldn't imagine me as a beginner being help much with that.

If I had only 60 seconds, I would recommend a beginner to do it with leiningen. So, install leiningen, clone the neanderthal hello-world project, as recommended in the article, run lein repl (or, better yet, if you are using CIDER, do the M-x cider-jack-in), connect your favorite editing environment (Emacs+CIDER, Idea + Cursive, etc.) to that repl process, and program by sending the expressions that you write to the live REPL interactively.

I understand that this info is too brief, I hope other users will give you more detailed instructions.

[kwccoin]

Thanks for the long comment.

Sorry I am not saying I spare only 60s on this. I have in fact read those part for 2 days now and start trying to do this for whole morning and still get to no where.

The 60s thingI did get me the lein repl and the sample project running. I have done Java since 1.0 and Lisp and python for 10 years. I get the repl. But somehow I just missed something related to how to install this even if one used Intel library (ignore NVIDIA for the moment; and I would try that later using my Windows PC). In particular, these two statement has confused me quite a bit:

` Installation Add Neanderthal jars to your classpath (from the Clojars). <-- where is the jars, how to download it and where to put it ... or the default will auto compile one version as leiningen work like make/brew/... ?

1a. If you are running on Java 9 or higher, you need to enable the java.base module. Add this to your JVM options (:jvm-opts in leiningen): "--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED". (You can find these in the project.clj i.e.

" ;;also replaces lein's default JVM argument TieredStopAtLevel=1 :jvm-opts ^:replace ["-Dclojure.compiler.direct-linking=true" "-XX:MaxDirectMemorySize=16g" "-XX:+UseLargePages"

_"--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED"]"

<-- now as the sample hello world cli does not have these how these statement would be used or the sample hello world native.clj somehow cascade and link to the core project.cli? Not sure how and confused how to make the java base work.

To use the native engine: install Intel’s MKL on your system following Native Engine Requirements). <-- I install the MKL and read from another thread and also in the instruction that one has to copy or link to another location, it is fine but would it work

`

`With Leiningen The most straightforward way to include Neanderthal in your project is with Leiningen. Add the following dependency to your project.clj, just like in the Hello World project:

<-- how ? `

I think we need a step by step installation so hello world work for say Intel library, and/or under Windows under cuda ... it would help a lot for newbie like me. I was attracted by the web pages and also those video made. Once the installation is done, I could then concentrate learn those vector/matrix/netural network part.

(For that matter I have read the little online book 2 years ago and implement easily using Python, done PIAP using Mac Lisp and struggle a bit then in installation; Java I work from 1.0 to now. Hence I would think I have the background to begin this journey. It may be all just installation issue I guess.)

[kwccoin]

I finally get N installed by doing this video:

https://www.youtube.com/watch?v=mDwNgQhh0hY

[kwccoin]

There are some problem but I guess I would try to handle it onwards. My steps are:

1. lein new app hw_c2
2. cd hw_c2
3. lein run ;;; or lein test
4. ;;; then modify project.clj

` :dependencies [[org.clojure/clojure "1.10.0"] [uncomplicate/neanderthal "0.22.1"] ; -Dennis- need this ]

:exclusions [[org.jcuda/jcuda-natives :classifier "apple-x86_64"] [org.jcuda/jcublas-natives :classifier "apple-x86_64"]]

;;-dennis- copy from project.cli and assumes needed

;;also replaces lein's default JVM argument TieredStopAtLevel=1 :jvm-opts ^:replace ["-Dclojure.compiler.direct-linking=true" "-XX:MaxDirectMemorySize=16g" "-XX:+UseLargePages"

_"--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED"]

;;-Dennis- get a message ;; "OpenJDK 64-Bit Server VM warning: -XX:+UseLargePages not supported in this VM" `

5. lein run ;;; should be lein deps but run seems ok

Retrieving uncomplicate/neanderthal/0.22.1/neanderthal-0.22.1.pom from clojars Retrieving uncomplicate/commons/0.7.2/commons-0.7.2.pom from clojars Retrieving uncomplicate/fluokitten/0.9.1/fluokitten-0.9.1.pom from clojars Retrieving uncomplicate/neanderthal-native/0.21.0/neanderthal-native-0.21.0.pom from clojars Retrieving uncomplicate/clojurecl/0.10.6/clojurecl-0.10.6.pom from clojars Retrieving org/jocl/jocl/2.0.1/jocl-2.0.1.pom from central Retrieving org/clojure/core.async/0.4.490/core.async-0.4.490.pom from central Retrieving org/clojure/tools.analyzer.jvm/0.7.2/tools.analyzer.jvm-0.7.2.pom from central Retrieving org/clojure/tools.analyzer/0.6.9/tools.analyzer-0.6.9.pom from central Retrieving org/clojure/clojure/1.5.1/clojure-1.5.1.pom from central Retrieving org/clojure/core.memoize/0.5.9/core.memoize-0.5.9.pom from central Retrieving org/clojure/core.cache/0.6.5/core.cache-0.6.5.pom from central Retrieving org/clojure/data.priority-map/0.0.7/data.priority-map-0.0.7.pom from central Retrieving org/clojure/clojure/1.4.0/clojure-1.4.0.pom from central Retrieving org/ow2/asm/asm-all/4.2/asm-all-4.2.pom from central Retrieving org/ow2/asm/asm-parent/4.2/asm-parent-4.2.pom from central Retrieving org/ow2/ow2/1.3/ow2-1.3.pom from central Retrieving org/clojure/tools.reader/1.0.0-beta4/tools.reader-1.0.0-beta4.pom from central Retrieving org/clojure/pom.contrib/0.2.0/pom.contrib-0.2.0.pom from central Retrieving org/clojure/clojure/1.7.0/clojure-1.7.0.pom from central Retrieving org/jocl/jocl-blast/1.5.0/jocl-blast-1.5.0.pom from central Retrieving org/jocl/jocl/2.0.0/jocl-2.0.0.pom from central Retrieving uncomplicate/clojurecuda/0.6.2/clojurecuda-0.6.2.pom from clojars Retrieving org/jcuda/jcuda/10.0.0/jcuda-10.0.0.pom from central Retrieving org/jcuda/jcuda-parent/10.0.0/jcuda-parent-10.0.0.pom from central Retrieving org/jcuda/jcublas/10.0.0/jcublas-10.0.0.pom from central Retrieving org/apache/commons/commons-math3/3.6.1/commons-math3-3.6.1.pom from central Retrieving org/apache/commons/commons-parent/39/commons-parent-39.pom from central Retrieving org/apache/apache/16/apache-16.pom from central Retrieving org/jocl/jocl/2.0.1/jocl-2.0.1.jar from central Retrieving org/clojure/core.async/0.4.490/core.async-0.4.490.jar from central Retrieving org/clojure/core.memoize/0.5.9/core.memoize-0.5.9.jar from central Retrieving org/clojure/tools.analyzer.jvm/0.7.2/tools.analyzer.jvm-0.7.2.jar from central Retrieving org/clojure/tools.analyzer/0.6.9/tools.analyzer-0.6.9.jar from central Retrieving org/clojure/core.cache/0.6.5/core.cache-0.6.5.jar from central Retrieving org/ow2/asm/asm-all/4.2/asm-all-4.2.jar from central Retrieving org/clojure/data.priority-map/0.0.7/data.priority-map-0.0.7.jar from central Retrieving org/clojure/tools.reader/1.0.0-beta4/tools.reader-1.0.0-beta4.jar from central Retrieving org/jcuda/jcuda/10.0.0/jcuda-10.0.0.jar from central Retrieving org/jcuda/jcublas/10.0.0/jcublas-10.0.0.jar from central Retrieving org/jocl/jocl-blast/1.5.0/jocl-blast-1.5.0.jar from central Retrieving org/apache/commons/commons-math3/3.6.1/commons-math3-3.6.1.jar from central Retrieving uncomplicate/fluokitten/0.9.1/fluokitten-0.9.1.jar from clojars Retrieving uncomplicate/commons/0.7.2/commons-0.7.2.jar from clojars Retrieving uncomplicate/neanderthal-native/0.21.0/neanderthal-native-0.21.0.jar from clojars Retrieving uncomplicate/clojurecl/0.10.6/clojurecl-0.10.6.jar from clojars Retrieving uncomplicate/neanderthal/0.22.1/neanderthal-0.22.1.jar from clojars Retrieving uncomplicate/clojurecuda/0.6.2/clojurecuda-0.6.2.jar from clojars OpenJDK 64-Bit Server VM warning: -XX:+UseLargePages not supported in this VM

6. Finally the real error comes along and that will be my starting point I guess:

see next post

[kwccoin]

Still cannot do any addition or multiplication both based on example and the web page:

` (ns hw-c2.core ;hello-world.native ; ;(:gen-class) (:require [uncomplicate.neanderthal.core :refer :all] [uncomplicate.neanderthal.native :refer :all]))

;; We create two matrices... (def a (dge 2 3 [1 2 3 4 5 6])) (def b (dge 3 2 [1 3 5 7 9 11]))

;; ... and multiply them BUT CANNOT <---

;(mm a b)

; (pxy a a)

;(ns hw-c2.core

(defn -main "I don't do a whole lot ... yet." [& args] (println "Hello, World bb!") (println (+ 22 33)) ;; We create two matrices... (def c (dge 2 3 [1 2 3 4 5 6])) (def d (dge 3 2 [1 3 5 7 9 11])) ;; We add two matrices... BUT CANNOT <---- ;(xpy c d)

)

; fail above and hence not trying to do any real matrix yet

;
; (require '[uncomplicate.commons.core :refer [with-release]] ; '[uncomplicate.neanderthal ; [native :refer [dv dge]] ; [core :refer [mv!]]]) ; )

; (with-release [x (dv 0.3 0.9) ; w1 (dge 4 2 [0.3 0.6 ; 0.1 2.0 ; 0.9 3.7 ; 0.0 1.0] ; {:layout :row}) ; h1 (dv 4)] ;( println (mv! w1 x h1)))

`

But at least this can run and seems need some help to find an example of how to use Neanderthal and not installation I guess

[blueberry]

Installation Add Neanderthal jars to your classpath (from the Clojars). <-- where is the jars, how to download it and where to put it ... or the default will auto compile one version as leiningen work like make/brew/... ?

It is done automatically by leiningen (or maven, or clojure deps). It is enough to specify neanderthal as one of the project's dependencies. See the Hello World: https://github.com/uncomplicate/neanderthal/blob/4fd0db19b283757b68288d11ac8fd3d8b1d195bf/examples/hello-world/project.clj#L5

1a. If you are running on Java 9 or higher, you need to enable the java.base module. Add this to your JVM options (:jvm-opts in leiningen): "--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED". (You can find these in the project.clj i.e.

" ;;also replaces lein's default JVM argument TieredStopAtLevel=1 :jvm-opts ^:replace ["-Dclojure.compiler.direct-linking=true" "-XX:MaxDirectMemorySize=16g" "-XX:+UseLargePages"

_"--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED"]"

This does not cascade, it is per-project (and is commented out by #_ in the main project since I'm on Java 8).

Add this:

:jvm-opts ^:replace ["--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED"]

To use the native engine: install Intel’s MKL on your system following Native Engine Requirements). <-- I install the MKL and read from another thread and also in the instruction that one has to copy or link to another location, it is fine but would it work

`

`With Leiningen The most straightforward way to include Neanderthal in your project is with Leiningen. Add the following dependency to your project.clj, just like in the Hello World project:

<-- how ?

You literally do the same as the hello world does:

 :dependencies [[org.clojure/clojure "1.10.0"]
                           [uncomplicate/neanderthal "0.22.1"]]

That's it.

`

I think we need a step by step installation so hello world work for say Intel library, and/or under Windows under cuda ... it would help a lot for newbie like me. I was attracted by the web pages and also those video made. Once the installation is done, I could then concentrate learn those vector/matrix/netural network part.

Step by step is literally:

1. Make sure Intel MKN .dylib is somewhere on your path (id doesn't matter whether you use intel's gui, or just copy/paste the .dylib files somewhere on your DYLD_LIBRARY_PATH)
2. clone neanderhtal hello-world
3. cd to the hello-world directory
4. run lein repl

(For that matter I have read the little online book 2 years ago and implement easily using Python, done PIAP using Mac Lisp and struggle a bit then in installation; Java I work from 1.0 to now. Hence I would think I have the background to begin this journey. It may be all just installation issue I guess.)

The issue I think is that this is much simpler in Clojure than it is in other environments, so you unintentionally overcomplicated the instructions. I would suggest that you first make yourself comfortable with leiningen by

1. creating some simple lein project yourself and make sure that the REPL workflow works for you
2. add some simple pure library as dependency to see how it works with clojars (try uncomplicate/fluokitten)
3. add neanderhtal and mkl

[blueberry]

Still cannot do any addition or multiplication both based on example and the web page:

But at least this can run and seems need some help to find an example of how to use Neanderthal and not installation I guess

As the first step, I'd try to work in the hello world project. Don't use lein test (as there are no tests there), don't use main, don't experiment. Just clone hello world, cd into it and run lein repl. Then go to the appropriate namespace (native, I suppose) and evaluate it.