problem compiling uberjar while requiring uncomplicate.neanderthal.native

[erkkikeranen]

I am using Lein 2.9.4 JDK 11.0.5 Mac OSX 10.15.7

If I have included neanderthal in a project, and try to run

lein uberjar

I get the output

Compiling cljml.benchmark.vectors
Compiling cljml.core
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Compiling cljml.native
Compiling cljml.system.info

but then the process just hangs forever, consuming no CPU and no jars are created.

I am actually not doing anything special, I have a -main function that prints hello world, doesn't even do any calculations.

If I remove

(:require [uncomplicate.neanderthal.core :as core]
               [uncomplicate.neanderthal.native :as native])

uberjar is built, and I get my huge jar (but of course nothing useful except my hello world print)

Compiling cljml.benchmark.vectors
Compiling cljml.core
Compiling cljml.native
Compiling cljml.system.info
Created /cljml/target/uberjar/cljml-0.1.0-SNAPSHOT.jar
Created /cljml/target/uberjar/cljml-0.1.0-SNAPSHOT-standalone.jar

Here's my project.clj

(defproject cljml "0.1.0-SNAPSHOT"
  :description "..."
  :license {:name "EPL-2.0 OR GPL-2.0-or-later WITH Classpath-exception-2.0"
            :url "https://www.eclipse.org/legal/epl-2.0/"}
  :dependencies [[org.clojure/clojure "1.10.1"]

                 [uncomplicate/neanderthal "0.38.0" 
                  :exclusions ; because https://github.com/uncomplicate/neanderthal/issues/41
                  [org.jcuda/jcuda-natives, org.jcuda/jcublas-natives]]

                 [org.bytedeco/mkl-platform-redist "2020.1-1.5.3"]]

  :jvm-opts ^:replace ["--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED"]

  :plugins [[lein-ancient "0.6.15"]]

  :main ^:skip-aot cljml.core

  :source-paths ["src"]

  :repl-options {:init-ns cljml.user}

  :target-path "target/%s"

  :profiles {:uberjar {:aot :all
                       :jvm-opts ["-Dclojure.compiler.direct-linking=true"]}
             :repl {:dependencies [[org.clojure/tools.namespace "1.0.0"]]
                    :source-paths ["dev" "src"]}})

lein repl works just fine and I can even run calculations.

Thanks!

[blueberry]

Have you seen #102 ?

If that doesn't help, it may be the agent I run before the native namespace to workaround some MacOS shenanigans that users reported. That is not strictly necessary, but I can't confirm since I can't reproduce their problems. Anyway, some users of non-lein build tools experienced this hang in CI, probably because their process is waiting for the agents thread to be shut down, which is a clojure thing, so I guess if the tool is not handling it itself, you should try to call that shutdown in code to test if that's the culprit. Another option is to replace the agent with explicit load of javacpp MKL class, and see whether it helps (in that case, I can fix this upstream)...

[erkkikeranen]

Actually I realized that https://neanderthal.uncomplicate.org/articles/getting_started.html mentioned version [org.bytedeco/mkl-platform-redist "2020.1-1.5.3"] for the dependency that instead was mentioned in issue #102 as[org.bytedeco/mkl-platform-redist "2020.3-1.5.4"].

I tried to change this but this didn't change anything..

I am a bit unsure, where should I exactly call the shutdown code and how?

[blueberry]

See here: https://clojure.org/reference/agents

Note that use of Agents starts a pool of non-daemon background threads that will prevent shutdown of the JVM. Use shutdown-agents to terminate these threads and allow shutdown.

Do it at the shutdown of your application, or whenever you feel you don't need agents any more.

Also see https://stackoverflow.com/questions/7173376/leiningen-uberjar-empty-running-time

[erkkikeranen]

Thanks, got a little progress.

I don't think I've intentionally started any agents, so I don't have a feeling that I need to shutdown my agents currently.

I found out something myself with the info you provided so I manage to compile the uberjar, but I don't fully understand it:

If I remove the require to [uncomplicate.neanderthal.native], uberjar compiles (and the SLF4J logging is not printed, i.e. something is not evaluated (unintentionally?) upon requiring?

this is my entry point

(ns cljml.core
  (:require [uncomplicate.neanderthal.core]
            [uncomplicate.neanderthal.native]

            [cljml.benchmark.vectors :as vs]
            [cljml.system.info :as system-info])
  (:gen-class))
;; adding this will make it possible to compile uberjar
;; is it a problem that it's evaluated now? I haven't implemented any useful code yet so I don't know if it cripples anything
;; My code doesn't currently cause the reason to call this, but the requiring of uncomplicate.neanderthal.native causes
(shutdown-agents)

(defn -main
  "I don't do a whole lot ... yet."
  [& args]
  (system-info/print!)) ;; basically a function that prints out some info on screen, so it's a hello world

[blueberry]

But I told you: the native namespace runs an agent during the initialization. Than's why you should shutdown it.

[erkkikeranen]

Thanks, I think can now live with the (shutdown-agents) for now.

But in case the upstream fix can do some magic, that would also be nice.