Open asimjalis opened 7 years ago
So I've dug into this a bit and I'm pretty unclear why it works from the REPL. There's clearly something interesting going on.
Interestingly, even when you run the above script in the REPL via copy-paste, there is no AOT compilation happening. Indeed, completely skipping the attempt at AOT has no change in the behavior from either location. This is because the boot aot
task only looks for namespaces in the fileset and unless you use the uber
task, clojure namespaces from dependencies won't be in the fileset (and thus can't be directly AOT compiled).
I had one suspicion that perhaps the important difference between the REPL and the command line was that the REPL is running inside of a pod. But I tried to pod-ify your spark test code and it didn't seem to make a difference. I've included that attempt below in case that's useful to you.
(set-env!
:dependencies '[[org.apache.spark/spark-core_2.10 "2.0.2"]
[org.apache.spark/spark-sql_2.10 "2.0.2"]
[gorillalabs/sparkling "1.2.5"]])
(require '[boot.pod :as pod]
'[boot.util :as util]
'[sparkling.conf :as conf]
'[sparkling.core :as spark]
'[sparkling.serialization]
'[sparkling.destructuring])
(task-options!
aot {:namespace '#{sparkling.serialization
sparkling.destructuring}})
(deftask spark-test []
(with-pass-thru fs
(util/info "NSes %s" (pr-str (fileset-namespaces fs)))
(let [spark-pod (pod/make-pod (get-env))]
(pod/with-eval-in spark-pod
(require '[sparkling.conf :as conf]
'[sparkling.core :as spark]
'[sparkling.serialization]
'[sparkling.destructuring])
(defonce sc (-> (conf/spark-conf)
(conf/app-name "My App")
(conf/master "local")
(spark/spark-context))))
(pod/with-eval-in spark-pod
(->> (range 100)
(spark/parallelize sc)
(spark/collect))))))
Also interestingly, I can't actually successfully AOT compile those two nses from sparkling even in the REPL.
(boot (uber) (aot :namespace #{'sparkling.serialization 'sparkling.destructuring}))
Adding uberjar entries...
17/02/15 19:02:39 INFO BlockManagerInfo: Removed broadcast_0_piece0 on 192.168.1.8:33736 in memory (size: 1002.0 B, free: 1955.7 MB)
17/02/15 19:02:39 INFO BlockManagerInfo: Removed broadcast_1_piece0 on 192.168.1.8:33736 in memory (size: 999.0 B, free: 1955.7 MB)
boot.user=> clojure.lang.Compiler$CompilerException: java.lang.NoSuchMethodError: clojure.lang.Compiler$InstanceMethodExpr.<init>(Ljava/lang/String;IILclojure/lang/Symbol;Lclojure/lang/Compiler$Expr;Ljava/lang/String;Lclojure/lang/IPersistentVector;)V, compiling:(NO_SOURCE_PATH:0:0)
java.lang.NoSuchMethodError: clojure.lang.Compiler$InstanceMethodExpr.<init>(Ljava/lang/String;IILclojure/lang/Symbol;Lclojure/lang/Compiler$Expr;Ljava/lang/String;Lclojure/lang/IPersistentVector;)V
When I use
(boot (aot :namespace '#{ns1 ns2}))
in the REPL this works fine. When I run the same code as a script the AOT does not happen, thesparkling.serialization.Registrator
class does not compile, and I get these AOT-related exceptions:Here is the code. To reproduce this issue save this to a file called
chk.clj
. Thenchmod 755 chk.clj
. Then run./chk.clj
To see the code work start
boot repl
and paste this file into it.Output of
boot --version
.