AOT works on REPL but not in script

When I use (boot (aot :namespace '#{ns1 ns2})) in the REPL this works fine. When I run the same code as a script the AOT does not happen, the sparkling.serialization.Registrator class does not compile, and I get these AOT-related exceptions:

org.apache.spark.SparkException: Failed to register classes with Kryo
java.lang.ClassNotFoundException: sparkling.serialization.Registrator

Here is the code. To reproduce this issue save this to a file called chk.clj. Then chmod 755 chk.clj. Then run ./chk.clj

To see the code work start boot repl and paste this file into it.

#!/usr/bin/env boot
(set-env! 
  :dependencies '[
    [org.apache.spark/spark-core_2.10 "2.0.2"]
    [org.apache.spark/spark-sql_2.10 "2.0.2"]
    [gorillalabs/sparkling "1.2.5"]])

(require '[sparkling.conf :as conf])
(require '[sparkling.core :as spark])
(require '[sparkling.serialization])
(require '[sparkling.destructuring])

(boot (aot :namespace '#{sparkling.serialization 
                         sparkling.destructuring } ))

(defonce sc (-> (conf/spark-conf)
                (conf/app-name "My App")
                (conf/master "local")
                (spark/spark-context)))

(->> (range 100) (spark/parallelize sc) (spark/collect))

Output of boot --version.

#http://boot-clj.com
#Wed Nov 30 01:20:40 PST 2016
BOOT_CLOJURE_NAME=org.clojure/clojure
BOOT_CLOJURE_VERSION=1.7.0
BOOT_VERSION=2.6.0

So I've dug into this a bit and I'm pretty unclear why it works from the REPL. There's clearly something interesting going on.

Interestingly, even when you run the above script in the REPL via copy-paste, there is no AOT compilation happening. Indeed, completely skipping the attempt at AOT has no change in the behavior from either location. This is because the boot aot task only looks for namespaces in the fileset and unless you use the uber task, clojure namespaces from dependencies won't be in the fileset (and thus can't be directly AOT compiled).

I had one suspicion that perhaps the important difference between the REPL and the command line was that the REPL is running inside of a pod. But I tried to pod-ify your spark test code and it didn't seem to make a difference. I've included that attempt below in case that's useful to you.

(set-env!
 :dependencies '[[org.apache.spark/spark-core_2.10 "2.0.2"]
                 [org.apache.spark/spark-sql_2.10 "2.0.2"]
                 [gorillalabs/sparkling "1.2.5"]])

(require '[boot.pod  :as pod]
         '[boot.util  :as util]
         '[sparkling.conf :as conf]
         '[sparkling.core :as spark]
         '[sparkling.serialization]
         '[sparkling.destructuring])

(task-options!
 aot {:namespace '#{sparkling.serialization
                    sparkling.destructuring}})

(deftask spark-test []
  (with-pass-thru fs
    (util/info "NSes %s" (pr-str (fileset-namespaces fs)))
    (let [spark-pod (pod/make-pod (get-env))]
      (pod/with-eval-in spark-pod
        (require '[sparkling.conf :as conf]
                 '[sparkling.core :as spark]
                 '[sparkling.serialization]
                 '[sparkling.destructuring])
        (defonce sc (-> (conf/spark-conf)
                        (conf/app-name "My App")
                        (conf/master "local")
                        (spark/spark-context))))
      (pod/with-eval-in spark-pod
        (->> (range 100)
             (spark/parallelize sc)
             (spark/collect))))))

Also interestingly, I can't actually successfully AOT compile those two nses from sparkling even in the REPL.

(boot (uber) (aot :namespace #{'sparkling.serialization 'sparkling.destructuring}))
Adding uberjar entries...
17/02/15 19:02:39 INFO BlockManagerInfo: Removed broadcast_0_piece0 on 192.168.1.8:33736 in memory (size: 1002.0 B, free: 1955.7 MB)
17/02/15 19:02:39 INFO BlockManagerInfo: Removed broadcast_1_piece0 on 192.168.1.8:33736 in memory (size: 999.0 B, free: 1955.7 MB)

boot.user=> clojure.lang.Compiler$CompilerException: java.lang.NoSuchMethodError: clojure.lang.Compiler$InstanceMethodExpr.<init>(Ljava/lang/String;IILclojure/lang/Symbol;Lclojure/lang/Compiler$Expr;Ljava/lang/String;Lclojure/lang/IPersistentVector;)V, compiling:(NO_SOURCE_PATH:0:0)
            java.lang.NoSuchMethodError: clojure.lang.Compiler$InstanceMethodExpr.<init>(Ljava/lang/String;IILclojure/lang/Symbol;Lclojure/lang/Compiler$Expr;Ljava/lang/String;Lclojure/lang/IPersistentVector;)V

boot-clj / boot

AOT works on REPL but not in script #534