search

HCADatalab / powderkeg

Live-coding the cluster!
Eclipse Public License 1.0
159 stars 23 forks source link

can't serialize field alter the var to unbound. #30

Open clojurians-org opened 7 years ago

clojurians-org commented 7 years ago

when i run the following code, i get var a, then i run "into" on it. it say: Warning: can't serialize #'bolome.ods.d_bolome_product_category/a it won't be sent to workers then the var a turn into unbound var.

  (def a                                                                                                                                                  
    (->> (keg/join (keg/by-key [[:a 3] [:b 4]]) :or nil (keg/by-key [[:a 5] [:b 7]]) :or nil)                                                             
         .cache))                                                                                                                                         

  (into [] a)
cgrand commented 7 years ago

What's your setup? I tested with spark 2.10 in local and standalone cluster and it worked, I get [[:b [4 7]] [:a [3 5]]].

clojurians-org commented 7 years ago

i run it in standalone mode in spark 2.10.

it's not about run without right result. rather than, after you got result, the var a turn into the unbound

  (def a => #object[clojure.lang.Var$Unbound 0x31ee0bf8 "Unbound: #'bolome.ods.d_bolome_product_category/a"]                           
    (->> (keg/join (keg/by-key [[:a 3] [:b 4]]) :or nil (keg/by-key [[:a 5] [:b 7]]) :or nil)                                          
         .cache))