HCADatalab / powderkeg

Live-coding the cluster!
Eclipse Public License 1.0
159 stars 23 forks source link

transducer invoke function modification didn't take affect after (keg/connect! "local") #27

Closed clojurians-org closed 7 years ago

clojurians-org commented 7 years ago

after i connect to local spark, then i modify xtest function(comment portition), it didn't take affect. i need to close and start keg/sc, it will work.

(comment                                                                                                                                                               
  (def data [{:a 3 :b 4 :c 5}                                                                                                                                          
             {:a 33 :b 44 :c 55}                                                                                                                                       
             {:a 3 :b 11 :c 22}])                                                                                                                                      

  (keg/connect! "local")                                                                                                                                               
  (.close keg/*sc*)                                                                                                                                                    
  (defn xtest                                                                                                                                                          
    ([] nil)                                                                                                                                                           
    ([acc x] (if (sequential? x)                                                                                                                                       
               (concat acc x)                                                                                                                                          
               (conj acc x)))                                                                                                                                          
    #_([x] x)                                                                                                                                                          
    ([x] (vec x)) )                                                                                                                                                    

  (as->                                                                                                                                                                
      (keg/rdd data                                                                                                                                                    
               (map #(vector (:a %) %)))                                                                                                                               
      $                                                                                                                                                                
    (keg/by-key $                                                                                                                                                      
                (x/reduce xtest))                                                                                                                                      
    (into [] $)) => [[3 ({:a 3, :b 11, :c 22} {:a 3, :b 4, :c 5})] [33 ({:a 33, :b 44, :c 55})]]                                                                       
  )                                                                                                                                                                    
cgrand commented 7 years ago

On first analysis I think this issue only appears in local mode (to confirm) and is tied to the naming of the classes emitted by the defn. I think replacing (defn xtest (....)) by (def xtest (fn xtest (...))) should provide a workaround.

cgrand commented 7 years ago

Fixed, please confirm on your side.