HCADatalab / powderkeg

Live-coding the cluster!
Eclipse Public License 1.0
159 stars 23 forks source link

How to call a side-effect function for each element of a rdd (foreach) #43

Closed dpom closed 6 years ago

dpom commented 6 years ago

I defined an rdd with: (def occs (-> orders (keg/rdd (map read-string) (map #(comb/combinations % 2)) (mapcat identity) (map sort)) (keg/by-key :key identity :pre x/count :post (x/reduce +))))

How I call a side-effect function (write to a db) for each element of the occs (an equivalent of spark's foreach)?

cgrand commented 6 years ago

See do-rdd. The binding vector is for resources management (e.g. db connection).

dpom commented 6 years ago

Thanks, that's exactly what I needed.