HCADatalab / powderkeg

Live-coding the cluster!
Eclipse Public License 1.0
159 stars 23 forks source link

how to implement doseq in do-rdd? #36

Closed clojurians-org closed 7 years ago

clojurians-org commented 7 years ago

i want to write rdd to jdbc database. so i need to do it in foreachPartition function. i seek and find the do-rdd function, it seems to be the key, but doseq is not a transducer, and the do-rdd example material is missing. i have no ideas how to use it do things like following.

rdd.foreachPartition { it =>
  val conn = sql.DriverManager.getConnection("jdbc:mysql://mysql.example.com/?user=batman&password=alfred")
  val del = conn.prepareStatement("DELETE FROM BOOKS WHERE BOOK_TITLE = ?")
  for (bookTitle <- it) {
    del.setString(1, bookTitle)
    del.executeUpdate
  }
}
cgrand commented 7 years ago

Something like:

(keg/do-rdd rdd
  [conn ... del ...]
  (map #(doto del (.setString 1 %) .executeUpdate)))

I think a slightly better API could be found.

clojurians-org commented 7 years ago

thanks. i works, i just forget to add the [] parameter list.

it's very easy to forget to add [] parameter when no parameter needed, then nothing happen.

  (keg/do-rdd (range 1 10)                                                                                                   
              #_[]                                                                                                           
              (map prn))