search

nerdammer / spark-hbase-connector

Connect Spark to HBase for reading and writing data with ease
Apache License 2.0
297 stars 107 forks source link

Spark Streaming context #6

Closed tarekabouzeid closed 9 years ago

tarekabouzeid commented 9 years ago

i am using spark streaming context to take flume stream and process it using spark , then want to save the Dstream into Hbase , is it possible to do so using your library ? Thanks in advance

nicolaferraro commented 9 years ago

I will provide this functionality with next release.

tarekabouzeid commented 9 years ago

thanks so much

nicolaferraro commented 9 years ago

Fixed with pull request https://github.com/nerdammer/spark-hbase-connector/pull/7.

Version 0.9.4 released in Maven central.

nicolaferraro commented 9 years ago

Starting with version 1.0.0, the streaming code looks like:

stream.foreachRDD(rdd =>
    rdd.toHBaseTable("table")
      .inColumnFamily("cf")
      .toColumns("col1")
      .save()
    )

This allowed to remove the direct dependency from Spark-Streaming and having a cleaner code.