search

yu-iskw / spark-streaming-with-google-cloud-example

an example of integrating Spark Streaming with Google Pub/Sub and Google Datastore
Apache License 2.0
17 stars 4 forks source link

readme

Example to Integrate Spark Streaming with Google Cloud at Scale

This is an example to integrate Spark Streaming with Google Cloud products. The streaming application pulls messages from Google Pub/Sub directly without Kafka, using custom receivers. When the streaming application is running, it can get entities from Google Datastore and put ones to Datastore.

What I want to show here is that we can be free from managing some big data products, such as Kafka and Cassandra. We data scientist can focus on implementing logics with Spark Streaming.

Spark Streaming with Google Cloud

Requirments

How to Run

  1. Create Google Pub/Sub topic/subscription 
    • gcloud beta pubsub topics create $TOPIC_NAME`
  2. Create a Spark cluster on Google Dataproc with make create-dataproc-cluster 
    • gcloud dataproc clusters create $CLUSTER_NAME \
--zone="us-central1-a" \
--image-version=1.1 \
--master-machine-type="n1-standard-4" \
--num-workers=2 \
--worker-machine-type="n1-standard-4" \
--scopes=https://www.googleapis.com/auth/pubsub,https://www.googleapis.com/auth/datastore,https://www.googleapis.com/auth/bigquery,https://www.googleapis.com/auth/devstorage.read_write,https://www.googleapis.com/auth/logging.write
  3. Create a JAR file of the project 
    • make assembly
  4. Submit a Spark Streaming job to Google Dataproc 
    • ./bin/submit-streaming-job.sh $GOOGLE_PROJECT_ID $PUBSUB_TOPIC
  5. Send Pub/Sub messages 
    • ./bin/send-pubsub-messages.sh $GOOGLE_PROJECT_ID $PUBSUB_TOPIC

Please make sure to delete a Pub/Sub topic and a Dataproc cluster.

Appendix: Price Estimation for Google Pub/Sub and Google Datastore

As you know, Google Cloud Platform Pricing Calculator allows us to estimate how much cost we need to run systems. I would like to show an example cost to use Google Pug/Sub and Google Datastore. Of course, we need extras cost to use Spark Streaming on Google Dataproc.

According to the calculator, the estimated amount is just $22.20 per 1 month. From my perspective, it is much more reasonable than the cost of having an own Kafka cluster and a Cassandra cluster.

The cost was estimated with the following condition:

Appendix: How to Avoid Conflicts on protobuf-java Versions

Apache Spark 2.0.2 uses protobuf-java 2.5. Meanwhile, we need to protobuf-java 3.0 or later for Google Datastore. We can't avoid this conflict on protobuf-java event with any eviction.

In stead of any eviction, sbt-assembly offers an excellent feature to resolve this issue. Please take a look ./build.sbt in the repository.

assemblyShadeRules in assembly := Seq(
  ShadeRule.rename("com.google.protobuf.*" -> "shadedproto.@1").inAll
)

Reference