11. Spark 개요 : Spark Components / Libraries

[tonykang22]

Spark Components / Libraries

MapReduce Limitations

• MapReduce use cases showed two major limitations
  • difficulty of programming directly in MR
  • performance bottlenecks, or batch not fitting the use cases
• MR doesn't compose well for large applications.
• Therefore, people built specialized systems as workarounds.

Spark's Approach

• Instead of various specialized systems, generalize MapReduce to support new apps within same engine.
• Two additions
  • Fast data sharing
  • General task DAGs
• Benefits of Unification
  • More efficient for engine
  • Much simpler for users

Spark's Code Size

Meaning...

• No copying or ETL of data between systems.
• Combine processing types in one program.

Combining Processing Types

• Example
  • Spark SQL + MLlib + Spark Streaming ... : in one program

// Spark SQL
df.createOrReplaceTempView("historic_tweets")
val points = spark.sql("select latitude, longitude from historic_tweets")

// MLlib
val kmeans = new KMeans().setK(10).setSeed(1L)
val model = kmeans.fit(points)

// Spark Streaming...
TwitterUtils.createStream(ssc, None)
                    .map(t => (model.predict(t.location), 1))
                    .reduceByKeyAndWindow(_+_, Seconds(60))

Spark Components (Libraries)

• Benefits of a Unified Platform
  • No copying or ETL of data between systems
  • Combine processing types in one program
  • Code reuse
  • One system to learn
  • One system to maintain