Spark Basics

[zjplab]

Why Scala? Why Spark?

• Normally, Data science and analytics is done “in the small”, in R/Python/MATLAB, etc
• If your dataset ever gets too large to fit into memory, these languages/frameworks won’t allow you to scale. You’ve to re-implement everything in some other language or system.
• By using a language like Scala, it's easier to scale your small problem to the large with Spark, whose API is almost 1-to-1 with Scala's collections. That is, by working in Scala, in a functional style, you can quickly scale your problem from one node to tens, hundreds, or even thousands by leveraging Spark, successful and performant large-scale data processing framework which looks a and feels a lot like Scala Collections!

Spark is more:

• More expressive. APIs modeled after Scala collections. Look like functional lists! Richer, more composable operations possible than in MapReduce.
• Performant. Not only performant in terms of running time… But also in terms of developer productivity! Interactive!
• Good for data science Not just because of performance, but because it enables iteration, which is required by most algorithms in a data scientist’s toolbox.

[zjplab]

Data-Parallel to Distributed Data-Parallel

Distributed Data-Parallelism:

• Split the data over several nodes.
• Nodes independently operate on the data shards in parallel.
• Combine when done ( if necessary).

Data-Parallel to Distributed Data-Parallel

[zjplab]

Latency

• Data parallelism in a distributed setting .
• Distributed collections abstraction from Apache Spark as an implementation of this paradigm.

Distribution

Distribution introduces important concerns beyond what we had to worry about when dealing with parallelism in the shared memory case:

• Partial failure: crash failures of a subset of the machines involved in a distributed computation .
• Latency: certain operations have a much higher latency than other operations due to network communication.

Latency cannot be masked completely; it will be an important aspect that also impacts the programming model.

Latency