10. Spark 개요 : How Spark works

[tonykang22]

How Spark works

• User application create RDDs, transform them, and run actions.
• This results in a DAG of operators.
• DAG is compiled into stages.
• Each stage is executed as a series of Task.
  • One Task for each Partition.

DAG Scheduler

• General task graphs
• Automatic piplelines functions
• Data locality aware
• Partitioning aware to avoid shuffles

Execution Plan

• Stages are sequences of RDDs, that don't have a Shuffle in between.

Stage Execution

1. Read HDFS split
2. Apply both the maps
3. Start partial reduce
4. Write shuffle data

• Create a task for each Partition in the new RDD.
• Serialize the Task.
  • Read HDFS > Maps > Partial Reduce > Write Shuffle data
• Schedule and ship Tasks to Executors.
• All happens internally. You don't need to do anything.

Executor, Task

• Example
  • Executor(JVM process) with 3 Cores
  • Executor executes 3 Tasks concurrently
  • run parallel

Summary