09. Spark 개요 : RDD - Rault Tolerance

[tonykang22]

RDD - Fault Tolerance

Lineage

• RDDs track the series of transformations used to build them to recompute lost data.

val msgs = sc.textFile("hdfs://...").filter(s => s.startsWith("ERROR"))
                                                        .map(s => s.split("\t"(2))

1. MappedRDD
   • func = split(...)
2. FilteredRDD
   • func = startsWith(...)
3. HadoopRDD
   • path = hdfs://...

Fault Recovery Test

Example

• Cache > Iterate Actions > Crash(Fault) > Recompute

• 1st Action : Actually caching stage
• 6th Action : Only damaged cache data is lineaged.
  • But since it is not caching all of the data, it is faster than 1st Action.