01. Spark 개요

[tonykang22]

Spark 개요

Apache Spark

• Unified Engine : Support end-to-end applications
• High-level APIs : Easy to use, rich optimizations
• Integrate Broadly : Stroage systems, libraries, etc

Apache Spark란?

• Apache Spark는 대용량 데이터 프로세싱을 위한 빠르고 범용적인 인메모리 기반 클러스터 컴퓨팅 엔진
• 분산 메모리 기반의 빠른 분산 병렬 처리
• 배치, 대화형 쿼리, 스트리밍, 머신러닝과 같은 다양한 작업 타입을 지원하는 범용 엔진으로 Apache Hadoop과 호환
• Scala, Java, Python, R 기반 High-level APIs 제공

특징

• In-Memory 컴퓨팅
  • Disk 기반도 가능
• RDD(Resilient Distributed Dataset) 데이터 모델
• 다양한 개발 언어 지원
  • Scala, Java, Python, R, SQL
• Rich APIs 제공
  • 80여개 이상, 2 ~ 10x Less Code
• General execution graphs -> DAG (Directed Acyclic Graph) -> Multiple stages of map & reduce
• Hadoop과 유연한 관계
  • HDFS, HBase, YARN, others
• 빠른 데이터 processing
  • In-Memory cached RRD, Up to 100x faster
• 대화형 질의를 위한 Interactive shell
  • Scala, Python, R Interpreter
• 실시간(Real-time) Stream Processing
  • vs. MapReduce for stored Data
• 배치, SQL 쿼리, 스트리밍, 머신러닝과 같은 다양한 작업을 하나의 애플리케이션에서 하나의 워크플로우로 결합 가능
• Both fast to write and run

RDD(Resilient Distributed Dataset)

• Dataset
  • 메모리나 디스크에 분산 저장된 변경 불가능한 데이터 객체들의 모음
• Distributed
  • RDD에 있는 데이터는 클러스터에 자동 분배 및 병렬 연산 수행
• Resilient
  • 클러스터의 한 노드가 실패하더라도 다른 노드가 작업 처리
  • RDD Lineage, Automatically rebuilt on failure
• Immutable
  • RDD는 수정이 불가하다.
  • 변경을 통해 새로운 RDD를 생성 가능
• Operation APIs
  • Transformations
  • 데이터 변형 : map, filter, groupBy, join
  • Actions
  • 결과 연산 리턴/저장 : count, collect, save
• Lazy Evaluation
  • All transformations
  • Action 실행 때까지
• Controllable Persistence
  • Cache in RAM/Disk 가능
  • 반복 연산에 유리

RDD : 생성 > 변형 > 연산

[tonykang22]

Spark 개요 (지원 언어, Interactive Shell)

Spark Language Support

• Standalone Programs
  • Scala, Java, Python, R
• Interactive Shells
  • Scala, Python, R
• Performance
  • Scala/Java are faster due to static typing
  • But Python is often fine

val lines = sc.textFile(/logs/*.log)
lines.filter(x => x.contains("ERROR")).count()

JavaRDD<String> lines = sc.textFile("/logs/*.log");
lines.filter(x -> x.contains("ERROR")).count();

lines = sc.textFile(/logs/*.log)
lines.filter(lambda x: "ERROR" in x).count()

Interactive Shell

Interactive Analysis

• The fastest way to learn Spark
• Available in Python, Scala, R
• Runs as an application on an existing Spark Cluster
  • or can run LOCALLY

[tonykang22]

03. Spark 개요 (Web Notebook, Zepplin/Jupyter/RStudio)

Web Notebook

• Web-based Notebook with Spark
  • Apache Zepplin
  • Jupyter Notebook
  • RStudio Server

[tonykang22]

04. Spark 개요 (Web UI, Driver/Cluster Manager)

Administrative Web UIs

• Driver (Spark Application)
• History Server
• Cluster Manager (Cluster Resource Manager)

Driver (Spark Application) Web UI

• Application's dashboard Web UI
• SparkContext launches a Web UI
• A list of scheduler stages and tasks
• A summary of RDD sizes and memory usage
• Environmental information
• Information about the running executors
• default : http://<Driver Node>:4040

History Server Web UI

• Application's Web UI is only availiable for the duration of the application
• History Server shows application's Web UI after the fact
• default : http://<History Server>:18080

Spark Standalone Web UI

• Resource Manager in Spark
• A simple cluster manager included with Spark that makes it easy to set up a cluster
• Only Spark Application
• default : http://<Standalone Master>:8080

Hadoop YARN Web UI

• Resource Manager in Hadoop 2.0
• Yet Another Resource Negotiator
• MapReduce(MR)
• Other Data Processing Frameworks
  • Hive, HBase, Presto, Flink, others
• default : http://<ResouceManager>:8088

[tonykang22]

05. Spark 개요 (Spark vs. MapReduce)

Hadoop - on List : Limitations

• Slow due to replication, serialization, and disk IO
• Inefficient for
  • Interactive algorithms
  • Machine Learning, Graphs & Network Analysis
  • Interactive Data Mining
  • R, Excel, Ad-hoc Reporting, Searching

Spark - In Memory + DAG : Solutions?

• Load Data into Memory
• 10 - 100x faster than disk
• Not tied to 2 stage Map/Reduce paradigm
  • Extract a working set (ex. flatMap > filter > map > reduceByKey > ~ )
  • Cache it (cache, persist)
  • Query it repeatedly

Spark vs Hadoop : SPEED

• Spark won the 2014 Daytona Gray Sort 100TB Benchmark
• 3x faster using 10x fewer machines ON DISK

• Spark won the 2016 CloudSort Benchmark (Cost to sort 100TB of data)
• 3x more cost-efficient than the previous world record

• Iterative algorithm used in machine learning
  • in Memory

Spark vs. Hadoop : Ease of Use

• 2 - 5x Less code

• Low level API vs. High level API