search

josephmachado / efficient_data_processing_spark

Code for "Efficient Data Processing in Spark" Course
https://josephmachado.podia.com/efficient-data-processing-in-spark
243 stars 54 forks source link
apache-spark data-engineering data-pipeline minio pyspark pyspark-notebook

readme

Code for my Efficient Data Processing in Spark course.

Efficient Data Processing in Spark

Repository for examples and exercises from the "Efficient Data Processing in Spark" course (under data-processing-spark). The capstone project is also present in this repository (under capstone/rainforest).

Setup

In order to run the project you'll need to install the following:

  1. git version >= 2.37.1
  2. Docker version >= 20.10.17 and Docker compose v2 version >= v2.10.2.

Windows users: please setup WSL and a local Ubuntu Virtual machine following the instructions here. Install the above prerequisites on your ubuntu terminal; if you have trouble installing docker, follow the steps here (only Step 1 is necessary). Please install the make command with sudo apt install make -y (if its not already present).

All the commands shown below are to be run via the terminal (use the Ubuntu terminal for WSL users). The make commands in this book should be run in the efficient_data_processing_spark folder. We will use docker to set up our containers. Clone and move into the lab repository, as shown below.

Note: If you are using mac M1 or later, please replace the "FROM deltaio/delta-docker:latest" in data-processing-spark/1-lab-setup/containers/spark/Dockerfile with "FROM deltaio/delta-docker:latest_arm64"

git clone https://github.com/josephmachado/efficient_data_processing_spark.git
cd efficient_data_processing_spark
# Start docker containers and create data for exercises and capstone project
# If you are using mac M1, please replace the "FROM deltaio/delta-docker:latest" 
# in data-processing-spark/1-lab-setup/containers/spark/Dockerfile
# with "FROM deltaio/delta-docker:latest_arm64"
make restart && make setup

Create aliases for long commands with a Makefile

Makefile lets you define shortcuts for commands that you might want to run, E.g., in our Makefile, we set the alias spark-sql for the command that opens us a spark sql session.

We have some helpful make commands for working with our systems. Shown below are the make commands and their definitions

  1. make restart: Stops running docker containers(if any) and starts new containers for our data infra.
  2. make setup: Generates data and loads them into tables and starts spark histroy server where we can see logs/Spark UI for already completed jobs.
  3. make spark-sql: Open a spark sql session; Use exit to quit the cli. This is where you will type your SQL queries.
  4. make cr: To run our pyspark code by pasting the relative path of exercise/example problems under data-processing-spark folder. See example image shown below.
  5. make rainforest: Runs our rainforest capstone project, the entry point for this code is here

This is how you run pyspark exercise files: make cr example

You can see the commands in this Makefile. If your terminal does not support make commands, please use the commands in the Makefile directly. All the commands in this book assume that you have the docker containers running.

You can test and run the capstone project as:

make pytest # to run all test cases
make ci # to run linting, formatting, and type checks
make rainforest # to run our ETL and create the final reports

Run a Jupyter notebook

Use the following command to start a jupyter server:

make notebook

You will see a link displayed with the format http://127.0.0.1:3000/?token=your-token, click it to open the jupyter notebook on your browser. You can use local jupyter notebook sample to get started.

You can stop the jupyter server with ctrl + c.

Infrastructure

We have three major services that run together, they are

  1. Postgres database: We use a postgres data base to simulate an upstream application database for our rainforest capstone project.
  2. Spark cluster: We create a spark cluster with a master and 2 workers which is where the data is processed. The spark cluster also includes a history server, which displays the logs and resource utilization (Spark UI) for completed/failed spark applications.
  3. Minio: Minio is an open source software that has fully compatable API with AWS S3 cloud storage system. We use minio to replicate S3 locally.

Infra

All our Spark images are built from the official Spark Delta image, and have the necessary modules installed. You can find the docker files defined here