The examples in this repository are support to the Spark in Action, 2nd edition book by Jean-Georges Perrin and published by Manning. Find out more about the book on Manning's website.
Welcome to Spark in Action, 2nd edition, chapter 3. This chapter is all about the dataframe, understand its role and royalty!
This code is designed to work with Apache Spark v3.1.2.
Each chapter has one or more labs. Labs are examples used for teaching in the book. You are encouraged to take ownership of the code and modify it, experiment with it, hence the use of the term lab.
IngestionSchemaManipulationApp
: ingestion of a CSV file, manipulation of schema structure post-ingestion
For information on running the Java lab, see chapter 3 in Spark in Action, 2nd edition.
You will need:
git
.Clone this project
git clone https://github.com/jgperrin/net.jgp.books.spark.ch03
Go to the lab in the Python directory
Here, we show lab #200.
cd net.jgp.books.spark.ch03/src/main/python/lab200_ingestion_schema_manipulation/
Execute the following spark-submit command to create a jar file to our this application
spark-submit ingestionSchemaManipulationApp.py
You will need:
git
.Clone this project
git clone https://github.com/jgperrin/net.jgp.books.spark.ch03
Go to the directory
cd net.jgp.books.spark.ch03
Package application using sbt command
sbt clean assembly
Run Spark/Scala application using spark-submit command as shown below:
spark-submit --class net.jgp.books.spark.ch03.lab200_ingestion_schema_manipulation.IngestionSchemaManipulationScalaApp target/scala-2.12/SparkInAction2-Chapter03-assembly-1.0.0.jar
Follow me on Twitter to get updates about the book and Apache Spark: @jgperrin. Join the book's community on Facebook or in Manning's community site.