Demo trame - Githubissues

activeviam / par-student-spark-atoti

Project with students from CentraleSupelec to explore Spark API in order to power atoti with Spark

1 stars 0 forks source link

Demo trame #21

Open arnaudframmery opened 2 years ago

arnaudframmery commented 2 years ago

On premise (small dataset) :

load a csv as a dataframe (CsvReader.read)
Show columns and types (Discovery.discoverDataframe)
Show a dataframe slice (ListQuery.list)
Select some rows with a condition (AggregateQuery.aggregate)
aggregate data (AggregateQuery.aggregate)

On cluster (big dataset) :

Show columns and types to check the connection (Discovery.discoverDataframe)
Perform a vector computation ()
Increase the worker number on the cluster to accelerate the computation (DatabricksManager.resize)
Perform a vector computation again ()
Reset the worker number (DatabricksManager.resize)