Support for cloud Apache Spark Clusters

GoogleCloudPlatform / PerfKitBenchmarker

PerfKit Benchmarker (PKB) contains a set of benchmarks to measure and compare cloud offerings. The benchmarks use default settings to reflect what most users will see. PerfKit Benchmarker is licensed under the Apache 2 license terms. Please make sure to read, understand and agree to the terms of the LICENSE and CONTRIBUTING files before proceeding.

https://googlecloudplatform.github.io/PerfKitBenchmarker/

Apache License 2.0

1.89k stars 480 forks source link

Support for cloud Apache Spark Clusters #1010

Open hildrum opened 8 years ago

hildrum commented 8 years ago

[x] Support AWS's EMR Service for the Apache Spark benchmark.
[ ] Support PKB-created Apache Spark Clusters
[x] Pass arguments to SubmitJob
[x] Get output from SubmitJob
[ ] Support setting Spark Cluster properties

voellm commented 8 years ago

This is cool to see. Congrats.

voellm commented 7 years ago

What is the status of this?

hildrum commented 7 years ago

The three checked-off items are done. Pull request 1104 adds support for hadoop jobs and uses PKB-created hadoop clusters. It also reworks the hadoop_terasort_benchmark to use the framework.

The next step is to add a PKB-installed spark, and the support for setting cluster properties.

It'd be super-nice to add support for HDInsights (currently just AWS and GCP are supported), but I don't know if I'm going to have time.

mateusz-blaszkowski commented 6 years ago

Hi,

Is there any progress on 'PKB-installed spark' item?

Regards, Mateusz