EC528-Fall-2023 / Kata-Containers-for-SPARK

Apache License 2.0
7 stars 1 forks source link

Create an Spark Experiment application #11

Closed jwz16 closed 1 year ago

jwz16 commented 1 year ago

DEPRECATED

A Spark application is required as workloads while performing benchmark.

This is the blocker of benchmark suite.

Since this Spark Experiment application will be used for benchmark, it at least needs the following features:

  1. A source input generator, it should keeping generating random texts until some constraints are reached, e.g. experiment time or max number of text lines.
  2. Allow configuring experiment parameters, e.g. text line size, experiment time, contain malicious codes or not etc.
  3. Must contain two kinds of workloads:
    1. Normal Spark workloads. (e.g. word count)
    2. Malicious Spark workloads which contains both the normal workloads and the security attack codes.
  4. Must output the start and the end timestamps for each text line to the sink (either a file sink or database sink), so that we can measure the latency and throughput after the experiment is completed.
jwz16 commented 1 year ago

Don't think we need this type of experiment application anymore since we are evaluation the e2e performance of the YARN cluster. We can just use a simple example Spark application instead.