kubebench

The goal of Kubebench is to make it easy to run benchmark jobs on Kubeflow with various system and model settings. Kubebench enables benchmarks by leveraging Kubeflow's capability of managing TFJobs, as well as Argo based workflows.

Quick Start

NOTE: the quick start guide serves as a demo that helps you quickly go through a Kubebench Job. The components installed may not be suitable for production use. Please refer to detailed user guide for proper configuration of Kubebench Jobs.

Prerequisites

Kubernetes >= 1.8
Ksonnet >= 0.11
Kubeflow >= 0.3
- Required modules: argo, tf-operator
For the quick-starter installation, Kubernetes nodes need to support NFS mounting

Installation

Install Dependencies (Kubebench depends on an existing Kubeflow deployment. For details about using Kubeflow, please refer to Kubeflow documentation)

Install Kubebench quick-starter package

In your Ksonnet app root, run the followings:

export KB_VERSION=master
export KB_ENV=default

curl https://raw.githubusercontent.com/kubeflow/kubebench/master/scripts/install_quickstarter.sh | bash

View the Kubebench directory contents

The installer comes with a simple file server that allows you to view the contents of Kubebench directory through browser. You may find details of the file server service through:
```
kubectl get svc kubebench-nfs-file-server-svc -o wide
```
Alternatively, you can also access the deployed NFS service directly. You may find details of the nfs service through:
```
kubectl get svc kubebench-nfs-svc -o wide
```

Run a Kubebench Job

Create a kubebench-job

JOB_NAME="my-benchmark"

ks pkg install kubeflow/kubebench@${KB_VERSION}
ks generate kubebench-job ${JOB_NAME}

ks apply ${KB_ENV} -c ${JOB_NAME}

Track the status of your job

The Kubebench Job will be deployed as an Argo Workflow, you may go to Argo dashboard to track the progress of the job.

Alternatively, you can also use the followings in command line:
```
kubectl get -o yaml workflows ${JOB_NAME}
```

View results

Once the job is finished, you can find the results under the experiment directory in the NFS, the details of the particular experiment is located at /experiments/<EXPERIMENT_UID>. You may also see a csv file at /experiments/report.csv, if you run multiple experiments, the aggregated results will be recorded here.

Cleanups

Delete the kubebench-job
```
ks delete ${KB_ENV} -c ${JOB_NAME}
```

Uninstall quickstarter

curl https://raw.githubusercontent.com/kubeflow/kubebench/master/scripts/uninstall_quickstarter.sh | bash

Other Examples

MPI Job Example

Design Document

For additional information about motivation and design for this project please refer to kubebench_design.md

Development

Ensure you run $ make verify before submitting PRs.

// TODO post detailed development guide.

kubeflow / kubebench

readme