Make framework configurable to run hierarchical suite of workloads

ganesank-git commented 3 years ago

The user should be able to provide a folder, subfolder or a test name to run.

Example: ./run.sh -t dataplane ./run.sh -t dataplane/ai/classification/resnet50 ./run.sh -t dataplane/ai

ndn97 commented 3 years ago

added -c option which takes folder/sub-folder that contains test names,

implementing folders in testnames (-t) option will not have clear identifcation of path/testnames, this change doesnt breaks old way of running.. some tests are given as follows

to run list of tests in custom path

ndn@issubuntu01:~/k-bench$ ./run.sh -c ./dataplane -t "custom_ai ai" Running test custom_ai found in ./dataplane and results redirected to "./results_run_11-Apr-2021-09-54-57-pm/custom_ai" Starting benchmark, writing logs to results_run_11-Apr-2021-09-54-57-pm/custom_ai/kbench.log...

if custom path is not provided , default custom path ./config dir is taken (original behavior)

ndn@issubuntu01:~/k-bench$ ./run.sh Usage: ./run.sh -r [-t -c -o ] Example: ./run.sh -r "kbench-run-on-XYZ-cluster" -t "cp_heavy16,dp_netperf_internode,dp_fio" -o "./"

Valid test names:

all || all_control_plane || all_data_plane || Since no tests specified, I am running the default workload: config/default Running test default found in ./config/ and results redirected to "./results_run_11-Apr-2021-09-55-55-pm/default" Starting benchmark, writing logs to results_run_11-Apr-2021-09-55-55-pm/default/kbench.log... Running workload, please check kbench log for details...

original behavior of the tests,works

ndn@issubuntu01:~/k-bench$ ./run.sh -t dp_redis Running test dp_redis found in ./config/ and results redirected to "./results_run_11-Apr-2021-09-57-09-pm/dp_redis" Starting benchmark, writing logs to results_run_11-Apr-2021-09-57-09-pm/dp_redis/kbench.log...

original behavior, run all control plane tests

ndn@issubuntu01:~/k-bench$ ./run.sh -t all_control_plane Running test default found in ./config/ and results redirected to "./results_run_11-Apr-2021-09-58-16-pm/default" Starting benchmark, writing logs to results_run_11-Apr-2021-09-58-16-pm/default/kbench.log...

original behavior, run all control plane tests

ndn@issubuntu01:~/k-bench$ ./run.sh -t all_data_plane Running test dp_fio found in ./config/ and results redirected to "./results_run_11-Apr-2021-09-59-50-pm/dp_fio" persistentvolumeclaim/fio-block-pvc created

ganesank-git commented 3 years ago

Hi Nagesh, thanks for the change. I understand why you think we should have two separate options for test name and config directory. But, in the case of an ask where the user has multiple tests he wants to run in a particular directory, the user still have to use the top level directory as the test name. My suggestion would be to remove test names altogether and just use one directory name. If the directory has a config.json, it will be run. If not, we recursively look at the subdirectories for config.json files and execute them one by one.

vmware-tanzu / k-bench