support multiple developer groups, update README

[qiliRedHat]

Changes in this PR

• https://issues.redhat.com/browse/OCPQE-13993 support multiple dev groups
• Enhanced the -t parameter parsing
• Updated readme
• Download dittybopper to the common util folder for reuse
• Minor fixes about logging dir

[qiliRedHat]

@svetsa-rh @paigerube14 PTAL Test logs

2023-02-14 07:58:08,418 - INFO - Will run the following groups concurrently [{'name': 'admin-1', 'persona': 'admin', 'users': 1, 'loops': 'forever', 'trigger': 600, 'jitter': 60, 'interval': 10, 'tasks': ['func check_operators', 'oc get project -l purpose=reliability', 'func check_nodes', 'kubectl get pods -A -o wide | egrep -v "Completed|Running"']}, {'name': 'developer-1', 'persona': 'developer', 'user_start': 0, 'user_end': 15, 'loops': 'forever', 'trigger': 60, 'jitter': 300, 'interval': 10, 'tasks': ['func delete_all_projects', 'func new_project 2', 'func check_all_projects', 'func new_app 2', 'func load_app 2 10', 'func build 1', 'func scale_up 2', 'func scale_down 1', 'func check_pods 2', 'func delete_project 2']}, {'name': 'developer-2', 'persona': 'developer', 'user_start': 15, 'user_end': 16, 'loops': 1, 'tasks': ['func new_project 2', 'func new_app 2']}]

The namespaces's objects long running in group developer-2

# oc get po -n testuser-15-1
NAME                                  READY   STATUS      RESTARTS   AGE
cakephp-mysql-persistent-1-build      0/1     Completed   0          17m
cakephp-mysql-persistent-1-cv8pj      1/1     Running     0          11m
cakephp-mysql-persistent-1-deploy     0/1     Completed   0          11m
cakephp-mysql-persistent-1-hook-pre   0/1     Completed   0          11m
mysql-1-584rp                         1/1     Running     0          16m
mysql-1-deploy                        0/1     Completed   0          17m
# oc get po -n testuser-15-0
NAME                                  READY   STATUS      RESTARTS   AGE
cakephp-mysql-persistent-1-4ft7r      1/1     Running     0          18m
cakephp-mysql-persistent-1-build      0/1     Completed   0          19m
cakephp-mysql-persistent-1-deploy     0/1     Completed   0          18m
cakephp-mysql-persistent-1-hook-pre   0/1     Completed   0          18m
mysql-1-deploy                        0/1     Completed   0          19m
mysql-1-pnz6c                         1/1     Running     0          19m

[qiliRedHat]

@svetsa-rh @paigerube14 Please help to review.

[svetsa-rh]

/lgtm

[svetsa-rh]

@qiliRedHat The code changes look good to me. Is there a passing run though? I see latest run has some failures reported, probably not related to these code changes though.

[qiliRedHat]

@svetsa-rh yes, the link you pasted is the latest test run with the new changed code. The failures are due the test case itself, like Internal error occurred: etcdserver: request timed out - I put a very big load the the cluster so etcd had problem ,that's not related to the reliability tool it self.

The following output verified the code change works well.

Group developer-1 will run tasks with 15 users for forever loops. Adding a jitter of 300s before group test started, wait 60s between loops, wait 10s between tasks.

Group developer-2 will run tasks with 1 users for 1 loops. Adding a jitter of 0s before group test started, wait 0s between loops, wait 60s between tasks.

Group admin-1 will run tasks with 1 users for forever loops. Adding a jitter of 60s before group test started, wait 600s between loops, wait 10s between tasks.

And the long running Group developer-2 with testsuser-15 works well.

% oc get po -n testuser-15-0
NAME                                  READY   STATUS      RESTARTS   AGE
cakephp-mysql-persistent-1-build      0/1     Completed   0          17h
cakephp-mysql-persistent-1-deploy     0/1     Completed   0          17h
cakephp-mysql-persistent-1-gczqf      1/1     Running     0          17h
cakephp-mysql-persistent-1-hook-pre   0/1     Completed   0          17h
mysql-1-cgd5n                         1/1     Running     0          17h
mysql-1-deploy                        0/1     Completed   0          17h

% oc get po -n testuser-15-1
NAME                                    READY   STATUS      RESTARTS   AGE
nodejs-postgresql-persistent-1-build    0/1     Completed   0          17h
nodejs-postgresql-persistent-1-deploy   0/1     Completed   0          17h
nodejs-postgresql-persistent-1-s6phl    1/1     Running     0          17h
postgresql-1-6nlnd                      1/1     Running     0          17h
postgresql-1-deploy                     0/1     Completed   0          17h