search

openshift / svt

Apache License 2.0
123 stars 105 forks source link

Desch node utilization with kube burner #767

Closed paigerube14 closed 1 year ago

paigerube14 commented 1 year ago

Want to add a version of the NodeUtilizaiton descheduler test case taht uses kube burner. I have altered the test slightly to use the cakephp template to not duplciate more yaml files

I have also created a separate script for the large version because I need more nodes and deschedule only a few of them 

% ./node_mem_utilization_large.sh 
++ set +ex
wanted profiles LifecycleAndUtilization
All profiles were properly set for descheduler
Prepare worker nodes
counter 0
node/p-***-worker-b-8kgpt.c.openshift-qe.internal already cordoned
node/p-***-worker-b-ntmc6.c.openshift-qe.internal already cordoned
node/p-***-worker-c-jjtc2.c.openshift-qe.internal already cordoned
======Use kube-burner to load the cluster with test objects - mem-util-desched======
###############################################
Workload: custom
Workload template: /Users/prubenda/PycharmProjects/svt/perfscale_regression_ci/scripts/scalability/descheduler/../../../kubeburner-object-templates/openshift-template/cakephp/cakephp-mysql-persistent.yaml
Metrics profile: metrics-profiles/metrics.yaml
Alerts profile: 
QPS: 20
Burst: 20
UUID: FE0431D7-8548-459C-BFAE-E8CA3C5CCA7F
Job iterations: 110
###############################################
Tue Sep  5 21:44:54 UTC 2023 Indexing enabled, using metrics from metrics-profiles/metrics.yaml
~/PycharmProjects/svt/perfscale_regression_ci/kubeburner-object-templates/openshift-template/cakephp ~/PycharmProjects/svt/perfscale_regression_ci/scripts/scalability/descheduler/e2e-benchmarking/workloads/kube-burner
INFO[2023-09-05 17:44:55] 🔥 Starting kube-burner (0.17.3@c38fe7eb37c62686b68e2b64bdc8311a4d73d8f1) with UUID FE0431D7-8548-459C-BFAE-E8CA3C5CCA7F 
INFO[2023-09-05 17:44:55] 📈 Creating measurement factory               
INFO[2023-09-05 17:44:55] Preparing create job: mem-util-desched       
INFO[2023-09-05 17:44:55] Job mem-util-desched: 110 iterations with 1 Secret replicas 
INFO[2023-09-05 17:44:55] Job mem-util-desched: 110 iterations with 1 ImageStream replicas 
INFO[2023-09-05 17:44:55] Job mem-util-desched: 110 iterations with 1 BuildConfig replicas 
INFO[2023-09-05 17:44:55] Job mem-util-desched: 110 iterations with 1 PersistentVolumeClaim replicas 
INFO[2023-09-05 17:44:55] Job mem-util-desched: 110 iterations with 1 DeploymentConfig replicas 
INFO[2023-09-05 17:44:55] Job mem-util-desched: 110 iterations with 1 Service replicas 
INFO[2023-09-05 17:44:55] Job mem-util-desched: 110 iterations with 1 DeploymentConfig replicas 
INFO[2023-09-05 17:44:55] Job mem-util-desched: 110 iterations with 1 Service replicas 
INFO[2023-09-05 17:44:55] Job mem-util-desched: 110 iterations with 1 Route replicas 
INFO[2023-09-05 17:44:55] Triggering job: mem-util-desched             
INFO[2023-09-05 17:44:55] QPS: 20                                      
INFO[2023-09-05 17:44:55] Burst: 20                                    
INFO[2023-09-05 17:44:55] Running job mem-util-desched                 
INFO[2023-09-05 17:44:55] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:44:55] Actions in namespace mem-util-desched-1 completed 
INFO[2023-09-05 17:44:55] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:44:55] Actions in namespace mem-util-desched-2 completed 
W0905 17:44:55.345522   56518 warnings.go:70] apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
W0905 17:44:55.352006   56518 warnings.go:70] apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
W0905 17:44:55.420076   56518 warnings.go:70] apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
W0905 17:44:55.434628   56518 warnings.go:70] apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
W0905 17:44:55.534866   56518 warnings.go:70] apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
INFO[2023-09-05 17:44:55] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:44:55] Actions in namespace mem-util-desched-3 completed 
W0905 17:44:55.623824   56518 warnings.go:70] apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
W0905 17:44:55.917684   56518 warnings.go:70] apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
W0905 17:44:56.037798   56518 warnings.go:70] apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
INFO[2023-09-05 17:44:56] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:44:56] Actions in namespace mem-util-desched-4 completed 
W0905 17:44:56.360030   56518 warnings.go:70] apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
W0905 17:44:56.472149   56518 warnings.go:70] apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
INFO[2023-09-05 17:44:56] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:44:56] Actions in namespace mem-util-desched-5 completed 
W0905 17:44:56.809017   56518 warnings.go:70] apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
W0905 17:44:56.954453   56518 warnings.go:70] apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
INFO[2023-09-05 17:44:56] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:44:56] Actions in namespace mem-util-desched-6 completed 
W0905 17:44:57.279472   56518 warnings.go:70] apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
W0905 17:44:57.389791   56518 warnings.go:70] apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
INFO[2023-09-05 17:44:57] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:44:57] Actions in namespace mem-util-desched-7 completed 
W0905 17:44:57.736141   56518 warnings.go:70] apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
INFO[2023-09-05 17:44:57] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:44:57] Actions in namespace mem-util-desched-8 completed 
W0905 17:44:57.892215   56518 warnings.go:70] apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
W0905 17:44:58.175203   56518 warnings.go:70] apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
W0905 17:44:58.261549   56518 warnings.go:70] apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
INFO[2023-09-05 17:44:58] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:44:58] Actions in namespace mem-util-desched-9 completed 
W0905 17:44:58.611691   56518 warnings.go:70] apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
W0905 17:44:58.722588   56518 warnings.go:70] apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
INFO[2023-09-05 17:44:58] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:44:58] Actions in namespace mem-util-desched-10 completed 
W0905 17:44:59.066876   56518 warnings.go:70] apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
W0905 17:44:59.193715   56518 warnings.go:70] apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
INFO[2023-09-05 17:44:59] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:44:59] Actions in namespace mem-util-desched-11 completed 
W0905 17:44:59.511546   56518 warnings.go:70] apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
W0905 17:44:59.611525   56518 warnings.go:70] apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
INFO[2023-09-05 17:44:59] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:44:59] Actions in namespace mem-util-desched-12 completed 
W0905 17:44:59.969741   56518 warnings.go:70] apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
INFO[2023-09-05 17:45:00] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:00] Actions in namespace mem-util-desched-13 completed 
INFO[2023-09-05 17:45:00] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:00] Actions in namespace mem-util-desched-14 completed 
INFO[2023-09-05 17:45:01] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:01] Actions in namespace mem-util-desched-15 completed 
INFO[2023-09-05 17:45:01] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:01] Actions in namespace mem-util-desched-16 completed 
INFO[2023-09-05 17:45:01] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:01] Actions in namespace mem-util-desched-17 completed 
INFO[2023-09-05 17:45:02] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:02] Actions in namespace mem-util-desched-18 completed 
INFO[2023-09-05 17:45:02] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:02] Actions in namespace mem-util-desched-19 completed 
INFO[2023-09-05 17:45:03] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:03] Actions in namespace mem-util-desched-20 completed 
INFO[2023-09-05 17:45:03] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:03] Actions in namespace mem-util-desched-21 completed 
INFO[2023-09-05 17:45:04] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:04] Actions in namespace mem-util-desched-22 completed 
INFO[2023-09-05 17:45:04] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:04] Actions in namespace mem-util-desched-23 completed 
INFO[2023-09-05 17:45:05] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:05] Actions in namespace mem-util-desched-24 completed 
INFO[2023-09-05 17:45:05] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:05] Actions in namespace mem-util-desched-25 completed 
INFO[2023-09-05 17:45:05] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:05] Actions in namespace mem-util-desched-26 completed 
INFO[2023-09-05 17:45:06] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:06] Actions in namespace mem-util-desched-27 completed 
INFO[2023-09-05 17:45:06] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:06] Actions in namespace mem-util-desched-28 completed 
INFO[2023-09-05 17:45:07] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:07] Actions in namespace mem-util-desched-29 completed 
INFO[2023-09-05 17:45:07] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:07] Actions in namespace mem-util-desched-30 completed 
INFO[2023-09-05 17:45:08] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:08] Actions in namespace mem-util-desched-31 completed 
INFO[2023-09-05 17:45:08] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:08] Actions in namespace mem-util-desched-32 completed 
INFO[2023-09-05 17:45:09] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:09] Actions in namespace mem-util-desched-33 completed 
INFO[2023-09-05 17:45:09] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:09] Actions in namespace mem-util-desched-34 completed 
INFO[2023-09-05 17:45:10] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:10] Actions in namespace mem-util-desched-35 completed 
INFO[2023-09-05 17:45:10] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:10] Actions in namespace mem-util-desched-36 completed 
INFO[2023-09-05 17:45:10] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:10] Actions in namespace mem-util-desched-37 completed 
INFO[2023-09-05 17:45:11] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:11] Actions in namespace mem-util-desched-38 completed 
INFO[2023-09-05 17:45:11] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:11] Actions in namespace mem-util-desched-39 completed 
INFO[2023-09-05 17:45:12] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:12] Actions in namespace mem-util-desched-40 completed 
INFO[2023-09-05 17:45:12] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:12] Actions in namespace mem-util-desched-41 completed 
INFO[2023-09-05 17:45:13] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:13] Actions in namespace mem-util-desched-42 completed 
INFO[2023-09-05 17:45:13] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:13] Actions in namespace mem-util-desched-43 completed 
INFO[2023-09-05 17:45:14] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:14] Actions in namespace mem-util-desched-44 completed 
INFO[2023-09-05 17:45:14] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:14] Actions in namespace mem-util-desched-45 completed 
INFO[2023-09-05 17:45:14] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:14] Actions in namespace mem-util-desched-46 completed 
INFO[2023-09-05 17:45:15] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:15] Actions in namespace mem-util-desched-47 completed 
INFO[2023-09-05 17:45:15] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:15] Actions in namespace mem-util-desched-48 completed 
INFO[2023-09-05 17:45:16] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:16] Actions in namespace mem-util-desched-49 completed 
INFO[2023-09-05 17:45:16] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:16] Actions in namespace mem-util-desched-50 completed 
INFO[2023-09-05 17:45:17] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:17] Actions in namespace mem-util-desched-51 completed 
INFO[2023-09-05 17:45:17] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:17] Actions in namespace mem-util-desched-52 completed 
INFO[2023-09-05 17:45:18] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:18] Actions in namespace mem-util-desched-53 completed 
INFO[2023-09-05 17:45:18] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:18] Actions in namespace mem-util-desched-54 completed 
INFO[2023-09-05 17:45:19] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:19] Actions in namespace mem-util-desched-55 completed 
INFO[2023-09-05 17:45:19] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:19] Actions in namespace mem-util-desched-56 completed 
INFO[2023-09-05 17:45:19] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:19] Actions in namespace mem-util-desched-57 completed 
INFO[2023-09-05 17:45:20] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:20] Actions in namespace mem-util-desched-58 completed 
INFO[2023-09-05 17:45:20] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:20] Actions in namespace mem-util-desched-59 completed 
INFO[2023-09-05 17:45:21] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:21] Actions in namespace mem-util-desched-60 completed 
INFO[2023-09-05 17:45:21] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:21] Actions in namespace mem-util-desched-61 completed 
INFO[2023-09-05 17:45:22] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:22] Actions in namespace mem-util-desched-62 completed 
INFO[2023-09-05 17:45:22] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:22] Actions in namespace mem-util-desched-63 completed 
INFO[2023-09-05 17:45:23] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:23] Actions in namespace mem-util-desched-64 completed 
INFO[2023-09-05 17:45:23] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:23] Actions in namespace mem-util-desched-65 completed 
INFO[2023-09-05 17:45:23] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:23] Actions in namespace mem-util-desched-66 completed 
INFO[2023-09-05 17:45:24] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:24] Actions in namespace mem-util-desched-67 completed 
INFO[2023-09-05 17:45:24] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:24] Actions in namespace mem-util-desched-68 completed 
INFO[2023-09-05 17:45:25] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:25] Actions in namespace mem-util-desched-69 completed 
INFO[2023-09-05 17:45:25] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:25] Actions in namespace mem-util-desched-70 completed 
INFO[2023-09-05 17:45:26] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:26] Actions in namespace mem-util-desched-71 completed 
INFO[2023-09-05 17:45:26] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:26] Actions in namespace mem-util-desched-72 completed 
INFO[2023-09-05 17:45:27] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:27] Actions in namespace mem-util-desched-73 completed 
INFO[2023-09-05 17:45:27] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:27] Actions in namespace mem-util-desched-74 completed 
INFO[2023-09-05 17:45:28] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:28] Actions in namespace mem-util-desched-75 completed 
INFO[2023-09-05 17:45:28] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:28] Actions in namespace mem-util-desched-76 completed 
INFO[2023-09-05 17:45:28] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:28] Actions in namespace mem-util-desched-77 completed 
INFO[2023-09-05 17:45:29] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:29] Actions in namespace mem-util-desched-78 completed 
INFO[2023-09-05 17:45:29] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:29] Actions in namespace mem-util-desched-79 completed 
INFO[2023-09-05 17:45:30] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:30] Actions in namespace mem-util-desched-80 completed 
INFO[2023-09-05 17:45:30] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:30] Actions in namespace mem-util-desched-81 completed 
INFO[2023-09-05 17:45:31] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:31] Actions in namespace mem-util-desched-82 completed 
INFO[2023-09-05 17:45:31] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:31] Actions in namespace mem-util-desched-83 completed 
INFO[2023-09-05 17:45:32] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:32] Actions in namespace mem-util-desched-84 completed 
INFO[2023-09-05 17:45:32] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:32] Actions in namespace mem-util-desched-85 completed 
INFO[2023-09-05 17:45:32] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:32] Actions in namespace mem-util-desched-86 completed 
INFO[2023-09-05 17:45:33] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:33] Actions in namespace mem-util-desched-87 completed 
INFO[2023-09-05 17:45:33] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:33] Actions in namespace mem-util-desched-88 completed 
INFO[2023-09-05 17:45:34] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:34] Actions in namespace mem-util-desched-89 completed 
INFO[2023-09-05 17:45:34] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:34] Actions in namespace mem-util-desched-90 completed 
INFO[2023-09-05 17:45:35] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:35] Actions in namespace mem-util-desched-91 completed 
INFO[2023-09-05 17:45:35] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:35] Actions in namespace mem-util-desched-92 completed 
INFO[2023-09-05 17:45:36] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:36] Actions in namespace mem-util-desched-93 completed 
INFO[2023-09-05 17:45:36] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:36] Actions in namespace mem-util-desched-94 completed 
INFO[2023-09-05 17:45:37] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:37] Actions in namespace mem-util-desched-95 completed 
INFO[2023-09-05 17:45:37] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:37] Actions in namespace mem-util-desched-96 completed 
INFO[2023-09-05 17:45:37] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:37] Actions in namespace mem-util-desched-97 completed 
INFO[2023-09-05 17:45:38] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:38] Actions in namespace mem-util-desched-98 completed 
INFO[2023-09-05 17:45:38] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:38] Actions in namespace mem-util-desched-99 completed 
INFO[2023-09-05 17:45:39] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:39] Actions in namespace mem-util-desched-100 completed 
INFO[2023-09-05 17:45:39] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:39] Actions in namespace mem-util-desched-101 completed 
INFO[2023-09-05 17:45:40] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:40] Actions in namespace mem-util-desched-102 completed 
INFO[2023-09-05 17:45:40] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:40] Actions in namespace mem-util-desched-103 completed 
INFO[2023-09-05 17:45:41] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:41] Actions in namespace mem-util-desched-104 completed 
INFO[2023-09-05 17:45:41] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:41] Actions in namespace mem-util-desched-105 completed 
INFO[2023-09-05 17:45:41] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:41] Actions in namespace mem-util-desched-106 completed 
INFO[2023-09-05 17:45:42] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:42] Actions in namespace mem-util-desched-107 completed 
INFO[2023-09-05 17:45:42] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:42] Actions in namespace mem-util-desched-108 completed 
INFO[2023-09-05 17:45:43] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:43] Actions in namespace mem-util-desched-109 completed 
INFO[2023-09-05 17:45:43] Waiting up to 1h0m0s all job actions to be completed 
INFO[2023-09-05 17:45:43] Actions in namespace mem-util-desched-110 completed 
INFO[2023-09-05 17:45:43] Finished the create job in 49s               
INFO[2023-09-05 17:45:43] Verifying created objects                    
INFO[2023-09-05 17:45:44] secrets found: 110 Expected: 110             
INFO[2023-09-05 17:45:44] imagestreams found: 110 Expected: 110        
INFO[2023-09-05 17:45:44] buildconfigs found: 110 Expected: 110        
INFO[2023-09-05 17:45:44] persistentvolumeclaims found: 110 Expected: 110 
INFO[2023-09-05 17:45:44] deploymentconfigs found: 110 Expected: 110   
INFO[2023-09-05 17:45:44] services found: 110 Expected: 110            
INFO[2023-09-05 17:45:44] deploymentconfigs found: 110 Expected: 110   
INFO[2023-09-05 17:45:44] services found: 110 Expected: 110            
INFO[2023-09-05 17:45:44] routes found: 110 Expected: 110              
INFO[2023-09-05 17:45:44] Job mem-util-desched took 49.93 seconds      
INFO[2023-09-05 17:45:44] Finished execution with UUID: FE0431D7-8548-459C-BFAE-E8CA3C5CCA7F 
INFO[2023-09-05 17:45:44] 👋 Exiting kube-burner                        
~/PycharmProjects/svt/perfscale_regression_ci/scripts/scalability/descheduler/e2e-benchmarking/workloads/kube-burner
Tue Sep  5 21:45:48 UTC 2023 Indexing benchmark metadata to https://search-perfscale-dev-**.us-west-2.es.amazonaws.com:443/ripsaw-kube-burner
Tue Sep  5 21:45:49 UTC 2023 benchmark
Tue Sep  5 21:45:49 UTC 2023 Installing touchstone
openshift-ovn-kubernetes
final csv /tmp/custom-FE0431D7-8548-459C-BFAE-E8CA3C5CCA7F/FE0431D7-8548-459C-BFAE-E8CA3C5CCA7F.csv
config /Users/prubenda/PycharmProjects/svt/perfscale_regression_ci/scripts/scalability/descheduler/e2e-benchmarking/utils/touchstone-configs/clusterVersion.json
comparison output
Tue Sep  5 21:46:07 UTC 2023 Querying results
Tue Sep  5 21:46:07 UTC 2023 Running: touchstone_compare --database elasticsearch -url https://search-perfscale-dev-**.us-west-2.es.amazonaws.com:443 -u FE0431D7-8548-459C-BFAE-E8CA3C5CCA7F --config /tmp/custom-FE0431D7-8548-459C-BFAE-E8CA3C5CCA7F/clusterVersion.json
2023-09-05 17:46:18,956 - touchstone - ERROR - Error: Issue capturing results from elasticsearch using config {'filter': {'metricName.keyword': 'clusterVersion'}, 'buckets': ['labels.version.keyword'], 'aggregations': {'timestamp': ['max']}}
Tue Sep  5 21:46:19 UTC 2023 python csv modifier
Tue Sep  5 21:46:19 UTC 2023 Removing touchstone
node/p-***-worker-a-f9nxm.c.openshift-qe.internal already uncordoned
node/p-***-worker-a-gngq2.c.openshift-qe.internal already uncordoned
node/p-***-worker-a-q5gm8.c.openshift-qe.internal already uncordoned
node/p-***-worker-a-xt4gg.c.openshift-qe.internal already uncordoned
node/p-***-worker-b-8kgpt.c.openshift-qe.internal uncordoned
node/p-***-worker-b-ntmc6.c.openshift-qe.internal uncordoned
node/p-***-worker-c-jjtc2.c.openshift-qe.internal uncordoned
Wait for descheduler to run
waiting for 10 seconds for descheduler to run
Get descheduler evicted, want to be more than 100
qiliRedHat commented 1 year ago

@paigerube14 In my test using the cakephp template, I found after kube-burner run, there could be some build/deploy error, application not running issue. I have to do some tricky fix after running kube-burner to make all objects up and running as expected. https://github.com/openshift/svt/blob/master/perfscale_regression_ci/scripts/registry/conc_registry_pull.sh#L70-L95 Some detail of the issues I met were recorded in https://docs.google.com/document/d/148Q-pIZlkZlyqdMDBI3Zr_I10_IDwMcKiBpuwP14lZw/edit#heading=h.91xp0heqzdq5.

paigerube14 commented 1 year ago

@qiliRedHat thanks for the information! I'm not too worried about all pods coming up but would you mind if I put that to a more common script/place that I could use it to try to get as many pods running as possible

I updated a bunch of other descheduler scripts because I was using the newest kube-burner version with regression testing and found a few small issues

qiliRedHat commented 1 year ago

@paigerube14 yes, please feel free to make it more common.

paigerube14 commented 1 year ago

@qiliRedHat I changed the objects I created and so I didn't have to more your fix functions

Successful run: https://mastern-jenkins-csb-openshift-qe.apps.ocp-c1.prod.psi.redhat.com/job/scale-ci/job/e2e-benchmarking-multibranch-pipeline/job/regression-test/390/console

qiliRedHat commented 1 year ago

/lgtm