Update sleep time to 3000.
Sometimes we got bellow error, from the log we can see autoscaler scale down automaticly before we delete payload. Workload pod is in Completed status. Increase sleep time to make sure pod are in running status. @jhou1 @miyadav Please help to take a look.
[07:42:21] INFO> Exit Status: 0
Machineset machineset-clone-20108-2 has 3 running machines, expected 2 (RuntimeError)
/home/sunny/code/verification-tests/features/step_definitions/machine_set.rb:56:in `/^the machineset should have expected number of running machines$/'
I0319 07:36:18.245507 1 scale_up.go:274] 8 other pods are also unschedulable
I0319 07:36:18.250258 1 scale_up.go:431] Best option to resize: openshift-machine-api/machineset-clone-20108-2
I0319 07:36:18.250289 1 scale_up.go:435] Estimated 6 nodes needed in openshift-machine-api/machineset-clone-20108-2
I0319 07:36:18.250412 1 scale_up.go:532] Splitting scale-up between 2 similar node groups: {openshift-machine-api/machineset-clone-20108-2, openshift-machine-api/machineset-clone-20108}
I0319 07:36:18.250437 1 scale_up.go:540] Final scale-up plan: [{openshift-machine-api/machineset-clone-20108-2 1->3 (max: 3)} {openshift-machine-api/machineset-clone-20108 1->3 (max: 3)}]
I0319 07:36:18.250460 1 scale_up.go:701] Scale-up: setting group openshift-machine-api/machineset-clone-20108-2 size to 3
I03
...
I0319 07:36:48.358331 1 static_autoscaler.go:343] No unschedulable pods
I0319 07:36:58.379419 1 static_autoscaler.go:269] 4 unregistered nodes present
...
I0319 07:39:28.823036 1 scale_up.go:271] Pod openshift-machine-api/workload-tgtfm is unschedulable
I0319 07:39:28.823042 1 scale_up.go:271] Pod openshift-machine-api/workload-ttkkx is unschedulable
I0319 07:39:28.823407 1 scale_up.go:423] No expansion options
...
I0319 07:41:19.061043 1 delete.go:102] Successfully added DeletionCandidateTaint on node machineset-clone-20108-2-gf8vj.c.openshift-qe.internal
I0319 07:41:29.084074 1 static_autoscaler.go:343] No unschedulable pods
I0319 07:41:29.086340 1 scale_down.go:827] Scale-down: removing node machineset-clone-20108-2-gf8vj.c.openshift-qe.internal, utilization: {0.45485714285714285 0.15678298221414386 0 cpu 0.45485714285714285}, pods to reschedule: openshift-machine-api/workload-988sw,openshift-machine-api/workload-vxstl
$ oc get po
NAME READY STATUS RESTARTS AGE
cluster-autoscaler-default-c5d4857bd-k5xv4 1/1 Running 0 12m
cluster-autoscaler-operator-f5bbcd64c-tc4mk 2/2 Running 0 4h36m
machine-api-controllers-5cbc9f58fd-m97m7 4/4 Running 0 4h56m
machine-api-operator-765f498db5-69lx9 2/2 Running 0 5h3m
workload-28t8d 0/1 Completed 0 12m
workload-2knrj 1/1 Running 0 81s
workload-2rxwg 0/1 Completed 0 12m
workload-5cjwv 0/1 Completed 0 6m16s
workload-5sv4b 0/1 Completed 0 12m
Update sleep time to 3000. Sometimes we got bellow error, from the log we can see autoscaler scale down automaticly before we delete payload. Workload pod is in Completed status. Increase sleep time to make sure pod are in running status. @jhou1 @miyadav Please help to take a look.