[k8s.io] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] Scaling should happen in predictable order and halt if any stateful pod is unhealthy 21m14s

[deads2k]

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/41821/pull-kubernetes-e2e-gce-gci/18753/

go run hack/e2e.go -v -test --test_args='--ginkgo.focus=StatefulSet\s\[k8s\.io\]\sBasic\sStatefulSet\sfunctionality\s\[StatefulSetBasic\]\sScaling\sshould\shappen\sin\spredictable\sorder\sand\shalt\sif\sany\sstateful\spod\sis\sunhealthy$'

@ncdc do you know the statefulset owners?

[ncdc]

cc @foxish @kow3ns @smarterclayton

[smarterclayton]

I did not see this happening until recently.

[0xmichalis]

The controller misses that the first pod transitioned to unready and proceeds to create the second pod. The test uses a client to hit the api server and will scale up the statefulset as soon as the broken pod is observed but the controller uses caches so it may observe the scaled up statefulset quicker than the updated pod.

[kow3ns]

I will update the test to ensure that we wait until the unreadiness is observed on StatefulSet.Status.Replicas field prior to the scaling operation

[0xmichalis]

SGTM