dualstack-azure-e2e is flaky

CecileRobertMichon commented 3 years ago

https://testgrid.k8s.io/provider-azure-1.19-signal#dualstack-azure-e2e-1-19 https://testgrid.k8s.io/provider-azure-1.20-signal#dualstack-azure-e2e-1-20 https://testgrid.k8s.io/provider-azure-1.21-signal#dualstack-azure-e2e-1-21

Categories of recent failures:

Quota exceeded issues
- eg: https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-dualstack-azure-e2e-1-20/1396543275631906816
- should be resolved now, due to failing cleanup job in test sub

Check APIReachability fails

https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-dualstack-azure-e2e-1-20/1395456073367818240

error starting kubectl --match-server-version=false version: exec: already started | error starting kubectl --match-server-version=false version: exec: already started

Services Secondary IP Family should function for pod-Service time out
- eg: https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-dualstack-azure-e2e-1-20/1395818473350762496

Kubernetes e2e suite: [sig-network] [Feature:IPv6DualStackAlphaFeature] [LinuxOnly] Granular Checks: Services Secondary IP Family should function for pod-Service: sctp [Feature:SCTPConnectivity][Disruptive] expand_less	5m2s
/home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/network/dual_stack.go:464 May 21 20:12:50.232: Unexpected error: <*errors.errorString \| 0xc0002801f0>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/network/utils.go:836

- BeforeSuite fails
  - eg: https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-dualstack-azure-e2e-1-19/1395456073346846720

_output/local/go/src/k8s.io/kubernetes/test/e2e/e2e.go:76 May 20 19:25:24.142: Error waiting for all pods to be running and ready: 11 / 15 pods in namespace "kube-system" are NOT in RUNNING and READY state in 10m0s There are too many bad pods. Please check log for details. _output/local/go/src/k8s.io/kubernetes/test/e2e/e2e.go:284


- Other kubetest tests failing (less frequent):
  - Kubernetes e2e suite.[sig-network] [Feature:IPv6DualStackAlphaFeature] [LinuxOnly] Granular Checks: Services Secondary IP Family should function for node-Service: udp
  - Kubernetes e2e suite.[sig-network] [Feature:IPv6DualStackAlphaFeature] [LinuxOnly] Granular Checks: Services Secondary IP Family should update endpoints: udp
  - Kubernetes e2e suite.[sig-storage] Downward API volume should provide container's memory limit [NodeConformance] [Conformance]
  - Kubernetes e2e suite.[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should mutate configmap [Conformance]
  - Kubernetes e2e suite.[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should mutate custom resource with different stored version [Conformance]Kubernetes e2e suite.[sig-network] Proxy version v1 should proxy through a service and a pod [Conformance]
  - Kubernetes e2e suite.[sig-network] [Feature:IPv6DualStackAlphaFeature] [LinuxOnly] Granular Checks: Services Secondary IP Family should function for pod-Service: udp

CecileRobertMichon commented 3 years ago

cc @aramase

CecileRobertMichon commented 3 years ago

/close

this is looking a lot better https://testgrid.k8s.io/provider-azure-1.19-signal#dualstack-azure-e2e-1-19

Azure / container-upstream

dualstack-azure-e2e is flaky #102