4 / 432 e2e test failures on k8s v1.11.3

[oomichi]

v1.10 での結果 https://github.com/oomichi/try-kubernetes/issues/17 の続き

• v1.10: 70 / 426 failures: 16.4%
• v1.11.1: 74 / 463 failures 15.9%, 12h47m
• v1.11.1 (+HPA修正2018/8/7): 71 / 463 failures 15.3%, 12h38m
• v1.11.1 (Above + StorageClass 2018/8/10): 64 / 473 failures 13.5%, 12h31m
• v1.11.1 (Above + Disruptiveスキップ 2018/8/15): 46 / 417 failures 11.0%, 10h35m
• v1.11.1 (Above + PV Protection修正 2018/8/16): 45 / 417 failures 10.7%, 10h27m
• v1.11.1 (Above + nfs-commonインストール 2018/9/19): 45 / 421 failures 10.6%, 14h58m
• v1.11.1 (Above + Internal IP対応 2018/9/21): 49 / 421 failures 11.6%, 14h19m
• v1.11.3 (Above + v1.11.3再構築 2018/9/28): 18 / 417 failures 4.3%, 9h8m
• v1.11.3 (Above + MountFlags=shared対応 2018/10/2): 15 / 417 failures 3.5%, 8h48m
• v1.11.3 (Above + ノード追加 + distroオプション追加 2018/10/3): 30 / 438 failures 6.8%, 9h58m
• v1.11.3 (Above + 追加ノードへのMountFlags=shared対応 2018/10/4): 27 / 438 failures 6.1%, 9h36m
• v1.11.3 (Above + nf_conntrackスキップ対応 2018/10/10): 26 / 436 failures 5.9%, 10h8m
• v1.11.3 (Above + gluster-client対応 2018/10/15): 11 / 436 failures 2.5%, 9h0m
• v1.11.3 (Above + LBテストスキップ対応 2018/11/05): 6 / 432 failures 1.3%, 7h26m
• v1.11.3 (Above + HPA 安定化パッチバックポート対応 2018/11/07): 4 / 432 failures 0.9%, 7h50m

[oomichi]

74項目をソート

$ tail -n 1000 all-tests-20180802.txt | grep "\[Fail\]" | sort
[Fail] [k8s.io] [sig-node] Mount propagation [It] should propagate mounts to the host
[Fail] [sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] [It] should adopt matching orphans and release non-matching pods
[Fail] [sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] [It] should not deadlock when a pod's predecessor fails
[Fail] [sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] [It] should provide basic identity
[Fail] [sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: CPU) [sig-autoscaling] [Serial] [Slow] Deployment [It] Should scale from 1 pod to 3 pods and from 3 to 5
[Fail] [sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: CPU) [sig-autoscaling] [Serial] [Slow] Deployment [It] Should scale from 5 pods to 3 pods and from 3 to 1
[Fail] [sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: CPU) [sig-autoscaling] [Serial] [Slow] ReplicaSet [It] Should scale from 5 pods to 3 pods and from 3 to 1
[Fail] [sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: CPU) [sig-autoscaling] [Serial] [Slow] ReplicationController [It] Should scale from 1 pod to 3 pods and from 3 to 5 and verify decision stability
[Fail] [sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: CPU) [sig-autoscaling] [Serial] [Slow] ReplicationController [It] Should scale from 5 pods to 3 pods and from 3 to 1 and verify decision stability
[Fail] [sig-instrumentation] MetricsGrabber [It] should grab all metrics from a ControllerManager.
[Fail] [sig-instrumentation] MetricsGrabber [It] should grab all metrics from a Scheduler.
[Fail] [sig-network] Network [It] should set TCP CLOSE_WAIT timeout
[Fail] [sig-network] Services [It] should be able to switch session affinity for LoadBalancer service with ESIPP off [Slow] [DisabledForLargeClusters]
[Fail] [sig-network] Services [It] should be able to switch session affinity for LoadBalancer service with ESIPP on [Slow] [DisabledForLargeClusters]
[Fail] [sig-network] Services [It] should have session affinity work for LoadBalancer service with ESIPP off [Slow] [DisabledForLargeClusters]
[Fail] [sig-network] Services [It] should have session affinity work for LoadBalancer service with ESIPP on [Slow] [DisabledForLargeClusters]
[Fail] [sig-scheduling] SchedulerPredicates [Serial] [It] validates MaxPods limit number of pods that are allowed to run [Slow]
[Fail] [sig-scheduling] SchedulerPredicates [Serial] [It] validates resource limits of pods that are allowed to run  [Conformance]
[Fail] [sig-storage] CSI Volumes CSI plugin test using CSI driver: hostPath [It] should provision storage
[Fail] [sig-storage] Dynamic Provisioning [k8s.io] GlusterDynamicProvisioner [It] should create and delete persistent volumes [fast]
[Fail] [sig-storage] PersistentVolumes NFS when invoking the Recycle reclaim policy [It] should test that a PV becomes Available and is clean after the PVC is deleted.
[Fail] [sig-storage] PersistentVolumes NFS with multiple PVs and PVCs all in same ns [It] should create 2 PVs and 4 PVCs: test write access
[Fail] [sig-storage] PersistentVolumes NFS with multiple PVs and PVCs all in same ns [It] should create 3 PVs and 3 PVCs: test write access
[Fail] [sig-storage] PersistentVolumes NFS with multiple PVs and PVCs all in same ns [It] should create 4 PVs and 2 PVCs: test write access [Slow]
[Fail] [sig-storage] PersistentVolumes NFS with Single PV - PVC pairs [It] create a PV and a pre-bound PVC: test write access
[Fail] [sig-storage] PersistentVolumes NFS with Single PV - PVC pairs [It] create a PVC and a pre-bound PV: test write access
[Fail] [sig-storage] PersistentVolumes NFS with Single PV - PVC pairs [It] create a PVC and non-pre-bound PV: test write access
[Fail] [sig-storage] PersistentVolumes NFS with Single PV - PVC pairs [It] should create a non-pre-bound PV and PVC: test write access
[Fail] [sig-storage] PVC Protection [BeforeEach] Verify "immediate" deletion of a PVC that is not in active use by a pod
[Fail] [sig-storage] PVC Protection [BeforeEach] Verify that PVC in active use by a pod is not removed immediately
[Fail] [sig-storage] PVC Protection [BeforeEach] Verify that scheduling of a pod that uses PVC that is being deleted fails and the pod becomes Unschedulable
[Fail] [sig-storage] PV Protection [BeforeEach] Verify "immediate" deletion of a PV that is not bound to a PVC
[Fail] [sig-storage] PV Protection [BeforeEach] Verify that PV bound to a PVC is not removed immediately
[Fail] [sig-storage] Subpath [Volume type: emptyDir] [It] should unmount if pod is force deleted while kubelet is down [Disruptive][Slow]
[Fail] [sig-storage] Subpath [Volume type: emptyDir] [It] should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow]
[Fail] [sig-storage] Subpath [Volume type: hostPath] [It] should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow]
[Fail] [sig-storage] Subpath [Volume type: hostPathSymlink] [It] should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow]
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should fail for new directories when readOnly specified in the volumeSource
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should fail if non-existent subpath is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should fail if subpath directory is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should fail if subpath file is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should fail if subpath with backstepping is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support creating multiple subpath from same volumes [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support existing directories when readOnly specified in the volumeSource
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support existing directory
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support existing single file
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support file as subpath
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support non-existent path
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support readOnly directory specified in the volumeMount
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support readOnly file specified in the volumeMount
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support restarting containers using directory as subpath [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support restarting containers using file as subpath [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should unmount if pod is force deleted while kubelet is down [Disruptive][Slow]
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should fail for new directories when readOnly specified in the volumeSource
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should fail if non-existent subpath is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should fail if subpath directory is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should fail if subpath file is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should fail if subpath with backstepping is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support creating multiple subpath from same volumes [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support existing directories when readOnly specified in the volumeSource
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support existing directory
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support existing single file
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support file as subpath
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support non-existent path
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support readOnly directory specified in the volumeMount
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support readOnly file specified in the volumeMount
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support restarting containers using directory as subpath [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support restarting containers using file as subpath [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should unmount if pod is force deleted while kubelet is down [Disruptive][Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow]
[Fail] [sig-storage] Volume plugin streaming [Slow] NFS [It] should write files of various sizes, verify size, validate content
[Fail] [sig-storage] Volumes NFS [It] should be mountable
[Fail] [sig-ui] Kubernetes Dashboard [It] should check that the kubernetes-dashboard instance is alive

[oomichi]

2018/8/7 v1.11.1 + HPA 修正適用後

[Fail] [k8s.io] [sig-node] Kubelet [Serial] [Slow] [k8s.io] [sig-node] regular resource usage tracking [It] resource tracking for 0 pods per node
[Fail] [k8s.io] [sig-node] Kubelet [Serial] [Slow] [k8s.io] [sig-node] regular resource usage tracking [It] resource tracking for 100 pods per node
[Fail] [k8s.io] [sig-node] Mount propagation [It] should propagate mounts to the host
[Fail] [sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] [It] should adopt matching orphans and release non-matching pods
[Fail] [sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] [It] should not deadlock when a pod's predecessor fails
[Fail] [sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] [It] should provide basic identity
[Fail] [sig-instrumentation] MetricsGrabber [It] should grab all metrics from a ControllerManager.
[Fail] [sig-instrumentation] MetricsGrabber [It] should grab all metrics from a Scheduler.
[Fail] [sig-network] Network [It] should set TCP CLOSE_WAIT timeout
[Fail] [sig-network] Services [It] should be able to switch session affinity for LoadBalancer service with ESIPP off [Slow] [DisabledForLargeClusters]
[Fail] [sig-network] Services [It] should be able to switch session affinity for LoadBalancer service with ESIPP on [Slow] [DisabledForLargeClusters]
[Fail] [sig-network] Services [It] should have session affinity work for LoadBalancer service with ESIPP off [Slow] [DisabledForLargeClusters]
[Fail] [sig-network] Services [It] should have session affinity work for LoadBalancer service with ESIPP on [Slow] [DisabledForLargeClusters]
[Fail] [sig-scheduling] SchedulerPredicates [Serial] [It] validates MaxPods limit number of pods that are allowed to run [Slow]
[Fail] [sig-scheduling] SchedulerPredicates [Serial] [It] validates resource limits of pods that are allowed to run  [Conformance]
[Fail] [sig-storage] CSI Volumes CSI plugin test using CSI driver: hostPath [It] should provision storage
[Fail] [sig-storage] Dynamic Provisioning [k8s.io] GlusterDynamicProvisioner [It] should create and delete persistent volumes [fast]
[Fail] [sig-storage] PersistentVolumes NFS when invoking the Recycle reclaim policy [It] should test that a PV becomes Available and is clean after the PVC is deleted.
[Fail] [sig-storage] PersistentVolumes NFS with multiple PVs and PVCs all in same ns [It] should create 2 PVs and 4 PVCs: test write access
[Fail] [sig-storage] PersistentVolumes NFS with multiple PVs and PVCs all in same ns [It] should create 3 PVs and 3 PVCs: test write access
[Fail] [sig-storage] PersistentVolumes NFS with multiple PVs and PVCs all in same ns [It] should create 4 PVs and 2 PVCs: test write access [Slow]
[Fail] [sig-storage] PersistentVolumes NFS with Single PV - PVC pairs [It] create a PV and a pre-bound PVC: test write access
[Fail] [sig-storage] PersistentVolumes NFS with Single PV - PVC pairs [It] create a PVC and a pre-bound PV: test write access
[Fail] [sig-storage] PersistentVolumes NFS with Single PV - PVC pairs [It] create a PVC and non-pre-bound PV: test write access
[Fail] [sig-storage] PersistentVolumes NFS with Single PV - PVC pairs [It] should create a non-pre-bound PV and PVC: test write access
[Fail] [sig-storage] PVC Protection [BeforeEach] Verify "immediate" deletion of a PVC that is not in active use by a pod
[Fail] [sig-storage] PVC Protection [BeforeEach] Verify that PVC in active use by a pod is not removed immediately
[Fail] [sig-storage] PVC Protection [BeforeEach] Verify that scheduling of a pod that uses PVC that is being deleted fails and the pod becomes Unschedulable
[Fail] [sig-storage] PV Protection [BeforeEach] Verify "immediate" deletion of a PV that is not bound to a PVC
[Fail] [sig-storage] PV Protection [BeforeEach] Verify that PV bound to a PVC is not removed immediately
[Fail] [sig-storage] Subpath [Volume type: emptyDir] [It] should unmount if pod is force deleted while kubelet is down [Disruptive][Slow]
[Fail] [sig-storage] Subpath [Volume type: emptyDir] [It] should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow]
[Fail] [sig-storage] Subpath [Volume type: hostPath] [It] should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow]
[Fail] [sig-storage] Subpath [Volume type: hostPathSymlink] [It] should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow]
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should fail for new directories when readOnly specified in the volumeSource
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should fail if non-existent subpath is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should fail if subpath directory is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should fail if subpath file is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should fail if subpath with backstepping is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support creating multiple subpath from same volumes [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support existing directories when readOnly specified in the volumeSource
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support existing directory
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support existing single file
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support file as subpath
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support non-existent path
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support readOnly directory specified in the volumeMount
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support readOnly file specified in the volumeMount
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support restarting containers using directory as subpath [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support restarting containers using file as subpath [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should unmount if pod is force deleted while kubelet is down [Disruptive][Slow]
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should fail for new directories when readOnly specified in the volumeSource
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should fail if non-existent subpath is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should fail if subpath directory is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should fail if subpath file is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should fail if subpath with backstepping is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support creating multiple subpath from same volumes [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support existing directories when readOnly specified in the volumeSource
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support existing directory
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support existing single file
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support file as subpath
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support non-existent path
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support readOnly directory specified in the volumeMount
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support readOnly file specified in the volumeMount
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support restarting containers using directory as subpath [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support restarting containers using file as subpath [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should unmount if pod is force deleted while kubelet is down [Disruptive][Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow]
[Fail] [sig-storage] Volume plugin streaming [Slow] NFS [It] should write files of various sizes, verify size, validate content
[Fail] [sig-storage] Volumes NFS [It] should be mountable
[Fail] [sig-ui] Kubernetes Dashboard [It] should check that the kubernetes-dashboard instance is alive

[oomichi]

2018/8/10 v1.11.1 + HPA 修正適用後 + StorageClass適用後

[Fail] [k8s.io] [sig-node] Kubelet [Serial] [Slow] [k8s.io] [sig-node] regular resource usage tracking [It] resource tracking for 0 pods per node
[Fail] [k8s.io] [sig-node] Mount propagation [It] should propagate mounts to the host
[Fail] [sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] [It] should not deadlock when a pod's predecessor fails
[Fail] [sig-instrumentation] MetricsGrabber [It] should grab all metrics from a ControllerManager.
[Fail] [sig-instrumentation] MetricsGrabber [It] should grab all metrics from a Scheduler.
[Fail] [sig-network] Network [It] should set TCP CLOSE_WAIT timeout
[Fail] [sig-network] Services [It] should be able to switch session affinity for LoadBalancer service with ESIPP off [Slow] [DisabledForLargeClusters]
[Fail] [sig-network] Services [It] should be able to switch session affinity for LoadBalancer service with ESIPP on [Slow] [DisabledForLargeClusters]
[Fail] [sig-network] Services [It] should have session affinity work for LoadBalancer service with ESIPP off [Slow] [DisabledForLargeClusters]
[Fail] [sig-network] Services [It] should have session affinity work for LoadBalancer service with ESIPP on [Slow] [DisabledForLargeClusters]
[Fail] [sig-scheduling] SchedulerPredicates [Serial] [It] validates MaxPods limit number of pods that are allowed to run [Slow]
[Fail] [sig-scheduling] SchedulerPredicates [Serial] [It] validates resource limits of pods that are allowed to run  [Conformance]
[Fail] [sig-storage] CSI Volumes CSI plugin test using CSI driver: hostPath [It] should provision storage
[Fail] [sig-storage] PersistentVolumes NFS when invoking the Recycle reclaim policy [It] should test that a PV becomes Available and is clean after the PVC is deleted.
[Fail] [sig-storage] PersistentVolumes NFS with multiple PVs and PVCs all in same ns [It] should create 2 PVs and 4 PVCs: test write access
[Fail] [sig-storage] PersistentVolumes NFS with multiple PVs and PVCs all in same ns [It] should create 3 PVs and 3 PVCs: test write access
[Fail] [sig-storage] PersistentVolumes NFS with multiple PVs and PVCs all in same ns [It] should create 4 PVs and 2 PVCs: test write access [Slow]
[Fail] [sig-storage] PersistentVolumes NFS with Single PV - PVC pairs [It] create a PV and a pre-bound PVC: test write access
[Fail] [sig-storage] PersistentVolumes NFS with Single PV - PVC pairs [It] create a PVC and a pre-bound PV: test write access
[Fail] [sig-storage] PersistentVolumes NFS with Single PV - PVC pairs [It] create a PVC and non-pre-bound PV: test write access
[Fail] [sig-storage] PersistentVolumes NFS with Single PV - PVC pairs [It] should create a non-pre-bound PV and PVC: test write access
[Fail] [sig-storage] PV Protection [BeforeEach] Verify "immediate" deletion of a PV that is not bound to a PVC
[Fail] [sig-storage] PV Protection [BeforeEach] Verify that PV bound to a PVC is not removed immediately
[Fail] [sig-storage] Subpath [Volume type: emptyDir] [It] should unmount if pod is force deleted while kubelet is down [Disruptive][Slow]
[Fail] [sig-storage] Subpath [Volume type: emptyDir] [It] should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow]
[Fail] [sig-storage] Subpath [Volume type: hostPath] [It] should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow]
[Fail] [sig-storage] Subpath [Volume type: hostPathSymlink] [It] should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow]
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should fail for new directories when readOnly specified in the volumeSource
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should fail if non-existent subpath is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should fail if subpath directory is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should fail if subpath file is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should fail if subpath with backstepping is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support creating multiple subpath from same volumes [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support existing directories when readOnly specified in the volumeSource
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support existing directory
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support existing single file
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support file as subpath
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support non-existent path
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support readOnly directory specified in the volumeMount
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support readOnly file specified in the volumeMount
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support restarting containers using directory as subpath [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support restarting containers using file as subpath [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should unmount if pod is force deleted while kubelet is down [Disruptive][Slow]
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should fail for new directories when readOnly specified in the volumeSource
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should fail if non-existent subpath is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should fail if subpath directory is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should fail if subpath file is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should fail if subpath with backstepping is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support creating multiple subpath from same volumes [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support existing directories when readOnly specified in the volumeSource
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support existing directory
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support existing single file
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support file as subpath
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support non-existent path
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support readOnly directory specified in the volumeMount
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support readOnly file specified in the volumeMount
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support restarting containers using directory as subpath [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support restarting containers using file as subpath [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should unmount if pod is force deleted while kubelet is down [Disruptive][Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow]
[Fail] [sig-storage] Volume plugin streaming [Slow] NFS [It] should write files of various sizes, verify size, validate content
[Fail] [sig-storage] Volumes NFS [It] should be mountable
[Fail] [sig-ui] Kubernetes Dashboard [It] should check that the kubernetes-dashboard instance is alive

[oomichi]

2018/8/15 v1.11.1 + HPA 修正適用後 + StorageClass適用後 + Disruptiveスキップ

[Fail] [k8s.io] [sig-node] Kubelet [Serial] [Slow] [k8s.io] [sig-node] regular resource usage tracking [It] resource tracking for 0 pods per node
[Fail] [k8s.io] [sig-node] Kubelet [Serial] [Slow] [k8s.io] [sig-node] regular resource usage tracking [It] resource tracking for 100 pods per node
[Fail] [k8s.io] [sig-node] Mount propagation [It] should propagate mounts to the host
[Fail] [sig-instrumentation] MetricsGrabber [It] should grab all metrics from a ControllerManager.
[Fail] [sig-instrumentation] MetricsGrabber [It] should grab all metrics from a Scheduler.
[Fail] [sig-network] Network [It] should set TCP CLOSE_WAIT timeout
[Fail] [sig-network] Services [It] should be able to switch session affinity for LoadBalancer service with ESIPP off [Slow] [DisabledForLargeClusters]
[Fail] [sig-network] Services [It] should be able to switch session affinity for LoadBalancer service with ESIPP on [Slow] [DisabledForLargeClusters]
[Fail] [sig-network] Services [It] should have session affinity work for LoadBalancer service with ESIPP off [Slow] [DisabledForLargeClusters]
[Fail] [sig-network] Services [It] should have session affinity work for LoadBalancer service with ESIPP on [Slow] [DisabledForLargeClusters]
[Fail] [sig-scheduling] SchedulerPredicates [Serial] [It] validates MaxPods limit number of pods that are allowed to run [Slow]
[Fail] [sig-scheduling] SchedulerPredicates [Serial] [It] validates resource limits of pods that are allowed to run  [Conformance]
[Fail] [sig-storage] CSI Volumes CSI plugin test using CSI driver: hostPath [It] should provision storage
[Fail] [sig-storage] PV Protection [BeforeEach] Verify "immediate" deletion of a PV that is not bound to a PVC
[Fail] [sig-storage] PV Protection [BeforeEach] Verify that PV bound to a PVC is not removed immediately
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should fail for new directories when readOnly specified in the volumeSource
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should fail if non-existent subpath is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should fail if subpath directory is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should fail if subpath file is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should fail if subpath with backstepping is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support creating multiple subpath from same volumes [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support existing directories when readOnly specified in the volumeSource
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support existing directory
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support existing single file
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support file as subpath
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support non-existent path
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support readOnly directory specified in the volumeMount
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support readOnly file specified in the volumeMount
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support restarting containers using directory as subpath [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support restarting containers using file as subpath [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should fail for new directories when readOnly specified in the volumeSource
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should fail if non-existent subpath is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should fail if subpath directory is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should fail if subpath file is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should fail if subpath with backstepping is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support creating multiple subpath from same volumes [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support existing directories when readOnly specified in the volumeSource
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support existing directory
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support existing single file
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support file as subpath
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support non-existent path
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support readOnly directory specified in the volumeMount
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support readOnly file specified in the volumeMount
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support restarting containers using directory as subpath [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support restarting containers using file as subpath [Slow]
[Fail] [sig-ui] Kubernetes Dashboard [It] should check that the kubernetes-dashboard instance is alive

[oomichi]

2018/8/15 v1.11.1 + HPA 修正適用後 + StorageClass適用後 + Disruptiveスキップ + PV Protection修正適用後

[Fail] [k8s.io] [sig-node] Kubelet [Serial] [Slow] [k8s.io] [sig-node] regular resource usage tracking [It] resource tracking for 0 pods per node
[Fail] [k8s.io] [sig-node] Kubelet [Serial] [Slow] [k8s.io] [sig-node] regular resource usage tracking [It] resource tracking for 100 pods per node
[Fail] [k8s.io] [sig-node] Mount propagation [It] should propagate mounts to the host
[Fail] [sig-instrumentation] MetricsGrabber [It] should grab all metrics from a ControllerManager.
[Fail] [sig-instrumentation] MetricsGrabber [It] should grab all metrics from a Scheduler.
[Fail] [sig-network] Network [It] should set TCP CLOSE_WAIT timeout
[Fail] [sig-network] Services [It] should be able to switch session affinity for LoadBalancer service with ESIPP off [Slow] [DisabledForLargeClusters]
[Fail] [sig-network] Services [It] should be able to switch session affinity for LoadBalancer service with ESIPP on [Slow] [DisabledForLargeClusters]
[Fail] [sig-network] Services [It] should have session affinity work for LoadBalancer service with ESIPP off [Slow] [DisabledForLargeClusters]
[Fail] [sig-network] Services [It] should have session affinity work for LoadBalancer service with ESIPP on [Slow] [DisabledForLargeClusters]
[Fail] [sig-scheduling] SchedulerPredicates [Serial] [It] validates MaxPods limit number of pods that are allowed to run [Slow]
[Fail] [sig-scheduling] SchedulerPredicates [Serial] [It] validates resource limits of pods that are allowed to run  [Conformance]
[Fail] [sig-storage] CSI Volumes CSI plugin test using CSI driver: hostPath [It] should provision storage
[Fail] [sig-storage] PV Protection [BeforeEach] Verify "immediate" deletion of a PV that is not bound to a PVC
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should fail for new directories when readOnly specified in the volumeSource
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should fail if non-existent subpath is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should fail if subpath directory is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should fail if subpath file is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should fail if subpath with backstepping is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support creating multiple subpath from same volumes [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support existing directories when readOnly specified in the volumeSource
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support existing directory
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support existing single file
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support file as subpath
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support non-existent path
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support readOnly directory specified in the volumeMount
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support readOnly file specified in the volumeMount
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support restarting containers using directory as subpath [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should support restarting containers using file as subpath [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should fail for new directories when readOnly specified in the volumeSource
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should fail if non-existent subpath is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should fail if subpath directory is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should fail if subpath file is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should fail if subpath with backstepping is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support creating multiple subpath from same volumes [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support existing directories when readOnly specified in the volumeSource
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support existing directory
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support existing single file
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support file as subpath
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support non-existent path
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support readOnly directory specified in the volumeMount
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support readOnly file specified in the volumeMount
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support restarting containers using directory as subpath [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support restarting containers using file as subpath [Slow]
[Fail] [sig-ui] Kubernetes Dashboard [It] should check that the kubernetes-dashboard instance is alive

[oomichi]

v1.11.1 (Above + nfs-commonインストール 2018/9/19): 45 / 421 failures 10.6%, 14h58m

[Fail] [k8s.io] Pods [It] should contain environment variables for services [NodeConformance] [Conformance]
[Fail] [k8s.io] Pods [It] should get a host IP [NodeConformance] [Conformance]
[Fail] [k8s.io] Pods [It] should support remote command execution over websockets [NodeConformance]
[Fail] [k8s.io] [sig-node] Kubelet [Serial] [Slow] [k8s.io] [sig-node] regular resource usage tracking [It] resource tracking for 0 pods per node
[Fail] [k8s.io] [sig-node] Kubelet [Serial] [Slow] [k8s.io] [sig-node] regular resource usage tracking [It] resource tracking for 100 pods per node
[Fail] [k8s.io] [sig-node] Mount propagation [It] should propagate mounts to the host
[Fail] [sig-api-machinery] Secrets [It] should be consumable via the environment [NodeConformance] [Conformance]
[Fail] [sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: CPU) [sig-autoscaling] ReplicationController light [It] Should scale from 1 pod to 2 pods
[Fail] [sig-cli] Kubectl client [k8s.io] Update Demo [It] should do a rolling update of a replication controller  [Conformance]
[Fail] [sig-cli] Kubectl Port forwarding [k8s.io] With a server listening on localhost [It] should support forwarding over websockets
[Fail] [sig-instrumentation] MetricsGrabber [It] should grab all metrics from a ControllerManager.
[Fail] [sig-network] Network [It] should set TCP CLOSE_WAIT timeout
[Fail] [sig-network] Networking Granular Checks: Pods [It] should function for intra-pod communication: udp [NodeConformance] [Conformance]
[Fail] [sig-network] Networking Granular Checks: Pods [It] should function for node-pod communication: http [NodeConformance] [Conformance]
[Fail] [sig-network] Networking Granular Checks: Pods [It] should function for node-pod communication: udp [NodeConformance] [Conformance]
[Fail] [sig-network] Networking Granular Checks: Services [Slow] [It] should function for client IP based session affinity: http
[Fail] [sig-network] Networking Granular Checks: Services [Slow] [It] should function for endpoint-Service: http
[Fail] [sig-network] Services [It] should be able to switch session affinity for LoadBalancer service with ESIPP off [Slow] [DisabledForLargeClusters]
[Fail] [sig-network] Services [It] should be able to switch session affinity for LoadBalancer service with ESIPP on [Slow] [DisabledForLargeClusters]
[Fail] [sig-network] Services [It] should be able to switch session affinity for NodePort service
[Fail] [sig-network] Services [It] should have session affinity work for LoadBalancer service with ESIPP off [Slow] [DisabledForLargeClusters]
[Fail] [sig-network] Services [It] should have session affinity work for LoadBalancer service with ESIPP on [Slow] [DisabledForLargeClusters]
[Fail] [sig-network] Services [It] should serve a basic endpoint from pods  [Conformance]
[Fail] [sig-scheduling] SchedulerPredicates [Serial] [It] validates MaxPods limit number of pods that are allowed to run [Slow]
[Fail] [sig-scheduling] SchedulerPredicates [Serial] [It] validates resource limits of pods that are allowed to run  [Conformance]
[Fail] [sig-storage] CSI Volumes CSI plugin test using CSI driver: hostPath [It] should provision storage
[Fail] [sig-storage] Downward API volume [It] should provide node allocatable (memory) as default memory limit if the limit is not set [NodeConformance] [Conformance]
[Fail] [sig-storage] Downward API volume [It] should update labels on modification [NodeConformance] [Conformance]
[Fail] [sig-storage] EmptyDir volumes [It] should support (root,0644,tmpfs) [NodeConformance] [Conformance]
[Fail] [sig-storage] EmptyDir wrapper volumes [It] should not cause race condition when used for configmaps [Serial] [Slow]
[Fail] [sig-storage] HostPath [It] should support existing directory subPath
[Fail] [sig-storage] HostPath [It] should support existing single file subPath
[Fail] [sig-storage] Projected [It] should set DefaultMode on files [NodeConformance] [Conformance]
[Fail] [sig-storage] Secrets [It] should be consumable from pods in volume with mappings and Item Mode set [NodeConformance] [Conformance]
[Fail] [sig-storage] Subpath Atomic writer volumes [It] should support subpaths with secret pod
[Fail] [sig-storage] Subpath [Volume type: emptyDir] [It] should fail if subpath with backstepping is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: emptyDir] [It] should support existing directory
[Fail] [sig-storage] Subpath [Volume type: hostPath] [It] should fail if subpath with backstepping is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: hostPath] [It] should support existing directory
[Fail] [sig-storage] Subpath [Volume type: hostPath] [It] should support restarting containers using file as subpath [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfs] [BeforeEach] should support readOnly file specified in the volumeMount
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [BeforeEach] should fail if subpath file is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [BeforeEach] should support creating multiple subpath from same volumes [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [It] should support existing single file
[Fail] [sig-ui] Kubernetes Dashboard [It] should check that the kubernetes-dashboard instance is alive

[oomichi]

v1.11.1 (Above + Internal IP対応 2018/9/21): 49 / 421 failures 11.6%, 14h19m 今まで失敗しなかったテストも失敗している。

[Fail] [k8s.io] Pods [It] should be submitted and removed [NodeConformance] [Conformance]
[Fail] [k8s.io] Probing container [It] with readiness probe should not be ready before initial delay and never restart [NodeConformance] [Conformance]
[Fail] [k8s.io] [sig-node] Kubelet [Serial] [Slow] [k8s.io] [sig-node] regular resource usage tracking [It] resource tracking for 0 pods per node
[Fail] [k8s.io] [sig-node] Kubelet [Serial] [Slow] [k8s.io] [sig-node] regular resource usage tracking [It] resource tracking for 100 pods per node
[Fail] [k8s.io] [sig-node] Mount propagation [It] should propagate mounts to the host
[Fail] [k8s.io] Variable Expansion [It] should allow substituting values in a container's command [NodeConformance] [Conformance]
[Fail] [sig-api-machinery] Secrets [It] should be consumable from pods in env vars [NodeConformance] [Conformance]
[Fail] [sig-apps] Deployment [It] deployment should support rollback
[Fail] [sig-apps] ReplicaSet [It] should serve a basic image on each replica with a public image  [Conformance]
[Fail] [sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: CPU) [sig-autoscaling] [Serial] [Slow] ReplicaSet [It] Should scale from 1 pod to 3 pods and from 3 to 5
[Fail] [sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: CPU) [sig-autoscaling] [Serial] [Slow] ReplicationController [It] Should scale from 1 pod to 3 pods and from 3 to 5 and verify decision stability
[Fail] [sig-cli] Kubectl client [k8s.io] Kubectl run rc [It] should create an rc from an image  [Conformance]
[Fail] [sig-cli] Kubectl client [k8s.io] Update Demo [It] should scale a replication controller  [Conformance]
[Fail] [sig-instrumentation] MetricsGrabber [It] should grab all metrics from a ControllerManager.
[Fail] [sig-network] DNS [It] should provide DNS for the cluster  [Conformance]
[Fail] [sig-network] DNS configMap federations [It] should be able to change federation configuration [Slow][Serial]
[Fail] [sig-network] DNS configMap nameserver Change stubDomain [It] should be able to change stubDomain configuration [Slow][Serial]
[Fail] [sig-network] Network [It] should set TCP CLOSE_WAIT timeout
[Fail] [sig-network] Networking Granular Checks: Pods [It] should function for intra-pod communication: http [NodeConformance] [Conformance]
[Fail] [sig-network] Networking Granular Checks: Services [Slow] [It] should function for client IP based session affinity: http
[Fail] [sig-network] Networking Granular Checks: Services [Slow] [It] should function for client IP based session affinity: udp
[Fail] [sig-network] Services [It] should be able to switch session affinity for LoadBalancer service with ESIPP off [Slow] [DisabledForLargeClusters]
[Fail] [sig-network] Services [It] should be able to switch session affinity for LoadBalancer service with ESIPP on [Slow] [DisabledForLargeClusters]
[Fail] [sig-network] Services [It] should have session affinity work for LoadBalancer service with ESIPP off [Slow] [DisabledForLargeClusters]
[Fail] [sig-network] Services [It] should have session affinity work for LoadBalancer service with ESIPP on [Slow] [DisabledForLargeClusters]
[Fail] [sig-scheduling] SchedulerPredicates [Serial] [It] validates MaxPods limit number of pods that are allowed to run [Slow]
[Fail] [sig-scheduling] SchedulerPredicates [Serial] [It] validates resource limits of pods that are allowed to run  [Conformance]
[Fail] [sig-storage] ConfigMap [It] should be consumable from pods in volume [NodeConformance] [Conformance]
[Fail] [sig-storage] CSI Volumes CSI plugin test using CSI driver: hostPath [It] should provision storage
[Fail] [sig-storage] Downward API volume [It] should provide container's cpu request [NodeConformance] [Conformance]
[Fail] [sig-storage] EmptyDir volumes [It] should support (root,0666,tmpfs) [NodeConformance] [Conformance]
[Fail] [sig-storage] EmptyDir wrapper volumes [It] should not cause race condition when used for configmaps [Serial] [Slow]
[Fail] [sig-storage] EmptyDir wrapper volumes [It] should not cause race condition when used for git_repo [Serial] [Slow]
[Fail] [sig-storage] Projected [It] optional updates should be reflected in volume [NodeConformance] [Conformance]
[Fail] [sig-storage] Projected [It] should be consumable in multiple volumes in the same pod [NodeConformance] [Conformance]
[Fail] [sig-storage] Projected [It] should project all components that make up the projection API [Projection][NodeConformance] [Conformance]
[Fail] [sig-storage] Projected [It] should provide container's memory limit [NodeConformance] [Conformance]
[Fail] [sig-storage] PVC Protection [It] Verify that scheduling of a pod that uses PVC that is being deleted fails and the pod becomes Unschedulable
[Fail] [sig-storage] Secrets [It] should be able to mount in a volume regardless of a different secret existing with same name in different namespace [NodeConformance]
[Fail] [sig-storage] Secrets [It] should be consumable from pods in volume with defaultMode set [NodeConformance] [Conformance]
[Fail] [sig-storage] Secrets [It] should be consumable in multiple volumes in a pod [NodeConformance] [Conformance]
[Fail] [sig-storage] Subpath [Volume type: emptyDir] [It] should fail if non-existent subpath is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: emptyDir] [It] should fail if subpath directory is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: emptyDir] [It] should fail if subpath file is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: emptyDir] [It] should fail if subpath with backstepping is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: hostPath] [It] should fail if non-existent subpath is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfs] [It] should fail if subpath with backstepping is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: nfsPVC] [BeforeEach] should support restarting containers using directory as subpath [Slow]
[Fail] [sig-ui] Kubernetes Dashboard [It] should check that the kubernetes-dashboard instance is alive

[oomichi]

• v1.11.3 (Above + v1.11.3再構築 2018/9/28): 18 / 417 failures 4.3%, 9h8m

  [Fail] [k8s.io] EquivalenceCache [Serial] [It] validates pod anti-affinity works properly when new replica pod is scheduled
  [Fail] [k8s.io] [sig-node] Mount propagation [It] should propagate mounts to the host
  [Fail] [sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: CPU) [sig-autoscaling] [Serial] [Slow] Deployment [It] Should scale from 1 pod to 3 pods and from 3 to 5
  [Fail] [sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: CPU) [sig-autoscaling] [Serial] [Slow] Deployment [It] Should scale from 5 pods to 3 pods and from 3 to 1
  [Fail] [sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: CPU) [sig-autoscaling] [Serial] [Slow] ReplicaSet [It] Should scale from 1 pod to 3 pods and from 3 to 5
  [Fail] [sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: CPU) [sig-autoscaling] [Serial] [Slow] ReplicaSet [It] Should scale from 5 pods to 3 pods and from 3 to 1
  [Fail] [sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: CPU) [sig-autoscaling] [Serial] [Slow] ReplicationController [It] Should scale from 1 pod to 3 pods and from 3 to 5 and verify decision stability
  [Fail] [sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: CPU) [sig-autoscaling] [Serial] [Slow] ReplicationController [It] Should scale from 5 pods to 3 pods and from 3 to 1 and verify decision stability
  [Fail] [sig-cli] Kubectl client [AfterEach] [k8s.io] Kubectl taint [Serial] should remove all the taints with the same key off a node
  [Fail] [sig-instrumentation] MetricsGrabber [It] should grab all metrics from a ControllerManager.
  [Fail] [sig-instrumentation] MetricsGrabber [It] should grab all metrics from a Scheduler.
  [Fail] [sig-network] Services [It] should be able to switch session affinity for LoadBalancer service with ESIPP off [Slow] [DisabledForLargeClusters]
  [Fail] [sig-network] Services [It] should be able to switch session affinity for LoadBalancer service with ESIPP on [Slow] [DisabledForLargeClusters]
  [Fail] [sig-network] Services [It] should have session affinity work for LoadBalancer service with ESIPP off [Slow] [DisabledForLargeClusters]
  [Fail] [sig-network] Services [It] should have session affinity work for LoadBalancer service with ESIPP on [Slow] [DisabledForLargeClusters]
  [Fail] [sig-scheduling] SchedulerPriorities [Serial] [It] Pod should be schedule to node that don't match the PodAntiAffinity terms
  [Fail] [sig-storage] CSI Volumes CSI plugin test using CSI driver: hostPath [It] should provision storage
  [Fail] [sig-ui] Kubernetes Dashboard [It] should check that the kubernetes-dashboard instance is alive

[oomichi]

• v1.11.3 (Above + MountFlags=shared対応 2018/10/2): 15 / 417 failures 3.5%, 8h48m

  [Fail] [k8s.io] EquivalenceCache [Serial] [It] validates pod anti-affinity works properly when new replica pod is scheduled
  [Fail] [sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: CPU) [sig-autoscaling] [Serial] [Slow] Deployment [It] Should scale from 1 pod to 3 pods and from 3 to 5
  [Fail] [sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: CPU) [sig-autoscaling] [Serial] [Slow] Deployment [It] Should scale from 5 pods to 3 pods and from 3 to 1
  [Fail] [sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: CPU) [sig-autoscaling] [Serial] [Slow] ReplicaSet [It] Should scale from 1 pod to 3 pods and from 3 to 5
  [Fail] [sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: CPU) [sig-autoscaling] [Serial] [Slow] ReplicaSet [It] Should scale from 5 pods to 3 pods and from 3 to 1
  [Fail] [sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: CPU) [sig-autoscaling] [Serial] [Slow] ReplicationController [It] Should scale from 1 pod to 3 pods and from 3 to 5 and verify decision stability
  [Fail] [sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: CPU) [sig-autoscaling] [Serial] [Slow] ReplicationController [It] Should scale from 5 pods to 3 pods and from 3 to 1 and verify decision stability
  [Fail] [sig-instrumentation] MetricsGrabber [It] should grab all metrics from a ControllerManager.
  [Fail] [sig-instrumentation] MetricsGrabber [It] should grab all metrics from a Scheduler.
  [Fail] [sig-network] Services [It] should be able to switch session affinity for LoadBalancer service with ESIPP off [Slow] [DisabledForLargeClusters]
  [Fail] [sig-network] Services [It] should be able to switch session affinity for LoadBalancer service with ESIPP on [Slow] [DisabledForLargeClusters]
  [Fail] [sig-network] Services [It] should have session affinity work for LoadBalancer service with ESIPP off [Slow] [DisabledForLargeClusters]
  [Fail] [sig-network] Services [It] should have session affinity work for LoadBalancer service with ESIPP on [Slow] [DisabledForLargeClusters]
  [Fail] [sig-scheduling] SchedulerPriorities [Serial] [It] Pod should be schedule to node that don't match the PodAntiAffinity terms
  [Fail] [sig-ui] Kubernetes Dashboard [It] should check that the kubernetes-dashboard instance is alive

[oomichi]

HPAが失敗しているのは実行ノード不足が原因と予想。 制御ノード1、実行ノード2（元は1）にして再テストする。

[oomichi]

• v1.11.3 (Above + ノード追加 + distroオプション追加 2018/10/3): 30 / 438 failures 6.8%, 9h58m 実行コマンド

  $ go run hack/e2e.go --provider=skeleton --test --test_args="--ginkgo.skip=.*(Feature)|(NFS)|(StatefulSet)|\[Disruptive\].* --master-os-distro=ubuntu --node-os-distro=ubuntu" --check-version-skew=false

[oomichi]

[Fail] [k8s.io] [sig-node] Kubelet [Serial] [Slow] [k8s.io] [sig-node] regular resource usage tracking [It] resource tracking for 0 pods per node
[Fail] [k8s.io] [sig-node] Kubelet [Serial] [Slow] [k8s.io] [sig-node] regular resource usage tracking [It] resource tracking for 100 pods per node
[Fail] [sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: CPU) [sig-autoscaling] [Serial] [Slow] ReplicationController [It] Should scale from 1 pod to 3 pods and from 3 to 5 and verify decision stability
[Fail] [sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: CPU) [sig-autoscaling] [Serial] [Slow] ReplicationController [It] Should scale from 5 pods to 3 pods and from 3 to 1 and verify decision stability
[Fail] [sig-instrumentation] MetricsGrabber [It] should grab all metrics from a ControllerManager.
[Fail] [sig-instrumentation] MetricsGrabber [It] should grab all metrics from a Scheduler.
[Fail] [sig-network] Network [It] should set TCP CLOSE_WAIT timeout
[Fail] [sig-network] Services [It] should be able to switch session affinity for LoadBalancer service with ESIPP off [Slow] [DisabledForLargeClusters]
[Fail] [sig-network] Services [It] should be able to switch session affinity for LoadBalancer service with ESIPP on [Slow] [DisabledForLargeClusters]
[Fail] [sig-network] Services [It] should have session affinity work for LoadBalancer service with ESIPP off [Slow] [DisabledForLargeClusters]
[Fail] [sig-network] Services [It] should have session affinity work for LoadBalancer service with ESIPP on [Slow] [DisabledForLargeClusters]
[Fail] [sig-storage] CSI Volumes CSI plugin test using CSI driver: hostPath [It] should provision storage
[Fail] [sig-storage] GCP Volumes GlusterFS [It] should be mountable
[Fail] [sig-storage] Subpath [Volume type: gluster] [It] should fail for new directories when readOnly specified in the volumeSource
[Fail] [sig-storage] Subpath [Volume type: gluster] [It] should fail if non-existent subpath is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: gluster] [It] should fail if subpath directory is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: gluster] [It] should fail if subpath file is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: gluster] [It] should fail if subpath with backstepping is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: gluster] [It] should support creating multiple subpath from same volumes [Slow]
[Fail] [sig-storage] Subpath [Volume type: gluster] [It] should support existing directories when readOnly specified in the volumeSource
[Fail] [sig-storage] Subpath [Volume type: gluster] [It] should support existing directory
[Fail] [sig-storage] Subpath [Volume type: gluster] [It] should support existing single file
[Fail] [sig-storage] Subpath [Volume type: gluster] [It] should support file as subpath
[Fail] [sig-storage] Subpath [Volume type: gluster] [It] should support non-existent path
[Fail] [sig-storage] Subpath [Volume type: gluster] [It] should support readOnly directory specified in the volumeMount
[Fail] [sig-storage] Subpath [Volume type: gluster] [It] should support readOnly file specified in the volumeMount
[Fail] [sig-storage] Subpath [Volume type: gluster] [It] should support restarting containers using directory as subpath [Slow]
[Fail] [sig-storage] Subpath [Volume type: gluster] [It] should support restarting containers using file as subpath [Slow]
[Fail] [sig-storage] Volumes GlusterFS [It] should be mountable
[Fail] [sig-ui] Kubernetes Dashboard [It] should check that the kubernetes-dashboard instance is alive

[oomichi]

• v1.11.3 (Above + 追加ノードへのMountFlags=shared対応 2018/10/4): 27 / 438 failures 6.1%, 9h36m

  [Fail] [k8s.io] [sig-node] Kubelet [Serial] [Slow] [k8s.io] [sig-node] regular resource usage tracking [It] resource tracking for 100 pods per node
  [Fail] [sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: CPU) [sig-autoscaling] [Serial] [Slow] ReplicationController [It] Should scale from 1 pod to 3 pods and from 3 to 5 and verify decision stability
  [Fail] [sig-instrumentation] MetricsGrabber [It] should grab all metrics from a ControllerManager.
  [Fail] [sig-instrumentation] MetricsGrabber [It] should grab all metrics from a Scheduler.
  [Fail] [sig-network] Network [It] should set TCP CLOSE_WAIT timeout
  [Fail] [sig-network] Services [It] should be able to switch session affinity for LoadBalancer service with ESIPP off [Slow] [DisabledForLargeClusters]
  [Fail] [sig-network] Services [It] should be able to switch session affinity for LoadBalancer service with ESIPP on [Slow] [DisabledForLargeClusters]
  [Fail] [sig-network] Services [It] should have session affinity work for LoadBalancer service with ESIPP off [Slow] [DisabledForLargeClusters]
  [Fail] [sig-network] Services [It] should have session affinity work for LoadBalancer service with ESIPP on [Slow] [DisabledForLargeClusters]
  [Fail] [sig-storage] GCP Volumes GlusterFS [It] should be mountable
  [Fail] [sig-storage] Subpath [Volume type: gluster] [It] should fail for new directories when readOnly specified in the volumeSource
  [Fail] [sig-storage] Subpath [Volume type: gluster] [It] should fail if non-existent subpath is outside the volume [Slow]
  [Fail] [sig-storage] Subpath [Volume type: gluster] [It] should fail if subpath directory is outside the volume [Slow]
  [Fail] [sig-storage] Subpath [Volume type: gluster] [It] should fail if subpath file is outside the volume [Slow]
  [Fail] [sig-storage] Subpath [Volume type: gluster] [It] should fail if subpath with backstepping is outside the volume [Slow]
  [Fail] [sig-storage] Subpath [Volume type: gluster] [It] should support creating multiple subpath from same volumes [Slow]
  [Fail] [sig-storage] Subpath [Volume type: gluster] [It] should support existing directories when readOnly specified in the volumeSource
  [Fail] [sig-storage] Subpath [Volume type: gluster] [It] should support existing directory
  [Fail] [sig-storage] Subpath [Volume type: gluster] [It] should support existing single file
  [Fail] [sig-storage] Subpath [Volume type: gluster] [It] should support file as subpath
  [Fail] [sig-storage] Subpath [Volume type: gluster] [It] should support non-existent path
  [Fail] [sig-storage] Subpath [Volume type: gluster] [It] should support readOnly directory specified in the volumeMount
  [Fail] [sig-storage] Subpath [Volume type: gluster] [It] should support readOnly file specified in the volumeMount
  [Fail] [sig-storage] Subpath [Volume type: gluster] [It] should support restarting containers using directory as subpath [Slow]
  [Fail] [sig-storage] Subpath [Volume type: gluster] [It] should support restarting containers using file as subpath [Slow]
  [Fail] [sig-storage] Volumes GlusterFS [It] should be mountable
  [Fail] [sig-ui] Kubernetes Dashboard [It] should check that the kubernetes-dashboard instance is alive

[oomichi]

v1.11.3 (Above + nf_conntrackスキップ対応 2018/10/10): 26 / 436 failures 5.9%, 10h8m

[Fail] [k8s.io] [sig-node] Kubelet [Serial] [Slow] [k8s.io] [sig-node] regular resource usage tracking [It] resource tracking for 0 pods per node
[Fail] [k8s.io] [sig-node] Kubelet [Serial] [Slow] [k8s.io] [sig-node] regular resource usage tracking [It] resource tracking for 100 pods per node
[Fail] [sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: CPU) [sig-autoscaling] [Serial] [Slow] ReplicationController [It] Should scale from 5 pods to 3 pods and from 3 to 1 and verify decision stability
[Fail] [sig-instrumentation] MetricsGrabber [It] should grab all metrics from a ControllerManager.
[Fail] [sig-instrumentation] MetricsGrabber [It] should grab all metrics from a Scheduler.
[Fail] [sig-network] Services [It] should be able to switch session affinity for LoadBalancer service with ESIPP off [Slow] [DisabledForLargeClusters]
[Fail] [sig-network] Services [It] should be able to switch session affinity for LoadBalancer service with ESIPP on [Slow] [DisabledForLargeClusters]
[Fail] [sig-network] Services [It] should have session affinity work for LoadBalancer service with ESIPP off [Slow] [DisabledForLargeClusters]
[Fail] [sig-network] Services [It] should have session affinity work for LoadBalancer service with ESIPP on [Slow] [DisabledForLargeClusters]
[Fail] [sig-storage] GCP Volumes GlusterFS [It] should be mountable
[Fail] [sig-storage] Subpath [Volume type: gluster] [It] should fail for new directories when readOnly specified in the volumeSource
[Fail] [sig-storage] Subpath [Volume type: gluster] [It] should fail if non-existent subpath is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: gluster] [It] should fail if subpath directory is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: gluster] [It] should fail if subpath file is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: gluster] [It] should fail if subpath with backstepping is outside the volume [Slow]
[Fail] [sig-storage] Subpath [Volume type: gluster] [It] should support creating multiple subpath from same volumes [Slow]
[Fail] [sig-storage] Subpath [Volume type: gluster] [It] should support existing directories when readOnly specified in the volumeSource
[Fail] [sig-storage] Subpath [Volume type: gluster] [It] should support existing directory
[Fail] [sig-storage] Subpath [Volume type: gluster] [It] should support existing single file
[Fail] [sig-storage] Subpath [Volume type: gluster] [It] should support file as subpath
[Fail] [sig-storage] Subpath [Volume type: gluster] [It] should support non-existent path
[Fail] [sig-storage] Subpath [Volume type: gluster] [It] should support readOnly directory specified in the volumeMount
[Fail] [sig-storage] Subpath [Volume type: gluster] [It] should support readOnly file specified in the volumeMount
[Fail] [sig-storage] Subpath [Volume type: gluster] [It] should support restarting containers using directory as subpath [Slow]
[Fail] [sig-storage] Subpath [Volume type: gluster] [It] should support restarting containers using file as subpath [Slow]
[Fail] [sig-storage] Volumes GlusterFS [It] should be mountable

[oomichi]

• v1.11.3 (Above + gluster-client対応 2018/10/15): 11 / 436 failures 2.5%, 9h0m

  [Fail] [k8s.io] [sig-node] Kubelet [Serial] [Slow] [k8s.io] [sig-node] regular resource usage tracking [It] resource tracking for 0 pods per node
  [Fail] [k8s.io] [sig-node] Kubelet [Serial] [Slow] [k8s.io] [sig-node] regular resource usage tracking [It] resource tracking for 100 pods per node
  [Fail] [sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: CPU) [sig-autoscaling] [Serial] [Slow] ReplicationController [It] Should scale from 1 pod to 3 pods and from 3 to 5 and verify decision stability
  [Fail] [sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: CPU) [sig-autoscaling] [Serial] [Slow] ReplicationController [It] Should scale from 5 pods to 3 pods and from 3 to 1 and verify decision stability
  [Fail] [sig-instrumentation] MetricsGrabber [It] should grab all metrics from a ControllerManager.
  [Fail] [sig-instrumentation] MetricsGrabber [It] should grab all metrics from a Scheduler.
  [Fail] [sig-network] Services [It] should be able to switch session affinity for LoadBalancer service with ESIPP off [Slow] [DisabledForLargeClusters]
  [Fail] [sig-network] Services [It] should be able to switch session affinity for LoadBalancer service with ESIPP on [Slow] [DisabledForLargeClusters]
  [Fail] [sig-network] Services [It] should have session affinity work for LoadBalancer service with ESIPP off [Slow] [DisabledForLargeClusters]
  [Fail] [sig-network] Services [It] should have session affinity work for LoadBalancer service with ESIPP on [Slow] [DisabledForLargeClusters]
  [Fail] [sig-storage] Subpath [Volume type: hostPath] [It] should fail if non-existent subpath is outside the volume [Slow]

[oomichi]

• v1.11.3 (Above + LBテストスキップ対応 2018/11/05): 6 / 432 failures 1.3%, 7h26m

  [Fail] [k8s.io] [sig-node] Kubelet [Serial] [Slow] [k8s.io] [sig-node] regular resource usage tracking [It] resource tracking for 0 pods per node
  [Fail] [k8s.io] [sig-node] Kubelet [Serial] [Slow] [k8s.io] [sig-node] regular resource usage tracking [It] resource tracking for 100 pods per node
  [Fail] [sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: CPU) [sig-autoscaling] [Serial] [Slow] ReplicationController [It] Should scale from 1 pod to 3 pods and from 3 to 5 and verify decision stability
  [Fail] [sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: CPU) [sig-autoscaling] [Serial] [Slow] ReplicationController [It] Should scale from 5 pods to 3 pods and from 3 to 1 and verify decision stability
  [Fail] [sig-instrumentation] MetricsGrabber [It] should grab all metrics from a ControllerManager.
  [Fail] [sig-instrumentation] MetricsGrabber [It] should grab all metrics from a Scheduler.

[oomichi]

v1.11.3 (Above + HPA 安定化パッチバックポート対応 2018/11/07): 4 / 432 failures 0.9%, 7h50m

[Fail] [k8s.io] [sig-node] Kubelet [Serial] [Slow] [k8s.io] [sig-node] regular resource usage tracking [It] resource tracking for 0 pods per node
[Fail] [k8s.io] [sig-node] Kubelet [Serial] [Slow] [k8s.io] [sig-node] regular resource usage tracking [It] resource tracking for 100 pods per node
[Fail] [sig-instrumentation] MetricsGrabber [It] should grab all metrics from a ControllerManager.
[Fail] [sig-instrumentation] MetricsGrabber [It] should grab all metrics from a Scheduler.

[oomichi]

regular resource usage tracking については、実行ノードの性能が必要なのでスキップとした。