search

kubernetes-sigs / hydrophone

Hydrophone is a lightweight Kubernetes conformance tests runner
Apache License 2.0
65 stars 30 forks source link

Count of conformance tests not matching what is expected (380 vs 388) #42

Closed dims closed 9 months ago

dims commented 9 months ago

https://github.com/kubernetes/kubernetes/blob/master/test/conformance/testdata/conformance.yaml shows 388 tests. we seem to be running 380 only

dims commented 9 months ago

Count confirmed here: https://cs.k8s.io/?q=framework%5C.ConformanceIt&i=nope&files=%5Etest%5C%2Fe2e%5C%2F.*%5C.go&excludeFiles=&repos=kubernetes/kubernetes

rjsadow commented 9 months ago

Here are the 388 tests we're expecting to cover

[sig-api-machinery] API priority and fairness should support FlowSchema API operations [Conformance]
[sig-api-machinery] API priority and fairness should support PriorityLevelConfiguration API operations [Conformance]
[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] listing mutating webhooks should work [Conformance]
[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] listing validating webhooks should work [Conformance]
[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] patching/updating a mutating webhook should work [Conformance]
[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] patching/updating a validating webhook should work [Conformance]
[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should be able to deny attaching pod [Conformance]
[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should be able to deny custom resource creation, update and deletion [Conformance]
[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should be able to deny pod and configmap creation [Conformance]
[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should deny crd creation [Conformance]
[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should honor timeout [Conformance]
[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should include webhook resources in discovery documents [Conformance]
[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should mutate configmap [Conformance]
[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should mutate custom resource [Conformance]
[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should mutate custom resource with different stored version [Conformance]
[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should mutate custom resource with pruning [Conformance]
[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should mutate pod and apply defaults after mutation [Conformance]
[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should not be able to mutate or prevent deletion of webhook configuration objects [Conformance]
[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should unconditionally reject operations on fail closed webhook [Conformance]
[sig-api-machinery] Aggregator Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance]
[sig-api-machinery] CustomResourceConversionWebhook [Privileged:ClusterAdmin] should be able to convert a non homogeneous list of CRs [Conformance]
[sig-api-machinery] CustomResourceConversionWebhook [Privileged:ClusterAdmin] should be able to convert from CR v1 to CR v2 [Conformance]
[sig-api-machinery] CustomResourceDefinition Watch [Privileged:ClusterAdmin] CustomResourceDefinition Watch watch on custom resource definition objects [Conformance]
[sig-api-machinery] CustomResourceDefinition resources [Privileged:ClusterAdmin] Simple CustomResourceDefinition creating/deleting custom resource definition objects works [Conformance]
[sig-api-machinery] CustomResourceDefinition resources [Privileged:ClusterAdmin] Simple CustomResourceDefinition getting/updating/patching custom resource definition status sub-resource works [Conformance]
[sig-api-machinery] CustomResourceDefinition resources [Privileged:ClusterAdmin] Simple CustomResourceDefinition listing custom resource definition objects works [Conformance]
[sig-api-machinery] CustomResourceDefinition resources [Privileged:ClusterAdmin] custom resource defaulting for requests and from storage works [Conformance]
[sig-api-machinery] CustomResourceDefinition resources [Privileged:ClusterAdmin] should include custom resource definition resources in discovery documents [Conformance]
[sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin] removes definition from spec when one version gets changed to not be served [Conformance]
[sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin] updates the published spec when one version gets renamed [Conformance]
[sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin] works for CRD preserving unknown fields at the schema root [Conformance]
[sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin] works for CRD preserving unknown fields in an embedded object [Conformance]
[sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin] works for CRD with validation schema [Conformance]
[sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin] works for CRD without validation schema [Conformance]
[sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin] works for multiple CRDs of different groups [Conformance]
[sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin] works for multiple CRDs of same group and version but different kinds [Conformance]
[sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin] works for multiple CRDs of same group but different versions [Conformance]
[sig-api-machinery] Discovery should locate the groupVersion and a resource within each APIGroup [Conformance]
[sig-api-machinery] Discovery should validate PreferredVersion for each APIGroup [Conformance]
[sig-api-machinery] FieldValidation should create/apply a CR with unknown fields for CRD with no validation schema [Conformance]
[sig-api-machinery] FieldValidation should create/apply a valid CR for CRD with validation schema [Conformance]
[sig-api-machinery] FieldValidation should create/apply an invalid CR with extra properties for CRD with validation schema [Conformance]
[sig-api-machinery] FieldValidation should detect duplicates in a CR when preserving unknown fields [Conformance]
[sig-api-machinery] FieldValidation should detect unknown and duplicate fields of a typed object [Conformance]
[sig-api-machinery] FieldValidation should detect unknown metadata fields in both the root and embedded object of a CR [Conformance]
[sig-api-machinery] FieldValidation should detect unknown metadata fields of a typed object [Conformance]
[sig-api-machinery] Garbage collector should delete RS created by deployment when not orphaning [Conformance]
[sig-api-machinery] Garbage collector should delete pods created by rc when not orphaning [Conformance]
[sig-api-machinery] Garbage collector should keep the rc around until all its pods are deleted if the deleteOptions says so [Conformance]
[sig-api-machinery] Garbage collector should not be blocked by dependency circle [Conformance]
[sig-api-machinery] Garbage collector should not delete dependents that have both valid owner and owner that's waiting for dependents to be deleted [Conformance]
[sig-api-machinery] Garbage collector should orphan RS created by deployment when deleteOptions.PropagationPolicy is Orphan [Conformance]
[sig-api-machinery] Garbage collector should orphan pods created by rc if delete options say so [Conformance]
[sig-api-machinery] Namespaces [Serial] should apply a finalizer to a Namespace [Conformance]
[sig-api-machinery] Namespaces [Serial] should apply an update to a Namespace [Conformance]
[sig-api-machinery] Namespaces [Serial] should apply changes to a namespace status [Conformance]
[sig-api-machinery] Namespaces [Serial] should ensure that all pods are removed when a namespace is deleted [Conformance]
[sig-api-machinery] Namespaces [Serial] should ensure that all services are removed when a namespace is deleted [Conformance]
[sig-api-machinery] Namespaces [Serial] should patch a Namespace [Conformance]
[sig-api-machinery] ResourceQuota should apply changes to a resourcequota status [Conformance]
[sig-api-machinery] ResourceQuota should be able to update and delete ResourceQuota. [Conformance]
[sig-api-machinery] ResourceQuota should create a ResourceQuota and capture the life of a configMap. [Conformance]
[sig-api-machinery] ResourceQuota should create a ResourceQuota and capture the life of a pod. [Conformance]
[sig-api-machinery] ResourceQuota should create a ResourceQuota and capture the life of a replica set. [Conformance]
[sig-api-machinery] ResourceQuota should create a ResourceQuota and capture the life of a replication controller. [Conformance]
[sig-api-machinery] ResourceQuota should create a ResourceQuota and capture the life of a secret. [Conformance]
[sig-api-machinery] ResourceQuota should create a ResourceQuota and capture the life of a service. [Conformance]
[sig-api-machinery] ResourceQuota should create a ResourceQuota and ensure its status is promptly calculated. [Conformance]
[sig-api-machinery] ResourceQuota should manage the lifecycle of a ResourceQuota [Conformance]
[sig-api-machinery] ResourceQuota should verify ResourceQuota with best effort scope. [Conformance]
[sig-api-machinery] ResourceQuota should verify ResourceQuota with terminating scopes. [Conformance]
[sig-api-machinery] Servers with support for API chunking should return chunks of results for list calls [Conformance]
[sig-api-machinery] Servers with support for API chunking should support continue listing from the last key if the original version has been compacted away, though the list is inconsistent [Slow] [Conformance]
[sig-api-machinery] Servers with support for Table transformation should return a 406 for a backend which does not implement metadata [Conformance]
[sig-api-machinery] Watchers should be able to restart watching from the last resource version observed by the previous watch [Conformance]
[sig-api-machinery] Watchers should be able to start watching from a specific resource version [Conformance]
[sig-api-machinery] Watchers should observe add, update, and delete watch notifications on configmaps [Conformance]
[sig-api-machinery] Watchers should observe an object deletion if it stops meeting the requirements of the selector [Conformance]
[sig-api-machinery] Watchers should receive events on concurrent watches in same order [Conformance]
[sig-api-machinery] server version should find the server version [Conformance]
[sig-apps] ControllerRevision [Serial] should manage the lifecycle of a ControllerRevision [Conformance]
[sig-apps] CronJob should not schedule jobs when suspended [Slow] [Conformance]
[sig-apps] CronJob should not schedule new jobs when ForbidConcurrent [Slow] [Conformance]
[sig-apps] CronJob should replace jobs when ReplaceConcurrent [Conformance]
[sig-apps] CronJob should schedule multiple jobs concurrently [Conformance]
[sig-apps] CronJob should support CronJob API operations [Conformance]
[sig-apps] Daemon set [Serial] should list and delete a collection of DaemonSets [Conformance]
[sig-apps] Daemon set [Serial] should retry creating failed daemon pods [Conformance]
[sig-apps] Daemon set [Serial] should rollback without unnecessary restarts [Conformance]
[sig-apps] Daemon set [Serial] should run and stop complex daemon [Conformance]
[sig-apps] Daemon set [Serial] should run and stop simple daemon [Conformance]
[sig-apps] Daemon set [Serial] should update pod when spec was updated and update strategy is RollingUpdate [Conformance]
[sig-apps] Daemon set [Serial] should verify changes to a daemon set status [Conformance]
[sig-apps] Deployment Deployment should have a working scale subresource [Conformance]
[sig-apps] Deployment RecreateDeployment should delete old pods and create new ones [Conformance]
[sig-apps] Deployment RollingUpdateDeployment should delete old pods and create new ones [Conformance]
[sig-apps] Deployment deployment should delete old replica sets [Conformance]
[sig-apps] Deployment deployment should support proportional scaling [Conformance]
[sig-apps] Deployment deployment should support rollover [Conformance]
[sig-apps] Deployment should run the lifecycle of a Deployment [Conformance]
[sig-apps] Deployment should validate Deployment Status endpoints [Conformance]
[sig-apps] DisruptionController Listing PodDisruptionBudgets for all namespaces should list and delete a collection of PodDisruptionBudgets [Conformance]
[sig-apps] DisruptionController should block an eviction until the PDB is updated to allow it [Conformance]
[sig-apps] DisruptionController should create a PodDisruptionBudget [Conformance]
[sig-apps] DisruptionController should observe PodDisruptionBudget status updated [Conformance]
[sig-apps] DisruptionController should update/patch PodDisruptionBudget status [Conformance]
[sig-apps] Job should adopt matching orphans and release non-matching pods [Conformance]
[sig-apps] Job should apply changes to a job status [Conformance]
[sig-apps] Job should create pods for an Indexed job with completion indexes and specified hostname [Conformance]
[sig-apps] Job should delete a job [Conformance]
[sig-apps] Job should manage the lifecycle of a job [Conformance]
[sig-apps] Job should run a job to completion when tasks sometimes fail and are locally restarted [Conformance]
[sig-apps] ReplicaSet Replace and Patch tests [Conformance]
[sig-apps] ReplicaSet Replicaset should have a working scale subresource [Conformance]
[sig-apps] ReplicaSet should adopt matching pods on creation and release no longer matching pods [Conformance]
[sig-apps] ReplicaSet should list and delete a collection of ReplicaSets [Conformance]
[sig-apps] ReplicaSet should serve a basic image on each replica with a public image [Conformance]
[sig-apps] ReplicaSet should validate Replicaset Status endpoints [Conformance]
[sig-apps] ReplicationController should adopt matching pods on creation [Conformance]
[sig-apps] ReplicationController should get and update a ReplicationController scale [Conformance]
[sig-apps] ReplicationController should release no longer matching pods [Conformance]
[sig-apps] ReplicationController should serve a basic image on each replica with a public image [Conformance]
[sig-apps] ReplicationController should surface a failure condition on a common issue like exceeded quota [Conformance]
[sig-apps] ReplicationController should test the lifecycle of a ReplicationController [Conformance]
[sig-apps] StatefulSet Basic StatefulSet functionality [StatefulSetBasic] Burst scaling should run to completion even with unhealthy pods [Slow] [Conformance]
[sig-apps] StatefulSet Basic StatefulSet functionality [StatefulSetBasic] Scaling should happen in predictable order and halt if any stateful pod is unhealthy [Slow] [Conformance]
[sig-apps] StatefulSet Basic StatefulSet functionality [StatefulSetBasic] Should recreate evicted statefulset [Conformance]
[sig-apps] StatefulSet Basic StatefulSet functionality [StatefulSetBasic] should have a working scale subresource [Conformance]
[sig-apps] StatefulSet Basic StatefulSet functionality [StatefulSetBasic] should list, patch and delete a collection of StatefulSets [Conformance]
[sig-apps] StatefulSet Basic StatefulSet functionality [StatefulSetBasic] should perform canary updates and phased rolling updates of template modifications [Conformance]
[sig-apps] StatefulSet Basic StatefulSet functionality [StatefulSetBasic] should perform rolling updates and roll backs of template modifications [Conformance]
[sig-apps] StatefulSet Basic StatefulSet functionality [StatefulSetBasic] should validate Statefulset Status endpoints [Conformance]
[sig-architecture] Conformance Tests should have at least two untainted nodes [Conformance]
[sig-auth] Certificates API [Privileged:ClusterAdmin] should support CSR API operations [Conformance]
[sig-auth] ServiceAccounts ServiceAccountIssuerDiscovery should support OIDC discovery of service account issuer [Conformance]
[sig-auth] ServiceAccounts should allow opting out of API token automount [Conformance]
[sig-auth] ServiceAccounts should guarantee kube-root-ca.crt exist in any namespace [Conformance]
[sig-auth] ServiceAccounts should mount an API token into pods [Conformance]
[sig-auth] ServiceAccounts should mount projected service account token [Conformance]
[sig-auth] ServiceAccounts should run through the lifecycle of a ServiceAccount [Conformance]
[sig-auth] ServiceAccounts should update a ServiceAccount [Conformance]
[sig-auth] SubjectReview should support SubjectReview API operations [Conformance]
[sig-cli] Kubectl client Guestbook application should create and stop a working application [Conformance]
[sig-cli] Kubectl client Kubectl api-versions should check if v1 is in available api versions [Conformance]
[sig-cli] Kubectl client Kubectl cluster-info should check if Kubernetes control plane services is included in cluster-info [Conformance]
[sig-cli] Kubectl client Kubectl describe should check if kubectl describe prints relevant information for rc and pods [Conformance]
[sig-cli] Kubectl client Kubectl diff should check if kubectl diff finds a difference for Deployments [Conformance]
[sig-cli] Kubectl client Kubectl expose should create services for rc [Conformance]
[sig-cli] Kubectl client Kubectl label should update the label on a resource [Conformance]
[sig-cli] Kubectl client Kubectl patch should add annotations for pods in rc [Conformance]
[sig-cli] Kubectl client Kubectl replace should update a single-container pod's image [Conformance]
[sig-cli] Kubectl client Kubectl run pod should create a pod from an image when restart is Never [Conformance]
[sig-cli] Kubectl client Kubectl server-side dry-run should check if kubectl can dry-run update Pods [Conformance]
[sig-cli] Kubectl client Kubectl version should check is all data is printed [Conformance]
[sig-cli] Kubectl client Proxy server should support --unix-socket=/path [Conformance]
[sig-cli] Kubectl client Proxy server should support proxy with --port 0 [Conformance]
[sig-cli] Kubectl client Update Demo should create and stop a replication controller [Conformance]
[sig-cli] Kubectl client Update Demo should scale a replication controller [Conformance]
[sig-cli] Kubectl logs logs should be able to retrieve and filter logs [Conformance]
[sig-instrumentation] Events API should delete a collection of events [Conformance]
[sig-instrumentation] Events API should ensure that an event can be fetched, patched, deleted, and listed [Conformance]
[sig-instrumentation] Events should delete a collection of events [Conformance]
[sig-instrumentation] Events should manage the lifecycle of an event [Conformance]
[sig-network] DNS should provide /etc/hosts entries for the cluster [Conformance]
[sig-network] DNS should provide DNS for ExternalName services [Conformance]
[sig-network] DNS should provide DNS for pods for Hostname [Conformance]
[sig-network] DNS should provide DNS for pods for Subdomain [Conformance]
[sig-network] DNS should provide DNS for services [Conformance]
[sig-network] DNS should provide DNS for the cluster [Conformance]
[sig-network] DNS should resolve DNS of partial qualified names for services [LinuxOnly] [Conformance]
[sig-network] DNS should support configurable pod DNS nameservers [Conformance]
[sig-network] EndpointSlice should create Endpoints and EndpointSlices for Pods matching a Service [Conformance]
[sig-network] EndpointSlice should create and delete Endpoints and EndpointSlices for a Service with a selector specified [Conformance]
[sig-network] EndpointSlice should have Endpoints and EndpointSlices pointing to API Server [Conformance]
[sig-network] EndpointSlice should support creating EndpointSlice API operations [Conformance]
[sig-network] EndpointSliceMirroring should mirror a custom Endpoints resource through create update and delete [Conformance]
[sig-network] HostPort validates that there is no conflict between pods with same hostPort but different hostIP and protocol [LinuxOnly] [Conformance]
[sig-network] Ingress API should support creating Ingress API operations [Conformance]
[sig-network] IngressClass API should support creating IngressClass API operations [Conformance]
[sig-network] Networking Granular Checks: Pods should function for intra-pod communication: http [NodeConformance] [Conformance]
[sig-network] Networking Granular Checks: Pods should function for intra-pod communication: udp [NodeConformance] [Conformance]
[sig-network] Networking Granular Checks: Pods should function for node-pod communication: http [LinuxOnly] [NodeConformance] [Conformance]
[sig-network] Networking Granular Checks: Pods should function for node-pod communication: udp [LinuxOnly] [NodeConformance] [Conformance]
[sig-network] Proxy version v1 A set of valid responses are returned for both pod and service Proxy [Conformance]
[sig-network] Proxy version v1 A set of valid responses are returned for both pod and service ProxyWithPath [Conformance]
[sig-network] Proxy version v1 should proxy through a service and a pod [Conformance]
[sig-network] Service endpoints latency should not be very high [Conformance]
[sig-network] Services should be able to change the type from ClusterIP to ExternalName [Conformance]
[sig-network] Services should be able to change the type from ExternalName to ClusterIP [Conformance]
[sig-network] Services should be able to change the type from ExternalName to NodePort [Conformance]
[sig-network] Services should be able to change the type from NodePort to ExternalName [Conformance]
[sig-network] Services should be able to create a functioning NodePort service [Conformance]
[sig-network] Services should be able to switch session affinity for NodePort service [LinuxOnly] [Conformance]
[sig-network] Services should be able to switch session affinity for service with type clusterIP [LinuxOnly] [Conformance]
[sig-network] Services should complete a service status lifecycle [Conformance]
[sig-network] Services should delete a collection of services [Conformance]
[sig-network] Services should find a service from listing all namespaces [Conformance]
[sig-network] Services should have session affinity work for NodePort service [LinuxOnly] [Conformance]
[sig-network] Services should have session affinity work for service with type clusterIP [LinuxOnly] [Conformance]
[sig-network] Services should provide secure master service [Conformance]
[sig-network] Services should serve a basic endpoint from pods [Conformance]
[sig-network] Services should serve endpoints on same port and different protocols [Conformance]
[sig-network] Services should serve multiport endpoints from pods [Conformance]
[sig-network] Services should test the lifecycle of an Endpoint [Conformance]
[sig-node] ConfigMap should be consumable via environment variable [NodeConformance] [Conformance]
[sig-node] ConfigMap should be consumable via the environment [NodeConformance] [Conformance]
[sig-node] ConfigMap should fail to create ConfigMap with empty key [Conformance]
[sig-node] ConfigMap should run through a ConfigMap lifecycle [Conformance]
[sig-node] Container Lifecycle Hook when create a pod with lifecycle hook should execute poststart exec hook properly [NodeConformance] [Conformance]
[sig-node] Container Lifecycle Hook when create a pod with lifecycle hook should execute poststart http hook properly [NodeConformance] [Conformance]
[sig-node] Container Lifecycle Hook when create a pod with lifecycle hook should execute prestop exec hook properly [NodeConformance] [Conformance]
[sig-node] Container Lifecycle Hook when create a pod with lifecycle hook should execute prestop http hook properly [NodeConformance] [Conformance]
[sig-node] Container Runtime blackbox test on terminated container should report termination message as empty when pod succeeds and TerminationMessagePolicy FallbackToLogsOnError is set [NodeConformance] [Conformance]
[sig-node] Container Runtime blackbox test on terminated container should report termination message from file when pod succeeds and TerminationMessagePolicy FallbackToLogsOnError is set [NodeConformance] [Conformance]
[sig-node] Container Runtime blackbox test on terminated container should report termination message from log output if TerminationMessagePolicy FallbackToLogsOnError is set [NodeConformance] [Conformance]
[sig-node] Container Runtime blackbox test on terminated container should report termination message if TerminationMessagePath is set as non-root user and at a non-default path [NodeConformance] [Conformance]
[sig-node] Container Runtime blackbox test when starting a container that exits should run with the expected status [NodeConformance] [Conformance]
[sig-node] Containers should be able to override the image's default arguments (container cmd) [NodeConformance] [Conformance]
[sig-node] Containers should be able to override the image's default command (container entrypoint) [NodeConformance] [Conformance]
[sig-node] Containers should be able to override the image's default command and arguments [NodeConformance] [Conformance]
[sig-node] Containers should use the image defaults if command and args are blank [NodeConformance] [Conformance]
[sig-node] Downward API should provide container's limits.cpu/memory and requests.cpu/memory as env vars [NodeConformance] [Conformance]
[sig-node] Downward API should provide default limits.cpu/memory from node allocatable [NodeConformance] [Conformance]
[sig-node] Downward API should provide host IP as an env var [NodeConformance] [Conformance]
[sig-node] Downward API should provide pod UID as env vars [NodeConformance] [Conformance]
[sig-node] Downward API should provide pod name, namespace and IP address as env vars [NodeConformance] [Conformance]
[sig-node] Ephemeral Containers [NodeConformance] should update the ephemeral containers in an existing pod [Conformance]
[sig-node] Ephemeral Containers [NodeConformance] will start an ephemeral container in an existing pod [Conformance]
[sig-node] InitContainer [NodeConformance] should invoke init containers on a RestartAlways pod [Conformance]
[sig-node] InitContainer [NodeConformance] should invoke init containers on a RestartNever pod [Conformance]
[sig-node] InitContainer [NodeConformance] should not start app containers and fail the pod if init containers fail on a RestartNever pod [Conformance]
[sig-node] InitContainer [NodeConformance] should not start app containers if init containers fail on a RestartAlways pod [Conformance]
[sig-node] Kubelet when scheduling a busybox command in a pod should print the output to logs [NodeConformance] [Conformance]
[sig-node] Kubelet when scheduling a busybox command that always fails in a pod should be possible to delete [NodeConformance] [Conformance]
[sig-node] Kubelet when scheduling a busybox command that always fails in a pod should have an terminated reason [NodeConformance] [Conformance]
[sig-node] Kubelet when scheduling a read only busybox container should not write to root filesystem [LinuxOnly] [NodeConformance] [Conformance]
[sig-node] Kubelet when scheduling an agnhost Pod with hostAliases should write entries to /etc/hosts [NodeConformance] [Conformance]
[sig-node] KubeletManagedEtcHosts should test kubelet managed /etc/hosts file [LinuxOnly] [NodeConformance] [Conformance]
[sig-node] Lease lease API should be available [Conformance]
[sig-node] NoExecuteTaintManager Multiple Pods [Serial] evicts pods with minTolerationSeconds [Disruptive] [Conformance]
[sig-node] NoExecuteTaintManager Single Pod [Serial] removing taint cancels eviction [Disruptive] [Conformance]
[sig-node] PodTemplates should delete a collection of pod templates [Conformance]
[sig-node] PodTemplates should replace a pod template [Conformance]
[sig-node] PodTemplates should run the lifecycle of PodTemplates [Conformance]
[sig-node] Pods Extended Pods Set QOS Class should be set on Pods with matching resource requests and limits for memory and cpu [Conformance]
[sig-node] Pods should allow activeDeadlineSeconds to be updated [NodeConformance] [Conformance]
[sig-node] Pods should be submitted and removed [NodeConformance] [Conformance]
[sig-node] Pods should be updated [NodeConformance] [Conformance]
[sig-node] Pods should contain environment variables for services [NodeConformance] [Conformance]
[sig-node] Pods should delete a collection of pods [Conformance]
[sig-node] Pods should get a host IP [NodeConformance] [Conformance]
[sig-node] Pods should patch a pod status [Conformance]
[sig-node] Pods should run through the lifecycle of Pods and PodStatus [Conformance]
[sig-node] Pods should support remote command execution over websockets [NodeConformance] [Conformance]
[sig-node] Pods should support retrieving logs from the container over websockets [NodeConformance] [Conformance]
[sig-node] PreStop should call prestop when killing a pod [Conformance]
[sig-node] Probing container should *not* be restarted with a /healthz http liveness probe [NodeConformance] [Conformance]
[sig-node] Probing container should *not* be restarted with a GRPC liveness probe [NodeConformance] [Conformance]
[sig-node] Probing container should *not* be restarted with a exec "cat /tmp/health" liveness probe [NodeConformance] [Conformance]
[sig-node] Probing container should *not* be restarted with a tcp:8080 liveness probe [NodeConformance] [Conformance]
[sig-node] Probing container should be restarted with a /healthz http liveness probe [NodeConformance] [Conformance]
[sig-node] Probing container should be restarted with a GRPC liveness probe [NodeConformance] [Conformance]
[sig-node] Probing container should be restarted with a exec "cat /tmp/health" liveness probe [NodeConformance] [Conformance]
[sig-node] Probing container should have monotonically increasing restart count [NodeConformance] [Conformance]
[sig-node] Probing container with readiness probe should not be ready before initial delay and never restart [NodeConformance] [Conformance]
[sig-node] Probing container with readiness probe that fails should never be ready and never restart [NodeConformance] [Conformance]
[sig-node] RuntimeClass should reject a Pod requesting a deleted RuntimeClass [NodeConformance] [Conformance]
[sig-node] RuntimeClass should reject a Pod requesting a non-existent RuntimeClass [NodeConformance] [Conformance]
[sig-node] RuntimeClass should schedule a Pod requesting a RuntimeClass and initialize its Overhead [NodeConformance] [Conformance]
[sig-node] RuntimeClass should schedule a Pod requesting a RuntimeClass without PodOverhead [NodeConformance] [Conformance]
[sig-node] RuntimeClass should support RuntimeClasses API operations [Conformance]
[sig-node] Secrets should be consumable from pods in env vars [NodeConformance] [Conformance]
[sig-node] Secrets should be consumable via the environment [NodeConformance] [Conformance]
[sig-node] Secrets should fail to create secret due to empty secret key [Conformance]
[sig-node] Secrets should patch a secret [Conformance]
[sig-node] Security Context When creating a container with runAsUser should run the container with uid 65534 [LinuxOnly] [NodeConformance] [Conformance]
[sig-node] Security Context When creating a pod with privileged should run the container as unprivileged when false [LinuxOnly] [NodeConformance] [Conformance]
[sig-node] Security Context When creating a pod with readOnlyRootFilesystem should run the container with writable rootfs when readOnlyRootFilesystem=false [NodeConformance] [Conformance]
[sig-node] Security Context should support container.SecurityContext.RunAsUser And container.SecurityContext.RunAsGroup [LinuxOnly] [Conformance]
[sig-node] Security Context should support pod.Spec.SecurityContext.RunAsUser And pod.Spec.SecurityContext.RunAsGroup [LinuxOnly] [Conformance]
[sig-node] Security Context when creating containers with AllowPrivilegeEscalation should not allow privilege escalation when false [LinuxOnly] [NodeConformance] [Conformance]
[sig-node] Sysctls [LinuxOnly] [NodeConformance] should reject invalid sysctls [MinimumKubeletVersion:1.21] [Conformance]
[sig-node] Sysctls [LinuxOnly] [NodeConformance] should support sysctls [MinimumKubeletVersion:1.21] [Environment:NotInUserNS] [Conformance]
[sig-node] Variable Expansion should allow composing env vars into new env vars [NodeConformance] [Conformance]
[sig-node] Variable Expansion should allow substituting values in a container's args [NodeConformance] [Conformance]
[sig-node] Variable Expansion should allow substituting values in a container's command [NodeConformance] [Conformance]
[sig-node] Variable Expansion should allow substituting values in a volume subpath [Conformance]
[sig-node] Variable Expansion should fail substituting values in a volume subpath with absolute path [Slow] [Conformance]
[sig-node] Variable Expansion should fail substituting values in a volume subpath with backticks [Slow] [Conformance]
[sig-node] Variable Expansion should succeed in writing subpaths in container [Slow] [Conformance]
[sig-node] Variable Expansion should verify that a failing subpath expansion can be modified during the lifecycle of a container [Slow] [Conformance]
[sig-scheduling] LimitRange should create a LimitRange with defaults and ensure pod has those defaults applied. [Conformance]
[sig-scheduling] LimitRange should list, patch and delete a LimitRange by collection [Conformance]
[sig-scheduling] SchedulerPredicates [Serial] validates resource limits of pods that are allowed to run [Conformance]
[sig-scheduling] SchedulerPredicates [Serial] validates that NodeSelector is respected if matching [Conformance]
[sig-scheduling] SchedulerPredicates [Serial] validates that NodeSelector is respected if not matching [Conformance]
[sig-scheduling] SchedulerPredicates [Serial] validates that there exists conflict between pods with same hostPort and protocol but one using 0.0.0.0 hostIP [Conformance]
[sig-scheduling] SchedulerPreemption [Serial] PreemptionExecutionPath runs ReplicaSets to verify preemption running path [Conformance]
[sig-scheduling] SchedulerPreemption [Serial] PriorityClass endpoints verify PriorityClass endpoints can be operated with different HTTP methods [Conformance]
[sig-scheduling] SchedulerPreemption [Serial] validates basic preemption works [Conformance]
[sig-scheduling] SchedulerPreemption [Serial] validates lower priority pod preemption by critical pod [Conformance]
[sig-storage] CSIInlineVolumes should run through the lifecycle of a CSIDriver [Conformance]
[sig-storage] CSIInlineVolumes should support CSIVolumeSource in Pod API [Conformance]
[sig-storage] CSIStorageCapacity should support CSIStorageCapacities API operations [Conformance]
[sig-storage] ConfigMap binary data should be reflected in volume [NodeConformance] [Conformance]
[sig-storage] ConfigMap optional updates should be reflected in volume [NodeConformance] [Conformance]
[sig-storage] ConfigMap should be consumable from pods in volume [NodeConformance] [Conformance]
[sig-storage] ConfigMap should be consumable from pods in volume as non-root [NodeConformance] [Conformance]
[sig-storage] ConfigMap should be consumable from pods in volume with defaultMode set [LinuxOnly] [NodeConformance] [Conformance]
[sig-storage] ConfigMap should be consumable from pods in volume with mappings [NodeConformance] [Conformance]
[sig-storage] ConfigMap should be consumable from pods in volume with mappings and Item mode set [LinuxOnly] [NodeConformance] [Conformance]
[sig-storage] ConfigMap should be consumable from pods in volume with mappings as non-root [NodeConformance] [Conformance]
[sig-storage] ConfigMap should be consumable in multiple volumes in the same pod [NodeConformance] [Conformance]
[sig-storage] ConfigMap should be immutable if `immutable` field is set [Conformance]
[sig-storage] ConfigMap updates should be reflected in volume [NodeConformance] [Conformance]
[sig-storage] Downward API volume should provide container's cpu limit [NodeConformance] [Conformance]
[sig-storage] Downward API volume should provide container's cpu request [NodeConformance] [Conformance]
[sig-storage] Downward API volume should provide container's memory limit [NodeConformance] [Conformance]
[sig-storage] Downward API volume should provide container's memory request [NodeConformance] [Conformance]
[sig-storage] Downward API volume should provide node allocatable (cpu) as default cpu limit if the limit is not set [NodeConformance] [Conformance]
[sig-storage] Downward API volume should provide node allocatable (memory) as default memory limit if the limit is not set [NodeConformance] [Conformance]
[sig-storage] Downward API volume should provide podname only [NodeConformance] [Conformance]
[sig-storage] Downward API volume should set DefaultMode on files [LinuxOnly] [NodeConformance] [Conformance]
[sig-storage] Downward API volume should set mode on item file [LinuxOnly] [NodeConformance] [Conformance]
[sig-storage] Downward API volume should update annotations on modification [NodeConformance] [Conformance]
[sig-storage] Downward API volume should update labels on modification [NodeConformance] [Conformance]
[sig-storage] EmptyDir volumes pod should support shared volumes between containers [Conformance]
[sig-storage] EmptyDir volumes should support (non-root,0644,default) [LinuxOnly] [NodeConformance] [Conformance]
[sig-storage] EmptyDir volumes should support (non-root,0644,tmpfs) [LinuxOnly] [NodeConformance] [Conformance]
[sig-storage] EmptyDir volumes should support (non-root,0666,default) [LinuxOnly] [NodeConformance] [Conformance]
[sig-storage] EmptyDir volumes should support (non-root,0666,tmpfs) [LinuxOnly] [NodeConformance] [Conformance]
[sig-storage] EmptyDir volumes should support (non-root,0777,default) [LinuxOnly] [NodeConformance] [Conformance]
[sig-storage] EmptyDir volumes should support (non-root,0777,tmpfs) [LinuxOnly] [NodeConformance] [Conformance]
[sig-storage] EmptyDir volumes should support (root,0644,default) [LinuxOnly] [NodeConformance] [Conformance]
[sig-storage] EmptyDir volumes should support (root,0644,tmpfs) [LinuxOnly] [NodeConformance] [Conformance]
[sig-storage] EmptyDir volumes should support (root,0666,default) [LinuxOnly] [NodeConformance] [Conformance]
[sig-storage] EmptyDir volumes should support (root,0666,tmpfs) [LinuxOnly] [NodeConformance] [Conformance]
[sig-storage] EmptyDir volumes should support (root,0777,default) [LinuxOnly] [NodeConformance] [Conformance]
[sig-storage] EmptyDir volumes should support (root,0777,tmpfs) [LinuxOnly] [NodeConformance] [Conformance]
[sig-storage] EmptyDir volumes volume on default medium should have the correct mode [LinuxOnly] [NodeConformance] [Conformance]
[sig-storage] EmptyDir volumes volume on tmpfs should have the correct mode [LinuxOnly] [NodeConformance] [Conformance]
[sig-storage] EmptyDir wrapper volumes should not cause race condition when used for configmaps [Serial] [Conformance]
[sig-storage] EmptyDir wrapper volumes should not conflict [Conformance]
[sig-storage] PersistentVolumes CSI Conformance should apply changes to a pv/pvc status [Conformance]
[sig-storage] PersistentVolumes CSI Conformance should run through the lifecycle of a PV and a PVC [Conformance]
[sig-storage] Projected combined should project all components that make up the projection API [Projection] [NodeConformance] [Conformance]
[sig-storage] Projected configMap optional updates should be reflected in volume [NodeConformance] [Conformance]
[sig-storage] Projected configMap should be consumable from pods in volume [NodeConformance] [Conformance]
[sig-storage] Projected configMap should be consumable from pods in volume as non-root [NodeConformance] [Conformance]
[sig-storage] Projected configMap should be consumable from pods in volume with defaultMode set [LinuxOnly] [NodeConformance] [Conformance]
[sig-storage] Projected configMap should be consumable from pods in volume with mappings [NodeConformance] [Conformance]
[sig-storage] Projected configMap should be consumable from pods in volume with mappings and Item mode set [LinuxOnly] [NodeConformance] [Conformance]
[sig-storage] Projected configMap should be consumable from pods in volume with mappings as non-root [NodeConformance] [Conformance]
[sig-storage] Projected configMap should be consumable in multiple volumes in the same pod [NodeConformance] [Conformance]
[sig-storage] Projected configMap updates should be reflected in volume [NodeConformance] [Conformance]
[sig-storage] Projected downwardAPI should provide container's cpu limit [NodeConformance] [Conformance]
[sig-storage] Projected downwardAPI should provide container's cpu request [NodeConformance] [Conformance]
[sig-storage] Projected downwardAPI should provide container's memory limit [NodeConformance] [Conformance]
[sig-storage] Projected downwardAPI should provide container's memory request [NodeConformance] [Conformance]
[sig-storage] Projected downwardAPI should provide node allocatable (cpu) as default cpu limit if the limit is not set [NodeConformance] [Conformance]
[sig-storage] Projected downwardAPI should provide node allocatable (memory) as default memory limit if the limit is not set [NodeConformance] [Conformance]
[sig-storage] Projected downwardAPI should provide podname only [NodeConformance] [Conformance]
[sig-storage] Projected downwardAPI should set DefaultMode on files [LinuxOnly] [NodeConformance] [Conformance]
[sig-storage] Projected downwardAPI should set mode on item file [LinuxOnly] [NodeConformance] [Conformance]
[sig-storage] Projected downwardAPI should update annotations on modification [NodeConformance] [Conformance]
[sig-storage] Projected downwardAPI should update labels on modification [NodeConformance] [Conformance]
[sig-storage] Projected secret optional updates should be reflected in volume [NodeConformance] [Conformance]
[sig-storage] Projected secret should be consumable from pods in volume [NodeConformance] [Conformance]
[sig-storage] Projected secret should be consumable from pods in volume as non-root with defaultMode and fsGroup set [LinuxOnly] [NodeConformance] [Conformance]
[sig-storage] Projected secret should be consumable from pods in volume with defaultMode set [LinuxOnly] [NodeConformance] [Conformance]
[sig-storage] Projected secret should be consumable from pods in volume with mappings [NodeConformance] [Conformance]
[sig-storage] Projected secret should be consumable from pods in volume with mappings and Item Mode set [LinuxOnly] [NodeConformance] [Conformance]
[sig-storage] Projected secret should be consumable in multiple volumes in a pod [NodeConformance] [Conformance]
[sig-storage] Secrets optional updates should be reflected in volume [NodeConformance] [Conformance]
[sig-storage] Secrets should be able to mount in a volume regardless of a different secret existing with same name in different namespace [NodeConformance] [Conformance]
[sig-storage] Secrets should be consumable from pods in volume [NodeConformance] [Conformance]
[sig-storage] Secrets should be consumable from pods in volume as non-root with defaultMode and fsGroup set [LinuxOnly] [NodeConformance] [Conformance]
[sig-storage] Secrets should be consumable from pods in volume with defaultMode set [LinuxOnly] [NodeConformance] [Conformance]
[sig-storage] Secrets should be consumable from pods in volume with mappings [NodeConformance] [Conformance]
[sig-storage] Secrets should be consumable from pods in volume with mappings and Item Mode set [LinuxOnly] [NodeConformance] [Conformance]
[sig-storage] Secrets should be consumable in multiple volumes in a pod [NodeConformance] [Conformance]
[sig-storage] Secrets should be immutable if `immutable` field is set [Conformance]
[sig-storage] StorageClasses CSI Conformance should run through the lifecycle of a StorageClass [Conformance]
[sig-storage] Subpath Atomic writer volumes should support subpaths with configmap pod [Conformance]
[sig-storage] Subpath Atomic writer volumes should support subpaths with configmap pod with mountPath of existing file [Conformance]
[sig-storage] Subpath Atomic writer volumes should support subpaths with downward pod [Conformance]
[sig-storage] Subpath Atomic writer volumes should support subpaths with projected pod [Conformance]
[sig-storage] Subpath Atomic writer volumes should support subpaths with secret pod [Conformance]
dims commented 9 months ago

we are missing 9 tests and running an extra 1 test as well.

$ diff -Bbwu sorted-hydrophone.txt sorted-conformance.txt
--- sorted-hydrophone.txt   2023-12-22 16:31:39
+++ sorted-conformance.txt  2023-12-22 16:32:07
@@ -1,3 +1,5 @@
+[sig-api-machinery] API priority and fairness should support FlowSchema API operations [Conformance]
+[sig-api-machinery] API priority and fairness should support PriorityLevelConfiguration API operations [Conformance]
 [sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] listing mutating webhooks should work [Conformance]
 [sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] listing validating webhooks should work [Conformance]
 [sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] patching/updating a mutating webhook should work [Conformance]
@@ -67,6 +69,8 @@
 [sig-api-machinery] ResourceQuota should manage the lifecycle of a ResourceQuota [Conformance]
 [sig-api-machinery] ResourceQuota should verify ResourceQuota with best effort scope. [Conformance]
 [sig-api-machinery] ResourceQuota should verify ResourceQuota with terminating scopes. [Conformance]
+[sig-api-machinery] Servers with support for API chunking should return chunks of results for list calls [Conformance]
+[sig-api-machinery] Servers with support for API chunking should support continue listing from the last key if the original version has been compacted away, though the list is inconsistent [Slow] [Conformance]
 [sig-api-machinery] Servers with support for Table transformation should return a 406 for a backend which does not implement metadata [Conformance]
 [sig-api-machinery] Watchers should be able to restart watching from the last resource version observed by the previous watch [Conformance]
 [sig-api-machinery] Watchers should be able to start watching from a specific resource version [Conformance]
@@ -195,6 +199,7 @@
 [sig-network] Services should have session affinity work for service with type clusterIP [LinuxOnly] [Conformance]
 [sig-network] Services should provide secure master service [Conformance]
 [sig-network] Services should serve a basic endpoint from pods [Conformance]
+[sig-network] Services should serve endpoints on same port and different protocols [Conformance]
 [sig-network] Services should serve multiport endpoints from pods [Conformance]
 [sig-network] Services should test the lifecycle of an Endpoint [Conformance]
 [sig-node] ConfigMap should be consumable via environment variable [NodeConformance] [Conformance]
@@ -259,11 +264,11 @@
 [sig-node] Probing container should have monotonically increasing restart count [NodeConformance] [Conformance]
 [sig-node] Probing container with readiness probe should not be ready before initial delay and never restart [NodeConformance] [Conformance]
 [sig-node] Probing container with readiness probe that fails should never be ready and never restart [NodeConformance] [Conformance]
-[sig-node] RuntimeClass  should support RuntimeClasses API operations [Conformance]
 [sig-node] RuntimeClass should reject a Pod requesting a deleted RuntimeClass [NodeConformance] [Conformance]
 [sig-node] RuntimeClass should reject a Pod requesting a non-existent RuntimeClass [NodeConformance] [Conformance]
 [sig-node] RuntimeClass should schedule a Pod requesting a RuntimeClass and initialize its Overhead [NodeConformance] [Conformance]
 [sig-node] RuntimeClass should schedule a Pod requesting a RuntimeClass without PodOverhead [NodeConformance] [Conformance]
+[sig-node] RuntimeClass should support RuntimeClasses API operations [Conformance]
 [sig-node] Secrets should be consumable from pods in env vars [NodeConformance] [Conformance]
 [sig-node] Secrets should be consumable via the environment [NodeConformance] [Conformance]
 [sig-node] Secrets should fail to create secret due to empty secret key [Conformance]
@@ -336,6 +341,8 @@
 [sig-storage] EmptyDir volumes volume on tmpfs should have the correct mode [LinuxOnly] [NodeConformance] [Conformance]
 [sig-storage] EmptyDir wrapper volumes should not cause race condition when used for configmaps [Serial] [Conformance]
 [sig-storage] EmptyDir wrapper volumes should not conflict [Conformance]
+[sig-storage] PersistentVolumes CSI Conformance should apply changes to a pv/pvc status [Conformance]
+[sig-storage] PersistentVolumes CSI Conformance should run through the lifecycle of a PV and a PVC [Conformance]
 [sig-storage] Projected combined should project all components that make up the projection API [Projection] [NodeConformance] [Conformance]
 [sig-storage] Projected configMap optional updates should be reflected in volume [NodeConformance] [Conformance]
 [sig-storage] Projected configMap should be consumable from pods in volume [NodeConformance] [Conformance]
@@ -373,6 +380,7 @@
 [sig-storage] Secrets should be consumable from pods in volume with mappings and Item Mode set [LinuxOnly] [NodeConformance] [Conformance]
 [sig-storage] Secrets should be consumable in multiple volumes in a pod [NodeConformance] [Conformance]
 [sig-storage] Secrets should be immutable if `immutable` field is set [Conformance]
+[sig-storage] StorageClasses CSI Conformance should run through the lifecycle of a StorageClass [Conformance]
 [sig-storage] Subpath Atomic writer volumes should support subpaths with configmap pod [Conformance]
 [sig-storage] Subpath Atomic writer volumes should support subpaths with configmap pod with mountPath of existing file [Conformance]
 [sig-storage] Subpath Atomic writer volumes should support subpaths with downward pod [Conformance]
dims commented 9 months ago

Hmm, we are running 1.28.0 conformance image, let me try switching it to something newer

dims commented 9 months ago

Matches nicely with this diff to conformance yaml: https://github.com/kubernetes/kubernetes/compare/v1.28.0...v1.29.0#diff-96054f175317e3d3dac5c6f71ba8273ee991e7aba2f16a352fb400ebdb89895f

rjsadow commented 9 months ago

I think the kubernetes version we're running with kind is 1.27.3

Ref https://github.com/dims/hydrophone/actions/runs/7302952615/job/19902529745#step:5:16

rjsadow commented 9 months ago

We should probably add a variable to the jobs to ensure the conformance test and cluster version are the same.

dims commented 9 months ago

@rjsadow sounds good. let's update kind cluster to use 1.29.0 as well!

dims commented 9 months ago 
Ran 388 of 7407 Specs in 6555.118 seconds
SUCCESS! -- 388 Passed | 0 Failed | 0 Pending | 7019 Skipped
PASS