Closed Hokwang closed 4 years ago
The output from the tests isn't particularly helpful here, but I looked at the line where the error was triggered on the v1.17.4 branch for Kubernetes: https://github.com/kubernetes/kubernetes/blob/v1.17.4/test/e2e/e2e.go#L242
The condition that was not met for these tests was that not all the nodes were considered ready to schedule workloads on. This happens when there are nodes where the Ready condition is not true, or there are taints on the nodes. We have an answer in FAQ about this problem and how you can work around it to start your tests: https://sonobuoy.io/docs/v0.18.3/faq/#we-have-some-nodes-with-custom-taints-in-our-cluster-and-the-tests-wont-start-how-can-i-run-the-tests
Hopefully this helps you get the tests running! I am going to close this issue but if you still encounter problems, or need additional help feel free to reopen.
@zubron when I use ./sonobuoy run --wait --plugin-env=e2e.E2E_EXTRA_ARGS="--non-blocking-taints=node-role.kubernetes.io/master,<my_taint>"
command, this resolve my problem, thanks.
and here's one another question, now I have another error message. I think that this is http proxy problem cause I am behind the corporate proxy.
$ ./sonobuoy results $results
Plugin: e2e
Status: failed
Total: 4842
Passed: 258
Failed: 20
Skipped: 4564
Failed tests:
[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should mutate custom resource with pruning [Conformance]
[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should mutate configmap [Conformance]
[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should deny crd creation [Conformance]
[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should be able to deny pod and configmap creation [Conformance]
[sig-api-machinery] CustomResourceConversionWebhook [Privileged:ClusterAdmin] should be able to convert a non homogeneous list of CRs [Conformance]
[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should not be able to mutate or prevent deletion of webhook configuration objects [Conformance]
[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should mutate pod and apply defaults after mutation [Conformance]
[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should unconditionally reject operations on fail closed webhook [Conformance]
[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should be able to deny custom resource creation, update and deletion [Conformance]
[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should mutate custom resource with different stored version [Conformance]
[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] listing mutating webhooks should work [Conformance]
[sig-api-machinery] CustomResourceConversionWebhook [Privileged:ClusterAdmin] should be able to convert from CR v1 to CR v2 [Conformance]
[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should mutate custom resource [Conformance]
[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should honor timeout [Conformance]
[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should be able to deny attaching pod [Conformance]
[sig-network] Networking Granular Checks: Pods should function for intra-pod communication: udp [LinuxOnly] [NodeConformance] [Conformance]
[sig-apps] Daemon set [Serial] should run and stop simple daemon [Conformance]
[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] listing validating webhooks should work [Conformance]
[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] patching/updating a validating webhook should work [Conformance]
[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] patching/updating a mutating webhook should work [Conformance]
Plugin: systemd-logs
Status: passed
Total: 29
Passed: 29
Failed: 0
Skipped: 0
I am using cilium for CNI, is this related?
Hi @Hokwang. To determine the cause of the test failures, you will need to look at the logs from the tests as sonobuoy results
just outputs the name of the failed tests with no further details.
Please see our guide in the FAQ on how to debug test failures: https://sonobuoy.io/docs/v0.18.3/faq/#how-do-i-determine-why-my-tests-failed
@zubron I found https://github.com/vmware-tanzu/sonobuoy/issues/1084, so I said cilium.
I don't know what is the problem..
$ ./sonobuoy results $results --mode detailed | jq '. | select(.status == "failed") | .details'
{
"failure": "/workspace/anago-v1.17.4-beta.0.54+12bf0cb73007af/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:721\nJul 22 07:08:41.407: waiting for webhook configuration to be ready\nUnexpected error:\n <*errors.errorString | 0xc0000cb960>: {\n s: \"timed out waiting for the condition\",\n }\n timed out waiting for the condition\noccurred\n/workspace/anago-v1.17.4-beta.0.54+12bf0cb73007af/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apimachinery/webhook.go:1865",
"system-out": "[BeforeEach] [sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin]\n /workspace/anago-v1.17.4-beta.0.54+12bf0cb73007af/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:151\nSTEP: Creating a kubernetes client\nJul 22 07:08:05.700: INFO: >>> kubeConfig: /tmp/kubeconfig-616749498\nSTEP: Building a namespace api object, basename webhook\nSTEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in webhook-2186\nSTEP: Waiting for a default service account to be provisioned in namespace\n[BeforeEach] [sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin]\n /workspace/anago-v1.17.4-beta.0.54+12bf0cb73007af/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apimachinery/webhook.go:87\nSTEP: Setting up server cert\nSTEP: Create role binding to let webhook read extension-apiserver-authentication\nSTEP: Deploying the webhook pod\nSTEP: Wait for the deployment to be ready\nJul 22 07:08:06.077: INFO: deployment \"sample-webhook-deployment\" doesn't have the required revision set\nJul 22 07:08:08.087: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:\"Available\", Status:\"False\", LastUpdateTime:v1.Time{Time:time.Time{wall:0x0, ext:63730998486, loc:(*time.Location)(0x791d1c0)}}, LastTransitionTime:v1.Time{Time:time.Time{wall:0x0, ext:63730998486, loc:(*time.Location)(0x791d1c0)}}, Reason:\"MinimumReplicasUnavailable\", Message:\"Deployment does not have minimum availability.\"}, v1.DeploymentCondition{Type:\"Progressing\", Status:\"True\", LastUpdateTime:v1.Time{Time:time.Time{wall:0x0, ext:63730998486, loc:(*time.Location)(0x791d1c0)}}, LastTransitionTime:v1.Time{Time:time.Time{wall:0x0, ext:63730998486, loc:(*time.Location)(0x791d1c0)}}, Reason:\"ReplicaSetUpdated\", Message:\"ReplicaSet \\\"sample-webhook-deployment-5f65f8c764\\\" is progressing.\"}}, CollisionCount:(*int32)(nil)}\nSTEP: Deploying the webhook service\nSTEP: Verifying the service has paired with the endpoint\nJul 22 07:08:11.105: INFO: Waiting for amount of service:e2e-test-webhook endpoints to be 1\n[It] should mutate custom resource with pruning [Conformance]\n /workspace/anago-v1.17.4-beta.0.54+12bf0cb73007af/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:721\nJul 22 07:08:11.108: INFO: >>> kubeConfig: /tmp/kubeconfig-616749498\nSTEP: Registering the mutating webhook for custom resource e2e-test-webhook-7051-crds.webhook.example.com via the AdmissionRegistration API\nJul 22 07:08:11.176: INFO: Waiting for webhook configuration to be ready...\nJul 22 07:08:11.462: INFO: Waiting for webhook configuration to be ready...\nJul 22 07:08:11.513: INFO: Waiting for webhook configuration to be ready...\nJul 22 07:08:11.637: INFO: Waiting for webhook configuration to be ready...\nJul 22 07:08:11.727: INFO: Waiting for webhook configuration to be ready...\nJul 22 07:08:11.820: INFO: Waiting for webhook configuration to be ready...
<snip>
Hi @Hokwang. It's possible that the CNI is related, however I couldn't say that for sure as the Sonobuoy team are not the authors of these tests.
I would try to run just one of the failing tests again (e.g. using --e2e-focus="should mutate custom resource with pruning"
) and carefully examine the logs. Looking at the snippet you've sent it is this test that is failing when registering the webhook.
While the test is running, you could inspect some of the resources it's creating to try and determine why they don't become ready. Most of the tests have a fairly generous timeout value so you should have time to see and inspect the resources that they attempt to create.
@Hokwang Can you please share the sample of the command ? I am passing the taint , still I get same error "Before Suite". How exactly to pass the taint, is it like for example, ./sonobuoy run --wait --plugin-env=e2e.E2E_EXTRA_ARGS="--non-blocking-taints=node-role.kubernetes.io/master, xyz:NoSchedule" or ./sonobuoy run --wait --plugin-env=e2e.E2E_EXTRA_ARGS="--non-blocking-taints=node-role.kubernetes.io/master, workloadType=xyz:NoSchedule"
@BadigerAnu, any luck finding the right format to pass the taints to the --non-blocking-taints
option?
@zubron & @johnSchnake, the format to pass the --non-blocking-taints
is to be elaborated in the FAQ because the excerpt there doesn't actually give an idea about how to pass the taints.
Hello.
Today I run sonobuoy first time. I found e2e failed but don't know how to fix this. Can you help me?
Environment:
kubectl version
):/etc/os-release
): ubuntu 18.04