search

vmware-tanzu / sonobuoy

Sonobuoy is a diagnostic tool that makes it easier to understand the state of a Kubernetes cluster by running a set of Kubernetes conformance tests and other plugins in an accessible and non-destructive manner.
https://sonobuoy.io
Apache License 2.0
2.9k stars 343 forks source link

Can't find the result from tar.gz #212

Closed warmchang closed 6 years ago

warmchang commented 6 years ago

After running the confrom test, can't get ther e2e.log & junit.xml from tar.gz

root@sonobuoy:/tmp/sonobuoy# ls -l total 4 -rw-r--r-- 1 root root 1489 Mar 4 08:12 201803040642_sonobuoy_34d6c330-4616-483b-83d5-9541ed0a8a4e.tar.gz drwxr-xr-x 2 root root 63 Mar 4 08:12 meta drwxr-xr-x 3 root root 16 Mar 4 08:12 resources root@sonobuoy:/tmp/sonobuoy#

Execution method:

curl -L https://raw.githubusercontent.com/cncf/k8s-conformance/master/sonobuoy-conformance.yaml | kubectl apply -f -

sonobuoy logs:

[root@k8s-master-controller:/]$ kubectl logs sonobuoy -n sonobuoy time="2018-03-04T06:42:01Z" level=info msg="Scanning plugins in ./plugins.d (pwd: /)" time="2018-03-04T06:42:01Z" level=warning msg="Unsupported plugin file detected ..3983_04_03_14_41_53.042217552" time="2018-03-04T06:42:01Z" level=warning msg="Unsupported plugin file detected ..data" time="2018-03-04T06:42:02Z" level=info msg="Scanning plugins in /etc/sonobuoy/plugins.d (pwd: /)" time="2018-03-04T06:42:02Z" level=info msg="Directory (/etc/sonobuoy/plugins.d) does not exist" time="2018-03-04T06:42:02Z" level=info msg="Scanning plugins in ~/sonobuoy/plugins.d (pwd: /)" time="2018-03-04T06:42:02Z" level=info msg="Directory (~/sonobuoy/plugins.d) does not exist" time="2018-03-04T06:42:02Z" level=info msg="Loading plugin driver Job" time="2018-03-04T06:42:02Z" level=info msg="Namespace default Matched=true" time="2018-03-04T06:42:02Z" level=info msg="Namespace e2e-tests-proxy-zc2b9 Matched=true" time="2018-03-04T06:42:02Z" level=info msg="Namespace e2e-tests-proxy-zc758 Matched=true" time="2018-03-04T06:42:02Z" level=info msg="Namespace e2e-tests-sched-pred-6qpws Matched=true" time="2018-03-04T06:42:02Z" level=info msg="Namespace kube-public Matched=true" time="2018-03-04T06:42:02Z" level=info msg="Namespace kube-system Matched=true" time="2018-03-04T06:42:02Z" level=info msg="Namespace sonobuoy Matched=true" time="2018-03-04T06:42:02Z" level=info msg="Starting server Expected Results: [{ e2e}]" time="2018-03-04T06:42:02Z" level=info msg="Running (e2e) plugin" time="2018-03-04T06:42:02Z" level=info msg="Listening for incoming results on 0.0.0.0:8080\n" time="2018-03-04T08:12:02Z" level=error msg="error running plugins: timed out waiting for plugins, shutting down HTTP server" time="2018-03-04T08:12:02Z" level=info msg="Running non-ns query" time="2018-03-04T08:12:02Z" level=info msg="Running ns query (default)" time="2018-03-04T08:12:02Z" level=info msg="Running ns query (e2e-tests-proxy-zc2b9)" time="2018-03-04T08:12:02Z" level=info msg="Running ns query (e2e-tests-proxy-zc758)" time="2018-03-04T08:12:02Z" level=info msg="Running ns query (e2e-tests-sched-pred-6qpws)" time="2018-03-04T08:12:02Z" level=info msg="Running ns query (kube-public)" time="2018-03-04T08:12:02Z" level=info msg="Running ns query (kube-system)" time="2018-03-04T08:12:02Z" level=info msg="Running ns query (sonobuoy)" time="2018-03-04T08:12:02Z" level=info msg="Results available at /tmp/sonobuoy/201803040642_sonobuoy_34d6c330-4616-483b-83d5-9541ed0a8a4e.tar.gz" 2018/03/04 08:12:02 failed to open logfile: /tmp/sonobuoy/34d6c330-4616-483b-83d5-9541ed0a8a4e/meta/run.log open /tmp/sonobuoy/34d6c330-4616-483b-83d5-9541ed0a8a4e/meta/run.log: no such file or directory Failed to fire hook: open /tmp/sonobuoy/34d6c330-4616-483b-83d5-9541ed0a8a4e/meta/run.log: no such file or directory time="2018-03-04T08:12:02Z" level=info msg="no-exit was specified, sonobuoy is now blocking" [root@k8s-master-controller:/]$

K8S version:

[root@k8s-master-controller:/]$ kubectl version Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.5", GitCommit:"cce11c6a185279d037023e02ac5249e14daa22bf", GitTreeState:"archive", BuildDate:"2018-01-02T13:51:05Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.5", GitCommit:"cce11c6a185279d037023e02ac5249e14daa22bf", GitTreeState:"archive", BuildDate:"2018-01-02T13:43:49Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} [root@k8s-master-controller:/]$

Any suggestions to find what happend? Thanks!

warmchang commented 6 years ago

It looks similar to https://github.com/heptio/sonobuoy/issues/118, but in fact https://github.com/heptio/sonobuoy/issues/118 has been resolved and closed.

So it should be caused by different reasons.

And we tryed the yaml from "https://scanner.heptio.com/", but got the same error. :cry:

timothysc commented 6 years ago

@warmchang 

time="2018-03-04T08:12:02Z" level=error msg="error running plugins: timed out waiting for plugins, shutting down HTTP server"

Indicates that a test has wedged, to understand which test has wedged we need to look at the e2e log which will contain the details os which test has failed and why.

I also think we could script-wrap an upper timeout to kill the tests if they wedge.
/cc @chuckha @liztio

timothysc commented 6 years ago

So after chatting on slack yesterday it was determined that DNS was missing and was the root of the problem after looking at the logs.

We are moving to a cli-driven interface and sonobuoy status or sonobuoy logs should be able to surface the data with an easier UX. Closing this issue in favor of #208 and #209