Closed oomichi closed 5 years ago
~ Failure [6.451 seconds]
[sig-instrumentation] MetricsGrabber
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/instrumentation/common/framework.go:23
should grab all metrics from a Scheduler. [It]
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/instrumentation/monitoring/metrics_grabber.go:61
Expected error:
<*errors.StatusError | 0xc421a2c3f0>: {
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {SelfLink: "", ResourceVersion: "", Continue: ""},
Status: "Failure",
Message: "the server is currently unable to handle the request (get pods kube-scheduler-k8s-master:10251)",
Reason: "ServiceUnavailable",
Details: {
Name: "kube-scheduler-k8s-master:10251",
Group: "",
Kind: "pods",
UID: "",
Causes: [
{
Type: "UnexpectedServerResponse",
Message: "Error: 'dial tcp 192.168.1.108:10251: connect: connection refused'\nTrying to reach: 'http://192.168.1.108:10251/metrics'",
Field: "",
},
],
RetryAfterSeconds: 0,
},
Code: 503,
},
}
the server is currently unable to handle the request (get pods kube-scheduler-k8s-master:10251)
not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/instrumentation/monitoring/metrics_grabber.go:78
kube-scheduler が待ち合わせている事がわかる。 → localhost で LISTEN しているから、他のホストにある e2e テストからはつながらないよね・・
$ sudo netstat -anp | grep 10251
tcp 0 0 127.0.0.1:10251 0.0.0.0:* LISTEN 2491/kube-scheduler
以下がテストコード コメントでは「API Server 経由で Pod (ここでは kube-scheduler?) につなぐ」とある
61 gin.It("should grab all metrics from a Scheduler.", func() {
62 gin.By("Proxying to Pod through the API server")
63 // Check if master Node is registered
64 nodes, err := c.CoreV1().Nodes().List(metav1.ListOptions{})
65 framework.ExpectNoError(err)
66
67 var masterRegistered = false
68 for _, node := range nodes.Items {
69 if strings.HasSuffix(node.Name, "master") {
70 masterRegistered = true
71 }
72 }
73 if !masterRegistered {
74 framework.Logf("Master is node api.Registry. Skipping testing Scheduler metrics.")
75 return
76 }
77 response, err := grabber.GrabFromScheduler()
78 framework.ExpectNoError(err)
79 gom.Expect(response).NotTo(gom.BeEmpty())
80 })
問題の関数
126 func (g *MetricsGrabber) GrabFromScheduler() (SchedulerMetrics, error) {
127 if !g.registeredMaster {
128 return SchedulerMetrics{}, fmt.Errorf("Master's Kubelet is not registered. Skipping Scheduler's metrics gathering.")
129 }
130 output, err := g.getMetricsFromPod(g.client, fmt.Sprintf("%v-%v", "kube-scheduler", g.masterName), metav1.NamespaceSystem, ports.SchedulerPort)
131 if err != nil {
132 return SchedulerMetrics{}, err
133 }
134 return parseSchedulerMetrics(output)
135 }
さらに
233 func (g *MetricsGrabber) getMetricsFromPod(client clientset.Interface, podName string, namespace string, port int) (string, error) {
234 rawOutput, err := client.CoreV1().RESTClient().Get().
235 Namespace(namespace).
236 Resource("pods").
237 SubResource("proxy").
238 Name(fmt.Sprintf("%v:%v", podName, port)).
239 Suffix("metrics").
240 Do().Raw()
241 if err != nil {
242 return "", err
243 }
244 return string(rawOutput), nil
245 }
$ kubectl describe pod kube-scheduler-k8s-master -n=kube-system
Name: kube-scheduler-k8s-master
Namespace: kube-system
Priority: 2000000000
...
Liveness: http-get http://127.0.0.1:10251/healthz delay=15s timeout=15s period=10s #success=1 #failure=8
...
127.0.0.1 に curl すると成功。しかし 192.168.1.108 で curl すると失敗 上記のとおり、kube-scheduler が 127.0.0.1 で LISTEN しているから。
$ curl http://127.0.0.1:10251/healthz
ok
$
$ curl http://192.168.1.108:10251/healthz
curl: (7) Failed to connect to 192.168.1.108 port 10251: Connection refused
$
Message: "Error: 'dial tcp 192.168.1.108:10251: connect: connection refused'\nTrying to reach: 'http://192.168.1.108:10251/metrics'",
の 192.168.1.108 は何処から来ている?
問題となったAPI
I0822 02:20:16.133328 20404 round_trippers.go:383] GET https://192.168.1.108:6443/api/v1/namespaces/kube-system/pods/kube-scheduler-k8s-master:10251/proxy/metrics
I0822 02:20:16.133341 20404 round_trippers.go:390] Request Headers:
I0822 02:20:16.133345 20404 round_trippers.go:393] Accept: application/vnd.kubernetes.protobuf, */*
I0822 02:20:16.133349 20404 round_trippers.go:393] User-Agent: e2e.test/v1.11.1 (linux/amd64) kubernetes/d0b061a
I0822 02:20:16.136376 20404 round_trippers.go:408] Response Status: 503 Service Unavailable in 3 milliseconds
I0822 02:20:16.136387 20404 round_trippers.go:411] Response Headers:
I0822 02:20:16.136391 20404 round_trippers.go:414] Content-Type: text/plain; charset=utf-8
I0822 02:20:16.136394 20404 round_trippers.go:414] Content-Length: 120
I0822 02:20:16.136398 20404 round_trippers.go:414] Date: Wed, 22 Aug 2018 02:20:16 GMT
I0822 02:20:16.136412 20404 request.go:897] Response Body: Error: 'dial tcp 192.168.1.108:10251: connect: connection refused'
Trying to reach: 'http://192.168.1.108:10251/metrics'
一先ず、manifest を改造
$ sudo diff -u etc/kube-scheduler.yaml.orig /etc/kubernetes/manifests/kube-scheduler.yaml
--- etc/kube-scheduler.yaml.orig 2018-08-22 02:30:16.060204589 +0000
+++ /etc/kubernetes/manifests/kube-scheduler.yaml 2018-08-22 02:30:38.160555932 +0000
@@ -13,7 +13,7 @@
containers:
- command:
- kube-scheduler
- - --address=127.0.0.1
+ - --address=192.168.1.108
- --kubeconfig=/etc/kubernetes/scheduler.conf
- --leader-elect=true
image: k8s.gcr.io/kube-scheduler-amd64:v1.11.1
指定したアドレスで scheduler が LISTEN していることを確認
$ sudo netstat -anp | grep 10251
tcp 0 0 192.168.1.108:10251 0.0.0.0:* LISTEN 6468/kube-scheduler
テストを実行 → 通るようになったことを確認
$ go run hack/e2e.go -- --provider=skeleton --test --test_args="--ginkgo.focus=should\sgrab\sall\smetrics\sfrom\sa\sSche
duler" --check-version-skew=false
...
~ [SLOW TEST:6.274 seconds]
[sig-instrumentation] MetricsGrabber
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/instrumentation/common/framework.go:23
should grab all metrics from a Scheduler.
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/instrumentation/monitoring/metrics_grabber.go:61
------------------------------
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSAug 22 02:33:23.242: INFO: Running AfterSuite actions on all node
Aug 22 02:33:23.242: INFO: Running AfterSuite actions on node 1
Ran 1 of 999 Specs in 6.386 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 998 Skipped PASS
Ginkgo ran 1 suite in 6.678185057s
Test Suite Passed
2018/08/22 02:33:23 process.go:155: Step './hack/ginkgo-e2e.sh --ginkgo.focus=should\sgrab\sall\smetrics\sfrom\sa\sScheduler' finished in 6.716759429s
2018/08/22 02:33:23 e2e.go:83: Done
上記の変更でよいのか k/kubernetes/issues/67685 として問い合わせ中
下記のテストが引き続き失敗中のため
[Fail] [sig-instrumentation] MetricsGrabber [It] should grab all metrics from a ControllerManager.
まとめ