search

kubernetes / kubernetes

Production-Grade Container Scheduling and Management
https://kubernetes.io
Apache License 2.0
109.97k stars 39.36k forks source link

kubeadm init hang again on new version on ubuntu 16.04 #36460

Closed yesimages closed 7 years ago

yesimages commented 7 years ago

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): kubeadm init hang

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Kubernetes version (use kubectl version): 1.4.4

Environment:

What happened: We installed k8s by using kubeadm a week ago without problem. Here is output form kubectl version: Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.3", GitCommit:"4957b090e9a4f6a68b4a40375408fdc74a212260", GitTreeState:"clean", BuildDate:"2016-10-16T06:36:33Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.0", GitCommit:"a16c0a7f71a6f93c7e0f222d961f4675cd97a46b", GitTreeState:"clean", BuildDate:"2016-09-26T18:10:32Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

Yesterday kicked off another installation and "kubeadm init" hang when"waiting for the control plane to become ready". Here is the output from kubectl version: Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.4", GitCommit:"3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56", GitTreeState:"clean", BuildDate:"2016-10-21T02:48:38Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.4", GitCommit:"3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56", GitTreeState:"clean", BuildDate:"2016-10-21T02:42:39Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

It seemed only difference is to have a higher version 1.4.4 now.

What you expected to happen: kubeadm init waiting less than 1 minute to resume the setup.

How to reproduce it (as minimally and precisely as possible): following the step described on "Installing Kubernetes on Linux with kubeadm" http://kubernetes.io/docs/getting-started-guides/kubeadm/

Anything else do we need to know: Do you have another way to describe how to install/setup kubenetes other than kubeadm?

Thanks

WY

yesimages commented 7 years ago

kubeadm version kubeadm version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-alpha.2.421+a6bea3d79b8bba", GitCommit:"a6bea3d79b8bbaa5e8b57482c9fff9265d402708", GitTreeState:"clean", BuildDate:"2016-11-03T06:54:50Z", GoVersion:"go1.7.1", Compiler:"gc", Platform:"linux/amd64"}

Here is output from journalctl when kubeadm init hung:

Nov 11 21:50:36 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:36.936112    3492 kubelet.go:2370] SyncLoop (housekeeping)
Nov 11 21:50:37 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:37.944014    3492 prober.go:161] HTTP-Probe Host: http://127.0.0.1, Port: 2379, Path: /health
Nov 11 21:50:37 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:37.944069    3492 prober.go:164] HTTP-Probe Headers: map[]
Nov 11 21:50:37 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:37.949992    3492 prober.go:161] HTTP-Probe Host: http://127.0.0.1, Port: 8080, Path: /healthz
Nov 11 21:50:37 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:37.950043    3492 prober.go:164] HTTP-Probe Headers: map[]
Nov 11 21:50:37 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:37.951137    3492 http.go:82] Probe succeeded for http://127.0.0.1:8080/healthz, Response: {200 OK 200 HTT
Nov 11 21:50:37 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:37.951203    3492 prober.go:115] Liveness probe for "kube-apiserver-ngpyesstack11.fyre.ibm.com_kube-system
Nov 11 21:50:37 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:37.962711    3492 prober.go:161] HTTP-Probe Host: http://127.0.0.1, Port: 10252, Path: /healthz
Nov 11 21:50:37 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:37.962776    3492 prober.go:164] HTTP-Probe Headers: map[]
Nov 11 21:50:37 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:37.963692    3492 http.go:82] Probe succeeded for http://127.0.0.1:10252/healthz, Response: {200 OK 200 HT
Nov 11 21:50:37 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:37.963744    3492 prober.go:115] Liveness probe for "kube-controller-manager-ngpyesstack11.fyre.ibm.com_ku
Nov 11 21:50:37 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:37.975104    3492 prober.go:161] HTTP-Probe Host: http://127.0.0.1, Port: 10251, Path: /healthz
Nov 11 21:50:37 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:37.975147    3492 prober.go:164] HTTP-Probe Headers: map[]
Nov 11 21:50:37 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:37.976420    3492 http.go:82] Probe succeeded for http://127.0.0.1:10251/healthz, Response: {200 OK 200 HT
Nov 11 21:50:37 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:37.976500    3492 prober.go:115] Liveness probe for "kube-scheduler-ngpyesstack11.fyre.ibm.com_kube-system
Nov 11 21:50:38 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:38.445906    3492 http.go:82] Probe succeeded for http://127.0.0.1:2379/health, Response: {200 OK 200 HTTP
Nov 11 21:50:38 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:38.447877    3492 prober.go:115] Liveness probe for "etcd-ngpyesstack11.fyre.ibm.com_kube-system(a81a848a0
Nov 11 21:50:38 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:38.936076    3492 kubelet.go:2370] SyncLoop (housekeeping)
Nov 11 21:50:39 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:39.672675    3492 reflector.go:403] pkg/kubelet/kubelet.go:403: Watch close - *api.Node total 42 items rec
Nov 11 21:50:40 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:40.938116    3492 kubelet.go:2370] SyncLoop (housekeeping)
Nov 11 21:50:42 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:42.936064    3492 kubelet.go:2370] SyncLoop (housekeeping)
Nov 11 21:50:44 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:44.936187    3492 kubelet.go:2347] SyncLoop (SYNC): 2 pods; kube-apiserver-ngpyesstack11.fyre.ibm.com_kube
Nov 11 21:50:44 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:44.936548    3492 kubelet.go:2370] SyncLoop (housekeeping)
Nov 11 21:50:44 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:44.936500    3492 kubelet.go:2761] Generating status for "kube-apiserver-ngpyesstack11.fyre.ibm.com_kube-s
Nov 11 21:50:44 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:44.936848    3492 kubelet.go:2761] Generating status for "kube-scheduler-ngpyesstack11.fyre.ibm.com_kube-s
Nov 11 21:50:44 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:44.936941    3492 status_manager.go:312] Ignoring same status for pod "kube-apiserver-ngpyesstack11.fyre.i
Nov 11 21:50:44 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:44.938479    3492 volume_manager.go:324] Waiting for volumes to attach and mount for pod "kube-apiserver-n
Nov 11 21:50:44 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:44.938703    3492 status_manager.go:312] Ignoring same status for pod "kube-scheduler-ngpyesstack11.fyre.i
Nov 11 21:50:44 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:44.938954    3492 docker_manager.go:1892] Found pod infra container for "kube-scheduler-ngpyesstack11.fyre
Nov 11 21:50:44 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:44.940445    3492 docker_manager.go:1905] Pod infra container looks good, keep it "kube-scheduler-ngpyesst
Nov 11 21:50:44 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:44.940674    3492 docker_manager.go:1955] pod "kube-scheduler-ngpyesstack11.fyre.ibm.com_kube-system(87687
Nov 11 21:50:44 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:44.940891    3492 docker_manager.go:2040] Got container changes for pod "kube-scheduler-ngpyesstack11.fyre
Nov 11 21:50:45 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:45.198666    3492 conversion.go:133] failed to handle multiple devices for container. Skipping Filesystem
Nov 11 21:50:45 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:45.199118    3492 conversion.go:133] failed to handle multiple devices for container. Skipping Filesystem
Nov 11 21:50:45 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:45.228404    3492 summary.go:352] Missing default interface "eth0" for node:ngpyesstack11.fyre.ibm.com
Nov 11 21:50:45 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:45.228480    3492 summary.go:352] Missing default interface "eth0" for pod:kube-system_etcd-ngpyesstack11.
Nov 11 21:50:45 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:45.228492    3492 summary.go:352] Missing default interface "eth0" for pod:kube-system_kube-controller-man
Nov 11 21:50:45 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:45.228501    3492 summary.go:352] Missing default interface "eth0" for pod:kube-system_kube-apiserver-ngpy
Nov 11 21:50:45 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:45.228515    3492 summary.go:352] Missing default interface "eth0" for pod:kube-system_kube-scheduler-ngpy
Nov 11 21:50:45 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:45.228547    3492 eviction_manager.go:204] eviction manager: no resources are starved
Nov 11 21:50:45 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:45.238805    3492 volume_manager.go:353] All volumes are attached and mounted for pod "kube-apiserver-ngpy
Nov 11 21:50:45 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:45.239219    3492 docker_manager.go:1892] Found pod infra container for "kube-apiserver-ngpyesstack11.fyre
Nov 11 21:50:45 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:45.241940    3492 docker_manager.go:1905] Pod infra container looks good, keep it "kube-apiserver-ngpyesst
Nov 11 21:50:45 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:45.242087    3492 docker_manager.go:1955] pod "kube-apiserver-ngpyesstack11.fyre.ibm.com_kube-system(2bea9
Nov 11 21:50:45 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:45.242119    3492 docker_manager.go:2040] Got container changes for pod "kube-apiserver-ngpyesstack11.fyre
Nov 11 21:50:46 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:46.936135    3492 kubelet.go:2370] SyncLoop (housekeeping)
Nov 11 21:50:47 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:47.944020    3492 prober.go:161] HTTP-Probe Host: http://127.0.0.1, Port: 2379, Path: /health
Nov 11 21:50:47 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:47.944078    3492 prober.go:164] HTTP-Probe Headers: map[]
Nov 11 21:50:47 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:47.949991    3492 prober.go:161] HTTP-Probe Host: http://127.0.0.1, Port: 8080, Path: /healthz
Nov 11 21:50:47 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:47.950392    3492 prober.go:164] HTTP-Probe Headers: map[]
Nov 11 21:50:47 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:47.951486    3492 http.go:82] Probe succeeded for http://127.0.0.1:8080/healthz, Response: {200 OK 200 HTT
Nov 11 21:50:47 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:47.951916    3492 prober.go:115] Liveness probe for "kube-apiserver-ngpyesstack11.fyre.ibm.com_kube-system
Nov 11 21:50:47 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:47.962548    3492 prober.go:161] HTTP-Probe Host: http://127.0.0.1, Port: 10252, Path: /healthz
Nov 11 21:50:47 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:47.962609    3492 prober.go:164] HTTP-Probe Headers: map[]
Nov 11 21:50:47 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:47.963589    3492 http.go:82] Probe succeeded for http://127.0.0.1:10252/healthz, Response: {200 OK 200 HT
Nov 11 21:50:47 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:47.963633    3492 prober.go:115] Liveness probe for "kube-controller-manager-ngpyesstack11.fyre.ibm.com_ku
Nov 11 21:50:47 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:47.975207    3492 prober.go:161] HTTP-Probe Host: http://127.0.0.1, Port: 10251, Path: /healthz
Nov 11 21:50:47 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:47.975327    3492 prober.go:164] HTTP-Probe Headers: map[]
Nov 11 21:50:47 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:47.976093    3492 http.go:82] Probe succeeded for http://127.0.0.1:10251/healthz, Response: {200 OK 200 HT
Nov 11 21:50:47 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:47.976142    3492 prober.go:115] Liveness probe for "kube-scheduler-ngpyesstack11.fyre.ibm.com_kube-system
Nov 11 21:50:48 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:48.445850    3492 http.go:82] Probe succeeded for http://127.0.0.1:2379/health, Response: {200 OK 200 HTTP
Nov 11 21:50:48 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:48.445958    3492 prober.go:115] Liveness probe for "etcd-ngpyesstack11.fyre.ibm.com_kube-system(a81a848a0
Nov 11 21:50:48 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:48.936100    3492 kubelet.go:2370] SyncLoop (housekeeping)
Nov 11 21:50:49 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:49.872233    3492 reflector.go:403] pkg/kubelet/config/apiserver.go:43: Watch close - *api.Pod total 6 ite
Nov 11 21:50:50 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:50.936340    3492 kubelet.go:2370] SyncLoop (housekeeping)
Nov 11 21:50:52 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:52.936260    3492 kubelet.go:2370] SyncLoop (housekeeping)
Nov 11 21:50:52 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:52.989753    3492 file.go:135] Reading config file "/etc/kubernetes/manifests/etcd.json"
Nov 11 21:50:52 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:52.990959    3492 file.go:135] Reading config file "/etc/kubernetes/manifests/kube-apiserver.json"
Nov 11 21:50:52 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:52.991797    3492 file.go:135] Reading config file "/etc/kubernetes/manifests/kube-controller-manager.json
Nov 11 21:50:52 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:52.995168    3492 file.go:135] Reading config file "/etc/kubernetes/manifests/kube-scheduler.json"
Nov 11 21:50:52 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:52.995743    3492 config.go:281] Setting pods for source file
Nov 11 21:50:54 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:54.936022    3492 kubelet.go:2370] SyncLoop (housekeeping)
Nov 11 21:50:55 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:55.278702    3492 conversion.go:133] failed to handle multiple devices for container. Skipping Filesystem
Nov 11 21:50:55 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:55.278773    3492 conversion.go:133] failed to handle multiple devices for container. Skipping Filesystem
Nov 11 21:50:55 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:55.291481    3492 summary.go:352] Missing default interface "eth0" for node:ngpyesstack11.fyre.ibm.com
Nov 11 21:50:55 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:55.291990    3492 summary.go:352] Missing default interface "eth0" for pod:kube-system_kube-apiserver-ngpy
Nov 11 21:50:55 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:55.292174    3492 summary.go:352] Missing default interface "eth0" for pod:kube-system_etcd-ngpyesstack11.
Nov 11 21:50:55 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:55.292337    3492 summary.go:352] Missing default interface "eth0" for pod:kube-system_kube-scheduler-ngpy
Nov 11 21:50:55 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:55.292549    3492 summary.go:352] Missing default interface "eth0" for pod:kube-system_kube-controller-man
Nov 11 21:50:55 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:55.292722    3492 eviction_manager.go:204] eviction manager: no resources are starved
Nov 11 21:50:56 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:56.344931    3492 volume_stat_calculator.go:103] Failed to calculate volume metrics for pod kube-apiserver
Nov 11 21:50:56 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:56.344997    3492 volume_stat_calculator.go:103] Failed to calculate volume metrics for pod kube-apiserver
Nov 11 21:50:56 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:56.936155    3492 kubelet.go:2370] SyncLoop (housekeeping)
Nov 11 21:50:57 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:57.944075    3492 prober.go:161] HTTP-Probe Host: http://127.0.0.1, Port: 2379, Path: /health
Nov 11 21:50:57 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:57.944153    3492 prober.go:164] HTTP-Probe Headers: map[]
Nov 11 21:50:57 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:57.949966    3492 prober.go:161] HTTP-Probe Host: http://127.0.0.1, Port: 8080, Path: /healthz
Nov 11 21:50:57 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:57.950024    3492 prober.go:164] HTTP-Probe Headers: map[]
Nov 11 21:50:57 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:57.951332    3492 http.go:82] Probe succeeded for http://127.0.0.1:8080/healthz, Response: {200 OK 200 HTT
Nov 11 21:50:57 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:57.951398    3492 prober.go:115] Liveness probe for "kube-apiserver-ngpyesstack11.fyre.ibm.com_kube-system
Nov 11 21:50:57 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:57.962518    3492 prober.go:161] HTTP-Probe Host: http://127.0.0.1, Port: 10252, Path: /healthz
Nov 11 21:50:57 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:57.962585    3492 prober.go:164] HTTP-Probe Headers: map[]
Nov 11 21:50:57 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:57.963539    3492 http.go:82] Probe succeeded for http://127.0.0.1:10252/healthz, Response: {200 OK 200 HT
Nov 11 21:50:57 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:57.963610    3492 prober.go:115] Liveness probe for "kube-controller-manager-ngpyesstack11.fyre.ibm.com_ku
Nov 11 21:50:57 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:57.975191    3492 prober.go:161] HTTP-Probe Host: http://127.0.0.1, Port: 10251, Path: /healthz
Nov 11 21:50:57 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:57.975250    3492 prober.go:164] HTTP-Probe Headers: map[]
Nov 11 21:50:57 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:57.975974    3492 http.go:82] Probe succeeded for http://127.0.0.1:10251/healthz, Response: {200 OK 200 HT
Nov 11 21:50:57 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:57.976041    3492 prober.go:115] Liveness probe for "kube-scheduler-ngpyesstack11.fyre.ibm.com_kube-system
Nov 11 21:50:58 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:58.445684    3492 http.go:82] Probe succeeded for http://127.0.0.1:2379/health, Response: {200 OK 200 HTTP
Nov 11 21:50:58 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:58.445765    3492 prober.go:115] Liveness probe for "etcd-ngpyesstack11.fyre.ibm.com_kube-system(a81a848a0
Nov 11 21:50:58 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:50:58.936074    3492 kubelet.go:2370] SyncLoop (housekeeping)
Nov 11 21:51:00 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:00.936147    3492 kubelet.go:2370] SyncLoop (housekeeping)
Nov 11 21:51:02 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:02.936032    3492 kubelet.go:2370] SyncLoop (housekeeping)
Nov 11 21:51:03 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:03.936255    3492 kubelet.go:2347] SyncLoop (SYNC): 1 pods; kube-controller-manager-ngpyesstack11.fyre.ibm
Nov 11 21:51:03 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:03.937088    3492 kubelet.go:2761] Generating status for "kube-controller-manager-ngpyesstack11.fyre.ibm.c
Nov 11 21:51:03 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:03.937684    3492 status_manager.go:312] Ignoring same status for pod "kube-controller-manager-ngpyesstack
Nov 11 21:51:03 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:03.937865    3492 volume_manager.go:324] Waiting for volumes to attach and mount for pod "kube-controller-
Nov 11 21:51:04 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:04.238155    3492 volume_manager.go:353] All volumes are attached and mounted for pod "kube-controller-man
Nov 11 21:51:04 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:04.238220    3492 docker_manager.go:1892] Found pod infra container for "kube-controller-manager-ngpyessta
Nov 11 21:51:04 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:04.239297    3492 docker_manager.go:1905] Pod infra container looks good, keep it "kube-controller-manager
Nov 11 21:51:04 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:04.239456    3492 docker_manager.go:1955] pod "kube-controller-manager-ngpyesstack11.fyre.ibm.com_kube-sys
Nov 11 21:51:04 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:04.239496    3492 docker_manager.go:2040] Got container changes for pod "kube-controller-manager-ngpyessta
Nov 11 21:51:04 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:04.936050    3492 kubelet.go:2370] SyncLoop (housekeeping)
Nov 11 21:51:05 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:05.332402    3492 conversion.go:133] failed to handle multiple devices for container. Skipping Filesystem
Nov 11 21:51:05 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:05.332452    3492 conversion.go:133] failed to handle multiple devices for container. Skipping Filesystem
Nov 11 21:51:05 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:05.339307    3492 summary.go:352] Missing default interface "eth0" for node:ngpyesstack11.fyre.ibm.com
Nov 11 21:51:05 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:05.339359    3492 summary.go:352] Missing default interface "eth0" for pod:kube-system_etcd-ngpyesstack11.
Nov 11 21:51:05 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:05.339393    3492 summary.go:352] Missing default interface "eth0" for pod:kube-system_kube-scheduler-ngpy
Nov 11 21:51:05 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:05.339402    3492 summary.go:352] Missing default interface "eth0" for pod:kube-system_kube-apiserver-ngpy
Nov 11 21:51:05 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:05.339413    3492 summary.go:352] Missing default interface "eth0" for pod:kube-system_kube-controller-man
Nov 11 21:51:05 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:05.339566    3492 eviction_manager.go:204] eviction manager: no resources are starved
Nov 11 21:51:06 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:06.936087    3492 kubelet.go:2370] SyncLoop (housekeeping)
Nov 11 21:51:07 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:07.943944    3492 prober.go:161] HTTP-Probe Host: http://127.0.0.1, Port: 2379, Path: /health
Nov 11 21:51:07 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:07.944014    3492 prober.go:164] HTTP-Probe Headers: map[]
Nov 11 21:51:07 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:07.949931    3492 prober.go:161] HTTP-Probe Host: http://127.0.0.1, Port: 8080, Path: /healthz
Nov 11 21:51:07 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:07.949975    3492 prober.go:164] HTTP-Probe Headers: map[]
Nov 11 21:51:07 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:07.951079    3492 http.go:82] Probe succeeded for http://127.0.0.1:8080/healthz, Response: {200 OK 200 HTT
Nov 11 21:51:07 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:07.951132    3492 prober.go:115] Liveness probe for "kube-apiserver-ngpyesstack11.fyre.ibm.com_kube-system
Nov 11 21:51:07 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:07.962503    3492 prober.go:161] HTTP-Probe Host: http://127.0.0.1, Port: 10252, Path: /healthz
Nov 11 21:51:07 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:07.962560    3492 prober.go:164] HTTP-Probe Headers: map[]
Nov 11 21:51:07 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:07.963501    3492 http.go:82] Probe succeeded for http://127.0.0.1:10252/healthz, Response: {200 OK 200 HT
Nov 11 21:51:07 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:07.963553    3492 prober.go:115] Liveness probe for "kube-controller-manager-ngpyesstack11.fyre.ibm.com_ku
Nov 11 21:51:07 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:07.975060    3492 prober.go:161] HTTP-Probe Host: http://127.0.0.1, Port: 10251, Path: /healthz
Nov 11 21:51:07 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:07.975119    3492 prober.go:164] HTTP-Probe Headers: map[]
Nov 11 21:51:07 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:07.976195    3492 http.go:82] Probe succeeded for http://127.0.0.1:10251/healthz, Response: {200 OK 200 HT
Nov 11 21:51:07 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:07.976289    3492 prober.go:115] Liveness probe for "kube-scheduler-ngpyesstack11.fyre.ibm.com_kube-system
Nov 11 21:51:08 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:08.445780    3492 http.go:82] Probe succeeded for http://127.0.0.1:2379/health, Response: {200 OK 200 HTTP
Nov 11 21:51:08 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:08.445901    3492 prober.go:115] Liveness probe for "etcd-ngpyesstack11.fyre.ibm.com_kube-system(a81a848a0
Nov 11 21:51:08 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:08.936075    3492 kubelet.go:2370] SyncLoop (housekeeping)
Nov 11 21:51:10 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:10.936068    3492 kubelet.go:2370] SyncLoop (housekeeping)
Nov 11 21:51:12 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:12.935982    3492 kubelet.go:2370] SyncLoop (housekeeping)
Nov 11 21:51:12 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:12.996033    3492 file.go:135] Reading config file "/etc/kubernetes/manifests/etcd.json"
Nov 11 21:51:12 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:12.996782    3492 file.go:135] Reading config file "/etc/kubernetes/manifests/kube-apiserver.json"
Nov 11 21:51:12 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:12.997311    3492 file.go:135] Reading config file "/etc/kubernetes/manifests/kube-controller-manager.json
Nov 11 21:51:12 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:12.997762    3492 file.go:135] Reading config file "/etc/kubernetes/manifests/kube-scheduler.json"
Nov 11 21:51:12 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:12.998163    3492 config.go:281] Setting pods for source file
Nov 11 21:51:14 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:14.936086    3492 kubelet.go:2370] SyncLoop (housekeeping)
Nov 11 21:51:15 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:15.365747    3492 conversion.go:133] failed to handle multiple devices for container. Skipping Filesystem
Nov 11 21:51:15 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:15.365788    3492 conversion.go:133] failed to handle multiple devices for container. Skipping Filesystem
Nov 11 21:51:15 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:15.375240    3492 summary.go:352] Missing default interface "eth0" for node:ngpyesstack11.fyre.ibm.com
Nov 11 21:51:15 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:15.375292    3492 summary.go:352] Missing default interface "eth0" for pod:kube-system_kube-scheduler-ngpy
Nov 11 21:51:15 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:15.375308    3492 summary.go:352] Missing default interface "eth0" for pod:kube-system_kube-apiserver-ngpy
Nov 11 21:51:15 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:15.375327    3492 summary.go:352] Missing default interface "eth0" for pod:kube-system_kube-controller-man
Nov 11 21:51:15 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:15.375344    3492 summary.go:352] Missing default interface "eth0" for pod:kube-system_etcd-ngpyesstack11.
Nov 11 21:51:15 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:15.375387    3492 eviction_manager.go:204] eviction manager: no resources are starved
Nov 11 21:51:15 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:15.936072    3492 kubelet.go:2347] SyncLoop (SYNC): 1 pods; etcd-ngpyesstack11.fyre.ibm.com_kube-system(a8
Nov 11 21:51:15 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:15.936179    3492 kubelet.go:2761] Generating status for "etcd-ngpyesstack11.fyre.ibm.com_kube-system(a81a
Nov 11 21:51:15 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:15.936340    3492 status_manager.go:312] Ignoring same status for pod "etcd-ngpyesstack11.fyre.ibm.com_kub
Nov 11 21:51:15 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:15.936484    3492 volume_manager.go:324] Waiting for volumes to attach and mount for pod "etcd-ngpyesstack
Nov 11 21:51:16 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:16.236792    3492 volume_manager.go:353] All volumes are attached and mounted for pod "etcd-ngpyesstack11.
Nov 11 21:51:16 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:16.236842    3492 docker_manager.go:1892] Found pod infra container for "etcd-ngpyesstack11.fyre.ibm.com_k
Nov 11 21:51:16 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:16.237936    3492 docker_manager.go:1905] Pod infra container looks good, keep it "etcd-ngpyesstack11.fyre
Nov 11 21:51:16 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:16.238026    3492 docker_manager.go:1955] pod "etcd-ngpyesstack11.fyre.ibm.com_kube-system(a81a848a0bf7f0d
Nov 11 21:51:16 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:16.238047    3492 docker_manager.go:2040] Got container changes for pod "etcd-ngpyesstack11.fyre.ibm.com_k
Nov 11 21:51:16 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:16.936125    3492 kubelet.go:2370] SyncLoop (housekeeping)
Nov 11 21:51:17 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:17.943999    3492 prober.go:161] HTTP-Probe Host: http://127.0.0.1, Port: 2379, Path: /health
Nov 11 21:51:17 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:17.944051    3492 prober.go:164] HTTP-Probe Headers: map[]
Nov 11 21:51:17 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:17.949982    3492 prober.go:161] HTTP-Probe Host: http://127.0.0.1, Port: 8080, Path: /healthz
Nov 11 21:51:17 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:17.950032    3492 prober.go:164] HTTP-Probe Headers: map[]
Nov 11 21:51:17 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:17.951237    3492 http.go:82] Probe succeeded for http://127.0.0.1:8080/healthz, Response: {200 OK 200 HTT
Nov 11 21:51:17 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:17.951333    3492 prober.go:115] Liveness probe for "kube-apiserver-ngpyesstack11.fyre.ibm.com_kube-system
Nov 11 21:51:17 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:17.962911    3492 prober.go:161] HTTP-Probe Host: http://127.0.0.1, Port: 10252, Path: /healthz
Nov 11 21:51:17 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:17.962983    3492 prober.go:164] HTTP-Probe Headers: map[]
Nov 11 21:51:17 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:17.964147    3492 http.go:82] Probe succeeded for http://127.0.0.1:10252/healthz, Response: {200 OK 200 HT
Nov 11 21:51:17 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:17.964212    3492 prober.go:115] Liveness probe for "kube-controller-manager-ngpyesstack11.fyre.ibm.com_ku
Nov 11 21:51:17 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:17.975879    3492 prober.go:161] HTTP-Probe Host: http://127.0.0.1, Port: 10251, Path: /healthz
Nov 11 21:51:17 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:17.975917    3492 prober.go:164] HTTP-Probe Headers: map[]
Nov 11 21:51:17 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:17.977416    3492 http.go:82] Probe succeeded for http://127.0.0.1:10251/healthz, Response: {200 OK 200 HT
Nov 11 21:51:17 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:17.977500    3492 prober.go:115] Liveness probe for "kube-scheduler-ngpyesstack11.fyre.ibm.com_kube-system
Nov 11 21:51:18 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:18.445629    3492 http.go:82] Probe succeeded for http://127.0.0.1:2379/health, Response: {200 OK 200 HTTP
Nov 11 21:51:18 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:18.445716    3492 prober.go:115] Liveness probe for "etcd-ngpyesstack11.fyre.ibm.com_kube-system(a81a848a0
Nov 11 21:51:18 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:18.936131    3492 kubelet.go:2370] SyncLoop (housekeeping)
Nov 11 21:51:20 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:20.936237    3492 kubelet.go:2370] SyncLoop (housekeeping)
Nov 11 21:51:22 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:22.936026    3492 kubelet.go:2370] SyncLoop (housekeeping)
Nov 11 21:51:24 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:24.936201    3492 kubelet.go:2370] SyncLoop (housekeeping)
Nov 11 21:51:25 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:25.391965    3492 conversion.go:133] failed to handle multiple devices for container. Skipping Filesystem
Nov 11 21:51:25 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:25.392012    3492 conversion.go:133] failed to handle multiple devices for container. Skipping Filesystem
Nov 11 21:51:25 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:25.415653    3492 summary.go:352] Missing default interface "eth0" for node:ngpyesstack11.fyre.ibm.com
Nov 11 21:51:25 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:25.415760    3492 summary.go:352] Missing default interface "eth0" for pod:kube-system_kube-controller-man
Nov 11 21:51:25 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:25.415785    3492 summary.go:352] Missing default interface "eth0" for pod:kube-system_kube-apiserver-ngpy
Nov 11 21:51:25 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:25.415814    3492 summary.go:352] Missing default interface "eth0" for pod:kube-system_etcd-ngpyesstack11.
Nov 11 21:51:25 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:25.415824    3492 summary.go:352] Missing default interface "eth0" for pod:kube-system_kube-scheduler-ngpy
Nov 11 21:51:25 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:25.415865    3492 eviction_manager.go:204] eviction manager: no resources are starved
Nov 11 21:51:26 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:26.936201    3492 kubelet.go:2370] SyncLoop (housekeeping)
Nov 11 21:51:27 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:27.944088    3492 prober.go:161] HTTP-Probe Host: http://127.0.0.1, Port: 2379, Path: /health
Nov 11 21:51:27 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:27.944149    3492 prober.go:164] HTTP-Probe Headers: map[]
Nov 11 21:51:27 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:27.950000    3492 prober.go:161] HTTP-Probe Host: http://127.0.0.1, Port: 8080, Path: /healthz
Nov 11 21:51:27 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:27.950475    3492 prober.go:164] HTTP-Probe Headers: map[]
Nov 11 21:51:27 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:27.951961    3492 http.go:82] Probe succeeded for http://127.0.0.1:8080/healthz, Response: {200 OK 200 HTT
Nov 11 21:51:27 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:27.952044    3492 prober.go:115] Liveness probe for "kube-apiserver-ngpyesstack11.fyre.ibm.com_kube-system
Nov 11 21:51:27 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:27.962679    3492 prober.go:161] HTTP-Probe Host: http://127.0.0.1, Port: 10252, Path: /healthz
Nov 11 21:51:27 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:27.963180    3492 prober.go:164] HTTP-Probe Headers: map[]
Nov 11 21:51:27 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:27.965630    3492 http.go:82] Probe succeeded for http://127.0.0.1:10252/healthz, Response: {200 OK 200 HT
Nov 11 21:51:27 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:27.965720    3492 prober.go:115] Liveness probe for "kube-controller-manager-ngpyesstack11.fyre.ibm.com_ku
Nov 11 21:51:27 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:27.975094    3492 prober.go:161] HTTP-Probe Host: http://127.0.0.1, Port: 10251, Path: /healthz
Nov 11 21:51:27 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:27.975136    3492 prober.go:164] HTTP-Probe Headers: map[]
Nov 11 21:51:27 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:27.975936    3492 http.go:82] Probe succeeded for http://127.0.0.1:10251/healthz, Response: {200 OK 200 HT
Nov 11 21:51:27 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:27.975998    3492 prober.go:115] Liveness probe for "kube-scheduler-ngpyesstack11.fyre.ibm.com_kube-system
Nov 11 21:51:28 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:28.445773    3492 http.go:82] Probe succeeded for http://127.0.0.1:2379/health, Response: {200 OK 200 HTTP
Nov 11 21:51:28 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:28.445874    3492 prober.go:115] Liveness probe for "etcd-ngpyesstack11.fyre.ibm.com_kube-system(a81a848a0
Nov 11 21:51:28 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:28.936123    3492 kubelet.go:2370] SyncLoop (housekeeping)
Nov 11 21:51:30 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:30.936078    3492 kubelet.go:2370] SyncLoop (housekeeping)
Nov 11 21:51:32 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:32.936289    3492 kubelet.go:2370] SyncLoop (housekeeping)
Nov 11 21:51:32 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:32.998524    3492 file.go:135] Reading config file "/etc/kubernetes/manifests/etcd.json"
Nov 11 21:51:32 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:32.999436    3492 file.go:135] Reading config file "/etc/kubernetes/manifests/kube-apiserver.json"
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:32.999968    3492 file.go:135] Reading config file "/etc/kubernetes/manifests/kube-controller-manager.json
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.003624    3492 file.go:135] Reading config file "/etc/kubernetes/manifests/kube-scheduler.json"
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.004141    3492 config.go:281] Setting pods for source file
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170470    3492 factory.go:104] Error trying to work out if we can handle /system.slice/var-lib-docker-a
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170527    3492 factory.go:115] Factory "docker" was unable to handle container "/system.slice/var-lib-d
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170540    3492 factory.go:108] Factory "systemd" can handle container "/system.slice/var-lib-docker-auf
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170549    3492 manager.go:843] ignoring container "/system.slice/var-lib-docker-aufs.mount"
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170565    3492 factory.go:104] Error trying to work out if we can handle /system.slice/boot.mount: inva
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170571    3492 factory.go:115] Factory "docker" was unable to handle container "/system.slice/boot.moun
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170578    3492 factory.go:108] Factory "systemd" can handle container "/system.slice/boot.mount", but i
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170585    3492 manager.go:843] ignoring container "/system.slice/boot.mount"
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170595    3492 factory.go:104] Error trying to work out if we can handle /system.slice/proc-sys-fs-binf
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170600    3492 factory.go:115] Factory "docker" was unable to handle container "/system.slice/proc-sys-
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170607    3492 factory.go:108] Factory "systemd" can handle container "/system.slice/proc-sys-fs-binfmt
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170614    3492 manager.go:843] ignoring container "/system.slice/proc-sys-fs-binfmt_misc.mount"
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170625    3492 factory.go:104] Error trying to work out if we can handle /system.slice/sys-fs-fuse-conn
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170630    3492 factory.go:115] Factory "docker" was unable to handle container "/system.slice/sys-fs-fu
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170636    3492 factory.go:108] Factory "systemd" can handle container "/system.slice/sys-fs-fuse-connec
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170643    3492 manager.go:843] ignoring container "/system.slice/sys-fs-fuse-connections.mount"
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170652    3492 factory.go:104] Error trying to work out if we can handle /system.slice/dev-mqueue.mount
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170657    3492 factory.go:115] Factory "docker" was unable to handle container "/system.slice/dev-mqueu
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170663    3492 factory.go:108] Factory "systemd" can handle container "/system.slice/dev-mqueue.mount",
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170670    3492 manager.go:843] ignoring container "/system.slice/dev-mqueue.mount"
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170680    3492 factory.go:104] Error trying to work out if we can handle /system.slice/run-docker-netns
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170685    3492 factory.go:115] Factory "docker" was unable to handle container "/system.slice/run-docke
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170692    3492 factory.go:108] Factory "systemd" can handle container "/system.slice/run-docker-netns-d
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170703    3492 manager.go:843] ignoring container "/system.slice/run-docker-netns-default.mount"
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170713    3492 factory.go:104] Error trying to work out if we can handle /system.slice/dev-hugepages.mo
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170719    3492 factory.go:115] Factory "docker" was unable to handle container "/system.slice/dev-hugep
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170725    3492 factory.go:108] Factory "systemd" can handle container "/system.slice/dev-hugepages.moun
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170733    3492 manager.go:843] ignoring container "/system.slice/dev-hugepages.mount"
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170742    3492 factory.go:104] Error trying to work out if we can handle /system.slice/sys-kernel-debug
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170748    3492 factory.go:115] Factory "docker" was unable to handle container "/system.slice/sys-kerne
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170755    3492 factory.go:108] Factory "systemd" can handle container "/system.slice/sys-kernel-debug.m
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170762    3492 manager.go:843] ignoring container "/system.slice/sys-kernel-debug.mount"
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170771    3492 factory.go:104] Error trying to work out if we can handle /system.slice/run-user-0.mount
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170777    3492 factory.go:115] Factory "docker" was unable to handle container "/system.slice/run-user-
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170783    3492 factory.go:108] Factory "systemd" can handle container "/system.slice/run-user-0.mount",
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170789    3492 manager.go:843] ignoring container "/system.slice/run-user-0.mount"
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170798    3492 factory.go:104] Error trying to work out if we can handle /system.slice/var-lib-lxcfs.mo
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170803    3492 factory.go:115] Factory "docker" was unable to handle container "/system.slice/var-lib-l
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170810    3492 factory.go:108] Factory "systemd" can handle container "/system.slice/var-lib-lxcfs.moun
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170816    3492 manager.go:843] ignoring container "/system.slice/var-lib-lxcfs.mount"
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170825    3492 factory.go:104] Error trying to work out if we can handle /system.slice/run-rpc_pipefs.m
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170830    3492 factory.go:115] Factory "docker" was unable to handle container "/system.slice/run-rpc_p
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170837    3492 factory.go:108] Factory "systemd" can handle container "/system.slice/run-rpc_pipefs.mou
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170844    3492 manager.go:843] ignoring container "/system.slice/run-rpc_pipefs.mount"
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170852    3492 factory.go:104] Error trying to work out if we can handle /system.slice/-.mount: invalid
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170857    3492 factory.go:115] Factory "docker" was unable to handle container "/system.slice/-.mount"
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170864    3492 factory.go:108] Factory "systemd" can handle container "/system.slice/-.mount", but igno
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.170871    3492 manager.go:843] ignoring container "/system.slice/-.mount"
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.298381    3492 iptables.go:362] running iptables -N [KUBE-MARK-DROP -t nat]
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.304250    3492 iptables.go:362] running iptables -C [KUBE-MARK-DROP -t nat -j MARK --set-xmark 0x000080
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.307435    3492 iptables.go:362] running iptables -N [KUBE-FIREWALL -t filter]
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.313591    3492 iptables.go:362] running iptables -C [KUBE-FIREWALL -t filter -m comment --comment kuber
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.320341    3492 iptables.go:362] running iptables -C [OUTPUT -t filter -j KUBE-FIREWALL]
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.325391    3492 iptables.go:362] running iptables -C [INPUT -t filter -j KUBE-FIREWALL]
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.332309    3492 iptables.go:362] running iptables -N [KUBE-MARK-MASQ -t nat]
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.336352    3492 iptables.go:362] running iptables -N [KUBE-POSTROUTING -t nat]
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.338382    3492 iptables.go:362] running iptables -C [KUBE-MARK-MASQ -t nat -j MARK --set-xmark 0x000040
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.340576    3492 iptables.go:362] running iptables -C [POSTROUTING -t nat -m comment --comment kubernetes
Nov 11 21:51:33 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:33.343372    3492 iptables.go:362] running iptables -C [KUBE-POSTROUTING -t nat -m comment --comment kuber
Nov 11 21:51:34 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:34.936086    3492 kubelet.go:2370] SyncLoop (housekeeping)
Nov 11 21:51:35 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:35.428397    3492 conversion.go:133] failed to handle multiple devices for container. Skipping Filesystem
Nov 11 21:51:35 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:35.428445    3492 conversion.go:133] failed to handle multiple devices for container. Skipping Filesystem
Nov 11 21:51:35 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:35.457301    3492 summary.go:352] Missing default interface "eth0" for node:ngpyesstack11.fyre.ibm.com
Nov 11 21:51:35 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:35.457403    3492 summary.go:352] Missing default interface "eth0" for pod:kube-system_kube-controller-man
Nov 11 21:51:35 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:35.457431    3492 summary.go:352] Missing default interface "eth0" for pod:kube-system_kube-apiserver-ngpy
Nov 11 21:51:35 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:35.457454    3492 summary.go:352] Missing default interface "eth0" for pod:kube-system_etcd-ngpyesstack11.
Nov 11 21:51:35 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:35.457479    3492 summary.go:352] Missing default interface "eth0" for pod:kube-system_kube-scheduler-ngpy
Nov 11 21:51:35 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:35.457520    3492 eviction_manager.go:204] eviction manager: no resources are starved
Nov 11 21:51:35 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:35.785800    3492 volume_stat_calculator.go:103] Failed to calculate volume metrics for pod kube-controlle
Nov 11 21:51:35 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:35.785861    3492 volume_stat_calculator.go:103] Failed to calculate volume metrics for pod kube-controlle
Nov 11 21:51:36 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:36.936152    3492 kubelet.go:2370] SyncLoop (housekeeping)
Nov 11 21:51:37 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:37.944070    3492 prober.go:161] HTTP-Probe Host: http://127.0.0.1, Port: 2379, Path: /health
Nov 11 21:51:37 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:37.944123    3492 prober.go:164] HTTP-Probe Headers: map[]
Nov 11 21:51:37 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:37.950435    3492 prober.go:161] HTTP-Probe Host: http://127.0.0.1, Port: 8080, Path: /healthz
Nov 11 21:51:37 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:37.950483    3492 prober.go:164] HTTP-Probe Headers: map[]
Nov 11 21:51:37 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:37.951550    3492 http.go:82] Probe succeeded for http://127.0.0.1:8080/healthz, Response: {200 OK 200 HTT
Nov 11 21:51:37 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:37.951596    3492 prober.go:115] Liveness probe for "kube-apiserver-ngpyesstack11.fyre.ibm.com_kube-system
Nov 11 21:51:37 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:37.962614    3492 prober.go:161] HTTP-Probe Host: http://127.0.0.1, Port: 10252, Path: /healthz
Nov 11 21:51:37 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:37.962665    3492 prober.go:164] HTTP-Probe Headers: map[]
Nov 11 21:51:37 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:37.963505    3492 http.go:82] Probe succeeded for http://127.0.0.1:10252/healthz, Response: {200 OK 200 HT
Nov 11 21:51:37 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:37.963559    3492 prober.go:115] Liveness probe for "kube-controller-manager-ngpyesstack11.fyre.ibm.com_ku
Nov 11 21:51:37 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:37.975116    3492 prober.go:161] HTTP-Probe Host: http://127.0.0.1, Port: 10251, Path: /healthz
Nov 11 21:51:37 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:37.975169    3492 prober.go:164] HTTP-Probe Headers: map[]
Nov 11 21:51:37 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:37.976120    3492 http.go:82] Probe succeeded for http://127.0.0.1:10251/healthz, Response: {200 OK 200 HT
Nov 11 21:51:37 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:37.976171    3492 prober.go:115] Liveness probe for "kube-scheduler-ngpyesstack11.fyre.ibm.com_kube-system
Nov 11 21:51:38 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:38.446645    3492 http.go:82] Probe succeeded for http://127.0.0.1:2379/health, Response: {200 OK 200 HTTP
Nov 11 21:51:38 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:38.446754    3492 prober.go:115] Liveness probe for "etcd-ngpyesstack11.fyre.ibm.com_kube-system(a81a848a0
Nov 11 21:51:38 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:38.936184    3492 kubelet.go:2370] SyncLoop (housekeeping)
Nov 11 21:51:40 NGPYesStack11.fyre.ibm.com kubelet[3492]: I1111 21:51:40.936082    3492 kubelet.go:2370] SyncLoop (housekeeping)
lines 10968-10990/10990 (END)

Can someone give some advice?

The document does not have any detail manual steps can do for install and setup if those tools are not working.

luxas commented 7 years ago

Please reopen in the kubeadm repo (https://github.com/kubernetes/kubeadm/issues) if you can reproduce with kubeadm v1.6. Thanks!