search

luxas / kubeadm-workshop

Showcasing a bare-metal multi-platform kubeadm setup with persistent storage and monitoring
MIT License
680 stars 190 forks source link

kubeadm in self-hosted mode (1.6.1) fails to startup scheduler #8

Closed rhuss closed 7 years ago

rhuss commented 7 years ago

I just did a super fresh install of everything but now hit the following issue with kubeadm 1.6.1

A 

KUBE_HYPERKUBE_IMAGE=luxas/hyperkube:v1.6.0-and-PR-42911 kubeadm init --config /etc/kubernetes/kubeadm.yml

with /etc/kubernetes/kubeadm.yml (from the README except an extra predefined token)

kind: MasterConfiguration
apiVersion: kubeadm.k8s.io/v1alpha1
token: "3d63a1.d93b1c68af864b80"
networking:
  podSubnet: "10.1.0.0/16"
#  serviceSubnet: "10.200.100.0/24"
controllerManagerExtraArgs:
  controllers: "*,-persistentvolume-binder,bootstrapsigner,tokencleaner"
  horizontal-pod-autoscaler-use-rest-clients: "true"
  horizontal-pod-autoscaler-sync-period: "10s"
  node-monitor-grace-period: "10s"
apiServerExtraArgs:
  runtime-config: "api/all=true"
  feature-gates: "TaintBasedEvictions=true"
  proxy-client-cert-file: "/etc/kubernetes/pki/front-proxy-client.crt"
  proxy-client-key-file: "/etc/kubernetes/pki/front-proxy-client.key"
selfHosted: true

leads to 

root@n0:/home/pi# KUBE_HYPERKUBE_IMAGE=luxas/hyperkube:v1.6.0-and-PR-42911 kubeadm init --config /etc/kubernetes/kubeadm.yml
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[init] Using Kubernetes version: v1.6.0
[init] Using Authorization mode: RBAC
[preflight] Running pre-flight checks
[preflight] Starting the kubelet service
[certificates] Generated CA certificate and key.
[certificates] Generated API server certificate and key.
[certificates] API Server serving cert is signed for DNS names [n0 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.23.200]
[certificates] Generated API server kubelet client certificate and key.
[certificates] Generated service account token signing key and public key.
[certificates] Generated front-proxy CA certificate and key.
[certificates] Generated front-proxy client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[apiclient] Created API client, waiting for the control plane to become ready
[apiclient] All control plane components are healthy after 38.382559 seconds
[apiclient] Waiting for at least one node to register
[apiclient] First node has registered after 8.119716 seconds
[self-hosted] Creating self-hosted control plane...
[self-hosted] kube-apiserver DaemonSet current=0, desired=0
[self-hosted] Found 0 self-hosted-kube-apiserver pods
[self-hosted] Found 0 self-hosted-kube-apiserver pods
[self-hosted] Found 0 self-hosted-kube-apiserver pods
....
[self-hosted] Found 1 self-hosted-kube-apiserver pods
[self-hosted] Pod self-hosted-kube-apiserver-jx8s8 status: Pending
[self-hosted] Found 1 self-hosted-kube-apiserver pods
[self-hosted] Pod self-hosted-kube-apiserver-jx8s8 status: Pending
[self-hosted] Found 1 self-hosted-kube-apiserver pods
.....
[self-hosted] Pod self-hosted-kube-apiserver-jx8s8 status: Running
[apiclient] All control plane components are healthy after 19.398582 seconds
[self-hosted] self-hosted kube-apiserver ready after 41.020531 seconds
[self-hosted] Found 0 self-hosted-kube-scheduler pods
[self-hosted] Found 0 self-hosted-kube-scheduler pods
[self-hosted] Found 0 self-hosted-kube-scheduler pods
[self-hosted] Found 0 self-hosted-kube-scheduler pods
[self-hosted] Found 0 self-hosted-kube-scheduler pods
[self-hosted] Found 0 self-hosted-kube-scheduler pods
[self-hosted] Found 1 self-hosted-kube-scheduler pods
[self-hosted] Pod self-hosted-kube-scheduler-2660681130-r0php status: Pending
[self-hosted] Found 1 self-hosted-kube-scheduler pods
[self-hosted] Pod self-hosted-kube-scheduler-2660681130-r0php status: Pending
[self-hosted] Found 1 self-hosted-kube-scheduler pods
[self-hosted] Pod self-hosted-kube-scheduler-2660681130-r0php status: Pending
[self-hosted] Found 1 self-hosted-kube-scheduler pods
[self-hosted] Pod self-hosted-kube-scheduler-2660681130-r0php status: Pending
[self-hosted] Found 1 self-hosted-kube-scheduler pods
.....
(never ending)

which indicates that the scheduler pod doesn't come up. And indeed the problem is that the scheduler pod cannot be scheduled:

root@n0:/home/pi# kubectl get pods --all-namespaces
NAMESPACE     NAME                                          READY     STATUS    RESTARTS   AGE
kube-system   etcd-n0                                       1/1       Running   0          1m
kube-system   kube-controller-manager-n0                    1/1       Running   0          22s
kube-system   kube-scheduler-n0                             1/1       Running   1          1m
kube-system   self-hosted-kube-apiserver-jx8s8              1/1       Running   0          2m
kube-system   self-hosted-kube-scheduler-2660681130-r0php   0/1       Pending   0          1m

root@n0:/home/pi# kubectl describe pod self-hosted-kube-scheduler-2660681130-r0php --namespace=kube-system
Name:       self-hosted-kube-scheduler-2660681130-r0php
Namespace:  kube-system
Node:       /
Labels:     component=kube-scheduler
        k8s-app=self-hosted-kube-scheduler
        pod-template-hash=2660681130
        tier=control-plane
Annotations:    kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"kube-system","name":"self-hosted-kube-scheduler-2660681130","uid":"449d6cb5-1917-...
Status:     Pending
IP:
Controllers:    ReplicaSet/self-hosted-kube-scheduler-2660681130
Containers:
  self-hosted-kube-scheduler:
    Image:  luxas/hyperkube:v1.6.0-and-PR-42911
    Port:
    Command:
      /hyperkube
      scheduler
      --address=127.0.0.1
      --leader-elect=true
      --kubeconfig=/etc/kubernetes/scheduler.conf
    Requests:
      cpu:      100m
    Liveness:       http-get http://127.0.0.1:10251/healthz delay=15s timeout=15s period=10s #success=1 #failure=8
    Environment:    <none>
    Mounts:
      /etc/kubernetes/ from k8s (ro)
      /etc/ssl/certs from certs (rw)
      /var/lock from var-lock (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-4szsp (ro)
Conditions:
  Type      Status
  PodScheduled  False
Volumes:
  k8s:
    Type:   HostPath (bare host directory volume)
    Path:   /etc/kubernetes
  certs:
    Type:   HostPath (bare host directory volume)
    Path:   /etc/ssl/certs
  var-lock:
    Type:   HostPath (bare host directory volume)
    Path:   /var/lock
  default-token-4szsp:
    Type:   Secret (a volume populated by a Secret)
    SecretName: default-token-4szsp
    Optional:   false
QoS Class:  Burstable
Node-Selectors: node-role.kubernetes.io/master=
Tolerations:    node-role.kubernetes.io/master=:NoSchedule
        node.alpha.kubernetes.io/notReady=:Exists:NoExecute for 300s
        node.alpha.kubernetes.io/unreachable=:Exists:NoExecute for 300s
Events:
  FirstSeen LastSeen    Count   From            SubObjectPath   Type        Reason      Message
  --------- --------    -----   ----            -------------   --------    ------      -------
  1m        42s     8   default-scheduler           Warning     FailedScheduling    no nodes available to schedule pods

Looks like a catch-22 ;-)

rhuss commented 7 years ago

these are the logs of the Docker container:

root@n0:/home/pi# docker logs 9552a3fccc4b
I0404 09:16:00.523415       1 leaderelection.go:179] attempting to acquire leader lease...
I0404 09:16:01.711489       1 leaderelection.go:189] successfully acquired lease kube-system/kube-scheduler
I0404 09:16:01.714291       1 event.go:217] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"kube-scheduler", UID:"2907d1f8-1917-11e7-9f3c-b827ebadcfc2", APIVersion:"v1", ResourceVersion:"388", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' n0 became leader
I0404 09:16:01.715333       1 event.go:217] Event(v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"self-hosted-kube-scheduler-2660681130-r0php", UID:"4671c05b-1917-11e7-8dc6-b827ebadcfc2", APIVersion:"v1", ResourceVersion:"362", FieldPath:""}): type: 'Warning' reason: 'FailedScheduling' no nodes available to schedule pods
I0404 09:16:01.751626       1 event.go:217] Event(v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"self-hosted-kube-scheduler-2660681130-r0php", UID:"4671c05b-1917-11e7-8dc6-b827ebadcfc2", APIVersion:"v1", ResourceVersion:"389", FieldPath:""}): type: 'Warning' reason: 'FailedScheduling' no nodes available to schedule pods
W0404 09:16:01.751239       1 factory.go:533] Request for pod kube-system/self-hosted-kube-scheduler-2660681130-r0php already in flight, abandoning
I0404 09:16:02.734193       1 event.go:217] Event(v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"self-hosted-kube-scheduler-2660681130-r0php", UID:"4671c05b-1917-11e7-8dc6-b827ebadcfc2", APIVersion:"v1", ResourceVersion:"389", FieldPath:""}): type: 'Warning' reason: 'FailedScheduling' no nodes available to schedule pods
I0404 09:16:04.757063       1 event.go:217] Event(v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"self-hosted-kube-scheduler-2660681130-r0php", UID:"4671c05b-1917-11e7-8dc6-b827ebadcfc2", APIVersion:"v1", ResourceVersion:"389", FieldPath:""}): type: 'Warning' reason: 'FailedScheduling' no nodes available to schedule pods
....
rhuss commented 7 years ago

Maybe also interesting that there is also a stopped scheduler container these logs:

Full log ``` docker logs d59b24c9d1a1 E0404 09:14:27.925556 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:27.927174 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused I0404 09:14:27.928893 1 leaderelection.go:179] attempting to acquire leader lease... E0404 09:14:27.932526 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:27.933367 1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: Get https://192.168.23.200:6443/api/v1/namespaces/kube-system/endpoints/kube-scheduler: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:27.935295 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:27.938076 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:27.939356 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:27.941254 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:27.942252 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:27.943986 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:28.930698 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:28.933691 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:28.934629 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:28.937206 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:28.939908 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:28.942690 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:28.942680 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:28.944690 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:28.946494 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:29.933107 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:29.936630 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:29.937594 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:29.939295 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:29.943207 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:29.944911 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:29.946020 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:29.948113 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:29.948132 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:30.935302 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:30.938779 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:30.940880 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:30.941455 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:30.946629 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:30.947426 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:30.949145 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:30.950186 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:30.951138 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:30.973177 1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: Get https://192.168.23.200:6443/api/v1/namespaces/kube-system/endpoints/kube-scheduler: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:31.937689 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:31.940800 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:31.942910 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:31.944463 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:31.948670 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:31.949826 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:31.951432 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:31.953286 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:31.953504 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:32.939885 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:32.942699 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:32.945117 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:32.946964 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:32.951129 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:32.953431 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:32.955548 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:32.956720 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:32.957001 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:33.942088 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:33.944747 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:33.947487 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:33.949991 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:33.952923 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:33.955752 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:33.957823 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:33.959875 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:33.959875 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:34.944226 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:34.946945 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:34.949646 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:34.951733 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:34.954721 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:34.957681 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:34.959827 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:34.961891 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:34.963770 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:35.372042 1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: Get https://192.168.23.200:6443/api/v1/namespaces/kube-system/endpoints/kube-scheduler: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:35.946302 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:35.948921 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:35.951742 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:35.953808 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:35.956664 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:35.959510 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:35.961663 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:35.963735 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:35.965944 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:36.948552 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:36.950985 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:36.953846 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:36.955906 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:36.958620 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:36.961310 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:36.963601 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:36.965694 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:36.968070 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:37.950854 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:37.953029 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:37.957571 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:37.957901 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:37.960660 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:37.963205 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:37.965626 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:37.968097 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:37.969899 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:38.402619 1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: Get https://192.168.23.200:6443/api/v1/namespaces/kube-system/endpoints/kube-scheduler: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:38.953136 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:38.955059 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:38.960051 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:38.961204 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:38.963148 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:38.965336 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:38.967653 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:38.970227 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:38.972056 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:39.957319 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:39.958701 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:39.963426 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:39.964318 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:39.965881 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:39.968060 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:39.969932 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:39.972155 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:39.974368 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:40.959180 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:40.961141 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:40.965715 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:40.966965 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:40.968548 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:40.970307 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:40.971894 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:40.973985 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:40.976628 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:41.961205 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:41.963975 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:41.968542 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:41.969904 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:41.970593 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:41.972589 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:41.973810 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:41.975912 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:41.978747 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:42.278273 1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: Get https://192.168.23.200:6443/api/v1/namespaces/kube-system/endpoints/kube-scheduler: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:42.963240 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:42.966679 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:42.971044 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:42.973321 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:42.973520 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:42.976741 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:42.978891 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:42.980871 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:42.981876 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:43.965190 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:43.969088 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:43.973076 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:43.975219 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:43.976155 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:43.981512 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:43.982122 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:43.983368 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:43.984129 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:44.969243 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:44.971918 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:44.977625 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:44.975556 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:44.979487 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:44.989462 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:44.991187 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:44.991187 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:44.992362 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:45.971443 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:45.974337 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:45.979647 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:45.982397 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:45.987661 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:45.991601 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:45.993840 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:45.995735 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:45.997787 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:46.004426 1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: Get https://192.168.23.200:6443/api/v1/namespaces/kube-system/endpoints/kube-scheduler: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:14:48.915127 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: User "system:kube-scheduler" cannot list replicasets.extensions at the cluster scope. (get replicasets.extensions) E0404 09:14:48.915812 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: User "system:kube-scheduler" cannot list pods at the cluster scope. (get pods) E0404 09:14:48.942015 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: User "system:kube-scheduler" cannot list replicationcontrollers at the cluster scope. (get replicationcontrollers) E0404 09:14:48.942507 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: User "system:kube-scheduler" cannot list pods at the cluster scope. (get pods) E0404 09:14:48.942015 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: User "system:kube-scheduler" cannot list persistentvolumes at the cluster scope. (get persistentvolumes) E0404 09:14:48.942256 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: User "system:kube-scheduler" cannot list nodes at the cluster scope. (get nodes) E0404 09:14:48.948150 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: User "system:kube-scheduler" cannot list services at the cluster scope. (get services) E0404 09:14:48.949463 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: User "system:kube-scheduler" cannot list statefulsets.apps at the cluster scope. (get statefulsets.apps) E0404 09:14:48.958913 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: User "system:kube-scheduler" cannot list persistentvolumeclaims at the cluster scope. (get persistentvolumeclaims) E0404 09:14:49.415261 1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: User "system:kube-scheduler" cannot get endpoints in the namespace "kube-system". (get endpoints kube-scheduler) E0404 09:14:49.931453 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: User "system:kube-scheduler" cannot list replicasets.extensions at the cluster scope. (get replicasets.extensions) E0404 09:14:49.932451 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: User "system:kube-scheduler" cannot list pods at the cluster scope. (get pods) E0404 09:14:49.965471 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: User "system:kube-scheduler" cannot list replicationcontrollers at the cluster scope. (get replicationcontrollers) E0404 09:14:49.971820 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: User "system:kube-scheduler" cannot list persistentvolumes at the cluster scope. (get persistentvolumes) E0404 09:14:49.994315 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: User "system:kube-scheduler" cannot list statefulsets.apps at the cluster scope. (get statefulsets.apps) E0404 09:14:49.999929 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: User "system:kube-scheduler" cannot list pods at the cluster scope. (get pods) E0404 09:14:50.018599 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: User "system:kube-scheduler" cannot list services at the cluster scope. (get services) E0404 09:14:50.019916 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: User "system:kube-scheduler" cannot list nodes at the cluster scope. (get nodes) E0404 09:14:50.027406 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: User "system:kube-scheduler" cannot list persistentvolumeclaims at the cluster scope. (get persistentvolumeclaims) I0404 09:14:52.155166 1 leaderelection.go:189] successfully acquired lease kube-system/kube-scheduler I0404 09:14:52.155917 1 event.go:217] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"kube-scheduler", UID:"2907d1f8-1917-11e7-9f3c-b827ebadcfc2", APIVersion:"v1", ResourceVersion:"108", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' n0 became leader E0404 09:15:18.992518 1 reflector.go:304] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to watch *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=1&timeoutSeconds=331&watch=true: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:18.995190 1 reflector.go:304] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to watch *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=1&timeoutSeconds=522&watch=true: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:19.021403 1 reflector.go:304] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to watch *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=1&timeoutSeconds=320&watch=true: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:19.032207 1 reflector.go:304] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to watch *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=1&timeoutSeconds=486&watch=true: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:19.034051 1 reflector.go:304] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to watch *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=1&timeoutSeconds=514&watch=true: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:19.035817 1 reflector.go:304] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to watch *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=344&timeoutSeconds=464&watch=true: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:19.039588 1 reflector.go:304] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to watch *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=347&timeoutSeconds=428&watch=true: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:19.041034 1 reflector.go:304] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to watch *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=1&timeoutSeconds=370&watch=true: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:19.058208 1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: Get https://192.168.23.200:6443/api/v1/namespaces/kube-system/endpoints/kube-scheduler: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:19.059912 1 reflector.go:304] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to watch *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=29&timeoutSeconds=340&watch=true: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:19.996723 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:19.999419 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:20.024563 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:20.037968 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:20.042794 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:20.035975 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:20.049952 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:20.059146 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:20.072081 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:21.002340 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:21.007076 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:21.017813 1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: Get https://192.168.23.200:6443/api/v1/namespaces/kube-system/endpoints/kube-scheduler: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:21.026837 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:21.039929 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:21.046970 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:21.047142 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:21.051713 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:21.063411 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:21.077679 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:22.008886 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:22.015563 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:22.033615 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:22.043228 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:22.049021 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:22.051081 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:22.055648 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:22.067152 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:22.082464 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:23.019035 1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: Get https://192.168.23.200:6443/api/v1/namespaces/kube-system/endpoints/kube-scheduler: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:23.023464 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:23.024604 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:23.035792 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:23.045297 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:23.054780 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:23.056666 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:23.058399 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:23.069641 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:23.086056 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:24.028056 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:24.029482 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:24.039431 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:24.048866 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:24.058563 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:24.060888 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:24.061258 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:24.072249 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:24.089485 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:25.024785 1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: Get https://192.168.23.200:6443/api/v1/namespaces/kube-system/endpoints/kube-scheduler: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:25.031564 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:25.035043 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:25.041539 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:25.050891 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:25.064149 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:25.064748 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:25.066112 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:25.076926 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:25.091848 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:26.037657 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:26.038807 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused E0404 09:15:33.796928 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: User "system:kube-scheduler" cannot list services at the cluster scope. (get services) E0404 09:15:33.797857 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: User "system:kube-scheduler" cannot list persistentvolumes at the cluster scope. (get persistentvolumes) E0404 09:15:33.807645 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: User "system:kube-scheduler" cannot list pods at the cluster scope. (get pods) E0404 09:15:33.964452 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: User "system:kube-scheduler" cannot list persistentvolumeclaims at the cluster scope. (get persistentvolumeclaims) E0404 09:15:33.964748 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: User "system:kube-scheduler" cannot list nodes at the cluster scope. (get nodes) E0404 09:15:34.647041 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: User "system:kube-scheduler" cannot list replicationcontrollers at the cluster scope. (get replicationcontrollers) E0404 09:15:34.704806 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: User "system:kube-scheduler" cannot list replicasets.extensions at the cluster scope. (get replicasets.extensions) E0404 09:15:35.027778 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: User "system:kube-scheduler" cannot list statefulsets.apps at the cluster scope. (get statefulsets.apps) E0404 09:15:35.028179 1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: User "system:kube-scheduler" cannot list pods at the cluster scope. (get pods) E0404 09:15:35.028430 1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: User "system:kube-scheduler" cannot get endpoints in the namespace "kube-system". (get endpoints kube-scheduler) E0404 09:15:35.029417 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: User "system:kube-scheduler" cannot list services at the cluster scope. (get services) E0404 09:15:35.029694 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: User "system:kube-scheduler" cannot list persistentvolumes at the cluster scope. (get persistentvolumes) E0404 09:15:35.263828 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: User "system:kube-scheduler" cannot list persistentvolumeclaims at the cluster scope. (get persistentvolumeclaims) E0404 09:15:35.264118 1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: User "system:kube-scheduler" cannot get endpoints in the namespace "kube-system". (get endpoints kube-scheduler) E0404 09:15:35.265475 1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: User "system:kube-scheduler" cannot list nodes at the cluster scope. (get nodes) E0404 09:15:35.265388 1 event.go:259] Could not construct reference to: '&v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:""}, Subsets:[]v1.EndpointSubset(nil)}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'LeaderElection' 'n0 stopped leading' I0404 09:15:35.267394 1 leaderelection.go:208] failed to renew lease kube-system/kube-scheduler F0404 09:15:35.268121 1 server.go:134] lost master ```
rhuss commented 7 years ago

Actually this is only an issue when running in self-hosted mode. Without selfHosted: true in the config file it works.

Current status: I'm now back using plain kubeadm 1.6.1 (without your custom hyperkube image) and disabled some features in the config file (like proxy-client-cert-file) since these are not available in for vanilla apiserver 1.6.0.

Now it works, and even weave is happy (because of different machine-ids now).

Now going to try PVs and Ingress (as I guess they don't require your customisations in the adapter hyperkube image ...)

Let me know if I can help further in evaluating the issue with the self-hosted mode.

luxas commented 7 years ago

Known issue. Since the kubelet behaviour changed drastically between beta.4 and rc.1-ish and kubeadm broke fatally as a consequence, we fixed that by breaking self-hosting only in v1.6.1 ;)

It's on my list to do pretty soon, but well, school ;)

As you noted, it will just work fine to disable proxy* and self-hosting if you don't want the API aggregation thing.

Awesome to see you're working on this. When you find flaws, feel free to do a PR as well :+1:

rhuss commented 7 years ago

Thanks for the info and for your awesome work on kubeadm and kubernetes in general.

np at all, take your time. School is important. still ;-)

'hope to contribute more, still looking around to get a feeling of the project(s). And yes, spare time is an issue here, too :)

luxas commented 7 years ago

I could now move from self-hosting to using static pods as normal, so this should be ok now :+1: Thanks ;)