kubernetes v1.13 kubeadm init; kubelet has troube to communicate with api server - connection refused; no pod-network, flannel does not work - all control plane pods have the same host IP, but other pods get IP from docker0 brigde

[tunggad]

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version (use kubeadm version): kubeadm version: &version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"ddf47ac13c1a9483ea035a79cd7c10005ff21a6d", GitTreeState:"clean", BuildDate:"2018-12-03T21:02:01Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version): Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"ddf47ac13c1a9483ea035a79cd7c10005ff21a6d", GitTreeState:"clean", BuildDate:"2018-12-03T21:04:45Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.1", GitCommit:"eec55b9ba98609a46fee712359c7b5b365bdd920", GitTreeState:"clean", BuildDate:"2018-12-13T10:31:33Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
• Cloud provider or hardware configuration: It is a bare metal local machine, AMD Athlon X 640 / 8gb ram.
• OS (e.g. from /etc/os-release):

  NAME="Arch Linux"
  PRETTY_NAME="Arch Linux"
  ID=arch
  ID_LIKE=archlinux
  ANSI_COLOR="0;36"
  HOME_URL="https://www.archlinux.org/"
  SUPPORT_URL="https://bbs.archlinux.org/"
  BUG_REPORT_URL="https://bugs.archlinux.org/"

• Others: docker info:

  Containers: 36
  Running: 16
  Paused: 0
  Stopped: 20
  Images: 14
  Server Version: 18.09.0-ce
  Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: false
  Logging Driver: json-file
  Cgroup Driver: cgroupfs
  Plugins:
  Volume: local
  Network: bridge host macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
  Swarm: inactive
  Runtimes: runc
  Default Runtime: runc
  Init Binary: docker-init
  containerd version: 9f2e07b1fc1342d1c48fe4d7bbb94cb6d1bf278b.m
  runc version: 079817cc26ec5292ac375bb9f47f373d33574949
  init version: fec3683
  Security Options:
  seccomp
  Profile: default
  Kernel Version: 4.19.9-arch1-1-ARCH
  Operating System: Arch Linux
  OSType: linux
  Architecture: x86_64
  CPUs: 4
  Total Memory: 7.788GiB
  Name: archserver
  ID: E7N3:JTBI:QQCO:5LJE:ROSP:DYCI:4BEN:PS3O:EYQI:2UZY:YI5K:M6YN
  Docker Root Dir: /var/lib/docker
  Debug Mode (client): false
  Debug Mode (server): false
  Registry: https://index.docker.io/v1/
  Labels:
  Experimental: false
  Insecure Registries:
  127.0.0.0/8
  Live Restore Enabled: false

  iptables -S:

  
  -P INPUT ACCEPT
  -P FORWARD ACCEPT
  -P OUTPUT ACCEPT
  -N DOCKER
  -N DOCKER-ISOLATION-STAGE-1
  -N DOCKER-ISOLATION-STAGE-2
  -N DOCKER-USER
  -N KUBE-EXTERNAL-SERVICES
  -N KUBE-FIREWALL
  -N KUBE-FORWARD
  -N KUBE-SERVICES
  -A INPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes externally-visible service portals" -j KUBE-EXTERNAL-SERVICES
  -A INPUT -j KUBE-FIREWALL
  -A INPUT -i virbr1 -p udp -m udp --dport 53 -j ACCEPT
  -A INPUT -i virbr1 -p tcp -m tcp --dport 53 -j ACCEPT
  -A INPUT -i virbr1 -p udp -m udp --dport 67 -j ACCEPT
  -A INPUT -i virbr1 -p tcp -m tcp --dport 67 -j ACCEPT
  -A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
  -A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
  -A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
  -A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
  -A INPUT -i virbr2 -p udp -m udp --dport 53 -j ACCEPT
  -A INPUT -i virbr2 -p tcp -m tcp --dport 53 -j ACCEPT
  -A INPUT -i virbr2 -p udp -m udp --dport 67 -j ACCEPT
  -A INPUT -i virbr2 -p tcp -m tcp --dport 67 -j ACCEPT
  -A FORWARD -m comment --comment "kubernetes forwarding rules" -j KUBE-FORWARD
  -A FORWARD -j DOCKER-USER
  -A FORWARD -j DOCKER-ISOLATION-STAGE-1
  -A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
  -A FORWARD -o docker0 -j DOCKER
  -A FORWARD -i docker0 ! -o docker0 -j ACCEPT
  -A FORWARD -i docker0 -o docker0 -j ACCEPT
  -A FORWARD -d 192.168.42.0/24 -o virbr1 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
  -A FORWARD -s 192.168.42.0/24 -i virbr1 -j ACCEPT
  -A FORWARD -i virbr1 -o virbr1 -j ACCEPT
  -A FORWARD -o virbr1 -j REJECT --reject-with icmp-port-unreachable
  -A FORWARD -i virbr1 -j REJECT --reject-with icmp-port-unreachable
  -A FORWARD -d 192.168.122.0/24 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
  -A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT
  -A FORWARD -i virbr0 -o virbr0 -j ACCEPT
  -A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
  -A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
  -A FORWARD -i virbr2 -o virbr2 -j ACCEPT
  -A FORWARD -o virbr2 -j REJECT --reject-with icmp-port-unreachable
  -A FORWARD -i virbr2 -j REJECT --reject-with icmp-port-unreachable
  -A FORWARD -s 10.244.0.0/16 -j ACCEPT
  -A FORWARD -d 10.244.0.0/16 -j ACCEPT
  -A OUTPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
  -A OUTPUT -j KUBE-FIREWALL
  -A OUTPUT -o virbr1 -p udp -m udp --dport 68 -j ACCEPT
  -A OUTPUT -o virbr0 -p udp -m udp --dport 68 -j ACCEPT
  -A OUTPUT -o virbr2 -p udp -m udp --dport 68 -j ACCEPT
  -A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
  -A DOCKER-ISOLATION-STAGE-1 -j RETURN
  -A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
  -A DOCKER-ISOLATION-STAGE-2 -j RETURN
  -A DOCKER-USER -j RETURN
  -A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
  -A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT
  -A KUBE-FORWARD -s 10.244.0.0/16 -m comment --comment "kubernetes forwarding conntrack pod source rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
  -A KUBE-FORWARD -d 10.244.0.0/16 -m comment --comment "kubernetes forwarding conntrack pod destination rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

## What happened?
i want to setup a single node master with `kubeadm init --pod-network-cidr=10.244.0.0/16` and then install flannel as network addon with following commands:

sysctl net.bridge.bridge-nf-call-iptables=1

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/bc79dd1505b0c8681ece4de4c0d86c5cd2643275/Documentation/kube-flannel.yml

But somehow kubelet keeps complaint about that it has trouble to communicate with the running api server with `connect: connection refused`, here the kubelet journal log:

Dec 23 00:07:29 archserver kubelet[957]: I1223 00:07:29.625946 957 server.go:407] Version: v1.13.0 Dec 23 00:07:29 archserver kubelet[957]: I1223 00:07:29.626430 957 plugins.go:103] No cloud provider specified. Dec 23 00:07:29 archserver kubelet[957]: I1223 00:07:29.648378 957 certificate_store.go:130] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem". Dec 23 00:07:29 archserver kubelet[957]: I1223 00:07:29.839992 957 server.go:666] --cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to / Dec 23 00:07:29 archserver kubelet[957]: I1223 00:07:29.851745 957 container_manager_linux.go:248] container manager verified user specified cgroup-root exists: [] Dec 23 00:07:29 archserver kubelet[957]: I1223 00:07:29.851873 957 container_manager_linux.go:253] Creating Container Manager object based on Node Config: {RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:} {Signal:nodefs.available Operator:LessThan Value:{Quantity: Percentage:0.1} GracePeriod:0s MinReclaim:} {Signal:nodefs.inodesFree Operator:LessThan Value:{Quantity: Percentage:0.05} GracePeriod:0s MinReclaim:} {Signal:imagefs.available Operator:LessThan Value:{Quantity: Percentage:0.15} GracePeriod:0s MinReclaim:}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalCPUManagerReconcilePeriod:10s ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms} Dec 23 00:07:29 archserver kubelet[957]: I1223 00:07:29.852343 957 container_manager_linux.go:272] Creating device plugin manager: true Dec 23 00:07:29 archserver kubelet[957]: I1223 00:07:29.852609 957 state_mem.go:36] [cpumanager] initializing new in-memory state store Dec 23 00:07:29 archserver kubelet[957]: I1223 00:07:29.864934 957 state_mem.go:84] [cpumanager] updated default cpuset: "" Dec 23 00:07:29 archserver kubelet[957]: I1223 00:07:29.864992 957 state_mem.go:92] [cpumanager] updated cpuset assignments: "map[]" Dec 23 00:07:29 archserver kubelet[957]: I1223 00:07:29.866637 957 kubelet.go:281] Adding pod path: /etc/kubernetes/manifests Dec 23 00:07:29 archserver kubelet[957]: I1223 00:07:29.866750 957 kubelet.go:306] Watching apiserver Dec 23 00:07:29 archserver kubelet[957]: E1223 00:07:29.876803 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list v1.Node: Get https://192.168.0.25:6443/api/v1/nodes?fieldSelector=metadata.name%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:29 archserver kubelet[957]: E1223 00:07:29.877145 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list v1.Service: Get https://192.168.0.25:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:29 archserver kubelet[957]: I1223 00:07:29.878818 957 client.go:75] Connecting to docker on unix:///var/run/docker.sock Dec 23 00:07:29 archserver kubelet[957]: I1223 00:07:29.878881 957 client.go:104] Start docker client with request timeout=2m0s Dec 23 00:07:29 archserver kubelet[957]: W1223 00:07:29.908557 957 docker_service.go:540] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth" Dec 23 00:07:29 archserver kubelet[957]: I1223 00:07:29.908652 957 docker_service.go:236] Hairpin mode set to "hairpin-veth" Dec 23 00:07:29 archserver kubelet[957]: E1223 00:07:29.921059 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.0.25:6443/api/v1/pods?fieldSelector=spec.nodeName%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:29 archserver kubelet[957]: I1223 00:07:29.977344 957 docker_service.go:251] Docker cri networking managed by kubernetes.io/no-op Dec 23 00:07:30 archserver kubelet[957]: I1223 00:07:30.056143 957 docker_service.go:256] Docker Info: &{ID:E7N3:JTBI:QQCO:5LJE:ROSP:DYCI:4BEN:PS3O:EYQI:2UZY:YI5K:M6YN Containers:34 ContainersRunning:0 ContainersPaused:0 ContainersStopped:34 Images:14 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Native Overlay Diff false]] SystemStatus:[] Plugins:{Volume:[local] Network:[bridge host macvlan null overlay] Authorization:[] Log:[awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6tables:true Debug:false NFd:23 OomKillDisable:true NGoroutines:44 SystemTime:2018-12-23T00:07:29.980243745Z LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:0 KernelVersion:4.19.9-arch1-1-ARCH OperatingSystem:Arch Linux OSType:linux Architecture:x86_64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:0xc00033ec40 NCPU:4 MemTotal:8362233856 GenericResources:[] DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:archserver Labels:[] ExperimentalBuild:false ServerVersion:18.09.0-ce ClusterStore: ClusterAdvertise: Runtimes:map[runc:{Path:runc Args:[]}] DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:[] Nodes:0 Managers:0 Cluster:} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:9f2e07b1fc1342d1c48fe4d7bbb94cb6d1bf278b.m Expected:9f2e07b1fc1342d1c48fe4d7bbb94cb6d1bf278b.m} RuncCommit:{ID:079817cc26ec5292ac375bb9f47f373d33574949 Expected:079817cc26ec5292ac375bb9f47f373d33574949} InitCommit:{ID:fec3683 Expected:fec3683} SecurityOptions:[name=seccomp,profile=default]} Dec 23 00:07:30 archserver kubelet[957]: I1223 00:07:30.056376 957 docker_service.go:269] Setting cgroupDriver to cgroupfs Dec 23 00:07:30 archserver kubelet[957]: I1223 00:07:30.124677 957 kuberuntime_manager.go:198] Container runtime docker initialized, version: 18.09.0-ce, apiVersion: 1.39.0 Dec 23 00:07:30 archserver kubelet[957]: I1223 00:07:30.157085 957 server.go:999] Started kubelet Dec 23 00:07:30 archserver kubelet[957]: I1223 00:07:30.157662 957 server.go:137] Starting to listen on 0.0.0.0:10250 Dec 23 00:07:30 archserver kubelet[957]: I1223 00:07:30.170590 957 server.go:333] Adding debug handlers to kubelet server. Dec 23 00:07:30 archserver kubelet[957]: I1223 00:07:30.172036 957 fs_resource_analyzer.go:66] Starting FS ResourceAnalyzer Dec 23 00:07:30 archserver kubelet[957]: I1223 00:07:30.172140 957 status_manager.go:152] Starting to sync pod status with apiserver Dec 23 00:07:30 archserver kubelet[957]: I1223 00:07:30.172177 957 kubelet.go:1829] Starting kubelet main sync loop. Dec 23 00:07:30 archserver kubelet[957]: I1223 00:07:30.172219 957 kubelet.go:1846] skipping pod synchronization - [container runtime status check may not have completed yet PLEG is not healthy: pleg has yet to be successful] Dec 23 00:07:30 archserver kubelet[957]: I1223 00:07:30.173193 957 volume_manager.go:248] Starting Kubelet Volume Manager Dec 23 00:07:30 archserver kubelet[957]: I1223 00:07:30.193187 957 desired_state_of_world_populator.go:130] Desired state populator starts to run Dec 23 00:07:30 archserver kubelet[957]: E1223 00:07:30.158057 957 kubelet.go:1308] Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data in memory cache Dec 23 00:07:30 archserver kubelet[957]: E1223 00:07:30.273492 957 event.go:212] Unable to write event: 'Post https://192.168.0.25:6443/api/v1/namespaces/default/events: dial tcp 192.168.0.25:6443: connect: connection refused' (may retry after sleeping) Dec 23 00:07:30 archserver kubelet[957]: I1223 00:07:30.291766 957 kubelet.go:1846] skipping pod synchronization - [container runtime status check may not have completed yet] Dec 23 00:07:30 archserver kubelet[957]: I1223 00:07:30.291825 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:30 archserver kubelet[957]: E1223 00:07:30.298852 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:30 archserver kubelet[957]: I1223 00:07:30.323791 957 kubelet_node_status.go:72] Attempting to register node archserver Dec 23 00:07:30 archserver kubelet[957]: E1223 00:07:30.324297 957 kubelet_node_status.go:94] Unable to register node "archserver" with API server: Post https://192.168.0.25:6443/api/v1/nodes: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:30 archserver kubelet[957]: I1223 00:07:30.345117 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:30 archserver kubelet[957]: I1223 00:07:30.348352 957 cpu_manager.go:155] [cpumanager] starting with none policy Dec 23 00:07:30 archserver kubelet[957]: I1223 00:07:30.348388 957 cpu_manager.go:156] [cpumanager] reconciling every 10s Dec 23 00:07:30 archserver kubelet[957]: I1223 00:07:30.348406 957 policy_none.go:42] [cpumanager] none policy: Start Dec 23 00:07:30 archserver kubelet[957]: E1223 00:07:30.399179 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:30 archserver kubelet[957]: E1223 00:07:30.417710 957 eviction_manager.go:243] eviction manager: failed to get summary stats: failed to get node info: node "archserver" not found Dec 23 00:07:30 archserver kubelet[957]: E1223 00:07:30.499559 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:30 archserver kubelet[957]: I1223 00:07:30.512441 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:30 archserver kubelet[957]: I1223 00:07:30.524543 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:30 archserver kubelet[957]: I1223 00:07:30.528684 957 kubelet_node_status.go:72] Attempting to register node archserver Dec 23 00:07:30 archserver kubelet[957]: E1223 00:07:30.529321 957 kubelet_node_status.go:94] Unable to register node "archserver" with API server: Post https://192.168.0.25:6443/api/v1/nodes: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:30 archserver kubelet[957]: E1223 00:07:30.599840 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:30 archserver kubelet[957]: I1223 00:07:30.615134 957 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "etcd-certs" (UniqueName: "kubernetes.io/host-path/459a3e3ab367c3ffe0309150ad8995c1-etcd-certs") pod "etcd-archserver" (UID: "459a3e3ab367c3ffe0309150ad8995c1") Dec 23 00:07:30 archserver kubelet[957]: I1223 00:07:30.615551 957 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "etcd-data" (UniqueName: "kubernetes.io/host-path/459a3e3ab367c3ffe0309150ad8995c1-etcd-data") pod "etcd-archserver" (UID: "459a3e3ab367c3ffe0309150ad8995c1") Dec 23 00:07:30 archserver kubelet[957]: I1223 00:07:30.616001 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:30 archserver kubelet[957]: I1223 00:07:30.699763 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:30 archserver kubelet[957]: E1223 00:07:30.700052 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:30 archserver kubelet[957]: I1223 00:07:30.716403 957 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "ca-certs" (UniqueName: "kubernetes.io/host-path/d92cb816c7d721bf5d651ac4cc2e875a-ca-certs") pod "kube-apiserver-archserver" (UID: "d92cb816c7d721bf5d651ac4cc2e875a") Dec 23 00:07:30 archserver kubelet[957]: I1223 00:07:30.716838 957 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "etc-ca-certificates" (UniqueName: "kubernetes.io/host-path/d92cb816c7d721bf5d651ac4cc2e875a-etc-ca-certificates") pod "kube-apiserver-archserver" (UID: "d92cb816c7d721bf5d651ac4cc2e875a") Dec 23 00:07:30 archserver kubelet[957]: I1223 00:07:30.717007 957 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "k8s-certs" (UniqueName: "kubernetes.io/host-path/d92cb816c7d721bf5d651ac4cc2e875a-k8s-certs") pod "kube-apiserver-archserver" (UID: "d92cb816c7d721bf5d651ac4cc2e875a") Dec 23 00:07:30 archserver kubelet[957]: I1223 00:07:30.717140 957 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "usr-share-ca-certificates" (UniqueName: "kubernetes.io/host-path/d92cb816c7d721bf5d651ac4cc2e875a-usr-share-ca-certificates") pod "kube-apiserver-archserver" (UID: "d92cb816c7d721bf5d651ac4cc2e875a") Dec 23 00:07:30 archserver kubelet[957]: E1223 00:07:30.800292 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:30 archserver kubelet[957]: I1223 00:07:30.800346 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:30 archserver kubelet[957]: I1223 00:07:30.818047 957 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "flexvolume-dir" (UniqueName: "kubernetes.io/host-path/8db7fcf04315eec49ec5bc8348632e73-flexvolume-dir") pod "kube-controller-manager-archserver" (UID: "8db7fcf04315eec49ec5bc8348632e73") Dec 23 00:07:30 archserver kubelet[957]: I1223 00:07:30.818240 957 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "k8s-certs" (UniqueName: "kubernetes.io/host-path/8db7fcf04315eec49ec5bc8348632e73-k8s-certs") pod "kube-controller-manager-archserver" (UID: "8db7fcf04315eec49ec5bc8348632e73") Dec 23 00:07:30 archserver kubelet[957]: I1223 00:07:30.818342 957 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "usr-share-ca-certificates" (UniqueName: "kubernetes.io/host-path/8db7fcf04315eec49ec5bc8348632e73-usr-share-ca-certificates") pod "kube-controller-manager-archserver" (UID: "8db7fcf04315eec49ec5bc8348632e73") Dec 23 00:07:30 archserver kubelet[957]: I1223 00:07:30.818593 957 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "ca-certs" (UniqueName: "kubernetes.io/host-path/8db7fcf04315eec49ec5bc8348632e73-ca-certs") pod "kube-controller-manager-archserver" (UID: "8db7fcf04315eec49ec5bc8348632e73") Dec 23 00:07:30 archserver kubelet[957]: I1223 00:07:30.818708 957 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "etc-ca-certificates" (UniqueName: "kubernetes.io/host-path/8db7fcf04315eec49ec5bc8348632e73-etc-ca-certificates") pod "kube-controller-manager-archserver" (UID: "8db7fcf04315eec49ec5bc8348632e73") Dec 23 00:07:30 archserver kubelet[957]: I1223 00:07:30.818893 957 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "kubeconfig" (UniqueName: "kubernetes.io/host-path/8db7fcf04315eec49ec5bc8348632e73-kubeconfig") pod "kube-controller-manager-archserver" (UID: "8db7fcf04315eec49ec5bc8348632e73") Dec 23 00:07:30 archserver kubelet[957]: W1223 00:07:30.867768 957 pod_container_deletor.go:75] Container "63dc669fd46b821f44ae10133f842f80885ec33f018ed91c00848b5df9af1237" not found in pod's containers Dec 23 00:07:30 archserver kubelet[957]: W1223 00:07:30.867876 957 pod_container_deletor.go:75] Container "1dcb4ebeb6e4d7b8f01466b014285c4107290a888df2464ac322b13e09aae413" not found in pod's containers Dec 23 00:07:30 archserver kubelet[957]: W1223 00:07:30.875916 957 docker_sandbox.go:384] failed to read pod IP from plugin/docker: Couldn't find network status for kube-system/coredns-86c58d9df4-lrkrq through plugin: invalid network status for Dec 23 00:07:30 archserver kubelet[957]: E1223 00:07:30.878015 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list v1.Node: Get https://192.168.0.25:6443/api/v1/nodes?fieldSelector=metadata.name%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:30 archserver kubelet[957]: E1223 00:07:30.879009 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list v1.Service: Get https://192.168.0.25:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:30 archserver kubelet[957]: E1223 00:07:30.900644 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:30 archserver kubelet[957]: W1223 00:07:30.907981 957 docker_sandbox.go:384] failed to read pod IP from plugin/docker: Couldn't find network status for kube-system/coredns-86c58d9df4-lrkrq through plugin: invalid network status for Dec 23 00:07:30 archserver kubelet[957]: I1223 00:07:30.920217 957 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "kubeconfig" (UniqueName: "kubernetes.io/host-path/44b569a35761491825f4e7253fbf0543-kubeconfig") pod "kube-scheduler-archserver" (UID: "44b569a35761491825f4e7253fbf0543") Dec 23 00:07:30 archserver kubelet[957]: E1223 00:07:30.922416 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list v1.Pod: Get https://192.168.0.25:6443/api/v1/pods?fieldSelector=spec.nodeName%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:30 archserver kubelet[957]: I1223 00:07:30.929701 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:30 archserver kubelet[957]: I1223 00:07:30.935007 957 kubelet_node_status.go:72] Attempting to register node archserver Dec 23 00:07:30 archserver kubelet[957]: E1223 00:07:30.935943 957 kubelet_node_status.go:94] Unable to register node "archserver" with API server: Post https://192.168.0.25:6443/api/v1/nodes: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:31 archserver kubelet[957]: E1223 00:07:31.001063 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:31 archserver kubelet[957]: W1223 00:07:31.083991 957 pod_container_deletor.go:75] Container "6ca5ab4f8d77201b93ff09d53d76c7fe14844163f358b1a7b6e42c2702ac90a0" not found in pod's containers Dec 23 00:07:31 archserver kubelet[957]: W1223 00:07:31.084074 957 pod_container_deletor.go:75] Container "be52f3d91d9b3df4e90de8ebfcc1386be731a5d7a6a23f0fd37ca97928f5d5f4" not found in pod's containers Dec 23 00:07:31 archserver kubelet[957]: E1223 00:07:31.101456 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:31 archserver kubelet[957]: E1223 00:07:31.201767 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:31 archserver kubelet[957]: E1223 00:07:31.253886 957 remote_runtime.go:282] ContainerStatus "9529c9f46f597cd2ae77b8e53e8b940fd6b1643245695e6e17764f082d513bc1" from runtime service failed: rpc error: code = Unknown desc = Error: No such container: 9529c9f46f597cd2ae77b8e53e8b940fd6b1643245695e6e17764f082d513bc1 Dec 23 00:07:31 archserver kubelet[957]: E1223 00:07:31.254002 957 kuberuntime_container.go:397] ContainerStatus for 9529c9f46f597cd2ae77b8e53e8b940fd6b1643245695e6e17764f082d513bc1 error: rpc error: code = Unknown desc = Error: No such container: 9529c9f46f597cd2ae77b8e53e8b940fd6b1643245695e6e17764f082d513bc1 Dec 23 00:07:31 archserver kubelet[957]: E1223 00:07:31.254057 957 kuberuntime_manager.go:871] getPodContainerStatuses for pod "etcd-archserver_kube-system(459a3e3ab367c3ffe0309150ad8995c1)" failed: rpc error: code = Unknown desc = Error: No such container: 9529c9f46f597cd2ae77b8e53e8b940fd6b1643245695e6e17764f082d513bc1 Dec 23 00:07:31 archserver kubelet[957]: E1223 00:07:31.254130 957 generic.go:247] PLEG: Ignoring events for pod etcd-archserver/kube-system: rpc error: code = Unknown desc = Error: No such container: 9529c9f46f597cd2ae77b8e53e8b940fd6b1643245695e6e17764f082d513bc1 Dec 23 00:07:31 archserver kubelet[957]: E1223 00:07:31.254484 957 pod_workers.go:190] Error syncing pod 459a3e3ab367c3ffe0309150ad8995c1 ("etcd-archserver_kube-system(459a3e3ab367c3ffe0309150ad8995c1)"), skipping: rpc error: code = Unknown desc = Error: No such container: 9529c9f46f597cd2ae77b8e53e8b940fd6b1643245695e6e17764f082d513bc1 Dec 23 00:07:31 archserver kubelet[957]: E1223 00:07:31.302128 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:31 archserver kubelet[957]: E1223 00:07:31.402434 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:31 archserver kubelet[957]: E1223 00:07:31.421382 957 remote_runtime.go:282] ContainerStatus "c52ae5a12dce379ccd96f94c5c561406db92d4955210423421d21c47b6a3fbda" from runtime service failed: rpc error: code = Unknown desc = Error: No such container: c52ae5a12dce379ccd96f94c5c561406db92d4955210423421d21c47b6a3fbda Dec 23 00:07:31 archserver kubelet[957]: E1223 00:07:31.502766 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:31 archserver kubelet[957]: E1223 00:07:31.603137 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:31 archserver kubelet[957]: I1223 00:07:31.641660 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:31 archserver kubelet[957]: I1223 00:07:31.641789 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:31 archserver kubelet[957]: I1223 00:07:31.647040 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:31 archserver kubelet[957]: W1223 00:07:31.648731 957 status_manager.go:485] Failed to get status for pod "kube-controller-manager-archserver_kube-system(8db7fcf04315eec49ec5bc8348632e73)": Get https://192.168.0.25:6443/api/v1/namespaces/kube-system/pods/kube-controller-manager-archserver: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:31 archserver kubelet[957]: I1223 00:07:31.653887 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:31 archserver kubelet[957]: W1223 00:07:31.667236 957 pod_container_deletor.go:75] Container "55ed2d2974bb94cb63389a0f0255355687624e7237dd2bbedd77bcad50aa946e" not found in pod's containers Dec 23 00:07:31 archserver kubelet[957]: I1223 00:07:31.667723 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:31 archserver kubelet[957]: W1223 00:07:31.672583 957 pod_container_deletor.go:75] Container "f4720e772fdd3a40ffb39df61fe6699df47f2bf48abfe7e6a6d53d3388700c10" not found in pod's containers Dec 23 00:07:31 archserver kubelet[957]: E1223 00:07:31.687035 957 remote_runtime.go:282] ContainerStatus "631a2defbd60e3192289cb92a6c94b4cb0fb639efb8bab89e9c953f4f9b5f5c2" from runtime service failed: rpc error: code = Unknown desc = Error: No such container: 631a2defbd60e3192289cb92a6c94b4cb0fb639efb8bab89e9c953f4f9b5f5c2 Dec 23 00:07:31 archserver kubelet[957]: E1223 00:07:31.703469 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:31 archserver kubelet[957]: I1223 00:07:31.736184 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:31 archserver kubelet[957]: I1223 00:07:31.742043 957 kubelet_node_status.go:72] Attempting to register node archserver Dec 23 00:07:31 archserver kubelet[957]: E1223 00:07:31.742952 957 kubelet_node_status.go:94] Unable to register node "archserver" with API server: Post https://192.168.0.25:6443/api/v1/nodes: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:31 archserver kubelet[957]: E1223 00:07:31.803902 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:31 archserver kubelet[957]: E1223 00:07:31.879486 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list v1.Node: Get https://192.168.0.25:6443/api/v1/nodes?fieldSelector=metadata.name%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:31 archserver kubelet[957]: E1223 00:07:31.880140 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list v1.Service: Get https://192.168.0.25:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:31 archserver kubelet[957]: E1223 00:07:31.904377 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:31 archserver kubelet[957]: I1223 00:07:31.923857 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:31 archserver kubelet[957]: E1223 00:07:31.923965 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list v1.Pod: Get https://192.168.0.25:6443/api/v1/pods?fieldSelector=spec.nodeName%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:31 archserver kubelet[957]: I1223 00:07:31.924495 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:31 archserver kubelet[957]: I1223 00:07:31.930620 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:31 archserver kubelet[957]: W1223 00:07:31.935505 957 status_manager.go:485] Failed to get status for pod "kube-scheduler-archserver_kube-system(44b569a35761491825f4e7253fbf0543)": Get https://192.168.0.25:6443/api/v1/namespaces/kube-system/pods/kube-scheduler-archserver: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:31 archserver kubelet[957]: W1223 00:07:31.937982 957 pod_container_deletor.go:75] Container "cce0facaceae1461ba066e35eab3c405bb6fe07ee688d108abdefb32e1aa4a70" not found in pod's containers Dec 23 00:07:31 archserver kubelet[957]: I1223 00:07:31.938216 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:31 archserver kubelet[957]: W1223 00:07:31.944467 957 pod_container_deletor.go:75] Container "c7151655fae574a6fb15eff83bc631e9087eb1986ebfe806c2d0bacad29bc1fe" not found in pod's containers Dec 23 00:07:31 archserver kubelet[957]: I1223 00:07:31.944607 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:31 archserver kubelet[957]: W1223 00:07:31.950791 957 pod_container_deletor.go:75] Container "2b577e257835c0dd3b9aa47006c9aa3da9b83ba3d37c4a8bd6723ad4127a0e98" not found in pod's containers Dec 23 00:07:32 archserver kubelet[957]: E1223 00:07:32.004743 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:32 archserver kubelet[957]: E1223 00:07:32.105101 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:32 archserver kubelet[957]: E1223 00:07:32.205463 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:32 archserver kubelet[957]: E1223 00:07:32.305879 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:32 archserver kubelet[957]: E1223 00:07:32.406286 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:32 archserver kubelet[957]: E1223 00:07:32.506747 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:32 archserver kubelet[957]: E1223 00:07:32.607164 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:32 archserver kubelet[957]: E1223 00:07:32.707574 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:32 archserver kubelet[957]: E1223 00:07:32.807978 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:32 archserver kubelet[957]: E1223 00:07:32.881418 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list v1.Node: Get https://192.168.0.25:6443/api/v1/nodes?fieldSelector=metadata.name%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:32 archserver kubelet[957]: E1223 00:07:32.882078 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list v1.Service: Get https://192.168.0.25:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:32 archserver kubelet[957]: E1223 00:07:32.908353 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:32 archserver kubelet[957]: E1223 00:07:32.925186 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list v1.Pod: Get https://192.168.0.25:6443/api/v1/pods?fieldSelector=spec.nodeName%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:32 archserver kubelet[957]: I1223 00:07:32.962235 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:32 archserver kubelet[957]: I1223 00:07:32.962253 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:32 archserver kubelet[957]: I1223 00:07:32.969080 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:32 archserver kubelet[957]: W1223 00:07:32.971485 957 status_manager.go:485] Failed to get status for pod "kube-apiserver-archserver_kube-system(d92cb816c7d721bf5d651ac4cc2e875a)": Get https://192.168.0.25:6443/api/v1/namespaces/kube-system/pods/kube-apiserver-archserver: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:32 archserver kubelet[957]: I1223 00:07:32.975787 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:32 archserver kubelet[957]: W1223 00:07:32.981485 957 pod_container_deletor.go:75] Container "18b05af10425d8b1005ca90d3123a55b74f822faba5167742a82e1936c3756dc" not found in pod's containers Dec 23 00:07:32 archserver kubelet[957]: I1223 00:07:32.981625 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:32 archserver kubelet[957]: W1223 00:07:32.986712 957 pod_container_deletor.go:75] Container "e360c8432fc972c02b2f264aa37cdb5131a71d6d98b9eea48186843956d2c60d" not found in pod's containers Dec 23 00:07:33 archserver kubelet[957]: E1223 00:07:33.008794 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:33 archserver kubelet[957]: E1223 00:07:33.109045 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:33 archserver kubelet[957]: E1223 00:07:33.209343 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:33 archserver kubelet[957]: E1223 00:07:33.219009 957 remote_runtime.go:282] ContainerStatus "74dab4028261f40511b2080923e4f02973d8ca9394d35f0c3fccb8e7c2444186" from runtime service failed: rpc error: code = Unknown desc = Error: No such container: 74dab4028261f40511b2080923e4f02973d8ca9394d35f0c3fccb8e7c2444186 Dec 23 00:07:33 archserver kubelet[957]: E1223 00:07:33.309718 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:33 archserver kubelet[957]: I1223 00:07:33.343405 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:33 archserver kubelet[957]: I1223 00:07:33.350088 957 kubelet_node_status.go:72] Attempting to register node archserver Dec 23 00:07:33 archserver kubelet[957]: E1223 00:07:33.351198 957 kubelet_node_status.go:94] Unable to register node "archserver" with API server: Post https://192.168.0.25:6443/api/v1/nodes: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:33 archserver kubelet[957]: E1223 00:07:33.410159 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:33 archserver kubelet[957]: E1223 00:07:33.510557 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:33 archserver kubelet[957]: E1223 00:07:33.610942 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:33 archserver kubelet[957]: E1223 00:07:33.711318 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:33 archserver kubelet[957]: E1223 00:07:33.811633 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:33 archserver kubelet[957]: W1223 00:07:33.851227 957 pod_container_deletor.go:75] Container "109f80e09dc1a9befb678867ea28c67ee6776da2dd7950eabeb30035de583685" not found in pod's containers Dec 23 00:07:33 archserver kubelet[957]: E1223 00:07:33.882912 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list v1.Node: Get https://192.168.0.25:6443/api/v1/nodes?fieldSelector=metadata.name%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:33 archserver kubelet[957]: E1223 00:07:33.883587 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list v1.Service: Get https://192.168.0.25:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:33 archserver kubelet[957]: E1223 00:07:33.912046 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:33 archserver kubelet[957]: E1223 00:07:33.926153 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list v1.Pod: Get https://192.168.0.25:6443/api/v1/pods?fieldSelector=spec.nodeName%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:33 archserver kubelet[957]: W1223 00:07:33.936571 957 docker_sandbox.go:384] failed to read pod IP from plugin/docker: Couldn't find network status for kube-system/coredns-86c58d9df4-g9bfh through plugin: invalid network status for Dec 23 00:07:34 archserver kubelet[957]: W1223 00:07:34.012235 957 docker_sandbox.go:384] failed to read pod IP from plugin/docker: Couldn't find network status for kube-system/coredns-86c58d9df4-g9bfh through plugin: invalid network status for Dec 23 00:07:34 archserver kubelet[957]: E1223 00:07:34.012496 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:34 archserver kubelet[957]: E1223 00:07:34.112899 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:34 archserver kubelet[957]: E1223 00:07:34.213229 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:34 archserver kubelet[957]: E1223 00:07:34.313624 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:34 archserver kubelet[957]: E1223 00:07:34.414050 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:34 archserver kubelet[957]: E1223 00:07:34.514468 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:34 archserver kubelet[957]: W1223 00:07:34.599189 957 pod_container_deletor.go:75] Container "742f2d2086a4876f844160fc58f3b4a23c4b3cb1b8fa514e831d0a26ccf88f38" not found in pod's containers Dec 23 00:07:34 archserver kubelet[957]: W1223 00:07:34.599278 957 pod_container_deletor.go:75] Container "384b6b4338c4eeae40ac1f4330f24f07ce60f81943266b5385aace309fd9ee59" not found in pod's containers Dec 23 00:07:34 archserver kubelet[957]: E1223 00:07:34.602866 957 remote_runtime.go:282] ContainerStatus "631a2defbd60e3192289cb92a6c94b4cb0fb639efb8bab89e9c953f4f9b5f5c2" from runtime service failed: rpc error: code = Unknown desc = Error: No such container: 631a2defbd60e3192289cb92a6c94b4cb0fb639efb8bab89e9c953f4f9b5f5c2 Dec 23 00:07:34 archserver kubelet[957]: E1223 00:07:34.602986 957 kuberuntime_gc.go:126] Failed to remove container "631a2defbd60e3192289cb92a6c94b4cb0fb639efb8bab89e9c953f4f9b5f5c2": failed to get container status "631a2defbd60e3192289cb92a6c94b4cb0fb639efb8bab89e9c953f4f9b5f5c2": rpc error: code = Unknown desc = Error: No such container: 631a2defbd60e3192289cb92a6c94b4cb0fb639efb8bab89e9c953f4f9b5f5c2 Dec 23 00:07:34 archserver kubelet[957]: E1223 00:07:34.604585 957 remote_runtime.go:282] ContainerStatus "c52ae5a12dce379ccd96f94c5c561406db92d4955210423421d21c47b6a3fbda" from runtime service failed: rpc error: code = Unknown desc = Error: No such container: c52ae5a12dce379ccd96f94c5c561406db92d4955210423421d21c47b6a3fbda Dec 23 00:07:34 archserver kubelet[957]: E1223 00:07:34.604657 957 kuberuntime_gc.go:126] Failed to remove container "c52ae5a12dce379ccd96f94c5c561406db92d4955210423421d21c47b6a3fbda": failed to get container status "c52ae5a12dce379ccd96f94c5c561406db92d4955210423421d21c47b6a3fbda": rpc error: code = Unknown desc = Error: No such container: c52ae5a12dce379ccd96f94c5c561406db92d4955210423421d21c47b6a3fbda Dec 23 00:07:34 archserver kubelet[957]: E1223 00:07:34.606067 957 remote_runtime.go:282] ContainerStatus "74dab4028261f40511b2080923e4f02973d8ca9394d35f0c3fccb8e7c2444186" from runtime service failed: rpc error: code = Unknown desc = Error: No such container: 74dab4028261f40511b2080923e4f02973d8ca9394d35f0c3fccb8e7c2444186 Dec 23 00:07:34 archserver kubelet[957]: E1223 00:07:34.606205 957 kuberuntime_gc.go:126] Failed to remove container "74dab4028261f40511b2080923e4f02973d8ca9394d35f0c3fccb8e7c2444186": failed to get container status "74dab4028261f40511b2080923e4f02973d8ca9394d35f0c3fccb8e7c2444186": rpc error: code = Unknown desc = Error: No such container: 74dab4028261f40511b2080923e4f02973d8ca9394d35f0c3fccb8e7c2444186 Dec 23 00:07:34 archserver kubelet[957]: E1223 00:07:34.614754 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:34 archserver kubelet[957]: E1223 00:07:34.617945 957 remote_runtime.go:250] RemoveContainer "0ff41a6b324be80db64a946f8786ce8d7eeaa9d5b48a0d17a98180134e832cba" from runtime service failed: rpc error: code = Unknown desc = failed to remove container "0ff41a6b324be80db64a946f8786ce8d7eeaa9d5b48a0d17a98180134e832cba": Error response from daemon: removal of container 0ff41a6b324be80db64a946f8786ce8d7eeaa9d5b48a0d17a98180134e832cba is already in progress Dec 23 00:07:34 archserver kubelet[957]: E1223 00:07:34.618036 957 kuberuntime_gc.go:126] Failed to remove container "0ff41a6b324be80db64a946f8786ce8d7eeaa9d5b48a0d17a98180134e832cba": rpc error: code = Unknown desc = failed to remove container "0ff41a6b324be80db64a946f8786ce8d7eeaa9d5b48a0d17a98180134e832cba": Error response from daemon: removal of container 0ff41a6b324be80db64a946f8786ce8d7eeaa9d5b48a0d17a98180134e832cba is already in progress Dec 23 00:07:34 archserver kubelet[957]: E1223 00:07:34.715121 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:34 archserver kubelet[957]: E1223 00:07:34.815508 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:34 archserver kubelet[957]: E1223 00:07:34.884319 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list v1.Node: Get https://192.168.0.25:6443/api/v1/nodes?fieldSelector=metadata.name%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:34 archserver kubelet[957]: E1223 00:07:34.884972 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list v1.Service: Get https://192.168.0.25:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:34 archserver kubelet[957]: E1223 00:07:34.915946 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:34 archserver kubelet[957]: E1223 00:07:34.927329 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list v1.Pod: Get https://192.168.0.25:6443/api/v1/pods?fieldSelector=spec.nodeName%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:35 archserver kubelet[957]: E1223 00:07:35.016306 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:35 archserver kubelet[957]: E1223 00:07:35.116751 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:35 archserver kubelet[957]: E1223 00:07:35.217096 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:35 archserver kubelet[957]: E1223 00:07:35.317506 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:35 archserver kubelet[957]: E1223 00:07:35.417833 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:35 archserver kubelet[957]: E1223 00:07:35.437347 957 remote_runtime.go:282] ContainerStatus "0ff41a6b324be80db64a946f8786ce8d7eeaa9d5b48a0d17a98180134e832cba" from runtime service failed: rpc error: code = Unknown desc = Error: No such container: 0ff41a6b324be80db64a946f8786ce8d7eeaa9d5b48a0d17a98180134e832cba Dec 23 00:07:35 archserver kubelet[957]: E1223 00:07:35.438877 957 remote_runtime.go:282] ContainerStatus "176a264d561f36ad22dd30af5831e745fce6d32ccf6620c98dda2cce724ac3e8" from runtime service failed: rpc error: code = Unknown desc = Error: No such container: 176a264d561f36ad22dd30af5831e745fce6d32ccf6620c98dda2cce724ac3e8 Dec 23 00:07:35 archserver kubelet[957]: E1223 00:07:35.440632 957 remote_runtime.go:282] ContainerStatus "176a264d561f36ad22dd30af5831e745fce6d32ccf6620c98dda2cce724ac3e8" from runtime service failed: rpc error: code = Unknown desc = Error: No such container: 176a264d561f36ad22dd30af5831e745fce6d32ccf6620c98dda2cce724ac3e8 Dec 23 00:07:35 archserver kubelet[957]: E1223 00:07:35.518182 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:35 archserver kubelet[957]: E1223 00:07:35.655394 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:35 archserver kubelet[957]: E1223 00:07:35.755688 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:35 archserver kubelet[957]: E1223 00:07:35.855932 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:35 archserver kubelet[957]: E1223 00:07:35.885742 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list v1.Node: Get https://192.168.0.25:6443/api/v1/nodes?fieldSelector=metadata.name%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:35 archserver kubelet[957]: E1223 00:07:35.886771 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list v1.Service: Get https://192.168.0.25:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:35 archserver kubelet[957]: E1223 00:07:35.928624 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list v1.Pod: Get https://192.168.0.25:6443/api/v1/pods?fieldSelector=spec.nodeName%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:35 archserver kubelet[957]: E1223 00:07:35.956287 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:36 archserver kubelet[957]: E1223 00:07:36.056558 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:36 archserver kubelet[957]: E1223 00:07:36.157185 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:36 archserver kubelet[957]: E1223 00:07:36.257401 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:36 archserver kubelet[957]: E1223 00:07:36.357680 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:36 archserver kubelet[957]: E1223 00:07:36.458038 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:36 archserver kubelet[957]: I1223 00:07:36.551582 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:36 archserver kubelet[957]: E1223 00:07:36.558301 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:36 archserver kubelet[957]: I1223 00:07:36.558381 957 kubelet_node_status.go:72] Attempting to register node archserver Dec 23 00:07:36 archserver kubelet[957]: E1223 00:07:36.559324 957 kubelet_node_status.go:94] Unable to register node "archserver" with API server: Post https://192.168.0.25:6443/api/v1/nodes: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:36 archserver kubelet[957]: E1223 00:07:36.658663 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:36 archserver kubelet[957]: E1223 00:07:36.758920 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:36 archserver kubelet[957]: E1223 00:07:36.859337 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:36 archserver kubelet[957]: E1223 00:07:36.886868 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list v1.Node: Get https://192.168.0.25:6443/api/v1/nodes?fieldSelector=metadata.name%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:36 archserver kubelet[957]: E1223 00:07:36.887781 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list v1.Service: Get https://192.168.0.25:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:36 archserver kubelet[957]: E1223 00:07:36.930003 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list v1.Pod: Get https://192.168.0.25:6443/api/v1/pods?fieldSelector=spec.nodeName%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:36 archserver kubelet[957]: E1223 00:07:36.959660 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:37 archserver kubelet[957]: E1223 00:07:37.060053 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:37 archserver kubelet[957]: E1223 00:07:37.160301 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:37 archserver kubelet[957]: E1223 00:07:37.260521 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:37 archserver kubelet[957]: E1223 00:07:37.360691 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:37 archserver kubelet[957]: E1223 00:07:37.460950 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:37 archserver kubelet[957]: E1223 00:07:37.561223 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:37 archserver kubelet[957]: E1223 00:07:37.661543 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:37 archserver kubelet[957]: E1223 00:07:37.762593 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:37 archserver kubelet[957]: E1223 00:07:37.770534 957 kuberuntime_manager.go:857] PodSandboxStatus of sandbox "f4720e772fdd3a40ffb39df61fe6699df47f2bf48abfe7e6a6d53d3388700c10" for pod "kube-controller-manager-archserver_kube-system(8db7fcf04315eec49ec5bc8348632e73)" error: rpc error: code = Unknown desc = Error: No such container: f4720e772fdd3a40ffb39df61fe6699df47f2bf48abfe7e6a6d53d3388700c10 Dec 23 00:07:37 archserver kubelet[957]: E1223 00:07:37.770662 957 generic.go:247] PLEG: Ignoring events for pod kube-controller-manager-archserver/kube-system: rpc error: code = Unknown desc = Error: No such container: f4720e772fdd3a40ffb39df61fe6699df47f2bf48abfe7e6a6d53d3388700c10 Dec 23 00:07:37 archserver kubelet[957]: I1223 00:07:37.796661 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:37 archserver kubelet[957]: W1223 00:07:37.800971 957 pod_container_deletor.go:75] Container "2ec6df9260f34e1852d4174ccf132175b3ba193ebc7a260ddfbb0c23b359141f" not found in pod's containers Dec 23 00:07:37 archserver kubelet[957]: I1223 00:07:37.811018 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:37 archserver kubelet[957]: W1223 00:07:37.814595 957 docker_sandbox.go:384] failed to read pod IP from plugin/docker: Couldn't find network status for kube-system/coredns-86c58d9df4-g9bfh through plugin: invalid network status for Dec 23 00:07:37 archserver kubelet[957]: W1223 00:07:37.814838 957 pod_container_deletor.go:75] Container "82831bed7697b647c619f7adfce6f7667f30c406c1f4f9507136ad7cb4c60d29" not found in pod's containers Dec 23 00:07:37 archserver kubelet[957]: W1223 00:07:37.833003 957 docker_sandbox.go:384] failed to read pod IP from plugin/docker: Couldn't find network status for kube-system/coredns-86c58d9df4-lrkrq through plugin: invalid network status for Dec 23 00:07:37 archserver kubelet[957]: I1223 00:07:37.851280 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:37 archserver kubelet[957]: I1223 00:07:37.851423 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:37 archserver kubelet[957]: I1223 00:07:37.855223 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:37 archserver kubelet[957]: W1223 00:07:37.856234 957 status_manager.go:485] Failed to get status for pod "etcd-archserver_kube-system(459a3e3ab367c3ffe0309150ad8995c1)": Get https://192.168.0.25:6443/api/v1/namespaces/kube-system/pods/etcd-archserver: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:37 archserver kubelet[957]: W1223 00:07:37.862581 957 pod_container_deletor.go:75] Container "8a07fff2aae2c9d7ef27a417e67c839b3c1a1d5f2e807fcddab7e20a829bec93" not found in pod's containers Dec 23 00:07:37 archserver kubelet[957]: I1223 00:07:37.862682 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:37 archserver kubelet[957]: E1223 00:07:37.863325 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:37 archserver kubelet[957]: W1223 00:07:37.866239 957 pod_container_deletor.go:75] Container "c99f68b287d90a781fca92cf03a6d5dbc8a93400b7066974fcc14cb333347602" not found in pod's containers Dec 23 00:07:37 archserver kubelet[957]: E1223 00:07:37.887813 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list v1.Node: Get https://192.168.0.25:6443/api/v1/nodes?fieldSelector=metadata.name%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:37 archserver kubelet[957]: E1223 00:07:37.888668 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list v1.Service: Get https://192.168.0.25:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:37 archserver kubelet[957]: E1223 00:07:37.931321 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list v1.Pod: Get https://192.168.0.25:6443/api/v1/pods?fieldSelector=spec.nodeName%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:37 archserver kubelet[957]: E1223 00:07:37.963585 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:38 archserver kubelet[957]: E1223 00:07:38.063879 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:38 archserver kubelet[957]: E1223 00:07:38.164221 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:38 archserver kubelet[957]: E1223 00:07:38.264605 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:38 archserver kubelet[957]: E1223 00:07:38.364993 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:38 archserver kubelet[957]: E1223 00:07:38.465323 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:38 archserver kubelet[957]: E1223 00:07:38.565676 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:38 archserver kubelet[957]: E1223 00:07:38.665949 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:38 archserver kubelet[957]: E1223 00:07:38.766283 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:38 archserver kubelet[957]: E1223 00:07:38.866491 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:38 archserver kubelet[957]: E1223 00:07:38.888634 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list v1.Node: Get https://192.168.0.25:6443/api/v1/nodes?fieldSelector=metadata.name%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:38 archserver kubelet[957]: W1223 00:07:38.889205 957 docker_sandbox.go:384] failed to read pod IP from plugin/docker: Couldn't find network status for kube-system/coredns-86c58d9df4-lrkrq through plugin: invalid network status for Dec 23 00:07:38 archserver kubelet[957]: E1223 00:07:38.889609 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list v1.Service: Get https://192.168.0.25:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:38 archserver kubelet[957]: E1223 00:07:38.932537 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list v1.Pod: Get https://192.168.0.25:6443/api/v1/pods?fieldSelector=spec.nodeName%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:38 archserver kubelet[957]: E1223 00:07:38.951377 957 remote_runtime.go:282] ContainerStatus "77e7675f54b23752a78fbd98d61ba17f73d4f044f1ecc07240cfd3195eea0305" from runtime service failed: rpc error: code = Unknown desc = Error: No such container: 77e7675f54b23752a78fbd98d61ba17f73d4f044f1ecc07240cfd3195eea0305 Dec 23 00:07:38 archserver kubelet[957]: E1223 00:07:38.951496 957 kuberuntime_container.go:397] ContainerStatus for 77e7675f54b23752a78fbd98d61ba17f73d4f044f1ecc07240cfd3195eea0305 error: rpc error: code = Unknown desc = Error: No such container: 77e7675f54b23752a78fbd98d61ba17f73d4f044f1ecc07240cfd3195eea0305 Dec 23 00:07:38 archserver kubelet[957]: E1223 00:07:38.951533 957 kuberuntime_manager.go:871] getPodContainerStatuses for pod "kube-scheduler-archserver_kube-system(44b569a35761491825f4e7253fbf0543)" failed: rpc error: code = Unknown desc = Error: No such container: 77e7675f54b23752a78fbd98d61ba17f73d4f044f1ecc07240cfd3195eea0305 Dec 23 00:07:39 archserver kubelet[957]: E1223 00:07:38.951588 957 generic.go:247] PLEG: Ignoring events for pod kube-scheduler-archserver/kube-system: rpc error: code = Unknown desc = Error: No such container: 77e7675f54b23752a78fbd98d61ba17f73d4f044f1ecc07240cfd3195eea0305 Dec 23 00:07:39 archserver kubelet[957]: E1223 00:07:38.966658 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:39 archserver kubelet[957]: W1223 00:07:38.973944 957 docker_sandbox.go:384] failed to read pod IP from plugin/docker: Couldn't find network status for kube-system/coredns-86c58d9df4-g9bfh through plugin: invalid network status for Dec 23 00:07:39 archserver kubelet[957]: E1223 00:07:39.067027 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:39 archserver kubelet[957]: E1223 00:07:39.114640 957 event.go:212] Unable to write event: 'Post https://192.168.0.25:6443/api/v1/namespaces/default/events: dial tcp 192.168.0.25:6443: connect: connection refused' (may retry after sleeping) Dec 23 00:07:39 archserver kubelet[957]: E1223 00:07:39.167366 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:39 archserver kubelet[957]: E1223 00:07:39.267766 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:39 archserver kubelet[957]: E1223 00:07:39.368131 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:39 archserver kubelet[957]: E1223 00:07:39.468513 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:39 archserver kubelet[957]: E1223 00:07:39.568888 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:39 archserver kubelet[957]: E1223 00:07:39.669320 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:39 archserver kubelet[957]: E1223 00:07:39.769754 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:39 archserver kubelet[957]: E1223 00:07:39.870160 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:39 archserver kubelet[957]: E1223 00:07:39.890260 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list v1.Node: Get https://192.168.0.25:6443/api/v1/nodes?fieldSelector=metadata.name%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:39 archserver kubelet[957]: E1223 00:07:39.890792 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list v1.Service: Get https://192.168.0.25:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:39 archserver kubelet[957]: E1223 00:07:39.933878 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list v1.Pod: Get https://192.168.0.25:6443/api/v1/pods?fieldSelector=spec.nodeName%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:39 archserver kubelet[957]: E1223 00:07:39.970553 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:40 archserver kubelet[957]: E1223 00:07:40.070901 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:40 archserver kubelet[957]: E1223 00:07:40.171334 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:40 archserver kubelet[957]: E1223 00:07:40.271677 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:40 archserver kubelet[957]: E1223 00:07:40.372078 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:40 archserver kubelet[957]: E1223 00:07:40.418187 957 eviction_manager.go:243] eviction manager: failed to get summary stats: failed to get node info: node "archserver" not found Dec 23 00:07:40 archserver kubelet[957]: E1223 00:07:40.472481 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:40 archserver kubelet[957]: E1223 00:07:40.572898 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:40 archserver kubelet[957]: E1223 00:07:40.673357 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:40 archserver kubelet[957]: E1223 00:07:40.773674 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:40 archserver kubelet[957]: E1223 00:07:40.873993 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:40 archserver kubelet[957]: E1223 00:07:40.891380 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list v1.Node: Get https://192.168.0.25:6443/api/v1/nodes?fieldSelector=metadata.name%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:40 archserver kubelet[957]: E1223 00:07:40.892299 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list v1.Service: Get https://192.168.0.25:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:40 archserver kubelet[957]: E1223 00:07:40.935188 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list v1.Pod: Get https://192.168.0.25:6443/api/v1/pods?fieldSelector=spec.nodeName%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:40 archserver kubelet[957]: E1223 00:07:40.974388 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:41 archserver kubelet[957]: E1223 00:07:41.074794 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:41 archserver kubelet[957]: E1223 00:07:41.175186 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:41 archserver kubelet[957]: E1223 00:07:41.275611 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:41 archserver kubelet[957]: E1223 00:07:41.375918 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:41 archserver kubelet[957]: E1223 00:07:41.476294 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:41 archserver kubelet[957]: E1223 00:07:41.576656 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:41 archserver kubelet[957]: E1223 00:07:41.676888 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:41 archserver kubelet[957]: E1223 00:07:41.777215 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:41 archserver kubelet[957]: E1223 00:07:41.877615 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:41 archserver kubelet[957]: E1223 00:07:41.892952 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list v1.Node: Get https://192.168.0.25:6443/api/v1/nodes?fieldSelector=metadata.name%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:41 archserver kubelet[957]: E1223 00:07:41.893489 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list v1.Service: Get https://192.168.0.25:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:41 archserver kubelet[957]: E1223 00:07:41.936478 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list v1.Pod: Get https://192.168.0.25:6443/api/v1/pods?fieldSelector=spec.nodeName%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:41 archserver kubelet[957]: E1223 00:07:41.978000 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:42 archserver kubelet[957]: E1223 00:07:42.078360 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:42 archserver kubelet[957]: E1223 00:07:42.178693 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:42 archserver kubelet[957]: E1223 00:07:42.279020 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:42 archserver kubelet[957]: E1223 00:07:42.379340 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:42 archserver kubelet[957]: E1223 00:07:42.479606 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:42 archserver kubelet[957]: E1223 00:07:42.579994 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:42 archserver kubelet[957]: E1223 00:07:42.680405 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:42 archserver kubelet[957]: E1223 00:07:42.780681 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:42 archserver kubelet[957]: E1223 00:07:42.880990 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:42 archserver kubelet[957]: E1223 00:07:42.894024 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list v1.Node: Get https://192.168.0.25:6443/api/v1/nodes?fieldSelector=metadata.name%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:42 archserver kubelet[957]: E1223 00:07:42.895134 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list v1.Service: Get https://192.168.0.25:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:42 archserver kubelet[957]: E1223 00:07:42.937384 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list v1.Pod: Get https://192.168.0.25:6443/api/v1/pods?fieldSelector=spec.nodeName%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:42 archserver kubelet[957]: I1223 00:07:42.959689 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:42 archserver kubelet[957]: I1223 00:07:42.964086 957 kubelet_node_status.go:72] Attempting to register node archserver Dec 23 00:07:42 archserver kubelet[957]: E1223 00:07:42.964748 957 kubelet_node_status.go:94] Unable to register node "archserver" with API server: Post https://192.168.0.25:6443/api/v1/nodes: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:42 archserver kubelet[957]: E1223 00:07:42.981273 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:43 archserver kubelet[957]: E1223 00:07:43.081485 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:43 archserver kubelet[957]: E1223 00:07:43.181782 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:43 archserver kubelet[957]: E1223 00:07:43.282137 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:43 archserver kubelet[957]: E1223 00:07:43.382423 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:43 archserver kubelet[957]: E1223 00:07:43.482700 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:43 archserver kubelet[957]: E1223 00:07:43.583107 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:43 archserver kubelet[957]: E1223 00:07:43.683421 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:43 archserver kubelet[957]: I1223 00:07:43.690613 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:43 archserver kubelet[957]: W1223 00:07:43.699045 957 pod_container_deletor.go:75] Container "5e7a131a23a48749fbe6882ba3e30a75e275f468d3e8aa4052cfc6aee2763e70" not found in pod's containers Dec 23 00:07:43 archserver kubelet[957]: I1223 00:07:43.715205 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:43 archserver kubelet[957]: W1223 00:07:43.720528 957 status_manager.go:485] Failed to get status for pod "kube-controller-manager-archserver_kube-system(8db7fcf04315eec49ec5bc8348632e73)": Get https://192.168.0.25:6443/api/v1/namespaces/kube-system/pods/kube-controller-manager-archserver: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:43 archserver kubelet[957]: I1223 00:07:43.736349 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:43 archserver kubelet[957]: W1223 00:07:43.741370 957 status_manager.go:485] Failed to get status for pod "kube-scheduler-archserver_kube-system(44b569a35761491825f4e7253fbf0543)": Get https://192.168.0.25:6443/api/v1/namespaces/kube-system/pods/kube-scheduler-archserver: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:43 archserver kubelet[957]: I1223 00:07:43.760662 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:43 archserver kubelet[957]: W1223 00:07:43.767102 957 status_manager.go:485] Failed to get status for pod "kube-apiserver-archserver_kube-system(d92cb816c7d721bf5d651ac4cc2e875a)": Get https://192.168.0.25:6443/api/v1/namespaces/kube-system/pods/kube-apiserver-archserver: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:43 archserver kubelet[957]: E1223 00:07:43.783812 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:43 archserver kubelet[957]: E1223 00:07:43.884175 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:43 archserver kubelet[957]: E1223 00:07:43.895379 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list v1.Node: Get https://192.168.0.25:6443/api/v1/nodes?fieldSelector=metadata.name%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:44 archserver kubelet[957]: E1223 00:07:43.896362 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list v1.Service: Get https://192.168.0.25:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:44 archserver kubelet[957]: E1223 00:07:43.938473 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list v1.Pod: Get https://192.168.0.25:6443/api/v1/pods?fieldSelector=spec.nodeName%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:44 archserver kubelet[957]: E1223 00:07:43.984558 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:44 archserver kubelet[957]: E1223 00:07:44.084992 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:44 archserver kubelet[957]: E1223 00:07:44.185365 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:44 archserver kubelet[957]: E1223 00:07:44.285782 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:44 archserver kubelet[957]: E1223 00:07:44.386214 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:44 archserver kubelet[957]: E1223 00:07:44.486717 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:44 archserver kubelet[957]: E1223 00:07:44.587143 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:44 archserver kubelet[957]: E1223 00:07:44.687453 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:44 archserver kubelet[957]: E1223 00:07:44.787742 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:44 archserver kubelet[957]: I1223 00:07:44.820204 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:44 archserver kubelet[957]: I1223 00:07:44.841356 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:44 archserver kubelet[957]: I1223 00:07:44.841483 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:44 archserver kubelet[957]: E1223 00:07:44.887995 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:44 archserver kubelet[957]: E1223 00:07:44.896785 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list v1.Node: Get https://192.168.0.25:6443/api/v1/nodes?fieldSelector=metadata.name%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:44 archserver kubelet[957]: E1223 00:07:44.898249 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list v1.Service: Get https://192.168.0.25:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:44 archserver kubelet[957]: E1223 00:07:44.939696 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list v1.Pod: Get https://192.168.0.25:6443/api/v1/pods?fieldSelector=spec.nodeName%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:44 archserver kubelet[957]: E1223 00:07:44.988266 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:45 archserver kubelet[957]: E1223 00:07:45.088677 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:45 archserver kubelet[957]: E1223 00:07:45.189120 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:45 archserver kubelet[957]: E1223 00:07:45.289543 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:45 archserver kubelet[957]: E1223 00:07:45.390023 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:45 archserver kubelet[957]: E1223 00:07:45.490400 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:45 archserver kubelet[957]: E1223 00:07:45.590804 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:45 archserver kubelet[957]: E1223 00:07:45.691105 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:45 archserver kubelet[957]: E1223 00:07:45.791566 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:45 archserver kubelet[957]: E1223 00:07:45.891839 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:45 archserver kubelet[957]: E1223 00:07:45.897545 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list v1.Node: Get https://192.168.0.25:6443/api/v1/nodes?fieldSelector=metadata.name%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:45 archserver kubelet[957]: E1223 00:07:45.898963 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list v1.Service: Get https://192.168.0.25:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:45 archserver kubelet[957]: E1223 00:07:45.941118 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list v1.Pod: Get https://192.168.0.25:6443/api/v1/pods?fieldSelector=spec.nodeName%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:45 archserver kubelet[957]: E1223 00:07:45.992204 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:46 archserver kubelet[957]: E1223 00:07:46.092488 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:46 archserver kubelet[957]: E1223 00:07:46.192856 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:46 archserver kubelet[957]: E1223 00:07:46.293323 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:46 archserver kubelet[957]: W1223 00:07:46.338543 957 docker_container.go:202] Deleted previously existing symlink file: "/var/log/pods/459a3e3ab367c3ffe0309150ad8995c1/etcd/3.log" Dec 23 00:07:46 archserver kubelet[957]: E1223 00:07:46.393678 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:46 archserver kubelet[957]: E1223 00:07:46.493950 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:46 archserver kubelet[957]: E1223 00:07:46.594230 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:46 archserver kubelet[957]: E1223 00:07:46.694638 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:46 archserver kubelet[957]: E1223 00:07:46.795050 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:46 archserver kubelet[957]: E1223 00:07:46.895410 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:46 archserver kubelet[957]: E1223 00:07:46.898991 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list v1.Node: Get https://192.168.0.25:6443/api/v1/nodes?fieldSelector=metadata.name%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:46 archserver kubelet[957]: E1223 00:07:46.900145 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list v1.Service: Get https://192.168.0.25:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:46 archserver kubelet[957]: E1223 00:07:46.942641 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list v1.Pod: Get https://192.168.0.25:6443/api/v1/pods?fieldSelector=spec.nodeName%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:46 archserver kubelet[957]: E1223 00:07:46.995747 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:47 archserver kubelet[957]: E1223 00:07:47.096196 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:47 archserver kubelet[957]: E1223 00:07:47.196684 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:47 archserver kubelet[957]: E1223 00:07:47.297001 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:47 archserver kubelet[957]: E1223 00:07:47.397627 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:47 archserver kubelet[957]: I1223 00:07:47.408401 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:47 archserver kubelet[957]: W1223 00:07:47.415696 957 status_manager.go:485] Failed to get status for pod "etcd-archserver_kube-system(459a3e3ab367c3ffe0309150ad8995c1)": Get https://192.168.0.25:6443/api/v1/namespaces/kube-system/pods/etcd-archserver: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:47 archserver kubelet[957]: E1223 00:07:47.497857 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:47 archserver kubelet[957]: E1223 00:07:47.598174 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:47 archserver kubelet[957]: E1223 00:07:47.698490 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:47 archserver kubelet[957]: E1223 00:07:47.798795 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:47 archserver kubelet[957]: E1223 00:07:47.899159 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:47 archserver kubelet[957]: E1223 00:07:47.899941 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list v1.Node: Get https://192.168.0.25:6443/api/v1/nodes?fieldSelector=metadata.name%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:47 archserver kubelet[957]: E1223 00:07:47.901298 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list v1.Service: Get https://192.168.0.25:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:47 archserver kubelet[957]: E1223 00:07:47.944071 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list v1.Pod: Get https://192.168.0.25:6443/api/v1/pods?fieldSelector=spec.nodeName%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:47 archserver kubelet[957]: E1223 00:07:47.999594 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:48 archserver kubelet[957]: E1223 00:07:48.100021 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:48 archserver kubelet[957]: E1223 00:07:48.200404 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:48 archserver kubelet[957]: E1223 00:07:48.300781 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:48 archserver kubelet[957]: E1223 00:07:48.401031 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:48 archserver kubelet[957]: I1223 00:07:48.428065 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:48 archserver kubelet[957]: E1223 00:07:48.501317 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:48 archserver kubelet[957]: E1223 00:07:48.601657 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:48 archserver kubelet[957]: E1223 00:07:48.702031 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:48 archserver kubelet[957]: E1223 00:07:48.802493 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:48 archserver kubelet[957]: E1223 00:07:48.901201 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list v1.Node: Get https://192.168.0.25:6443/api/v1/nodes?fieldSelector=metadata.name%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:48 archserver kubelet[957]: E1223 00:07:48.902583 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list v1.Service: Get https://192.168.0.25:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:48 archserver kubelet[957]: E1223 00:07:48.902758 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:48 archserver kubelet[957]: E1223 00:07:48.945511 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list v1.Pod: Get https://192.168.0.25:6443/api/v1/pods?fieldSelector=spec.nodeName%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:49 archserver kubelet[957]: E1223 00:07:49.003002 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:49 archserver kubelet[957]: E1223 00:07:49.103335 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:49 archserver kubelet[957]: E1223 00:07:49.115619 957 event.go:212] Unable to write event: 'Post https://192.168.0.25:6443/api/v1/namespaces/default/events: dial tcp 192.168.0.25:6443: connect: connection refused' (may retry after sleeping) Dec 23 00:07:49 archserver kubelet[957]: E1223 00:07:49.204128 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:49 archserver kubelet[957]: E1223 00:07:49.304506 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:49 archserver kubelet[957]: E1223 00:07:49.404862 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:49 archserver kubelet[957]: E1223 00:07:49.505277 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:49 archserver kubelet[957]: E1223 00:07:49.605634 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:49 archserver kubelet[957]: E1223 00:07:49.706061 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:49 archserver kubelet[957]: E1223 00:07:49.806445 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:49 archserver kubelet[957]: E1223 00:07:49.902564 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list v1.Node: Get https://192.168.0.25:6443/api/v1/nodes?fieldSelector=metadata.name%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:49 archserver kubelet[957]: E1223 00:07:49.903597 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list v1.Service: Get https://192.168.0.25:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:49 archserver kubelet[957]: E1223 00:07:49.906761 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:49 archserver kubelet[957]: E1223 00:07:49.946665 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list v1.Pod: Get https://192.168.0.25:6443/api/v1/pods?fieldSelector=spec.nodeName%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:49 archserver kubelet[957]: I1223 00:07:49.965348 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:49 archserver kubelet[957]: I1223 00:07:49.971675 957 kubelet_node_status.go:72] Attempting to register node archserver Dec 23 00:07:50 archserver kubelet[957]: E1223 00:07:49.972638 957 kubelet_node_status.go:94] Unable to register node "archserver" with API server: Post https://192.168.0.25:6443/api/v1/nodes: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:50 archserver kubelet[957]: E1223 00:07:50.007098 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:50 archserver kubelet[957]: E1223 00:07:50.107470 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:50 archserver kubelet[957]: E1223 00:07:50.207852 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:50 archserver kubelet[957]: E1223 00:07:50.308265 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:50 archserver kubelet[957]: E1223 00:07:50.408765 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:50 archserver kubelet[957]: E1223 00:07:50.418685 957 eviction_manager.go:243] eviction manager: failed to get summary stats: failed to get node info: node "archserver" not found Dec 23 00:07:50 archserver kubelet[957]: E1223 00:07:50.509202 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:50 archserver kubelet[957]: E1223 00:07:50.609651 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:50 archserver kubelet[957]: E1223 00:07:50.710084 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:50 archserver kubelet[957]: E1223 00:07:50.810423 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:50 archserver kubelet[957]: E1223 00:07:50.903966 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list v1.Node: Get https://192.168.0.25:6443/api/v1/nodes?fieldSelector=metadata.name%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:50 archserver kubelet[957]: E1223 00:07:50.904793 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list v1.Service: Get https://192.168.0.25:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:50 archserver kubelet[957]: E1223 00:07:50.910814 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:50 archserver kubelet[957]: E1223 00:07:50.947912 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.0.25:6443/api/v1/pods?fieldSelector=spec.nodeName%3Darchserver&limit=500&resourceVersion=0: dial tcp 192.168.0.25:6443: connect: connection refused Dec 23 00:07:51 archserver kubelet[957]: E1223 00:07:51.011221 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:51 archserver kubelet[957]: E1223 00:07:51.111664 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:51 archserver kubelet[957]: E1223 00:07:51.212038 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:51 archserver kubelet[957]: E1223 00:07:51.312389 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:51 archserver kubelet[957]: E1223 00:07:51.412711 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:51 archserver kubelet[957]: E1223 00:07:51.513000 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:51 archserver kubelet[957]: E1223 00:07:51.613474 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:51 archserver kubelet[957]: E1223 00:07:51.713792 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:51 archserver kubelet[957]: E1223 00:07:51.814221 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:51 archserver kubelet[957]: E1223 00:07:51.914616 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:52 archserver kubelet[957]: E1223 00:07:52.014928 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:52 archserver kubelet[957]: E1223 00:07:52.115264 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:52 archserver kubelet[957]: E1223 00:07:52.215632 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:52 archserver kubelet[957]: E1223 00:07:52.316034 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:52 archserver kubelet[957]: E1223 00:07:52.416347 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:52 archserver kubelet[957]: E1223 00:07:52.517029 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:52 archserver kubelet[957]: E1223 00:07:52.617696 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:52 archserver kubelet[957]: E1223 00:07:52.718031 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:52 archserver kubelet[957]: E1223 00:07:52.818385 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:52 archserver kubelet[957]: E1223 00:07:52.918768 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:53 archserver kubelet[957]: E1223 00:07:53.019152 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:53 archserver kubelet[957]: E1223 00:07:53.119469 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:53 archserver kubelet[957]: E1223 00:07:53.219808 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:53 archserver kubelet[957]: E1223 00:07:53.320126 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:53 archserver kubelet[957]: E1223 00:07:53.420557 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:53 archserver kubelet[957]: E1223 00:07:53.520923 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:53 archserver kubelet[957]: E1223 00:07:53.621295 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:53 archserver kubelet[957]: E1223 00:07:53.721696 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:53 archserver kubelet[957]: E1223 00:07:53.822071 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:53 archserver kubelet[957]: E1223 00:07:53.922459 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:54 archserver kubelet[957]: E1223 00:07:54.022838 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:54 archserver kubelet[957]: E1223 00:07:54.123135 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:54 archserver kubelet[957]: E1223 00:07:54.223520 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:54 archserver kubelet[957]: E1223 00:07:54.323788 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:54 archserver kubelet[957]: E1223 00:07:54.424313 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:54 archserver kubelet[957]: E1223 00:07:54.524626 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:54 archserver kubelet[957]: E1223 00:07:54.625000 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:54 archserver kubelet[957]: E1223 00:07:54.725382 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:54 archserver kubelet[957]: E1223 00:07:54.825744 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:54 archserver kubelet[957]: E1223 00:07:54.926083 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:55 archserver kubelet[957]: E1223 00:07:55.026336 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:55 archserver kubelet[957]: E1223 00:07:55.126750 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:55 archserver kubelet[957]: E1223 00:07:55.227069 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:55 archserver kubelet[957]: E1223 00:07:55.327340 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:55 archserver kubelet[957]: E1223 00:07:55.428057 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:55 archserver kubelet[957]: E1223 00:07:55.528632 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:55 archserver kubelet[957]: E1223 00:07:55.628969 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:55 archserver kubelet[957]: E1223 00:07:55.729264 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:55 archserver kubelet[957]: E1223 00:07:55.829574 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:55 archserver kubelet[957]: E1223 00:07:55.930409 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:56 archserver kubelet[957]: E1223 00:07:56.030711 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:56 archserver kubelet[957]: E1223 00:07:56.131082 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:56 archserver kubelet[957]: E1223 00:07:56.231478 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:56 archserver kubelet[957]: E1223 00:07:56.331745 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:56 archserver kubelet[957]: E1223 00:07:56.431978 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:56 archserver kubelet[957]: E1223 00:07:56.532266 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:56 archserver kubelet[957]: E1223 00:07:56.632620 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:56 archserver kubelet[957]: E1223 00:07:56.732964 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:56 archserver kubelet[957]: E1223 00:07:56.833406 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:56 archserver kubelet[957]: E1223 00:07:56.933745 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:56 archserver kubelet[957]: I1223 00:07:56.973003 957 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach Dec 23 00:07:56 archserver kubelet[957]: I1223 00:07:56.980074 957 kubelet_node_status.go:72] Attempting to register node archserver Dec 23 00:07:57 archserver kubelet[957]: E1223 00:07:57.034019 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:57 archserver kubelet[957]: E1223 00:07:57.134356 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:57 archserver kubelet[957]: E1223 00:07:57.234814 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:57 archserver kubelet[957]: E1223 00:07:57.335182 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:57 archserver kubelet[957]: E1223 00:07:57.435559 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:57 archserver kubelet[957]: E1223 00:07:57.535944 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:57 archserver kubelet[957]: E1223 00:07:57.636231 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:57 archserver kubelet[957]: E1223 00:07:57.736984 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:57 archserver kubelet[957]: E1223 00:07:57.837360 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:57 archserver kubelet[957]: E1223 00:07:57.937733 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:58 archserver kubelet[957]: E1223 00:07:58.038088 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:58 archserver kubelet[957]: E1223 00:07:58.138431 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:58 archserver kubelet[957]: E1223 00:07:58.239237 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:58 archserver kubelet[957]: E1223 00:07:58.339628 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:58 archserver kubelet[957]: E1223 00:07:58.440050 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:58 archserver kubelet[957]: E1223 00:07:58.540416 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:58 archserver kubelet[957]: E1223 00:07:58.640776 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:58 archserver kubelet[957]: E1223 00:07:58.741114 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:58 archserver kubelet[957]: E1223 00:07:58.841461 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:58 archserver kubelet[957]: E1223 00:07:58.941843 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:59 archserver kubelet[957]: E1223 00:07:59.042213 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:59 archserver kubelet[957]: E1223 00:07:59.142458 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:59 archserver kubelet[957]: E1223 00:07:59.242880 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:59 archserver kubelet[957]: E1223 00:07:59.343369 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:59 archserver kubelet[957]: E1223 00:07:59.443698 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:59 archserver kubelet[957]: E1223 00:07:59.544117 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:59 archserver kubelet[957]: E1223 00:07:59.644459 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:59 archserver kubelet[957]: E1223 00:07:59.744949 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:59 archserver kubelet[957]: E1223 00:07:59.845345 957 kubelet.go:2266] node "archserver" not found Dec 23 00:07:59 archserver kubelet[957]: E1223 00:07:59.945731 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:00 archserver kubelet[957]: E1223 00:08:00.046546 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:00 archserver kubelet[957]: E1223 00:08:00.146934 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:00 archserver kubelet[957]: E1223 00:08:00.247193 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:00 archserver kubelet[957]: E1223 00:08:00.347569 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:00 archserver kubelet[957]: E1223 00:08:00.419171 957 eviction_manager.go:243] eviction manager: failed to get summary stats: failed to get node info: node "archserver" not found Dec 23 00:08:00 archserver kubelet[957]: E1223 00:08:00.447863 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:00 archserver kubelet[957]: E1223 00:08:00.548249 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:00 archserver kubelet[957]: E1223 00:08:00.648596 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:00 archserver kubelet[957]: E1223 00:08:00.749004 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:00 archserver kubelet[957]: E1223 00:08:00.849315 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:00 archserver kubelet[957]: E1223 00:08:00.949535 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:01 archserver kubelet[957]: E1223 00:08:01.049899 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:01 archserver kubelet[957]: E1223 00:08:01.150394 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:01 archserver kubelet[957]: E1223 00:08:01.250749 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:01 archserver kubelet[957]: E1223 00:08:01.351113 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:01 archserver kubelet[957]: E1223 00:08:01.451452 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:01 archserver kubelet[957]: E1223 00:08:01.551826 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:01 archserver kubelet[957]: E1223 00:08:01.652204 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:01 archserver kubelet[957]: E1223 00:08:01.752532 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:01 archserver kubelet[957]: E1223 00:08:01.852878 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:01 archserver kubelet[957]: E1223 00:08:01.905874 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list v1.Node: Get https://192.168.0.25:6443/api/v1/nodes?fieldSelector=metadata.name%3Darchserver&limit=500&resourceVersion=0: net/http: TLS handshake timeout Dec 23 00:08:01 archserver kubelet[957]: E1223 00:08:01.906729 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list v1.Service: Get https://192.168.0.25:6443/api/v1/services?limit=500&resourceVersion=0: net/http: TLS handshake timeout Dec 23 00:08:01 archserver kubelet[957]: E1223 00:08:01.949647 957 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.0.25:6443/api/v1/pods?fieldSelector=spec.nodeName%3Darchserver&limit=500&resourceVersion=0: net/http: TLS handshake timeout Dec 23 00:08:01 archserver kubelet[957]: E1223 00:08:01.953222 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:02 archserver kubelet[957]: E1223 00:08:02.053607 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:02 archserver kubelet[957]: E1223 00:08:02.153945 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:02 archserver kubelet[957]: E1223 00:08:02.254278 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:02 archserver kubelet[957]: E1223 00:08:02.354660 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:02 archserver kubelet[957]: E1223 00:08:02.455007 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:02 archserver kubelet[957]: E1223 00:08:02.555401 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:02 archserver kubelet[957]: E1223 00:08:02.656148 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:02 archserver kubelet[957]: E1223 00:08:02.756501 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:02 archserver kubelet[957]: E1223 00:08:02.856879 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:02 archserver kubelet[957]: E1223 00:08:02.957190 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:03 archserver kubelet[957]: E1223 00:08:03.057566 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:03 archserver kubelet[957]: E1223 00:08:03.157915 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:03 archserver kubelet[957]: E1223 00:08:03.258293 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:03 archserver kubelet[957]: E1223 00:08:03.358553 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:03 archserver kubelet[957]: E1223 00:08:03.459000 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:03 archserver kubelet[957]: E1223 00:08:03.559368 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:03 archserver kubelet[957]: E1223 00:08:03.659704 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:03 archserver kubelet[957]: E1223 00:08:03.760086 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:03 archserver kubelet[957]: E1223 00:08:03.860437 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:03 archserver kubelet[957]: E1223 00:08:03.960736 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:04 archserver kubelet[957]: E1223 00:08:04.061077 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:04 archserver kubelet[957]: E1223 00:08:04.161344 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:04 archserver kubelet[957]: E1223 00:08:04.262082 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:04 archserver kubelet[957]: E1223 00:08:04.362515 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:04 archserver kubelet[957]: E1223 00:08:04.462853 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:04 archserver kubelet[957]: E1223 00:08:04.563173 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:04 archserver kubelet[957]: E1223 00:08:04.663546 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:04 archserver kubelet[957]: E1223 00:08:04.763893 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:04 archserver kubelet[957]: E1223 00:08:04.864245 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:04 archserver kubelet[957]: E1223 00:08:04.964701 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:05 archserver kubelet[957]: E1223 00:08:05.065063 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:05 archserver kubelet[957]: E1223 00:08:05.165471 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:05 archserver kubelet[957]: E1223 00:08:05.266040 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:05 archserver kubelet[957]: E1223 00:08:05.366389 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:05 archserver kubelet[957]: E1223 00:08:05.467094 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:05 archserver kubelet[957]: E1223 00:08:05.567380 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:05 archserver kubelet[957]: E1223 00:08:05.667759 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:05 archserver kubelet[957]: E1223 00:08:05.768042 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:05 archserver kubelet[957]: E1223 00:08:05.868336 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:05 archserver kubelet[957]: E1223 00:08:05.968590 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:06 archserver kubelet[957]: E1223 00:08:06.068851 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:06 archserver kubelet[957]: E1223 00:08:06.169767 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:06 archserver kubelet[957]: E1223 00:08:06.270084 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:06 archserver kubelet[957]: E1223 00:08:06.370545 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:06 archserver kubelet[957]: E1223 00:08:06.471211 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:06 archserver kubelet[957]: E1223 00:08:06.571613 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:06 archserver kubelet[957]: E1223 00:08:06.671925 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:06 archserver kubelet[957]: E1223 00:08:06.772194 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:06 archserver kubelet[957]: E1223 00:08:06.872414 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:06 archserver kubelet[957]: E1223 00:08:06.972844 957 kubelet.go:2266] node "archserver" not found Dec 23 00:08:07 archserver kubelet[957]: I1223 00:08:07.073158 957 kuberuntime_manager.go:915] updating runtime config through cri with podcidr 10.244.0.0/24 Dec 23 00:08:07 archserver kubelet[957]: I1223 00:08:07.073733 957 docker_service.go:345] docker cri received runtime config &RuntimeConfig{NetworkConfig:&NetworkConfig{PodCidr:10.244.0.0/24,},} Dec 23 00:08:07 archserver kubelet[957]: I1223 00:08:07.074185 957 kubelet_network.go:77] Setting Pod CIDR: -> 10.244.0.0/24 Dec 23 00:08:07 archserver kubelet[957]: I1223 00:08:07.173753 957 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "lib-modules" (UniqueName: "kubernetes.io/host-path/b9bf1b00-063d-11e9-b166-001966fbc1d4-lib-modules") pod "kube-proxy-58pbt" (UID: "b9bf1b00-063d-11e9-b166-001966fbc1d4") Dec 23 00:08:07 archserver kubelet[957]: I1223 00:08:07.174170 957 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "kube-proxy-token-jjpr9" (UniqueName: "kubernetes.io/secret/b9bf1b00-063d-11e9-b166-001966fbc1d4-kube-proxy-token-jjpr9") pod "kube-proxy-58pbt" (UID: "b9bf1b00-063d-11e9-b166-001966fbc1d4") Dec 23 00:08:07 archserver kubelet[957]: I1223 00:08:07.174333 957 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "kube-proxy" (UniqueName: "kubernetes.io/configmap/b9bf1b00-063d-11e9-b166-001966fbc1d4-kube-proxy") pod "kube-proxy-58pbt" (UID: "b9bf1b00-063d-11e9-b166-001966fbc1d4") Dec 23 00:08:07 archserver kubelet[957]: I1223 00:08:07.174432 957 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "xtables-lock" (UniqueName: "kubernetes.io/host-path/b9bf1b00-063d-11e9-b166-001966fbc1d4-xtables-lock") pod "kube-proxy-58pbt" (UID: "b9bf1b00-063d-11e9-b166-001966fbc1d4") Dec 23 00:08:07 archserver kubelet[957]: I1223 00:08:07.275307 957 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "coredns-token-d2jmz" (UniqueName: "kubernetes.io/secret/b9cf29c2-063d-11e9-b166-001966fbc1d4-coredns-token-d2jmz") pod "coredns-86c58d9df4-g9bfh" (UID: "b9cf29c2-063d-11e9-b166-001966fbc1d4") Dec 23 00:08:07 archserver kubelet[957]: I1223 00:08:07.275735 957 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "config-volume" (UniqueName: "kubernetes.io/configmap/b9cf29c2-063d-11e9-b166-001966fbc1d4-config-volume") pod "coredns-86c58d9df4-g9bfh" (UID: "b9cf29c2-063d-11e9-b166-001966fbc1d4") Dec 23 00:08:07 archserver kubelet[957]: I1223 00:08:07.379344 957 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "config-volume" (UniqueName: "kubernetes.io/configmap/b9d3e42a-063d-11e9-b166-001966fbc1d4-config-volume") pod "coredns-86c58d9df4-lrkrq" (UID: "b9d3e42a-063d-11e9-b166-001966fbc1d4") Dec 23 00:08:07 archserver kubelet[957]: I1223 00:08:07.379629 957 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "coredns-token-d2jmz" (UniqueName: "kubernetes.io/secret/b9d3e42a-063d-11e9-b166-001966fbc1d4-coredns-token-d2jmz") pod "coredns-86c58d9df4-lrkrq" (UID: "b9d3e42a-063d-11e9-b166-001966fbc1d4") Dec 23 00:08:07 archserver kubelet[957]: I1223 00:08:07.480212 957 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "flannel-token-r7c47" (UniqueName: "kubernetes.io/secret/1b1816d8-0646-11e9-a90a-001966fbc1d4-flannel-token-r7c47") pod "kube-flannel-ds-amd64-dcpcm" (UID: "1b1816d8-0646-11e9-a90a-001966fbc1d4") Dec 23 00:08:07 archserver kubelet[957]: I1223 00:08:07.480576 957 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "cni" (UniqueName: "kubernetes.io/host-path/1b1816d8-0646-11e9-a90a-001966fbc1d4-cni") pod "kube-flannel-ds-amd64-dcpcm" (UID: "1b1816d8-0646-11e9-a90a-001966fbc1d4") Dec 23 00:08:07 archserver kubelet[957]: I1223 00:08:07.480813 957 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "flannel-cfg" (UniqueName: "kubernetes.io/configmap/1b1816d8-0646-11e9-a90a-001966fbc1d4-flannel-cfg") pod "kube-flannel-ds-amd64-dcpcm" (UID: "1b1816d8-0646-11e9-a90a-001966fbc1d4") Dec 23 00:08:07 archserver kubelet[957]: I1223 00:08:07.481186 957 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "run" (UniqueName: "kubernetes.io/host-path/1b1816d8-0646-11e9-a90a-001966fbc1d4-run") pod "kube-flannel-ds-amd64-dcpcm" (UID: "1b1816d8-0646-11e9-a90a-001966fbc1d4") Dec 23 00:08:07 archserver kubelet[957]: I1223 00:08:07.556347 957 kubelet_node_status.go:114] Node archserver was previously registered Dec 23 00:08:07 archserver kubelet[957]: I1223 00:08:07.556768 957 kubelet_node_status.go:75] Successfully registered node archserver Dec 23 00:08:07 archserver kubelet[957]: I1223 00:08:07.683056 957 reconciler.go:154] Reconciler: start to sync state Dec 23 00:08:10 archserver kubelet[957]: W1223 00:08:10.199532 957 pod_container_deletor.go:75] Container "c99998a295415ebc89eb30537af9af4ae28f9a6fd358928d776e3c32630c985a" not found in pod's containers Dec 23 00:08:10 archserver kubelet[957]: W1223 00:08:10.572939 957 kuberuntime_container.go:687] No ref for container {"docker" "0315de6800d9fabd7974fcc822c1557367c103e77cadc01770b0a401032cccd0"} Dec 23 00:08:12 archserver kubelet[957]: W1223 00:08:12.490577 957 pod_container_deletor.go:75] Container "71d6810c2b1efc0152ae57befdb2904bd90f2c833321894adf48c884db66dc8d" not found in pod's containers Dec 23 00:08:15 archserver kubelet[957]: W1223 00:08:15.788166 957 pod_container_deletor.go:75] Container "22993ef3c804b5f1788059f5242e2fa6e4212dc59b7a65609bec06029effd989" not found in pod's containers Dec 23 00:08:17 archserver kubelet[957]: W1223 00:08:17.278415 957 pod_container_deletor.go:75] Container "0315de6800d9fabd7974fcc822c1557367c103e77cadc01770b0a401032cccd0" not found in pod's containers Dec 23 00:08:17 archserver kubelet[957]: W1223 00:08:17.278615 957 pod_container_deletor.go:75] Container "760078cf574e62ff94bd02324cfb1b81bff7a5b544a01f9911779f3d4effbf28" not found in pod's containers

systemctl status kubelet:

● kubelet.service - Kubernetes Kubelet Server Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled) Active: active (running) since Sun 2018-12-23 00:07:27 UTC; 37min ago Docs: https://github.com/GoogleCloudPlatform/kubernetes Main PID: 957 (kubelet) Tasks: 23 (limit: 4915) Memory: 108.3M CGroup: /system.slice/kubelet.service └─957 /usr/bin/kubelet --logtostderr=true --v=0 --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml

Dec 23 00:08:07 archserver kubelet[957]: I1223 00:08:07.481186 957 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "run" (UniqueName: "kubernetes.io/host-path/1b1816d8-0646-11e9-a90a-001966fbc1d4-run") pod "kube-flannel-ds-amd64-dcpcm" (UID: "1b1816d8-0646-11e9-a90a-001966fbc1d4") Dec 23 00:08:07 archserver kubelet[957]: I1223 00:08:07.556347 957 kubelet_node_status.go:114] Node archserver was previously registered Dec 23 00:08:07 archserver kubelet[957]: I1223 00:08:07.556768 957 kubelet_node_status.go:75] Successfully registered node archserver Dec 23 00:08:07 archserver kubelet[957]: I1223 00:08:07.683056 957 reconciler.go:154] Reconciler: start to sync state Dec 23 00:08:10 archserver kubelet[957]: W1223 00:08:10.199532 957 pod_container_deletor.go:75] Container "c99998a295415ebc89eb30537af9af4ae28f9a6fd358928d776e3c32630c985a" not found in pod's containers Dec 23 00:08:10 archserver kubelet[957]: W1223 00:08:10.572939 957 kuberuntime_container.go:687] No ref for container {"docker" "0315de6800d9fabd7974fcc822c1557367c103e77cadc01770b0a401032cccd0"} Dec 23 00:08:12 archserver kubelet[957]: W1223 00:08:12.490577 957 pod_container_deletor.go:75] Container "71d6810c2b1efc0152ae57befdb2904bd90f2c833321894adf48c884db66dc8d" not found in pod's containers Dec 23 00:08:15 archserver kubelet[957]: W1223 00:08:15.788166 957 pod_container_deletor.go:75] Container "22993ef3c804b5f1788059f5242e2fa6e4212dc59b7a65609bec06029effd989" not found in pod's containers Dec 23 00:08:17 archserver kubelet[957]: W1223 00:08:17.278415 957 pod_container_deletor.go:75] Container "0315de6800d9fabd7974fcc822c1557367c103e77cadc01770b0a401032cccd0" not found in pod's containers Dec 23 00:08:17 archserver kubelet[957]: W1223 00:08:17.278615 957 pod_container_deletor.go:75] Container "760078cf574e62ff94bd02324cfb1b81bff7a5b544a01f9911779f3d4effbf28" not found in pod's containers

And it is really strange, all control plane pods got launched successfully and they seems to run, but they all got the same host IP (`192.168.0.25`). The the coredns-Pods somehow got the IPs from `docker0`-bridge, i guess so, because they begin with `172.17.0.x`. Here kubectl get pods -o wide:

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES coredns-86c58d9df4-g9bfh 1/1 Running 3 67m 172.17.0.2 archserver coredns-86c58d9df4-lrkrq 1/1 Running 3 67m 172.17.0.3 archserver etcd-archserver 1/1 Running 3 67m 192.168.0.25 archserver kube-apiserver-archserver 1/1 Running 2 48m 192.168.0.25 archserver kube-controller-manager-archserver 1/1 Running 4 66m 192.168.0.25 archserver kube-flannel-ds-amd64-dcpcm 1/1 Running 1 7m36s 192.168.0.25 archserver kube-proxy-58pbt 1/1 Running 3 67m 192.168.0.25 archserver kube-scheduler-archserver 1/1 Running 4 66m 192.168.0.25 archserver

The pods had got the host IP address like above already before i have install flannel, but it does not change after installing flannel and system reboot. All that you see above is the current state of the system after a fresh reboot. I have rebooted the system multiple times, but makes no differences.

## What you expected to happen?
Kubelet journal log does not show such strange error messages and all pods get some pod IP from the ip range specified while kubeadm init `--pod-network-cidr=10.244.0.0/16`

## How to reproduce it (as minimally and precisely as possible)?
Just execute `kubeadm init --pod-network-cidr=10.244.0.0/16` and then install flannel network addon.

## Anything else we need to know?
docper ps:

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f5ca8af32f08 f0fad859c909 "/opt/bin/flanneld -…" 2 minutes ago Up 2 minutes k8s_kube-flannel_kube-flannel-ds-amd64-dcpcm_kube-system_1b1816d8-0646-11e9-a90a-001966fbc1d4_1 89d69f474cf1 f59dcacceff4 "/coredns -conf /etc…" 2 minutes ago Up 2 minutes k8s_coredns_coredns-86c58d9df4-lrkrq_kube-system_b9d3e42a-063d-11e9-b166-001966fbc1d4_3 2d498ba70549 f59dcacceff4 "/coredns -conf /etc…" 3 minutes ago Up 2 minutes k8s_coredns_coredns-86c58d9df4-g9bfh_kube-system_b9cf29c2-063d-11e9-b166-001966fbc1d4_3 9287347fb490 fdb321fd30a0 "/usr/local/bin/kube…" 3 minutes ago Up 2 minutes k8s_kube-proxy_kube-proxy-58pbt_kube-system_b9bf1b00-063d-11e9-b166-001966fbc1d4_3 760078cf574e k8s.gcr.io/pause:3.1 "/pause" 3 minutes ago Up 2 minutes k8s_POD_kube-flannel-ds-amd64-dcpcm_kube-system_1b1816d8-0646-11e9-a90a-001966fbc1d4_1 22993ef3c804 k8s.gcr.io/pause:3.1 "/pause" 3 minutes ago Up 2 minutes k8s_POD_coredns-86c58d9df4-lrkrq_kube-system_b9d3e42a-063d-11e9-b166-001966fbc1d4_3 c99998a29541 k8s.gcr.io/pause:3.1 "/pause" 3 minutes ago Up 3 minutes k8s_POD_coredns-86c58d9df4-g9bfh_kube-system_b9cf29c2-063d-11e9-b166-001966fbc1d4_3 71d6810c2b1e k8s.gcr.io/pause:3.1 "/pause" 3 minutes ago Up 3 minutes k8s_POD_kube-proxy-58pbt_kube-system_b9bf1b00-063d-11e9-b166-001966fbc1d4_3 e7ba49b12335 3cab8e1b9802 "etcd --advertise-cl…" 3 minutes ago Up 3 minutes k8s_etcd_etcd-archserver_kube-system_459a3e3ab367c3ffe0309150ad8995c1_3 5e7a131a23a4 k8s.gcr.io/pause:3.1 "/pause" 3 minutes ago Up 3 minutes k8s_POD_etcd-archserver_kube-system_459a3e3ab367c3ffe0309150ad8995c1_3 5babfe2ae495 40a63db91ef8 "kube-apiserver --au…" 3 minutes ago Up 3 minutes k8s_kube-apiserver_kube-apiserver-archserver_kube-system_d92cb816c7d721bf5d651ac4cc2e875a_2 3736f9129fe8 26e6f1db2a52 "kube-controller-man…" 3 minutes ago Up 3 minutes k8s_kube-controller-manager_kube-controller-manager-archserver_kube-system_8db7fcf04315eec49ec5bc8348632e73_4 77e7675f54b2 ab81d7360408 "kube-scheduler --ad…" 3 minutes ago Up 3 minutes k8s_kube-scheduler_kube-scheduler-archserver_kube-system_44b569a35761491825f4e7253fbf0543_4 82831bed7697 k8s.gcr.io/pause:3.1 "/pause" 3 minutes ago Up 3 minutes k8s_POD_kube-apiserver-archserver_kube-system_d92cb816c7d721bf5d651ac4cc2e875a_2 2ec6df9260f3 k8s.gcr.io/pause:3.1 "/pause" 3 minutes ago Up 3 minutes k8s_POD_kube-scheduler-archserver_kube-system_44b569a35761491825f4e7253fbf0543_3 8d6507cc70f5 k8s.gcr.io/pause:3.1 "/pause" 3 minutes ago Up 3 minutes k8s_POD_kube-controller-manager-archserver_kube-system_8db7fcf04315eec49ec5bc8348632e73_3

kubectl logs kube-apiserver-archserver -n kube-system

Flag --insecure-port has been deprecated, This flag will be removed in a future version. I1223 00:07:51.863733 1 server.go:557] external host was not specified, using 192.168.0.25 I1223 00:07:51.864237 1 server.go:146] Version: v1.13.1 I1223 00:07:52.667343 1 plugins.go:158] Loaded 8 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,Priority,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook. I1223 00:07:52.667573 1 plugins.go:161] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota. I1223 00:07:52.679851 1 plugins.go:158] Loaded 8 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,Priority,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook. I1223 00:07:52.679908 1 plugins.go:161] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota. I1223 00:07:53.961437 1 master.go:228] Using reconciler: lease W1223 00:07:56.730713 1 genericapiserver.go:334] Skipping API batch/v2alpha1 because it has no resources. W1223 00:07:56.982264 1 genericapiserver.go:334] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources. W1223 00:07:56.998317 1 genericapiserver.go:334] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources. W1223 00:07:57.071412 1 genericapiserver.go:334] Skipping API storage.k8s.io/v1alpha1 because it has no resources. W1223 00:07:57.203701 1 genericapiserver.go:334] Skipping API admissionregistration.k8s.io/v1alpha1 because it has no resources. [restful] 2018/12/23 00:07:57 log.go:33: [restful/swagger] listing is available at https://192.168.0.25:6443/swaggerapi [restful] 2018/12/23 00:07:57 log.go:33: [restful/swagger] https://192.168.0.25:6443/swaggerui/ is mapped to folder /swagger-ui/ [restful] 2018/12/23 00:08:00 log.go:33: [restful/swagger] listing is available at https://192.168.0.25:6443/swaggerapi [restful] 2018/12/23 00:08:00 log.go:33: [restful/swagger] https://192.168.0.25:6443/swaggerui/ is mapped to folder /swagger-ui/ I1223 00:08:00.354847 1 plugins.go:158] Loaded 8 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,Priority,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook. I1223 00:08:00.354882 1 plugins.go:161] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota. I1223 00:08:06.751963 1 secure_serving.go:116] Serving securely on [::]:6443 I1223 00:08:06.752275 1 apiservice_controller.go:90] Starting APIServiceRegistrationController I1223 00:08:06.752506 1 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller I1223 00:08:06.752531 1 available_controller.go:283] Starting AvailableConditionController I1223 00:08:06.755054 1 cache.go:32] Waiting for caches to sync for AvailableConditionController controller I1223 00:08:06.752684 1 autoregister_controller.go:136] Starting autoregister controller I1223 00:08:06.755340 1 cache.go:32] Waiting for caches to sync for autoregister controller I1223 00:08:06.752780 1 controller.go:84] Starting OpenAPI AggregationController I1223 00:08:06.752821 1 crdregistration_controller.go:112] Starting crd-autoregister controller I1223 00:08:06.755473 1 controller_utils.go:1027] Waiting for caches to sync for crd-autoregister controller I1223 00:08:06.753356 1 crd_finalizer.go:242] Starting CRDFinalizer I1223 00:08:06.753403 1 customresource_discovery_controller.go:203] Starting DiscoveryController I1223 00:08:06.753438 1 naming_controller.go:284] Starting NamingConditionController I1223 00:08:06.753474 1 establishing_controller.go:73] Starting EstablishingController I1223 00:08:06.814359 1 log.go:172] http: TLS handshake error from 192.168.0.25:52356: EOF I1223 00:08:06.820388 1 log.go:172] http: TLS handshake error from 192.168.0.25:52358: EOF I1223 00:08:06.829604 1 log.go:172] http: TLS handshake error from 192.168.0.25:52360: EOF I1223 00:08:06.837240 1 log.go:172] http: TLS handshake error from 192.168.0.25:52362: EOF I1223 00:08:06.843615 1 log.go:172] http: TLS handshake error from 192.168.0.25:52364: EOF I1223 00:08:06.848556 1 log.go:172] http: TLS handshake error from 192.168.0.25:52366: EOF I1223 00:08:06.852710 1 log.go:172] http: TLS handshake error from 192.168.0.25:52368: EOF I1223 00:08:06.856882 1 log.go:172] http: TLS handshake error from 192.168.0.25:52370: EOF I1223 00:08:06.861038 1 log.go:172] http: TLS handshake error from 192.168.0.25:52372: EOF I1223 00:08:06.861715 1 controller_utils.go:1034] Caches are synced for crd-autoregister controller I1223 00:08:06.865196 1 log.go:172] http: TLS handshake error from 192.168.0.25:52374: EOF I1223 00:08:06.869384 1 log.go:172] http: TLS handshake error from 192.168.0.25:52376: EOF I1223 00:08:06.873971 1 log.go:172] http: TLS handshake error from 192.168.0.25:52378: EOF I1223 00:08:06.884057 1 log.go:172] http: TLS handshake error from 192.168.0.25:52380: EOF I1223 00:08:06.888331 1 log.go:172] http: TLS handshake error from 192.168.0.25:52384: EOF I1223 00:08:06.929848 1 cache.go:39] Caches are synced for autoregister controller I1223 00:08:06.955947 1 cache.go:39] Caches are synced for AvailableConditionController controller I1223 00:08:06.958226 1 cache.go:39] Caches are synced for APIServiceRegistrationController controller I1223 00:08:08.178730 1 storage_scheduling.go:100] all system priority classes are created successfully or already exist. I1223 00:08:08.656503 1 trace.go:76] Trace[2005574443]: "Get /api/v1/namespaces/kube-system/endpoints/kube-scheduler" (started: 2018-12-23 00:08:08.095684504 +0000 UTC m=+22.259121879) (total time: 560.704903ms): Trace[2005574443]: [560.520248ms] [560.414661ms] About to write a response I1223 00:08:08.658802 1 trace.go:76] Trace[1707036773]: "GuaranteedUpdate etcd3: *core.Event" (started: 2018-12-23 00:08:07.8786348 +0000 UTC m=+22.042072094) (total time: 780.064054ms): Trace[1707036773]: [348.978966ms] [348.978966ms] initial value restored Trace[1707036773]: [779.973308ms] [430.087985ms] Transaction committed I1223 00:08:08.659126 1 trace.go:76] Trace[1207649510]: "Patch /api/v1/namespaces/default/events/archserver.1572cd92498358fc" (started: 2018-12-23 00:08:07.878476311 +0000 UTC m=+22.041913628) (total time: 780.588916ms): Trace[1207649510]: [349.143207ms] [349.097113ms] About to apply patch Trace[1207649510]: [780.392909ms] [430.741322ms] Object stored in database I1223 00:08:22.447888 1 controller.go:608] quota admission added evaluator for: endpoints

kubectl logs etcd-archserver -n kube-system

2018-12-23 00:07:47.781821 I | etcdmain: etcd Version: 3.2.24 2018-12-23 00:07:47.795165 I | etcdmain: Git SHA: 420a45226 2018-12-23 00:07:47.795184 I | etcdmain: Go Version: go1.8.7 2018-12-23 00:07:47.795211 I | etcdmain: Go OS/Arch: linux/amd64 2018-12-23 00:07:47.795229 I | etcdmain: setting maximum number of CPUs to 4, total number of available CPUs is 4 2018-12-23 00:07:47.802892 N | etcdmain: the server is already initialized as member before, starting as etcd member... 2018-12-23 00:07:47.803006 I | embed: peerTLS: cert = /etc/kubernetes/pki/etcd/peer.crt, key = /etc/kubernetes/pki/etcd/peer.key, ca = , trusted-ca = /etc/kubernetes/pki/etcd/ca.crt, client-cert-auth = true 2018-12-23 00:07:47.918263 I | embed: listening for peers on https://192.168.0.25:2380 2018-12-23 00:07:47.918438 I | embed: listening for client requests on 127.0.0.1:2379 2018-12-23 00:07:47.918485 I | embed: listening for client requests on 192.168.0.25:2379 2018-12-23 00:07:48.392327 I | etcdserver: name = archserver 2018-12-23 00:07:48.393168 I | etcdserver: data dir = /var/lib/etcd 2018-12-23 00:07:48.393217 I | etcdserver: member dir = /var/lib/etcd/member 2018-12-23 00:07:48.393236 I | etcdserver: heartbeat = 100ms 2018-12-23 00:07:48.393300 I | etcdserver: election = 1000ms 2018-12-23 00:07:48.393317 I | etcdserver: snapshot count = 10000 2018-12-23 00:07:48.393359 I | etcdserver: advertise client URLs = https://192.168.0.25:2379 2018-12-23 00:07:51.975673 I | etcdserver: restarting member affd478dfebabc4b in cluster 5d6769b114066486 at commit index 5473 2018-12-23 00:07:51.977927 I | raft: affd478dfebabc4b became follower at term 4 2018-12-23 00:07:51.978018 I | raft: newRaft affd478dfebabc4b [peers: [], term: 4, commit: 5473, applied: 0, lastindex: 5473, lastterm: 4] 2018-12-23 00:07:52.135877 I | mvcc: restore compact to 4309 2018-12-23 00:07:52.160140 W | auth: simple token is not cryptographically signed 2018-12-23 00:07:52.240312 I | etcdserver: starting server... [version: 3.2.24, cluster version: to_be_decided] 2018-12-23 00:07:52.290038 I | embed: ClientTLS: cert = /etc/kubernetes/pki/etcd/server.crt, key = /etc/kubernetes/pki/etcd/server.key, ca = , trusted-ca = /etc/kubernetes/pki/etcd/ca.crt, client-cert-auth = true 2018-12-23 00:07:52.297865 I | etcdserver/membership: added member affd478dfebabc4b [https://192.168.0.25:2380] to cluster 5d6769b114066486 2018-12-23 00:07:52.298644 N | etcdserver/membership: set the initial cluster version to 3.2 2018-12-23 00:07:52.298800 I | etcdserver/api: enabled capabilities for version 3.2 2018-12-23 00:07:53.680245 I | raft: affd478dfebabc4b is starting a new election at term 4 2018-12-23 00:07:53.680340 I | raft: affd478dfebabc4b became candidate at term 5 2018-12-23 00:07:53.680400 I | raft: affd478dfebabc4b received MsgVoteResp from affd478dfebabc4b at term 5 2018-12-23 00:07:53.680447 I | raft: affd478dfebabc4b became leader at term 5 2018-12-23 00:07:53.680547 I | raft: raft.node: affd478dfebabc4b elected leader affd478dfebabc4b at term 5 2018-12-23 00:07:53.695571 I | etcdserver: published {Name:archserver ClientURLs:[https://192.168.0.25:2379]} to cluster 5d6769b114066486 2018-12-23 00:07:53.695643 I | embed: ready to serve client requests 2018-12-23 00:07:53.695843 I | embed: ready to serve client requests 2018-12-23 00:07:53.716480 I | embed: serving client requests on 127.0.0.1:2379 2018-12-23 00:07:53.716674 I | embed: serving client requests on 192.168.0.25:2379 2018-12-23 00:08:07.123802 W | etcdserver: read-only range request "key:\"/registry/ranges/serviceips\" " with result "range_response_count:1 size:118" took too long (177.921464ms) to execute proto: no coders for int proto: no encoder for ValueSize int [GetProperties] 2018-12-23 00:08:07.153678 W | etcdserver: request "header:<ID:13568052481467067269 username:\"kube-apiserver-etcd-client\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/events/default/archserver.1572cd923f9268b3\" mod_revision:0 > success:<request_put:<key:\"/registry/events/default/archserver.1572cd923f9268b3\" value_size:252 lease:4344680444612291459 >> failure:<>>" with result "size:16" took too long (161.994249ms) to execute 2018-12-23 00:08:07.155466 W | etcdserver: read-only range request "key:\"/registry/apiregistration.k8s.io/apiservices/v1.autoscaling\" " with result "range_response_count:1 size:624" took too long (179.245064ms) to execute 2018-12-23 00:08:07.189161 W | etcdserver: read-only range request "key:\"/registry/apiregistration.k8s.io/apiservices/v1beta1.apps\" " with result "range_response_count:1 size:617" took too long (209.280247ms) to execute 2018-12-23 00:08:07.190501 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:451" took too long (156.948786ms) to execute 2018-12-23 00:08:07.192799 W | etcdserver: read-only range request "key:\"/registry/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io\" " with result "range_response_count:1 size:654" took too long (209.402063ms) to execute 2018-12-23 00:08:07.194343 W | etcdserver: read-only range request "key:\"/registry/apiregistration.k8s.io/apiservices/v2beta2.autoscaling\" " with result "range_response_count:1 size:638" took too long (215.29691ms) to execute 2018-12-23 00:08:07.195416 W | etcdserver: read-only range request "key:\"/registry/apiregistration.k8s.io/apiservices/v1beta1.coordination.k8s.io\" " with result "range_response_count:1 size:662" took too long (217.677244ms) to execute 2018-12-23 00:08:07.340457 W | etcdserver: read-only range request "key:\"/registry/apiregistration.k8s.io/apiservices/v1beta1.storage.k8s.io\" " with result "range_response_count:1 size:647" took too long (176.100004ms) to execute 2018-12-23 00:08:07.340546 W | etcdserver: read-only range request "key:\"/registry/minions/archserver\" " with result "range_response_count:1 size:3507" took too long (181.96235ms) to execute 2018-12-23 00:08:07.340940 W | etcdserver: read-only range request "key:\"/registry/pods/kube-system/kube-proxy-58pbt\" " with result "range_response_count:1 size:2190" took too long (151.991401ms) to execute 2018-12-23 00:08:07.341547 W | etcdserver: read-only range request "key:\"/registry/apiregistration.k8s.io/apiservices/v1beta1.admissionregistration.k8s.io\" " with result "range_response_count:1 size:690" took too long (127.687534ms) to execute 2018-12-23 00:08:07.341917 W | etcdserver: read-only range request "key:\"/registry/apiregistration.k8s.io/apiservices/v1.apps\" " with result "range_response_count:1 size:603" took too long (130.538329ms) to execute 2018-12-23 00:08:07.343101 W | etcdserver: read-only range request "key:\"/registry/apiregistration.k8s.io/apiservices/v1.networking.k8s.io\" " with result "range_response_count:1 size:642" took too long (129.189826ms) to execute 2018-12-23 00:08:07.344207 W | etcdserver: read-only range request "key:\"/registry/services/specs/\" range_end:\"/registry/services/specs0\" " with result "range_response_count:2 size:729" took too long (128.572896ms) to execute 2018-12-23 00:08:07.345795 W | etcdserver: read-only range request "key:\"/registry/apiregistration.k8s.io/apiservices/v1.batch\" " with result "range_response_count:1 size:606" took too long (128.076229ms) to execute 2018-12-23 00:08:07.552379 W | etcdserver: read-only range request "key:\"/registry/apiregistration.k8s.io/apiservices/v1.rbac.authorization.k8s.io\" " with result "range_response_count:1 size:666" took too long (180.289305ms) to execute 2018-12-23 00:08:07.552503 W | etcdserver: read-only range request "key:\"/registry/minions/archserver\" " with result "range_response_count:1 size:3507" took too long (168.153019ms) to execute 2018-12-23 00:08:07.552573 W | etcdserver: read-only range request "key:\"/registry/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io\" " with result "range_response_count:1 size:651" took too long (180.447812ms) to execute 2018-12-23 00:08:07.552726 W | etcdserver: read-only range request "key:\"/registry/namespaces/default\" " with result "range_response_count:1 size:170" took too long (165.769097ms) to execute 2018-12-23 00:08:07.552960 W | etcdserver: read-only range request "key:\"/registry/pods/kube-system/coredns-86c58d9df4-g9bfh\" " with result "range_response_count:1 size:1759" took too long (105.090766ms) to execute 2018-12-23 00:08:07.553385 W | etcdserver: read-only range request "key:\"/registry/apiregistration.k8s.io/apiservices/v1.storage.k8s.io\" " with result "range_response_count:1 size:633" took too long (167.364683ms) to execute 2018-12-23 00:08:07.553626 W | etcdserver: read-only range request "key:\"/registry/events/default/archserver.1572cd924983260b\" " with result "range_response_count:1 size:389" took too long (168.46884ms) to execute 2018-12-23 00:08:07.554019 W | etcdserver: read-only range request "key:\"/registry/apiregistration.k8s.io/apiservices/v1beta1.policy\" " with result "range_response_count:1 size:623" took too long (182.139518ms) to execute 2018-12-23 00:08:07.554333 W | etcdserver: read-only range request "key:\"/registry/apiregistration.k8s.io/apiservices/v1beta1.authentication.k8s.io\" " with result "range_response_count:1 size:668" took too long (119.747896ms) to execute 2018-12-23 00:08:07.855759 W | etcdserver: read-only range request "key:\"/registry/services/specs/default/kubernetes\" " with result "range_response_count:1 size:291" took too long (284.769505ms) to execute 2018-12-23 00:08:07.856456 W | etcdserver: request "header:<ID:13568052481467067291 username:\"kube-apiserver-etcd-client\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/pods/kube-system/coredns-86c58d9df4-g9bfh\" mod_revision:1830 > success:<request_put:<key:\"/registry/pods/kube-system/coredns-86c58d9df4-g9bfh\" value_size:1783 >> failure:<request_range:<key:\"/registry/pods/kube-system/coredns-86c58d9df4-g9bfh\" > >>" with result "size:16" took too long (131.198128ms) to execute 2018-12-23 00:08:07.858597 W | etcdserver: read-only range request "key:\"/registry/apiregistration.k8s.io/apiservices/v1beta1.scheduling.k8s.io\" " with result "range_response_count:1 size:657" took too long (277.168919ms) to execute 2018-12-23 00:08:07.859320 W | etcdserver: read-only range request "key:\"/registry/apiregistration.k8s.io/apiservices/v2beta1.autoscaling\" " with result "range_response_count:1 size:638" took too long (277.933469ms) to execute 2018-12-23 00:08:07.859625 W | etcdserver: read-only range request "key:\"/registry/events/default/archserver.1572cd92498346be\" " with result "range_response_count:1 size:385" took too long (278.392057ms) to execute 2018-12-23 00:08:07.863434 W | etcdserver: read-only range request "key:\"/registry/apiregistration.k8s.io/apiservices/v1beta1.extensions\" " with result "range_response_count:1 size:635" took too long (290.29744ms) to execute 2018-12-23 00:08:07.864001 W | etcdserver: read-only range request "key:\"/registry/priorityclasses/system-node-critical\" " with result "range_response_count:1 size:293" took too long (105.871279ms) to execute 2018-12-23 00:08:07.864592 W | etcdserver: read-only range request "key:\"/registry/namespaces/kube-system\" " with result "range_response_count:1 size:178" took too long (106.501071ms) to execute 2018-12-23 00:08:07.866390 W | etcdserver: read-only range request "key:\"/registry/clusterroles/\" range_end:\"/registry/clusterroles0\" " with result "range_response_count:55 size:34259" took too long (110.022236ms) to execute 2018-12-23 00:08:07.867145 W | etcdserver: read-only range request "key:\"/registry/apiregistration.k8s.io/apiservices/v1beta2.apps\" " with result "range_response_count:1 size:617" took too long (283.736258ms) to execute 2018-12-23 00:08:08.172005 W | etcdserver: read-only range request "key:\"/registry/priorityclasses/system-cluster-critical\" " with result "range_response_count:1 size:328" took too long (297.083041ms) to execute 2018-12-23 00:08:08.224862 W | etcdserver: read-only range request "key:\"/registry/events/default/archserver.1572cd92498358fc\" " with result "range_response_count:1 size:383" took too long (343.460919ms) to execute 2018-12-23 00:08:08.224954 W | etcdserver: read-only range request "key:\"/registry/minions/archserver\" " with result "range_response_count:1 size:3507" took too long (337.388578ms) to execute 2018-12-23 00:08:08.225553 W | etcdserver: read-only range request "key:\"/registry/clusterrolebindings/\" range_end:\"/registry/clusterrolebindings0\" " with result "range_response_count:42 size:19581" took too long (322.323685ms) to execute 2018-12-23 00:08:08.654762 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-scheduler\" " with result "range_response_count:1 size:433" took too long (557.74089ms) to execute 2018-12-23 00:08:08.654831 W | etcdserver: read-only range request "key:\"/registry/pods/kube-system/kube-flannel-ds-amd64-dcpcm\" " with result "range_response_count:1 size:2735" took too long (468.408941ms) to execute 2018-12-23 00:08:08.655185 W | etcdserver: request "header:<ID:13568052481467067306 username:\"kube-apiserver-etcd-client\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/events/default/archserver.1572cd92498358fc\" mod_revision:4737 > success:<request_put:<key:\"/registry/events/default/archserver.1572cd92498358fc\" value_size:298 lease:4344680444612291459 >> failure:<request_range:<key:\"/registry/events/default/archserver.1572cd92498358fc\" > >>" with result "size:16" took too long (357.077663ms) to execute 2018-12-23 00:08:08.655591 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/default/kubernetes\" " with result "range_response_count:1 size:213" took too long (423.225451ms) to execute 2018-12-23 00:08:08.656118 W | etcdserver: read-only range request "key:\"/registry/clusterroles/system:aggregate-to-admin\" " with result "range_response_count:1 size:544" took too long (420.048177ms) to execute 2018-12-23 00:08:08.947068 W | etcdserver: read-only range request "key:\"/registry/services/specs/default/kubernetes\" " with result "range_response_count:1 size:291" took too long (216.503999ms) to execute 2018-12-23 00:08:08.947180 W | etcdserver: read-only range request "key:\"/registry/services/specs/\" range_end:\"/registry/services/specs0\" " with result "range_response_count:2 size:729" took too long (267.3985ms) to execute 2018-12-23 00:08:08.947560 W | etcdserver: read-only range request "key:\"/registry/namespaces/kube-public\" " with result "range_response_count:1 size:178" took too long (217.059193ms) to execute 2018-12-23 00:08:08.948116 W | etcdserver: read-only range request "key:\"/registry/events/default/archserver.1572cd924983260b\" " with result "range_response_count:1 size:389" took too long (267.11611ms) to execute 2018-12-23 00:08:08.948292 W | etcdserver: read-only range request "key:\"/registry/ranges/servicenodeports\" " with result "range_response_count:1 size:120" took too long (271.727473ms) to execute 2018-12-23 00:08:08.948881 W | etcdserver: read-only range request "key:\"/registry/clusterroles/system:aggregate-to-view\" " with result "range_response_count:1 size:1255" took too long (273.082886ms) to execute 2018-12-23 00:08:09.200819 W | etcdserver: read-only range request "key:\"/registry/masterleases/192.168.0.25\" " with result "range_response_count:1 size:133" took too long (244.897903ms) to execute 2018-12-23 00:08:09.200943 W | etcdserver: read-only range request "key:\"/registry/events/default/archserver.1572cd92498346be\" " with result "range_response_count:1 size:385" took too long (233.630665ms) to execute 2018-12-23 00:08:09.201000 W | etcdserver: read-only range request "key:\"/registry/ranges/serviceips\" " with result "range_response_count:1 size:118" took too long (243.830154ms) to execute 2018-12-23 00:08:09.201387 W | etcdserver: read-only range request "key:\"/registry/clusterroles/cluster-admin\" " with result "range_response_count:1 size:309" took too long (243.173682ms) to execute 2018-12-23 00:08:09.456135 W | etcdserver: read-only range request "key:\"/registry/events/default/archserver.1572cd924983260b\" " with result "range_response_count:1 size:389" took too long (185.545512ms) to execute 2018-12-23 00:08:09.456767 W | etcdserver: read-only range request "key:\"/registry/masterleases/\" range_end:\"/registry/masterleases0\" " with result "range_response_count:1 size:133" took too long (193.349442ms) to execute 2018-12-23 00:08:09.457193 W | etcdserver: read-only range request "key:\"/registry/clusterroles/edit\" " with result "range_response_count:1 size:2471" took too long (185.490077ms) to execute 2018-12-23 00:08:09.752254 W | etcdserver: read-only range request "key:\"/registry/clusterroles/system:aggregate-to-admin\" " with result "range_response_count:1 size:544" took too long (203.545494ms) to execute 2018-12-23 00:08:09.752556 W | etcdserver: read-only range request "key:\"/registry/events/default/archserver.1572cd92498358fc\" " with result "range_response_count:1 size:383" took too long (201.044945ms) to execute 2018-12-23 00:08:09.986223 W | etcdserver: read-only range request "key:\"/registry/clusterroles/system:aggregate-to-edit\" " with result "range_response_count:1 size:1562" took too long (227.603428ms) to execute 2018-12-23 00:08:09.986292 W | etcdserver: read-only range request "key:\"/registry/events/default/archserver.1572cd924983260b\" " with result "range_response_count:1 size:389" took too long (223.872011ms) to execute 2018-12-23 00:08:10.314629 W | etcdserver: read-only range request "key:\"/registry/clusterroles/system:aggregate-to-view\" " with result "range_response_count:1 size:1255" took too long (310.426151ms) to execute 2018-12-23 00:08:10.314873 W | etcdserver: request "header: lease_revoke:" with result "size:28" took too long (134.088817ms) to execute 2018-12-23 00:08:10.315180 W | etcdserver: read-only range request "key:\"/registry/events/default/archserver.1572cd92498346be\" " with result "range_response_count:1 size:385" took too long (309.862407ms) to execute 2018-12-23 00:08:10.315437 W | etcdserver: read-only range request "key:\"/registry/pods/kube-system/kube-controller-manager-archserver\" " with result "range_response_count:1 size:3000" took too long (139.065048ms) to execute 2018-12-23 00:08:10.510762 W | etcdserver: read-only range request "key:\"/registry/clusterroles/system:heapster\" " with result "range_response_count:1 size:394" took too long (185.443449ms) to execute 2018-12-23 00:08:10.568913 W | etcdserver: read-only range request "key:\"/registry/events/default/archserver.1572cd92498358fc\" " with result "range_response_count:1 size:383" took too long (241.297297ms) to execute 2018-12-23 00:08:10.730832 W | etcdserver: read-only range request "key:\"/registry/clusterroles/system:node-problem-detector\" " with result "range_response_count:1 size:396" took too long (149.773559ms) to execute 2018-12-23 00:08:10.731364 W | etcdserver: read-only range request "key:\"/registry/events/default/archserver.1572cd924983260b\" " with result "range_response_count:1 size:389" took too long (144.277355ms) to execute 2018-12-23 00:08:11.114179 W | etcdserver: read-only range request "key:\"/registry/clusterroles/system:kubelet-api-admin\" " with result "range_response_count:1 size:430" took too long (372.071453ms) to execute 2018-12-23 00:08:11.114363 W | etcdserver: read-only range request "key:\"/registry/events/default/archserver.1572cd92498346be\" " with result "range_response_count:1 size:385" took too long (367.421924ms) to execute 2018-12-23 00:08:11.602169 W | etcdserver: read-only range request "key:\"/registry/clusterroles/system:auth-delegator\" " with result "range_response_count:1 size:407" took too long (474.692356ms) to execute 2018-12-23 00:08:11.602272 W | etcdserver: read-only range request "key:\"/registry/pods/kube-system/etcd-archserver\" " with result "range_response_count:1 size:2445" took too long (478.847704ms) to execute 2018-12-23 00:08:11.602967 W | etcdserver: read-only range request "key:\"/registry/events/default/archserver.1572cd92498358fc\" " with result "range_response_count:1 size:383" took too long (471.962773ms) to execute 2018-12-23 00:08:11.971989 W | etcdserver: read-only range request "key:\"/registry/clusterroles/system:kube-controller-manager\" " with result "range_response_count:1 size:616" took too long (358.904578ms) to execute 2018-12-23 00:08:11.972611 W | etcdserver: request "header:<ID:13568052481467067368 username:\"kube-apiserver-etcd-client\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/pods/kube-system/etcd-archserver\" mod_revision:1807 > success:<request_put:<key:\"/registry/pods/kube-system/etcd-archserver\" value_size:2380 >> failure:<request_range:<key:\"/registry/pods/kube-system/etcd-archserver\" > >>" with result "size:16" took too long (181.821543ms) to execute 2018-12-23 00:08:11.974593 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:451" took too long (123.694459ms) to execute 2018-12-23 00:08:11.978658 W | etcdserver: read-only range request "key:\"/registry/events/default/archserver.1572cd924983260b\" " with result "range_response_count:1 size:389" took too long (362.134245ms) to execute 2018-12-23 00:08:12.210547 W | etcdserver: read-only range request "key:\"/registry/clusterroles/system:kube-dns\" " with result "range_response_count:1 size:332" took too long (226.853844ms) to execute 2018-12-23 00:08:12.210864 W | etcdserver: read-only range request "key:\"/registry/events/default/archserver.1572cd92498346be\" " with result "range_response_count:1 size:385" took too long (224.739792ms) to execute 2018-12-23 00:08:12.577526 W | etcdserver: read-only range request "key:\"/registry/clusterroles/system:persistent-volume-provisioner\" " with result "range_response_count:1 size:553" took too long (356.221472ms) to execute 2018-12-23 00:08:12.577631 W | etcdserver: read-only range request "key:\"/registry/events/default/archserver.1572cd92498358fc\" " with result "range_response_count:1 size:383" took too long (352.944283ms) to execute 2018-12-23 00:08:12.818855 W | etcdserver: read-only range request "key:\"/registry/events/default/archserver.1572cd92498358fc\" " with result "range_response_count:1 size:383" took too long (167.330074ms) to execute 2018-12-23 00:08:12.819120 W | etcdserver: read-only range request "key:\"/registry/clusterroles/system:aws-cloud-provider\" " with result "range_response_count:1 size:372" took too long (173.726951ms) to execute 2018-12-23 00:08:13.145247 W | etcdserver: read-only range request "key:\"/registry/clusterroles/system:certificates.k8s.io:certificatesigningrequests:nodeclient\" " with result "range_response_count:1 size:462" took too long (320.577189ms) to execute 2018-12-23 00:08:13.145409 W | etcdserver: request "header:<ID:13568052481467067386 username:\"kube-apiserver-etcd-client\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/events/kube-system/etcd-archserver.1572cd9280fa991a\" mod_revision:0 > success:<request_put:<key:\"/registry/events/kube-system/etcd-archserver.1572cd9280fa991a\" value_size:436 lease:4344680444612291459 >> failure:<>>" with result "size:16" took too long (223.30574ms) to execute 2018-12-23 00:08:13.145659 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-scheduler\" " with result "range_response_count:1 size:433" took too long (148.85647ms) to execute 2018-12-23 00:08:13.568783 W | etcdserver: read-only range request "key:\"/registry/clusterroles/system:csi-external-provisioner\" " with result "range_response_count:1 size:573" took too long (406.549687ms) to execute 2018-12-23 00:08:13.568847 W | etcdserver: request "header:<ID:13568052481467067392 username:\"kube-apiserver-etcd-client\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/events/kube-system/kube-scheduler-archserver.1572cd92aa384159\" mod_revision:0 > success:<request_put:<key:\"/registry/events/kube-system/kube-scheduler-archserver.1572cd92aa384159\" value_size:363 lease:4344680444612291459 >> failure:<>>" with result "size:16" took too long (152.502471ms) to execute 2018-12-23 00:08:13.843515 W | etcdserver: read-only range request "key:\"/registry/clusterroles/system:controller:clusterrole-aggregation-controller\" " with result "range_response_count:1 size:387" took too long (254.605744ms) to execute 2018-12-23 00:08:13.843895 W | etcdserver: request "header:<ID:13568052481467067396 username:\"kube-apiserver-etcd-client\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/events/kube-system/kube-apiserver-archserver.1572cd9405466d81\" mod_revision:0 > success:<request_put:<key:\"/registry/events/kube-system/kube-apiserver-archserver.1572cd9405466d81\" value_size:410 lease:4344680444612291459 >> failure:<>>" with result "size:16" took too long (102.024985ms) to execute 2018-12-23 00:08:14.061115 W | etcdserver: read-only range request "key:\"/registry/clusterroles/system:controller:daemon-set-controller\" " with result "range_response_count:1 size:699" took too long (209.459495ms) to execute 2018-12-23 00:08:14.062957 W | etcdserver: request "header:<ID:13568052481467067400 username:\"kube-apiserver-etcd-client\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/events/kube-system/kube-controller-manager-archserver.1572cd9405538999\" mod_revision:0 > success:<request_put:<key:\"/registry/events/kube-system/kube-controller-manager-archserver.1572cd9405538999\" value_size:447 lease:4344680444612291459 >> failure:<>>" with result "size:16" took too long (104.866798ms) to execute 2018-12-23 00:08:14.476728 W | etcdserver: read-only range request "key:\"/registry/clusterroles/system:controller:endpoint-controller\" " with result "range_response_count:1 size:495" took too long (396.568904ms) to execute 2018-12-23 00:08:14.476936 W | etcdserver: request "header:<ID:13568052481467067405 username:\"kube-apiserver-etcd-client\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/events/kube-system/kube-scheduler-archserver.1572cd948b37a541\" mod_revision:0 > success:<request_put:<key:\"/registry/events/kube-system/kube-scheduler-archserver.1572cd948b37a541\" value_size:350 lease:4344680444612291459 >> failure:<>>" with result "size:16" took too long (259.435162ms) to execute 2018-12-23 00:08:14.748980 W | etcdserver: read-only range request "key:\"/registry/clusterroles/system:controller:generic-garbage-collector\" " with result "range_response_count:1 size:434" took too long (264.081554ms) to execute 2018-12-23 00:08:14.749242 W | etcdserver: request "header:<ID:13568052481467067409 username:\"kube-apiserver-etcd-client\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/events/kube-system/kube-scheduler-archserver.1572cd9507b12536\" mod_revision:0 > success:<request_put:<key:\"/registry/events/kube-system/kube-scheduler-archserver.1572cd9507b12536\" value_size:350 lease:4344680444612291459 >> failure:<>>" with result "size:16" took too long (138.294326ms) to execute 2018-12-23 00:08:15.011939 W | etcdserver: read-only range request "key:\"/registry/clusterroles/system:controller:job-controller\" " with result "range_response_count:1 size:514" took too long (251.468028ms) to execute 2018-12-23 00:08:15.012203 W | etcdserver: request "header:<ID:13568052481467067413 username:\"kube-apiserver-etcd-client\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/events/kube-system/kube-controller-manager-archserver.1572cd952f074683\" mod_revision:0 > success:<request_put:<key:\"/registry/events/kube-system/kube-controller-manager-archserver.1572cd952f074683\" value_size:378 lease:4344680444612291459 >> failure:<>>" with result "size:16" took too long (139.532711ms) to execute 2018-12-23 00:08:15.402142 W | etcdserver: read-only range request "key:\"/registry/clusterroles/system:controller:node-controller\" " with result "range_response_count:1 size:492" took too long (371.195316ms) to execute 2018-12-23 00:08:15.402506 W | etcdserver: request "header:<ID:13568052481467067417 username:\"kube-apiserver-etcd-client\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/events/kube-system/etcd-archserver.1572cd9565b68dc9\" mod_revision:0 > success:<request_put:<key:\"/registry/events/kube-system/etcd-archserver.1572cd9565b68dc9\" value_size:369 lease:4344680444612291459 >> failure:<>>" with result "size:16" took too long (162.92613ms) to execute 2018-12-23 00:08:15.760460 W | etcdserver: read-only range request "key:\"/registry/clusterroles/system:controller:pod-garbage-collector\" " with result "range_response_count:1 size:390" took too long (253.952086ms) to execute 2018-12-23 00:08:16.232345 W | etcdserver: read-only range request "key:\"/registry/clusterroles/system:controller:replication-controller\" " with result "range_response_count:1 size:569" took too long (413.693011ms) to execute 2018-12-23 00:08:16.232924 W | etcdserver: request "header:<ID:13568052481467067427 username:\"kube-apiserver-etcd-client\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/events/kube-system/coredns-86c58d9df4-lrkrq.1572cd9afc91a52c\" mod_revision:0 > success:<request_put:<key:\"/registry/events/kube-system/coredns-86c58d9df4-lrkrq.1572cd9afc91a52c\" value_size:369 lease:4344680444612291459 >> failure:<>>" with result "size:16" took too long (296.363358ms) to execute 2018-12-23 00:08:16.385786 W | etcdserver: read-only range request "key:\"/registry/clusterroles/system:controller:route-controller\" " with result "range_response_count:1 size:416" took too long (142.49085ms) to execute 2018-12-23 00:08:16.626856 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-scheduler\" " with result "range_response_count:1 size:433" took too long (382.334166ms) to execute 2018-12-23 00:08:16.628759 W | etcdserver: request "header:<ID:13568052481467067434 username:\"kube-apiserver-etcd-client\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/events/kube-system/coredns-86c58d9df4-g9bfh.1572cd9ca7ed656b\" mod_revision:0 > success:<request_put:<key:\"/registry/events/kube-system/coredns-86c58d9df4-g9bfh.1572cd9ca7ed656b\" value_size:400 lease:4344680444612291459 >> failure:<>>" with result "size:16" took too long (138.078262ms) to execute 2018-12-23 00:08:16.629029 W | etcdserver: read-only range request "key:\"/registry/clusterroles/system:controller:service-account-controller\" " with result "range_response_count:1 size:416" took too long (235.653088ms) to execute 2018-12-23 00:08:16.857768 W | etcdserver: read-only range request "key:\"/registry/clusterroles/system:controller:statefulset-controller\" " with result "range_response_count:1 size:685" took too long (222.130625ms) to execute 2018-12-23 00:08:17.285679 W | etcdserver: read-only range request "key:\"/registry/clusterrolebindings/system:controller:disruption-controller\" " with result "range_response_count:1 size:495" took too long (355.658618ms) to execute 2018-12-23 00:08:17.493784 W | etcdserver: read-only range request "key:\"/registry/clusterrolebindings/system:controller:endpoint-controller\" " with result "range_response_count:1 size:487" took too long (202.191485ms) to execute 2018-12-23 00:08:18.660637 W | etcdserver: request "header:<ID:13568052481467067499 username:\"kube-apiserver-etcd-client\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/events/kube-system/coredns-86c58d9df4-g9bfh.1572cd9d7fa21f86\" mod_revision:0 > success:<request_put:<key:\"/registry/events/kube-system/coredns-86c58d9df4-g9bfh.1572cd9d7fa21f86\" value_size:349 lease:4344680444612291459 >> failure:<>>" with result "size:16" took too long (179.91317ms) to execute 2018-12-23 00:08:18.661188 W | etcdserver: read-only range request "key:\"/registry/pods/kube-system/kube-proxy-58pbt\" " with result "range_response_count:1 size:2304" took too long (141.751794ms) to execute 2018-12-23 00:08:19.344035 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:451" took too long (171.165999ms) to execute 2018-12-23 00:08:22.736240 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:451" took too long (234.684823ms) to execute 2018-12-23 00:09:35.065110 W | etcdserver: read-only range request "key:\"/registry/roles\" range_end:\"/registry/rolet\" count_only:true " with result "range_response_count:0 size:7" took too long (279.961004ms) to execute 2018-12-23 00:17:53.419372 I | mvcc: store.index: compact 5140 2018-12-23 00:17:53.426096 I | mvcc: finished scheduled compaction at 5140 (took 5.806978ms)

When i deploy a new pod (tomcat from docker hub) , the cluster got it up and running, but the new pod also gets an IP from `docker0` bridge, it's `172.17.0.4`. And again `docker ps` output looks really strange:
You see the `NAMES` Column?, the entries are somehow double, e.g. for the tomcat image which i have deployed with `kubectl run tomcat --image=tomcat --port=8080` there are two entries:
`228bc493391c        tomcat                 "catalina.sh run"        6 minutes ago       Up 6 minutes                            k8s_tomcat_tomcat-5d6455cc4-crkcw_default_7b543b37-0694-11e9-a3f6-001966fbc1d4_0` - which is running
`c6e5748a89be        k8s.gcr.io/pause:3.1   "/pause"                 7 minutes ago       Up 7 minutes                            k8s_POD_tomcat-5d6455cc4-crkcw_default_7b543b37-0694-11e9-a3f6-001966fbc1d4_0
` - which is always the image `k8s.gcr.io/pause:3.1`

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 228bc493391c tomcat "catalina.sh run" 6 minutes ago Up 6 minutes k8s_tomcat_tomcat-5d6455cc4-crkcw_default_7b543b37-0694-11e9-a3f6-001966fbc1d4_0 c6e5748a89be k8s.gcr.io/pause:3.1 "/pause" 7 minutes ago Up 7 minutes k8s_POD_tomcat-5d6455cc4-crkcw_default_7b543b37-0694-11e9-a3f6-001966fbc1d4_0 f5ca8af32f08 f0fad859c909 "/opt/bin/flanneld -…" 9 hours ago Up 9 hours k8s_kube-flannel_kube-flannel-ds-amd64-dcpcm_kube-system_1b1816d8-0646-11e9-a90a-001966fbc1d4_1 89d69f474cf1 f59dcacceff4 "/coredns -conf /etc…" 9 hours ago Up 9 hours k8s_coredns_coredns-86c58d9df4-lrkrq_kube-system_b9d3e42a-063d-11e9-b166-001966fbc1d4_3 2d498ba70549 f59dcacceff4 "/coredns -conf /etc…" 9 hours ago Up 9 hours k8s_coredns_coredns-86c58d9df4-g9bfh_kube-system_b9cf29c2-063d-11e9-b166-001966fbc1d4_3 9287347fb490 fdb321fd30a0 "/usr/local/bin/kube…" 9 hours ago Up 9 hours k8s_kube-proxy_kube-proxy-58pbt_kube-system_b9bf1b00-063d-11e9-b166-001966fbc1d4_3 760078cf574e k8s.gcr.io/pause:3.1 "/pause" 9 hours ago Up 9 hours k8s_POD_kube-flannel-ds-amd64-dcpcm_kube-system_1b1816d8-0646-11e9-a90a-001966fbc1d4_1 22993ef3c804 k8s.gcr.io/pause:3.1 "/pause" 9 hours ago Up 9 hours k8s_POD_coredns-86c58d9df4-lrkrq_kube-system_b9d3e42a-063d-11e9-b166-001966fbc1d4_3 c99998a29541 k8s.gcr.io/pause:3.1 "/pause" 9 hours ago Up 9 hours k8s_POD_coredns-86c58d9df4-g9bfh_kube-system_b9cf29c2-063d-11e9-b166-001966fbc1d4_3 71d6810c2b1e k8s.gcr.io/pause:3.1 "/pause" 9 hours ago Up 9 hours k8s_POD_kube-proxy-58pbt_kube-system_b9bf1b00-063d-11e9-b166-001966fbc1d4_3 e7ba49b12335 3cab8e1b9802 "etcd --advertise-cl…" 9 hours ago Up 9 hours k8s_etcd_etcd-archserver_kube-system_459a3e3ab367c3ffe0309150ad8995c1_3 5e7a131a23a4 k8s.gcr.io/pause:3.1 "/pause" 9 hours ago Up 9 hours k8s_POD_etcd-archserver_kube-system_459a3e3ab367c3ffe0309150ad8995c1_3 5babfe2ae495 40a63db91ef8 "kube-apiserver --au…" 9 hours ago Up 9 hours k8s_kube-apiserver_kube-apiserver-archserver_kube-system_d92cb816c7d721bf5d651ac4cc2e875a_2 3736f9129fe8 26e6f1db2a52 "kube-controller-man…" 9 hours ago Up 9 hours k8s_kube-controller-manager_kube-controller-manager-archserver_kube-system_8db7fcf04315eec49ec5bc8348632e73_4 77e7675f54b2 ab81d7360408 "kube-scheduler --ad…" 9 hours ago Up 9 hours k8s_kube-scheduler_kube-scheduler-archserver_kube-system_44b569a35761491825f4e7253fbf0543_4 82831bed7697 k8s.gcr.io/pause:3.1 "/pause" 9 hours ago Up 9 hours k8s_POD_kube-apiserver-archserver_kube-system_d92cb816c7d721bf5d651ac4cc2e875a_2 2ec6df9260f3 k8s.gcr.io/pause:3.1 "/pause" 9 hours ago Up 9 hours k8s_POD_kube-scheduler-archserver_kube-system_44b569a35761491825f4e7253fbf0543_3 8d6507cc70f5 k8s.gcr.io/pause:3.1 "/pause" 9 hours ago Up 9 hours k8s_POD_kube-controller-manager-archserver_kube-system_8db7fcf04315eec49ec5bc8348632e73_3

Is it my cluster setup properly like that and fully usable?

[fabriziopandini]

@tunggad thanks for the detailed report

all pods have the same host IP

Kubeadm deploys control plane pods using hostnetwork=true so this is expected, not a bug.

all the rest

It seems a sort of network problem, but from the logs above I can't grasp the source of it. Is it possible to see the kubeadm init output?

In the meantime, fyi there is bounch of of possible configurations that can create problems:

• local firewall should be disabled
• you have more than one network adapter
• flannel requires running sysctl net.bridge.bridge-nf-call-iptables=1

Usually, a good test is to do curl -k https://localhost:6443/apis immediately after init.

[tunggad]

@fabriziopandini thank you for your time and clarification about pod ip of control plance pods. Let come to your questions:

kubeadm init log: i have executed kubeadm reset and kubeadm init --pod-netwrok-cidr=xxx multiple times, i didn't see anything special in console output, it always looks the same as the kubernetes doc site: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/. With kubeadm init i only have the problem, that i have to start kubelet.service with kubectl start kubelet manually while kubeadm waiting control plane pod to come up, because somehow the 5 tries to start kubelet triggered by kubeadm init has trouble to get started, it complaints about that some certificate files it need does not exist, but they have been all generated and written to disk at right place by kubeadm init.

i don't have any local firewall, i think, i can reach the api server at https://192.168.0.25/apis from shell on the same host by executing curl -k https:/192.168.0.25/apis, i got back some JSON response.

No, i have 1 ethernet and 1 wlan adapter, but ony the ethernet has the IP 192.168.0.25 assigned, the wlan adapter does not have an IP assigned.

I have done the mentioned iptables-tweak needed by flannel before installing it, of cource.

With curl -k https://localhost:6443/apis i also go the same JSON response back as with https://192.168.0.25/apis

Do you need any further info?

[fabriziopandini]

@tunggad now I got it You didn't installed kubelet properly. If you use the official package it will provide a systemd drop in that will make kubelet work properly and use cni addon. Ñow this drop in is missing

[tunggad]

@fabriziopandini hi, thank you for insight. I tried to mimic the systemd unit file kubelet.service and the drop-in file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf (missing on Arch Linux) found in DEB or RPM package of other distros on my ArchLinux-box and kubelet seems to start automatically without restarting it manually while kubeadm init. And cni network addon Calico also works fine. The pods get IP from Calico pod cidr range and they can talk to each other. Flannel still had trouble that`s why i tried Calico.

My kubelet systemd unit /usr/lib/systemd/system/kubelet.service on Arch Linux

[Unit]
Description=Kubernetes Kubelet Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
WorkingDirectory=/var/lib/kubelet
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/kubelet
ExecStart=/usr/bin/kubelet \
        $KUBE_LOGTOSTDERR \
        $KUBE_LOG_LEVEL \
        $KUBELET_API_SERVER \
        $KUBELET_ADDRESS \
        $KUBELET_PORT \
        $KUBELET_HOSTNAME \
        $KUBE_ALLOW_PRIV \
        $KUBELET_ARGS
Restart=always
StartLimitInterval=0
RestartSec=10

[Install]
WantedBy=multi-user.target

And the EnvironmentFile/etc/kubernetes/kubelet

# Arguments to /usr/bin/kubelet started kubelet.service
KUBELET_ARGS="--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1"

systemctl status kubelet:

● kubelet.service - Kubernetes Kubelet Server
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2018-12-23 22:54:48 UTC; 11h ago
     Docs: https://github.com/GoogleCloudPlatform/kubernetes
 Main PID: 325 (kubelet)
    Tasks: 24 (limit: 4915)
   Memory: 119.3M
   CGroup: /system.slice/kubelet.service
           └─325 /usr/bin/kubelet --logtostderr=true --v=0 --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1

But the problem that kubelet keeps complaint about it can't communicate with api server due to some connection refused error still resists. as mentioned above in journal log of kubelet. I really dont know whether that will have negivate effects on the cluster later, in journal log it keeps printing some errors like following:

Dec 24 10:36:51 archserver kubelet[325]: E1224 10:36:51.637955     325 pod_workers.go:190] Error syncing pod 00ce97f9-0764-11e9-bbd7-001966fbc1d4 ("alpine-curl-84bc4c8978-9h27m_default(00ce97f9-0764-11e9-bbd7-001966fbc1d4)"), skipping: failed to "StartContainer" for "alpine-curl" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=alpine-curl pod=alpine-curl-84bc4c8978-9h27m_default(00ce97f9-0764-11e9-bbd7-001966fbc1d4)"
Dec 24 10:37:02 archserver kubelet[325]: E1224 10:37:02.639458     325 pod_workers.go:190] Error syncing pod 00ce97f9-0764-11e9-bbd7-001966fbc1d4 ("alpine-curl-84bc4c8978-9h27m_default(00ce97f9-0764-11e9-bbd7-001966fbc1d4)"), skipping: failed to "StartContainer" for "alpine-curl" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=alpine-curl pod=alpine-curl-84bc4c8978-9h27m_default(00ce97f9-0764-11e9-bbd7-001966fbc1d4)"
Dec 24 10:37:13 archserver kubelet[325]: E1224 10:37:13.637850     325 pod_workers.go:190] Error syncing pod 00ce97f9-0764-11e9-bbd7-001966fbc1d4 ("alpine-curl-84bc4c8978-9h27m_default(00ce97f9-0764-11e9-bbd7-001966fbc1d4)"), skipping: failed to "StartContainer" for "alpine-curl" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=alpine-curl pod=alpine-curl-84bc4c8978-9h27m_default(00ce97f9-0764-11e9-bbd7-001966fbc1d4)"
Dec 24 10:37:35 archserver kubelet[325]: W1224 10:37:35.811641     325 container.go:409] Failed to create summary reader for "/kubepods/besteffort/pod00ce97f9-0764-11e9-bbd7-001966fbc1d4/a94f26bfc48782000b533c26d5ab42cb1cf70412baa892e8037d138a6bfeb908": none of the resources are being tracked.
Dec 24 10:37:36 archserver kubelet[325]: E1224 10:37:36.488835     325 pod_workers.go:190] Error syncing pod 00ce97f9-0764-11e9-bbd7-001966fbc1d4 ("alpine-curl-84bc4c8978-9h27m_default(00ce97f9-0764-11e9-bbd7-001966fbc1d4)"), skipping: failed to "StartContainer" for "alpine-curl" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=alpine-curl pod=alpine-curl-84bc4c8978-9h27m_default(00ce97f9-0764-11e9-bbd7-001966fbc1d4)"
Dec 24 10:37:48 archserver kubelet[325]: E1224 10:37:48.637154     325 pod_workers.go:190] Error syncing pod 00ce97f9-0764-11e9-bbd7-001966fbc1d4 ("alpine-curl-84bc4c8978-9h27m_default(00ce97f9-0764-11e9-bbd7-001966fbc1d4)"), skipping: failed to "StartContainer" for "alpine-curl" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=alpine-curl pod=alpine-curl-84bc4c8978-9h27m_default(00ce97f9-0764-11e9-bbd7-001966fbc1d4)"
Dec 24 10:37:59 archserver kubelet[325]: E1224 10:37:59.636934     325 pod_workers.go:190] Error syncing pod 00ce97f9-0764-11e9-bbd7-001966fbc1d4 ("alpine-curl-84bc4c8978-9h27m_default(00ce97f9-0764-11e9-bbd7-001966fbc1d4)"), skipping: failed to "StartContainer" for "alpine-curl" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=alpine-curl pod=alpine-curl-84bc4c8978-9h27m_default(00ce97f9-0764-11e9-bbd7-001966fbc1d4)"
Dec 24 10:38:12 archserver kubelet[325]: E1224 10:38:12.639366     325 pod_workers.go:190] Error syncing pod 00ce97f9-0764-11e9-bbd7-001966fbc1d4 ("alpine-curl-84bc4c8978-9h27m_default(00ce97f9-0764-11e9-bbd7-001966fbc1d4)"), skipping: failed to "StartContainer" for "alpine-curl" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=alpine-curl pod=alpine-curl-84bc4c8978-9h27m_default(00ce97f9-0764-11e9-bbd7-001966fbc1d4)"
Dec 24 10:38:24 archserver kubelet[325]: E1224 10:38:24.638035     325 pod_workers.go:190] Error syncing pod 00ce97f9-0764-11e9-bbd7-001966fbc1d4 ("alpine-curl-84bc4c8978-9h27m_default(00ce97f9-0764-11e9-bbd7-001966fbc1d4)"), skipping: failed to "StartContainer" for "alpine-curl" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=alpine-curl pod=alpine-curl-84bc4c8978-9h27m_default(00ce97f9-0764-11e9-bbd7-001966fbc1d4)"
Dec 24 10:38:37 archserver kubelet[325]: E1224 10:38:37.637879     325 pod_workers.go:190] Error syncing pod 00ce97f9-0764-11e9-bbd7-001966fbc1d4 ("alpine-curl-84bc4c8978-9h27m_default(00ce97f9-0764-11e9-bbd7-001966fbc1d4)"), skipping: failed to "StartContainer" for "alpine-curl" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=alpine-curl pod=alpine-curl-84bc4c8978-9h27m_default(00ce97f9-0764-11e9-bbd7-001966fbc1d4)"

But for now i can work with the cluster fine, i can deploy pod, delete deployment, create service, let them talk together, everything seems to work. I'm too tired to debug and fix kubernetes setup problems, i want to learn using it.

[fabriziopandini]

@tunggad well done! In my experience, the easiest network plugin for getting started is weave net, because it just works out of the box without any additional settings.

I close the issue because the pending problem doesn't seem related to kubeadm, but probably it is something related to the unconventional installation. If there is something else please let us know /close

[k8s-ci-robot]

@fabriziopandini: Closing this issue.

In response to [this](https://github.com/kubernetes/kubeadm/issues/1329#issuecomment-449722208): >@tunggad well done! >In my experience, the easiest network plugin for getting started is weave net, because it just works out of the box without any additional settings. > >I close the issue because the pending problem doesn't seem related to kubeadm, but probably it is something related to the unconventional installation. >If there is something else please let us know >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.