search

coreos / coreos-kubernetes

CoreOS Container Linux+Kubernetes documentation & Vagrant installers
https://coreos.com/kubernetes/docs/latest/
Apache License 2.0
1.1k stars 466 forks source link

controller-install sctipt stuck at Waiting for Kubernetes API... #896

Open amit2277 opened 7 years ago

amit2277 commented 7 years ago

Hello I am new to kubernetes and want to install test environment with two controller and three worker nodes on 5 physical HP Prolient servers.

I have installed coreos on baremetal server and run controller-install.sh script but it stuck at waiting for kubernetes API (logs at below) -- it seems it some issue with my /etc/resolve.conf but i don't know where and what to change i have tried to change nameserver to 127.0.0.1 but wont succeed. currently it is 8.8.8.8 , Can anyone please guide me.

My current setup :

cat /etc/resolv.conf.

nameserver 8.8.8.8

cat /etc/hosts

127.0.0.1 localhost ::1 localhost 192.168.51.81 controller

systemctl status etcd2

● etcd2.service - etcd2 Loaded: loaded (/usr/lib/systemd/system/etcd2.service; disabled; vendor preset: disabled) Drop-In: /run/systemd/system/etcd2.service.d └─20-cloudinit.conf Active: active (running) since Fri 2017-07-14 08:48:38 UTC; 1h 27min ago Main PID: 990 (etcd2) Tasks: 23 (limit: 32768) Memory: 64.2M CPU: 19.540s CGroup: /system.slice/etcd2.service └─990 /usr/bin/etcd2

Jul 14 08:48:38 controller etcd2[990]: ce2a822cea30bfca is starting a new election at term 10 Jul 14 08:48:38 controller etcd2[990]: ce2a822cea30bfca became candidate at term 11 Jul 14 08:48:38 controller etcd2[990]: ce2a822cea30bfca received vote from ce2a822cea30bfca at term 11 Jul 14 08:48:38 controller etcd2[990]: ce2a822cea30bfca became leader at term 11 Jul 14 08:48:38 controller etcd2[990]: raft.node: ce2a822cea30bfca elected leader ce2a822cea30bfca at term 11 Jul 14 08:48:38 controller etcd2[990]: published {Name:controller ClientURLs:[http://0.0.0.0:4001 http://192.168.51.81:2379]} to cluster 7e27652122e8b2ae Jul 14 09:29:01 controller etcd2[990]: start to snapshot (applied: 230023, lastsnap: 220022) Jul 14 09:29:01 controller etcd2[990]: saved snapshot at index 230023 Jul 14 09:29:01 controller etcd2[990]: compacted raft log at 225023 Jul 14 09:29:08 controller etcd2[990]: purged file /var/lib/etcd2/member/snap/0000000000000009-000000000002bf32.snap successfully

systemctl status flanneld

● flanneld.service - flannel - Network fabric for containers (System Application Container) Loaded: loaded (/usr/lib/systemd/system/flanneld.service; enabled; vendor preset: disabled) Drop-In: /etc/systemd/system/flanneld.service.d └─40-ExecStartPre-symlink.conf.conf, 50-network-config.conf Active: active (running) since Fri 2017-07-14 08:48:41 UTC; 1h 28min ago Docs: https://github.com/coreos/flannel Main PID: 1068 (flanneld) Tasks: 22 (limit: 32768) Memory: 126.6M CPU: 1.489s CGroup: /system.slice/flanneld.service └─1068 /opt/bin/flanneld --ip-masq=true

Jul 14 08:48:41 controller flannel-wrapper[1068]: I0714 08:48:41.610498 1068 manager.go:124] Searching for interface using 192.168.51.81 Jul 14 08:48:41 controller flannel-wrapper[1068]: I0714 08:48:41.610840 1068 manager.go:149] Using interface with name enp2s0f0 and address 192.168.51.81 Jul 14 08:48:41 controller flannel-wrapper[1068]: I0714 08:48:41.610854 1068 manager.go:166] Defaulting external address to interface address (192.168.51.81) Jul 14 08:48:41 controller flannel-wrapper[1068]: I0714 08:48:41.661383 1068 local_manager.go:134] Found lease (10.10.81.0/24) for current IP (192.168.51.81), reusing Jul 14 08:48:41 controller flannel-wrapper[1068]: I0714 08:48:41.720174 1068 ipmasq.go:47] Adding iptables rule: -s 10.10.0.0/16 -d 10.10.0.0/16 -j RETURN Jul 14 08:48:41 controller flannel-wrapper[1068]: I0714 08:48:41.864413 1068 ipmasq.go:47] Adding iptables rule: -s 10.10.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE Jul 14 08:48:41 controller flannel-wrapper[1068]: I0714 08:48:41.902481 1068 ipmasq.go:47] Adding iptables rule: ! -s 10.10.0.0/16 -d 10.10.0.0/16 -j MASQUERADE Jul 14 08:48:41 controller flannel-wrapper[1068]: I0714 08:48:41.905284 1068 manager.go:250] Lease acquired: 10.10.81.0/24 Jul 14 08:48:41 controller flannel-wrapper[1068]: I0714 08:48:41.905587 1068 network.go:98] Watching for new subnet leases Jul 14 08:48:41 controller systemd[1]: Started flannel - Network fabric for containers (System Application Container).

systemctl status fleet
● fleet.service - fleet daemon Loaded: loaded (/usr/lib/systemd/system/fleet.service; disabled; vendor preset: disabled) Drop-In: /run/systemd/system/fleet.service.d └─20-cloudinit.conf Active: active (running) since Fri 2017-07-14 08:48:38 UTC; 1h 29min ago Main PID: 1036 (fleetd) Tasks: 7 (limit: 32768) Memory: 36.5M CPU: 1min 32.825s CGroup: /system.slice/fleet.service └─1036 /usr/bin/fleetd

Jul 14 08:48:38 controller systemd[1]: Started fleet daemon. Jul 14 08:48:39 controller fleetd[1036]: INFO fleetd.go:64: Starting fleetd version 0.11.8 Jul 14 08:48:39 controller fleetd[1036]: INFO fleetd.go:170: No provided or default config file found - proceeding without Jul 14 08:48:39 controller fleetd[1036]: INFO server.go:160: Establishing etcd connectivity Jul 14 08:48:39 controller fleetd[1036]: INFO server.go:171: Starting server components Jul 14 08:48:39 controller fleetd[1036]: INFO engine.go:185: Engine leadership acquired

fleetctl list-machines MACHINE IP METADATA a682ae32... 192.168.51.81 role=master

etcdctl cluster-health member ce2a822cea30bfca is healthy: got healthy result from http://0.0.0.0:4001 cluster is healthy

netstat -ntpul Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 192.168.51.81:7001 0.0.0.0: LISTEN 990/etcd2
tcp 0 0 127.0.0.1:10248 0.0.0.0: LISTEN 1248/kubelet
tcp 0 0 127.0.0.1:10249 0.0.0.0: LISTEN 1687/hyperkube
tcp 0 0 192.168.51.81:2380 0.0.0.0: LISTEN 990/etcd2
tcp6 0 0 :::22 ::: LISTEN 1/systemd
tcp6 0 0 :::4001 ::: LISTEN 990/etcd2
tcp6 0 0 :::4194 ::: LISTEN 1248/kubelet
tcp6 0 0 :::2375 ::: LISTEN 1/systemd
tcp6 0 0 :::10250 ::: LISTEN 1248/kubelet
tcp6 0 0 :::10251 ::: LISTEN 1610/hyperkube
tcp6 0 0 :::2379 ::: LISTEN 990/etcd2
tcp6 0 0 :::10252 ::: LISTEN 1720/hyperkube
tcp6 0 0 :::10255 ::: LISTEN 1248/kubelet
udp 0 0 192.168.51.81:8285 0.0.0.0: 1068/flanneld
udp6 0 0 fe80::6ab5:99ff:fec:546 :::* 9020/systemd-network

Can anyone help me where i am doing mistake.

Error: systemctl status kubelet -l ● kubelet.service Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled) Active: active (running) since Fri 2017-07-14 08:48:48 UTC; 1h 14min ago Main PID: 1248 (kubelet) Tasks: 28 (limit: 32768) Memory: 768.5M CPU: 2min 14.584s CGroup: /system.slice/kubelet.service ├─1248 /kubelet --api-servers=http://127.0.0.1:8080 --register-schedulable=false --cni-conf-dir=/etc/kubernetes/cni/net.d --network-plugin=cni --container-runtime=docker --rkt-path=/usr/bin/rkt --rkt-stage1-image=coreos.com/rk └─1433 journalctl -k -f

Jul 14 10:02:54 controller kubelet-wrapper[1248]: E0714 10:02:54.625731 1248 reflector.go:188] pkg/kubelet/kubelet.go:386: Failed to list api.Node: Get http://127.0.0.1:8080/api/v1/nodes?fieldSelector=metadata.name%3D192.168.51.81&re Jul 14 10:02:55 controller kubelet-wrapper[1248]: E0714 10:02:55.377911 1248 reflector.go:188] pkg/kubelet/kubelet.go:378: Failed to list api.Service: Get http://127.0.0.1:8080/api/v1/services?resourceVersion=0: dial tcp 127.0.0.1:80 Jul 14 10:02:55 controller kubelet-wrapper[1248]: E0714 10:02:55.378592 1248 reflector.go:188] pkg/kubelet/config/apiserver.go:44: Failed to list api.Pod: Get http://127.0.0.1:8080/api/v1/pods?fieldSelector=spec.nodeName%3D192.168.51 Jul 14 10:02:55 controller kubelet-wrapper[1248]: E0714 10:02:55.626339 1248 reflector.go:188] pkg/kubelet/kubelet.go:386: Failed to list api.Node: Get http://127.0.0.1:8080/api/v1/nodes?fieldSelector=metadata.name%3D192.168.51.81&re Jul 14 10:02:56 controller kubelet-wrapper[1248]: E0714 10:02:56.378822 1248 reflector.go:188] pkg/kubelet/kubelet.go:378: Failed to list api.Service: Get http://127.0.0.1:8080/api/v1/services?resourceVersion=0: dial tcp 127.0.0.1:80 Jul 14 10:02:56 controller kubelet-wrapper[1248]: E0714 10:02:56.379795 1248 reflector.go:188] pkg/kubelet/config/apiserver.go:44: Failed to list api.Pod: Get http://127.0.0.1:8080/api/v1/pods?fieldSelector=spec.nodeName%3D192.168.51 Jul 14 10:02:56 controller kubelet-wrapper[1248]: E0714 10:02:56.627007 1248 reflector.go:188] pkg/kubelet/kubelet.go:386: Failed to list api.Node: Get http://127.0.0.1:8080/api/v1/nodes?fieldSelector=metadata.name%3D192.168.51.81&re Jul 14 10:02:57 controller kubelet-wrapper[1248]: E0714 10:02:57.379581 1248 reflector.go:188] pkg/kubelet/kubelet.go:378: Failed to list api.Service: Get http://127.0.0.1:8080/api/v1/services?resourceVersion=0: dial tcp 127.0.0.1:80 Jul 14 10:02:57 controller kubelet-wrapper[1248]: E0714 10:02:57.380463 1248 reflector.go:188] pkg/kubelet/config/apiserver.go:44: Failed to list api.Pod: Get http://127.0.0.1:8080/api/v1/pods?fieldSelector=spec.nodeName%3D192.168.51 Jul 14 10:02:57 controller kubelet-wrapper[1248]: E0714 10:02:57.627616 1248 reflector.go:188] pkg/kubelet/kubelet.go:386: Failed to list api.Node: Get http://127.0.0.1:8080/api/v1/nodes?fieldSelector=metadata.name%3D192.168.51.81&re Jul 14 10:02:58 controller kubelet-wrapper[1248]: E0714 10:02:58.380454 1248 reflector.go:188] pkg/kubelet/kubelet.go:378: Failed to list api.Service: Get http://127.0.0.1:8080/api/v1/services?resourceVersion=0: dial tcp 127.0.0.1:80 Jul 14 10:02:58 controller kubelet-wrapper[1248]: E0714 10:02:58.381452 1248 reflector.go:188] pkg/kubelet/config/apiserver.go:44: Failed to list api.Pod: Get http://127.0.0.1:8080/api/v1/pods?fieldSelector=spec.nodeName%3D192.168.51 Jul 14 10:02:58 controller kubelet-wrapper[1248]: E0714 10:02:58.628355 1248 reflector.go:188] pkg/kubelet/kubelet.go:386: Failed to list *api.Node: Get http://127.0.0.1:8080/api/v1/nodes?fieldSelector=metadata.name%3D192.168.51.81&re lines 1-24/24 (END)

Logs::

Jul 14 10:30:10 controller kubelet-wrapper[1248]: E0714 10:30:10.516751 1248 reflector.go:188] pkg/kubelet/kubelet.go:378: Failed to list api.Service: Get http://127.0.0.1:8080/api/v1/services?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused Jul 14 10:30:10 controller kubelet-wrapper[1248]: E0714 10:30:10.517665 1248 reflector.go:188] pkg/kubelet/config/apiserver.go:44: Failed to list api.Pod: Get http://127.0.0.1:8080/api/v1/pods?fieldSelector=spec.nodeName%3D192.168.51.81&resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused Jul 14 10:30:10 controller kubelet-wrapper[1248]: E0714 10:30:10.663798 1248 reflector.go:188] pkg/kubelet/kubelet.go:386: Failed to list api.Node: Get http://127.0.0.1:8080/api/v1/nodes?fieldSelector=metadata.name%3D192.168.51.81&resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused Jul 14 10:30:10 controller kubelet-wrapper[1248]: I0714 10:30:10.882697 1248 kubelet_node_status.go:204] Setting node annotation to enable volume controller attach/detach Jul 14 10:30:10 controller kubelet-wrapper[1248]: I0714 10:30:10.883764 1248 kubelet_node_status.go:74] Attempting to register node 192.168.51.81 Jul 14 10:30:10 controller kubelet-wrapper[1248]: E0714 10:30:10.884014 1248 kubelet_node_status.go:98] Unable to register node "192.168.51.81" with API server: Post http://127.0.0.1:8080/api/v1/nodes: dial tcp 127.0.0.1:8080: getsockopt: connection refused Jul 14 10:30:11 controller kubelet-wrapper[1248]: E0714 10:30:11.517400 1248 reflector.go:188] pkg/kubelet/kubelet.go:378: Failed to list api.Service: Get http://127.0.0.1:8080/api/v1/services?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused Jul 14 10:30:11 controller kubelet-wrapper[1248]: E0714 10:30:11.518318 1248 reflector.go:188] pkg/kubelet/config/apiserver.go:44: Failed to list api.Pod: Get http://127.0.0.1:8080/api/v1/pods?fieldSelector=spec.nodeName%3D192.168.51.81&resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused Jul 14 10:30:11 controller kubelet-wrapper[1248]: E0714 10:30:11.664582 1248 reflector.go:188] pkg/kubelet/kubelet.go:386: Failed to list api.Node: Get http://127.0.0.1:8080/api/v1/nodes?fieldSelector=metadata.name%3D192.168.51.81&resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused Jul 14 10:30:11 controller kubelet-wrapper[1248]: E0714 10:30:11.857368 1248 cni.go:163] error updating cni config: No networks found in /etc/kubernetes/cni/net.d Jul 14 10:30:12 controller kubelet-wrapper[1248]: E0714 10:30:12.518148 1248 reflector.go:188] pkg/kubelet/kubelet.go:378: Failed to list api.Service: Get http://127.0.0.1:8080/api/v1/services?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused Jul 14 10:30:12 controller kubelet-wrapper[1248]: E0714 10:30:12.518996 1248 reflector.go:188] pkg/kubelet/config/apiserver.go:44: Failed to list api.Pod: Get http://127.0.0.1:8080/api/v1/pods?fieldSelector=spec.nodeName%3D192.168.51.81&resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused

thanks