search

ReSearchITEng / kubeadm-playbook

Fully fledged (HA) Kubernetes Cluster using official kubeadm, ansible and helm. Tested on RHEL/CentOS/Ubuntu with support of http_proxy, dashboard installed, ingress controller, heapster - using official helm charts
https://researchiteng.github.io/kubeadm-playbook/
The Unlicense
592 stars 102 forks source link

Setup failed on Centos 7 #64

Closed vilu closed 5 years ago

vilu commented 5 years ago

I'm not that familiar with many of these tools so I just post some output I hope can be helpful to diagnose. Let me know how I can help further.

TASK [tools : kubectl get nodes] **** FAILED - RETRYING: kubectl get nodes (40 retries left). FAILED - RETRYING: kubectl get nodes (39 retries left). FAILED - RETRYING: kubectl get nodes (38 retries left). FAILED - RETRYING: kubectl get nodes (37 retries left). FAILED - RETRYING: kubectl get nodes (36 retries left). FAILED - RETRYING: kubectl get nodes (35 retries left). FAILED - RETRYING: kubectl get nodes (34 retries left). FAILED - RETRYING: kubectl get nodes (33 retries left). FAILED - RETRYING: kubectl get nodes (32 retries left). FAILED - RETRYING: kubectl get nodes (31 retries left). FAILED - RETRYING: kubectl get nodes (30 retries left). FAILED - RETRYING: kubectl get nodes (29 retries left). FAILED - RETRYING: kubectl get nodes (28 retries left). FAILED - RETRYING: kubectl get nodes (27 retries left). FAILED - RETRYING: kubectl get nodes (26 retries left). FAILED - RETRYING: kubectl get nodes (25 retries left). FAILED - RETRYING: kubectl get nodes (24 retries left). FAILED - RETRYING: kubectl get nodes (23 retries left). FAILED - RETRYING: kubectl get nodes (22 retries left). FAILED - RETRYING: kubectl get nodes (21 retries left). FAILED - RETRYING: kubectl get nodes (20 retries left). FAILED - RETRYING: kubectl get nodes (19 retries left). FAILED - RETRYING: kubectl get nodes (18 retries left). FAILED - RETRYING: kubectl get nodes (17 retries left). FAILED - RETRYING: kubectl get nodes (16 retries left). FAILED - RETRYING: kubectl get nodes (15 retries left). FAILED - RETRYING: kubectl get nodes (14 retries left). FAILED - RETRYING: kubectl get nodes (13 retries left). FAILED - RETRYING: kubectl get nodes (12 retries left). FAILED - RETRYING: kubectl get nodes (11 retries left). FAILED - RETRYING: kubectl get nodes (10 retries left). FAILED - RETRYING: kubectl get nodes (9 retries left). FAILED - RETRYING: kubectl get nodes (8 retries left). FAILED - RETRYING: kubectl get nodes (7 retries left). FAILED - RETRYING: kubectl get nodes (6 retries left). FAILED - RETRYING: kubectl get nodes (5 retries left). FAILED - RETRYING: kubectl get nodes (4 retries left). FAILED - RETRYING: kubectl get nodes (3 retries left). FAILED - RETRYING: kubectl get nodes (2 retries left). FAILED - RETRYING: kubectl get nodes (1 retries left). fatal: [host1.viktorlund.com]: FAILED! => {"attempts": 40, "changed": false, "cmd": "kubectl get nodes --no-headers | grep -v -w 'Ready' || true", "delta": "0:00:00.095557", "end": "2019-04-14 15:30:11.830508", "rc": 0, "start": "2019-04-14 15:30:11.734951", "stderr": "", "stderr_lines": [], "stdout": "k8s-master-1 NotReady master 68m v1.13.4", "stdout_lines": ["k8s-master-1 NotReady master 68m v1.13.4"]} to retry, use: --limit @/home/vilu/code/infra/mono/ansible/kubeadm-playbook/site.retry

[root@k8s-master-1 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master-1 NotReady master 73m v1.13.4 [root@k8s-master-1 ~]# kubectl describe k8s-master-1 error: the server doesn't have a resource type "k8s-master-1" [root@k8s-master-1 ~]# kubectl describe nodes k8s-master-1 Name: k8s-master-1 Roles: master Labels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=linux kubernetes.io/hostname=k8s-master-1 node-role.kubernetes.io/master= Annotations: kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock node.alpha.kubernetes.io/ttl: 0 volumes.kubernetes.io/controller-managed-attach-detach: true CreationTimestamp: Sun, 14 Apr 2019 14:21:18 +0200 Taints: node-role.kubernetes.io/master:NoSchedule node.kubernetes.io/not-ready:NoSchedule Unschedulable: false Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message

MemoryPressure False Sun, 14 Apr 2019 15:35:01 +0200 Sun, 14 Apr 2019 14:21:18 +0200 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Sun, 14 Apr 2019 15:35:01 +0200 Sun, 14 Apr 2019 14:21:18 +0200 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Sun, 14 Apr 2019 15:35:01 +0200 Sun, 14 Apr 2019 14:21:18 +0200 KubeletHasSufficientPID kubelet has sufficient PID available Ready False Sun, 14 Apr 2019 15:35:01 +0200 Sun, 14 Apr 2019 14:21:18 +0200 KubeletNotReady runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized Addresses: InternalIP: 94.130.182.127 Hostname: k8s-master-1 Capacity: cpu: 2 ephemeral-storage: 78735916Ki hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 7814612Ki pods: 110 Allocatable: cpu: 2 ephemeral-storage: 72563020066 hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 7712212Ki pods: 110 System Info: Machine ID: 6337f8e638f44a0ab8e7d3fe010d88df System UUID: 30A07EC0-07C8-495D-B0C2-9AB5251D5020 Boot ID: a0bf2936-ff22-40da-9025-46cc064b8574 Kernel Version: 3.10.0-957.5.1.el7.x86_64 OS Image: CentOS Linux 7 (Core) Operating System: linux Architecture: amd64 Container Runtime Version: docker://1.13.1 Kubelet Version: v1.13.4 Kube-Proxy Version: v1.13.4 PodCIDR: 10.244.0.0/24 Non-terminated Pods: (5 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE

kube-system etcd-k8s-master-1 0 (0%) 0 (0%) 0 (0%) 0 (0%) 73m kube-system kube-apiserver-k8s-master-1 250m (12%) 0 (0%) 0 (0%) 0 (0%) 73m kube-system kube-controller-manager-k8s-master-1 200m (10%) 0 (0%) 0 (0%) 0 (0%) 73m kube-system kube-proxy-9nldk 0 (0%) 0 (0%) 0 (0%) 0 (0%) 73m kube-system kube-scheduler-k8s-master-1 100m (5%) 0 (0%) 0 (0%) 0 (0%) 73m Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits

cpu 550m (27%) 0 (0%) memory 0 (0%) 0 (0%) ephemeral-storage 0 (0%) 0 (0%) Events:

ReSearchITEng commented 5 years ago

Please share your setup (inventory, proxy, etc). You may want to see which pods are working, and which are not: kubectl get po --all-namespaces -o wide

If the flanneld network is not fine, you can try to redeploy it either with: ansible-playbook -vv -i hosts site.yml --tags post_deploy or simply run on the master: kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

vilu commented 5 years ago

Thanks for the quick suggestions. I'll try to give this another go next week.

ReSearchITEng commented 5 years ago

Hello, if there are no more updates, we plan to close it. Please reopen if required.