search

kubesphere / ks-installer

Install KubeSphere on existing Kubernetes cluster
https://kubesphere.io
Apache License 2.0
530 stars 744 forks source link

安装卡在了Initialize first master #540

Open zhangpfly opened 4 years ago

zhangpfly commented 4 years ago

系统环境：centos7 4核8G*3

执行安装脚本报错如下：

2020-01-06 11:26:35,182 p=79605 u=root |  TASK [kubernetes/master : kubeadm | Initialize first master] ********************************************************************************************************************************
2020-01-06 11:26:35,182 p=79605 u=root |  Monday 06 January 2020  11:26:35 +0800 (0:00:00.142)       0:04:13.304 ******** 
2020-01-06 11:31:35,412 p=79605 u=root |  FAILED - RETRYING: kubeadm | Initialize first master (3 retries left).
2020-01-06 11:36:40,626 p=79605 u=root |  FAILED - RETRYING: kubeadm | Initialize first master (2 retries left).
2020-01-06 11:41:45,818 p=79605 u=root |  FAILED - RETRYING: kubeadm | Initialize first master (1 retries left).
2020-01-06 11:46:51,024 p=79605 u=root |  fatal: test01]: FAILED! => {"attempts": 3, "changed": true, "cmd": ["timeout", "-k", "300s", "300s", "/usr/local/bin/kubeadm", "init", "--config=/etc/kubernetes/kubeadm-config.yaml", "--ignore-preflight-errors=all", "--skip-phases=addon/coredns", "--upload-certs"], "delta": "0:05:00.043001", "end": "2020-01-06 11:46:50.982595", "failed_when_result": true, "msg": "non-zero return code", "rc": 124, "start": "2020-01-06 11:41:50.939594", "stderr": "\t[WARNING Port-6443]: Port 6443 is in use\n\t[WARNING Port-10251]: Port 10251 is in use\n\t[WARNING Port-10252]: Port 10252 is in use\n\t[WARNING FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists\n\t[WARNING FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml]: /etc/kubernetes/manifests/kube-controller-manager.yaml already exists\n\t[WARNING FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml]: /etc/kubernetes/manifests/kube-scheduler.yaml already exists\n\t[WARNING Port-10250]: Port 10250 is in use", "stderr_lines": ["\t[WARNING Port-6443]: Port 6443 is in use", "\t[WARNING Port-10251]: Port 10251 is in use", "\t[WARNING Port-10252]: Port 10252 is in use", "\t[WARNING FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists", "\t[WARNING FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml]: /etc/kubernetes/manifests/kube-controller-manager.yaml already exists", "\t[WARNING FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml]: /etc/kubernetes/manifests/kube-scheduler.yaml already exists", "\t[WARNING Port-10250]: Port 10250 is in use"], "stdout": "[init] Using Kubernetes version: v1.15.5\n[preflight] Running pre-flight checks\n[preflight] Pulling images required for setting up a Kubernetes cluster\n[preflight] This might take a minute or two, depending on the speed of your internet connection\n[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'\n[kubelet-start] Writing kubelet environment file with flags to file \"/var/lib/kubelet/kubeadm-flags.env\"\n[kubelet-start] Writing kubelet configuration to file \"/var/lib/kubelet/config.yaml\"\n[kubelet-start] Activating the kubelet service\n[certs] Using certificateDir folder \"/etc/kubernetes/ssl\"\n[certs] External etcd mode: Skipping etcd/ca certificate authority generation\n[certs] External etcd mode: Skipping etcd/peer certificate authority generation\n[certs] External etcd mode: Skipping apiserver-etcd-client certificate authority generation\n[certs] External etcd mode: Skipping etcd/healthcheck-client certificate authority generation\n[certs] External etcd mode: Skipping etcd/server certificate authority generation\n[certs] Using existing ca certificate authority\n[certs] Using existing apiserver-kubelet-client certificate and key on disk\n[certs] Using existing apiserver certificate and key on disk\n[certs] Using existing front-proxy-ca certificate authority\n[certs] Using existing front-proxy-client certificate and key on disk\n[certs] Using the existing \"sa\" key\n[kubeconfig] Using kubeconfig folder \"/etc/kubernetes\"\n[kubeconfig] Using existing kubeconfig file: \"/etc/kubernetes/admin.conf\"\n[kubeconfig] Using existing kubeconfig file: \"/etc/kubernetes/kubelet.conf\"\n[kubeconfig] Using existing kubeconfig file: \"/etc/kubernetes/controller-manager.conf\"\n[kubeconfig] Using existing kubeconfig file: \"/etc/kubernetes/scheduler.conf\"\n[control-plane] Using manifest folder \"/etc/kubernetes/manifests\"\n[control-plane] Creating static Pod manifest for \"kube-apiserver\"\n[controlplane] Adding extra host path mount \"etc-pki-tls\" to \"kube-apiserver\"\n[controlplane] Adding extra host path mount \"etc-pki-ca-trust\" to \"kube-apiserver\"\n[control-plane] Creating static Pod manifest for \"kube-controller-manager\"\n[controlplane] Adding extra host path mount \"etc-pki-tls\" to \"kube-apiserver\"\n[controlplane] Adding extra host path mount \"etc-pki-ca-trust\" to \"kube-apiserver\"\n[control-plane] Creating static Pod manifest for \"kube-scheduler\"\n[controlplane] Adding extra host path mount \"etc-pki-tls\" to \"kube-apiserver\"\n[controlplane] Adding extra host path mount \"etc-pki-ca-trust\" to \"kube-apiserver\"\n[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory \"/etc/kubernetes/manifests\". This can take up to 5m0s\n[kubelet-check] Initial timeout of 40s passed.", "stdout_lines": ["[init] Using Kubernetes version: v1.15.5", "[preflight] Running pre-flight checks", "[preflight] Pulling images required for setting up a Kubernetes cluster", "[preflight] This might take a minute or two, depending on the speed of your internet connection", "[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'", "[kubelet-start] Writing kubelet environment file with flags to file \"/var/lib/kubelet/kubeadm-flags.env\"", "[kubelet-start] Writing kubelet configuration to file \"/var/lib/kubelet/config.yaml\"", "[kubelet-start] Activating the kubelet service", "[certs] Using certificateDir folder \"/etc/kubernetes/ssl\"", "[certs] External etcd mode: Skipping etcd/ca certificate authority generation", "[certs] External etcd mode: Skipping etcd/peer certificate authority generation", "[certs] External etcd mode: Skipping apiserver-etcd-client certificate authority generation", "[certs] External etcd mode: Skipping etcd/healthcheck-client certificate authority generation", "[certs] External etcd mode: Skipping etcd/server certificate authority generation", "[certs] Using existing ca certificate authority", "[certs] Using existing apiserver-kubelet-client certificate and key on disk", "[certs] Using existing apiserver certificate and key on disk", "[certs] Using existing front-proxy-ca certificate authority", "[certs] Using existing front-proxy-client certificate and key on disk", "[certs] Using the existing \"sa\" key", "[kubeconfig] Using kubeconfig folder \"/etc/kubernetes\"", "[kubeconfig] Using existing kubeconfig file: \"/etc/kubernetes/admin.conf\"", "[kubeconfig] Using existing kubeconfig file: \"/etc/kubernetes/kubelet.conf\"", "[kubeconfig] Using existing kubeconfig file: \"/etc/kubernetes/controller-manager.conf\"", "[kubeconfig] Using existing kubeconfig file: \"/etc/kubernetes/scheduler.conf\"", "[control-plane] Using manifest folder \"/etc/kubernetes/manifests\"", "[control-plane] Creating static Pod manifest for \"kube-apiserver\"", "[controlplane] Adding extra host path mount \"etc-pki-tls\" to \"kube-apiserver\"", "[controlplane] Adding extra host path mount \"etc-pki-ca-trust\" to \"kube-apiserver\"", "[control-plane] Creating static Pod manifest for \"kube-controller-manager\"", "[controlplane] Adding extra host path mount \"etc-pki-tls\" to \"kube-apiserver\"", "[controlplane] Adding extra host path mount \"etc-pki-ca-trust\" to \"kube-apiserver\"", "[control-plane] Creating static Pod manifest for \"kube-scheduler\"", "[controlplane] Adding extra host path mount \"etc-pki-tls\" to \"kube-apiserver\"", "[controlplane] Adding extra host path mount \"etc-pki-ca-trust\" to \"kube-apiserver\"", "[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory \"/etc/kubernetes/manifests\". This can take up to 5m0s", "[kubelet-check] Initial timeout of 40s passed."]}
2020-01-06 11:46:51,025 p=79605 u=root |  NO MORE HOSTS LEFT **************************************************************************************************************************************************************************

master启动的容器

# docker ps
CONTAINER ID        IMAGE                                    COMMAND                  CREATED             STATUS              PORTS               NAMES
736fc50b8dc5        bb10f0e78768                             "kube-controller-man…"   24 minutes ago      Up 24 minutes                           k8s_kube-controller-manager_kube-controller-manager-bjyz-va-hwebapp1_kube-system_5b1a70449d46c2016fe8b99df48022c2_0
f3122f94745e        bb10f0e78768                             "kube-scheduler --bi…"   24 minutes ago      Up 24 minutes                           k8s_kube-scheduler_kube-scheduler-bjyz-va-hwebapp1_kube-system_551c16d92b3d5f8d85004544b437c8ff_0
823ebb58e9be        bb10f0e78768                             "kube-apiserver --ad…"   24 minutes ago      Up 24 minutes                           k8s_kube-apiserver_kube-apiserver-bjyz-va-hwebapp1_kube-system_3bd9f5f57a05c15a86c88134fb4fc349_0
b009a6eaf303        mirrorgooglecontainers/pause-amd64:3.1   "/pause"                 24 minutes ago      Up 24 minutes                           k8s_POD_kube-scheduler-bjyz-va-hwebapp1_kube-system_551c16d92b3d5f8d85004544b437c8ff_0
87ff7c834d9c        mirrorgooglecontainers/pause-amd64:3.1   "/pause"                 24 minutes ago      Up 24 minutes                           k8s_POD_kube-controller-manager-bjyz-va-hwebapp1_kube-system_5b1a70449d46c2016fe8b99df48022c2_0
f53d420e809b        mirrorgooglecontainers/pause-amd64:3.1   "/pause"                 24 minutes ago      Up 24 minutes                           k8s_POD_kube-apiserver-bjyz-va-hwebapp1_kube-system_3bd9f5f57a05c15a86c88134fb4fc349_0
f941ce3a77a2        quay.azk8s.cn/coreos/etcd:v3.2.18        "/usr/local/bin/etcd"    26 minutes ago      Up 26 minutes                           etcd1

docker信息

Client:
 Debug Mode: false

Server:
 Containers: 7
  Running: 7
  Paused: 0
  Stopped: 0
 Images: 12
 Server Version: 18.09.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: systemd
 Plugins:
  Volume: local
  Network: bridge host macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 3.10.0-957.21.2.el7.x86_64
 Operating System: CentOS Linux 7 (Core)
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.62GiB
 Name: bjyz-va-hwebapp1
 ID: UEHC:A4G7:NYNF:F2GT:MNIR:2LJR:24R6:WDML:A2VK:CV5F:4HRM:2DE2
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Registry Mirrors:
  https://3x8fih0m.mirror.aliyuncs.com/
  https://docker.mirrors.ustc.edu.cn/
 Live Restore Enabled: false
 Product License: Community Engine

master节点，iptables

# iptables -vnL
Chain INPUT (policy ACCEPT 94092 packets, 8577K bytes)
 pkts bytes target     prot opt in     out     source               destination         
 236K   21M KUBE-FIREWALL  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER-USER  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 DOCKER-ISOLATION-STAGE-1  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 DOCKER     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 113K packets, 21M bytes)
 pkts bytes target     prot opt in     out     source               destination         
 276K   57M KUBE-FIREWALL  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER-ISOLATION-STAGE-2  all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all  --  *      docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain KUBE-FIREWALL (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000

麻烦看一下问题原因。非常感谢。

zhangpfly commented 4 years ago

端口信息

# netstat -tnulp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:62245           0.0.0.0:*               LISTEN      53930/rpc.statd     
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      94557/kubelet       
tcp        0      0 10.1.3.62:10250         0.0.0.0:*               LISTEN      94557/kubelet       
tcp        0      0 127.0.0.1:2379          0.0.0.0:*               LISTEN      89223/etcd          
tcp        0      0 10.1.3.62:2379          0.0.0.0:*               LISTEN      89223/etcd          
tcp        0      0 10.1.3.62:2380          0.0.0.0:*               LISTEN      89223/etcd          
tcp        0      0 10.1.3.62:10255         0.0.0.0:*               LISTEN      94557/kubelet       
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      1/systemd           
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      9254/sshd           
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      9678/master         
tcp        0      0 0.0.0.0:1377            0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:10050           0.0.0.0:*               LISTEN      9267/zabbix_agentd  
tcp        0      0 127.0.0.1:5060          0.0.0.0:*               LISTEN      94557/kubelet       
tcp6       0      0 :::20489                :::*                    LISTEN      -                   
tcp6       0      0 :::49386                :::*                    LISTEN      53930/rpc.statd     
tcp6       0      0 :::10251                :::*                    LISTEN      92644/kube-schedule 
tcp6       0      0 :::6443                 :::*                    LISTEN      92621/kube-apiserve 
tcp6       0      0 :::10252                :::*                    LISTEN      92651/kube-controll 
tcp6       0      0 :::111                  :::*                    LISTEN      1/systemd           
tcp6       0      0 :::10257                :::*                    LISTEN      92651/kube-controll 
tcp6       0      0 :::10259                :::*                    LISTEN      92644/kube-schedule 
tcp6       0      0 :::22                   :::*                    LISTEN      9254/sshd           
tcp6       0      0 ::1:25                  :::*                    LISTEN      9678/master         
tcp6       0      0 :::10050                :::*                    LISTEN      9267/zabbix_agentd  
udp        0      0 0.0.0.0:32802           0.0.0.0:*                           53930/rpc.statd     
udp        0      0 0.0.0.0:111             0.0.0.0:*                           1/systemd           
udp        0      0 0.0.0.0:123             0.0.0.0:*                           80338/chronyd       
udp        0      0 127.0.0.1:323           0.0.0.0:*                           80338/chronyd       
udp        0      0 127.0.0.1:703           0.0.0.0:*                           53930/rpc.statd     
udp        0      0 0.0.0.0:917             0.0.0.0:*                           8805/rpcbind        
udp        0      0 0.0.0.0:62816           0.0.0.0:*                           -                   
udp6       0      0 :::44316                :::*                                -                   
udp6       0      0 :::24570                :::*                                53930/rpc.statd     
udp6       0      0 :::111                  :::*                                1/systemd           
udp6       0      0 ::1:323                 :::*                                80338/chronyd       
udp6       0      0 :::917                  :::*                                8805/rpcbind
pixiake commented 4 years ago

检查下机器的防火墙