search

sortteam / HyKuFe

MIT License
4 stars 0 forks source link

Soma 서버 쿠버네티스 클러스터링 안됨. #13

Closed wonjong-yoo closed 5 years ago

wonjong-yoo commented 5 years ago

버그 설명 Kubespray로 클러스터링 도중 Cluster Join 앤서블 명령어가 끝날 생각을 안함. (거의 마지막 단계)

Ansible Log

TASK [kubernetes/kubeadm : Join to cluster if needed] *************************************************************************************
task path: /home/sort-server-1/onprem-kubespray/kubespray-2.8.5/roles/kubernetes/kubeadm/tasks/main.yml:60
skipping: [sort-server-1] => {
    "changed": false,
    "skip_reason": "Conditional result was False"
}
<172.16.100.101> ESTABLISH SSH CONNECTION FOR USER: sort-server-2
<172.16.100.101> SSH: EXEC sshpass -d13 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=7777 -o User=sort-server-2 -o ConnectTimeout=10 -o ControlPath=/home/sort-server-1/.ansible/cp/71329be8d9 172.16.100.101 '/bin/sh -c '"'"'echo ~sort-server-2 && sleep 0'"'"''
<172.16.100.101> (0, '/home/sort-server-2\n', '')
<172.16.100.101> ESTABLISH SSH CONNECTION FOR USER: sort-server-2
<172.16.100.101> SSH: EXEC sshpass -d13 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=7777 -o User=sort-server-2 -o ConnectTimeout=10 -o ControlPath=/home/sort-server-1/.ansible/cp/71329be8d9 172.16.100.101 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/sort-server-2/.ansible/tmp/ansible-tmp-1570201830.24-147225566909601 `" && echo ansible-tmp-1570201830.24-147225566909601="` echo /home/sort-server-2/.ansible/tmp/ansible-tmp-1570201830.24-147225566909601 `" ) && sleep 0'"'"''
<172.16.100.102> ESTABLISH SSH CONNECTION FOR USER: sort-server-3
<172.16.100.101> (0, 'ansible-tmp-1570201830.24-147225566909601=/home/sort-server-2/.ansible/tmp/ansible-tmp-1570201830.24-147225566909601\n', '')
<172.16.100.102> SSH: EXEC sshpass -d13 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=7777 -o User=sort-server-3 -o ConnectTimeout=10 -o ControlPath=/home/sort-server-1/.ansible/cp/a40d48ee74 172.16.100.102 '/bin/sh -c '"'"'echo ~sort-server-3 && sleep 0'"'"''
Using module file /home/sort-server-1/.local/lib/python2.7/site-packages/ansible/modules/commands/command.py
<172.16.100.101> PUT /home/sort-server-1/.ansible/tmp/ansible-local-30790NJeo5z/tmpYgzNy0 TO /home/sort-server-2/.ansible/tmp/ansible-tmp-1570201830.24-147225566909601/AnsiballZ_command.py
<172.16.100.101> SSH: EXEC sshpass -d13 sftp -o BatchMode=no -b - -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=7777 -o User=sort-server-2 -o ConnectTimeout=10 -o ControlPath=/home/sort-server-1/.ansible/cp/71329be8d9 '[172.16.100.101]'
<172.16.100.102> (0, '/home/sort-server-3\n', '')
<172.16.100.102> ESTABLISH SSH CONNECTION FOR USER: sort-server-3
<172.16.100.102> SSH: EXEC sshpass -d13 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=7777 -o User=sort-server-3 -o ConnectTimeout=10 -o ControlPath=/home/sort-server-1/.ansible/cp/a40d48ee74 172.16.100.102 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/sort-server-3/.ansible/tmp/ansible-tmp-1570201830.27-86933938630144 `" && echo ansible-tmp-1570201830.27-86933938630144="` echo /home/sort-server-3/.ansible/tmp/ansible-tmp-1570201830.27-86933938630144 `" ) && sleep 0'"'"''
<172.16.100.101> (0, 'sftp> put /home/sort-server-1/.ansible/tmp/ansible-local-30790NJeo5z/tmpYgzNy0 /home/sort-server-2/.ansible/tmp/ansible-tmp-1570201830.24-147225566909601/AnsiballZ_command.py\n', '')
<172.16.100.101> ESTABLISH SSH CONNECTION FOR USER: sort-server-2
<172.16.100.101> SSH: EXEC sshpass -d13 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=7777 -o User=sort-server-2 -o ConnectTimeout=10 -o ControlPath=/home/sort-server-1/.ansible/cp/71329be8d9 172.16.100.101 '/bin/sh -c '"'"'chmod u+x /home/sort-server-2/.ansible/tmp/ansible-tmp-1570201830.24-147225566909601/ /home/sort-server-2/.ansible/tmp/ansible-tmp-1570201830.24-147225566909601/AnsiballZ_command.py && sleep 0'"'"''
<172.16.100.102> (0, 'ansible-tmp-1570201830.27-86933938630144=/home/sort-server-3/.ansible/tmp/ansible-tmp-1570201830.27-86933938630144\n', '')
Using module file /home/sort-server-1/.local/lib/python2.7/site-packages/ansible/modules/commands/command.py
<172.16.100.102> PUT /home/sort-server-1/.ansible/tmp/ansible-local-30790NJeo5z/tmp7xH8zT TO /home/sort-server-3/.ansible/tmp/ansible-tmp-1570201830.27-86933938630144/AnsiballZ_command.py
<172.16.100.102> SSH: EXEC sshpass -d13 sftp -o BatchMode=no -b - -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=7777 -o User=sort-server-3 -o ConnectTimeout=10 -o ControlPath=/home/sort-server-1/.ansible/cp/a40d48ee74 '[172.16.100.102]'
<172.16.100.101> (0, '', '')
<172.16.100.101> ESTABLISH SSH CONNECTION FOR USER: sort-server-2
<172.16.100.101> SSH: EXEC sshpass -d13 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=7777 -o User=sort-server-2 -o ConnectTimeout=10 -o ControlPath=/home/sort-server-1/.ansible/cp/71329be8d9 -tt 172.16.100.101 '/bin/sh -c '"'"'sudo -H -S  -p "[sudo via ansible, key=tjjtvbjurtrcoofasbntghkaxprcwffp] password: " -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-tjjtvbjurtrcoofasbntghkaxprcwffp; PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin /usr/bin/python /home/sort-server-2/.ansible/tmp/ansible-tmp-1570201830.24-147225566909601/AnsiballZ_command.py'"'"'"'"'"'"'"'"' && sleep 0'"'"''
<172.16.100.102> (0, 'sftp> put /home/sort-server-1/.ansible/tmp/ansible-local-30790NJeo5z/tmp7xH8zT /home/sort-server-3/.ansible/tmp/ansible-tmp-1570201830.27-86933938630144/AnsiballZ_command.py\n', '')
<172.16.100.102> ESTABLISH SSH CONNECTION FOR USER: sort-server-3
<172.16.100.102> SSH: EXEC sshpass -d13 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=7777 -o User=sort-server-3 -o ConnectTimeout=10 -o ControlPath=/home/sort-server-1/.ansible/cp/a40d48ee74 172.16.100.102 '/bin/sh -c '"'"'chmod u+x /home/sort-server-3/.ansible/tmp/ansible-tmp-1570201830.27-86933938630144/ /home/sort-server-3/.ansible/tmp/ansible-tmp-1570201830.27-86933938630144/AnsiballZ_command.py && sleep 0'"'"''
<172.16.100.102> (0, '', '')
<172.16.100.102> ESTABLISH SSH CONNECTION FOR USER: sort-server-3
<172.16.100.102> SSH: EXEC sshpass -d13 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=7777 -o User=sort-server-3 -o ConnectTimeout=10 -o ControlPath=/home/sort-server-1/.ansible/cp/a40d48ee74 -tt 172.16.100.102 '/bin/sh -c '"'"'sudo -H -S  -p "[sudo via ansible, key=oxbtpurenwqfqbmhhybxfrcdjvoeqhlb] password: " -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-oxbtpurenwqfqbmhhybxfrcdjvoeqhlb; PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin /usr/bin/python /home/sort-server-3/.ansible/tmp/ansible-tmp-1570201830.27-86933938630144/AnsiballZ_command.py'"'"'"'"'"'"'"'"' && sleep 0'"'"''
Escalation succeeded
Escalation succeeded

Fatal Log

fatal: [sort-server-2]: FAILED! => {
    "ansible_job_id": "750381583864.4109",
    "changed": true,
    "cmd": [
        "/usr/local/bin/kubeadm",
        "join",
        "--config",
        "/etc/kubernetes/kubeadm-client.conf",
        "--ignore-preflight-errors=all"
    ],
    "delta": "0:00:00.015750",
    "end": "2019-10-05 16:28:43.459898",
    "finished": 1,
    "invocation": {
        "module_args": {
            "_raw_params": "/usr/local/bin/kubeadm join --config /etc/kubernetes/kubeadm-client.conf --ignore-preflight-errors=all",
            "_uses_shell": false,
            "argv": null,
            "chdir": null,
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "warn": true
        }
    },
    "msg": "non-zero return code",
    "rc": 1,
    "start": "2019-10-05 16:28:43.444148",
    "stderr": "unable to read config from \"/etc/kubernetes/kubeadm-client.conf\" [open /etc/kubernetes/kubeadm-client.conf: no such file or directory]",
    "stderr_lines": [
        "unable to read config from \"/etc/kubernetes/kubeadm-client.conf\" [open /etc/kubernetes/kubeadm-client.conf: no such file or directory]"
    ],
    "stdout": "",
    "stdout_lines": []
}
fatal: [sort-server-3]: FAILED! => {
    "ansible_job_id": "379111894176.29975",
    "changed": true,
    "cmd": [
        "/usr/local/bin/kubeadm",
        "join",
        "--config",
        "/etc/kubernetes/kubeadm-client.conf",
        "--ignore-preflight-errors=all"
    ],
    "delta": "0:00:00.016606",
    "end": "2019-10-05 16:28:43.521150",
    "finished": 1,
    "invocation": {
        "module_args": {
            "_raw_params": "/usr/local/bin/kubeadm join --config /etc/kubernetes/kubeadm-client.conf --ignore-preflight-errors=all",
            "_uses_shell": false,
            "argv": null,
            "chdir": null,
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "warn": true
        }
    },
    "msg": "non-zero return code",
    "rc": 1,
    "start": "2019-10-05 16:28:43.504544",
    "stderr": "unable to read config from \"/etc/kubernetes/kubeadm-client.conf\" [open /etc/kubernetes/kubeadm-client.conf: no such file or directory]",
    "stderr_lines": [
        "unable to read config from \"/etc/kubernetes/kubeadm-client.conf\" [open /etc/kubernetes/kubeadm-client.conf: no such file or directory]"
    ],
    "stdout": "",
    "stdout_lines": []
}

history

sudo ansible-playbook -i inventory/mycluster/hosts.ini --become --become-user=root kubespray-2.8.5/cluster.yml --extra-vars "ansible_sudo_pass=[서버1비밀번호]"

개발 버전

원인 파악

  1. Network 방화벽 문제?
wonjong-yoo commented 5 years ago

해당 ansible task 스크립트

- name: Join to cluster if needed
  command: >-
    {{ bin_dir }}/kubeadm join
    --config {{ kube_config_dir}}/kubeadm-client.{{ kubeadmConfig_api_version }}.conf
    --ignore-preflight-errors=all
  register: kubeadm_join
  when: not is_kube_master and (not kubelet_conf.stat.exists)
  environment:
    PATH: "{{ bin_dir }}:{{ ansible_env.PATH }}"
wonjong-yoo commented 5 years ago

1번 서버(master)의 netstat 결과

sort-server-1@sort-server-1:~/onprem-kubespray$ sudo netstat -lnp | grep 6443
tcp6       0      0 :::6443                 :::*                    LISTEN      9800/kube-apiserver

2번 서버에서 1번 서버의 6443 포트에 대해서 namp 결과 Open 되있는데??

sort-server-2@sort-server-2:~$ nmap -p 6443 172.16.100.100

Starting Nmap 7.60 ( https://nmap.org ) at 2019-10-05 16:33 KST
Nmap scan report for sort-server-1 (172.16.100.100)
Host is up (0.00023s latency).

PORT     STATE SERVICE
6443/tcp open  sun-sr-https
wonjong-yoo commented 5 years ago 
$ ansible-playbook -i inventory/mycluster/hosts.ini --become --become-user=root kubespray-2.8.5/reset.yml --extra-vars "ansible_sudo_pass=[비밀번호]" -vvvv

초기화 후 다시 실행하니까 됨.