search

kubernetes-sigs / kubespray

Deploy a Production Ready Kubernetes Cluster
Apache License 2.0
15.86k stars 6.41k forks source link

Status code was not [200]: Request failed: <urlopen error [Errno 111] Connection refused> #2794

Closed IvanBiv closed 6 years ago

IvanBiv commented 6 years ago

I see problem with kube-scheduler: Status code was not [200]: Request failed: <urlopen error [Errno 111] Connection refused>

hosts

k8s-m1.me ansible_host=192.168.88.11 ansible_become=yes
k8s-m2.me ansible_host=192.168.88.12 ansible_become=yes
k8s-m3.me ansible_host=192.168.88.10 ansible_become=yes #bootstrap_os=linux_mint

[kube-master]
k8s-m1.me
k8s-m2.me
k8s-m3.me

[etcd]
k8s-m1.me
#k8s-m2.me
#k8s-m3.me

[kube-node]
# k8s-s2.me

[kube-ingress]
# node2
# node3

[k8s-cluster:children]
kube-master
kube-node
kube-ingress

call playbook: # ansible-playbook -i my_inventory/inventory cluster.yml -b -v

log:

RUNNING HANDLER [kubernetes/master : Master | wait for kube-scheduler] ************************************************************************************************************************************************************
Friday 18 May 2018  16:03:23 +0300 (0:00:01.092)       0:02:47.453 ************ 
FAILED - RETRYING: Master | wait for kube-scheduler (60 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (60 retries left).
ok: [k8s-m1.me] => {"attempts": 2, "changed": false, "connection": "close", "content_length": "2", "content_type": "text/plain; charset=utf-8", "cookies": {}, "date": "Fri, 18 May 2018 13:03:29 GMT", "failed": false, "msg": "OK (2 bytes)", "redirected": false, "status": 200, "url": "http://localhost:10251/healthz"}
FAILED - RETRYING: Master | wait for kube-scheduler (59 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (58 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (57 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (56 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (55 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (54 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (53 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (52 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (51 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (50 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (49 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (48 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (47 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (46 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (45 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (44 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (43 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (42 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (41 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (40 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (39 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (38 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (37 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (36 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (35 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (34 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (33 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (32 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (31 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (30 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (29 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (28 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (27 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (26 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (25 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (24 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (23 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (22 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (21 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (20 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (19 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (18 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (17 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (16 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (15 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (14 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (13 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (12 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (11 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (10 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (9 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (8 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (7 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (6 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (5 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (4 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (3 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (2 retries left).
FAILED - RETRYING: Master | wait for kube-scheduler (1 retries left).
fatal: [k8s-m3.me]: FAILED! => {"attempts": 60, "changed": false, "content": "", "failed": true, "msg": "Status code was not [200]: Request failed: <urlopen error [Errno 111] Connection refused>", "redirected": false, "status": -1, "url": "http://localhost:10251/healthz"}

...
PLAY RECAP ************************************************************************************************************************************************************************************************************************
k8s-m1.me                  : ok=336  changed=29   unreachable=0    failed=0   
k8s-m2.me                  : ok=230  changed=8    unreachable=0    failed=0   
k8s-m3.me                  : ok=221  changed=14   unreachable=0    failed=1   
localhost                  : ok=2    changed=0    unreachable=0    failed=0   

Friday 18 May 2018  16:09:56 +0300 (0:00:00.034)       0:09:19.755 ************ 
=============================================================================== 
kubernetes/master : Master | wait for kube-scheduler --------------------------------------------------------------------------------------------------------------------------------------------------------------------- 321.09s
kubernetes/master : Master | wait for the apiserver to be running --------------------------------------------------------------------------------------------------------------------------------------------------------- 27.98s
bootstrap-os : Bootstrap | Install python 2.x and pip ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- 5.73s
kubernetes-apps/ansible : Kubernetes Apps | Start Resources ---------------------------------------------------------------------------------------------------------------------------------------------------------------- 5.38s
network_plugin/calico : Calico | Copy cni plugins from calico/cni container ------------------------------------------------------------------------------------------------------------------------------------------------ 4.37s
network_plugin/calico : Calico | Copy cni plugins from hyperkube ----------------------------------------------------------------------------------------------------------------------------------------------------------- 4.28s
kubernetes/master : Compare host kubectl with hyperkube container ---------------------------------------------------------------------------------------------------------------------------------------------------------- 4.01s
kubernetes/node : install | Compare host kubelet with hyperkube container -------------------------------------------------------------------------------------------------------------------------------------------------- 3.58s
kubernetes-apps/network_plugin/calico : Start Calico resources ------------------------------------------------------------------------------------------------------------------------------------------------------------- 3.48s
gather facts from all instances -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 3.44s
kubernetes-apps/ansible : Kubernetes Apps | Lay Down KubeDNS Template ------------------------------------------------------------------------------------------------------------------------------------------------------ 3.39s
kubernetes-apps/ansible : Kubernetes Apps | Delete old KubeDNS resources --------------------------------------------------------------------------------------------------------------------------------------------------- 3.06s
kubernetes-apps/helm : Helm | Compare host helm with hyperkube container --------------------------------------------------------------------------------------------------------------------------------------------------- 2.92s
kubernetes-apps/rotate_tokens : Rotate Tokens | Get default token name ----------------------------------------------------------------------------------------------------------------------------------------------------- 2.66s
docker : Ensure old versions of Docker are not installed. | Debian --------------------------------------------------------------------------------------------------------------------------------------------------------- 2.47s
network_plugin/calico : Calico | Create calico manifests ------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2.33s
network_plugin/calico : Calico | Set global as_num ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.80s
kubernetes-apps/helm : Helm | Apply Helm Manifests (RBAC) ------------------------------------------------------------------------------------------------------------------------------------------------------------------ 1.59s
download : Download items -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.44s
kubernetes-apps/ansible : Kubernetes Apps | Delete old CoreDNS resources --------------------------------------------------------------------------------------------------------------------------------------------------- 1.32s

Details log:

Using module file /usr/local/lib/python2.7/dist-packages/ansible/modules/net_tools/basics/uri.py
<192.168.88.10> ESTABLISH SSH CONNECTION FOR USER: ansible
<192.168.88.10> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=30m)(-o)(ConnectionAttempts=100)(-o)(UserKnownHostsFile=/dev/null)
<192.168.88.10> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)
<192.168.88.10> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<192.168.88.10> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=ansible)
<192.168.88.10> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<192.168.88.10> SSH: PlayContext set ssh_common_args: ()
<192.168.88.10> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/ivan/.ansible/cp/f657ebb9e8)
<192.168.88.10> SSH: EXEC ssh -vvv -o ControlMaster=auto -o ControlPersist=30m -o ConnectionAttempts=100 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ansible -o ConnectTimeout=10 -o ControlPath=/home/ivan/.ansible/cp/f657ebb9e8 192.168.88.10 '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-xhkfjhlbhmcilvfpubsaerbihzvpwzne; /usr/bin/python'"'"'"'"'"'"'"'"' && sleep 0'"'"''
<192.168.88.10> (1, '\n{"status": -1, "exception": "  File \\"/tmp/ansible_VY3EGx/ansible_module_uri.py\\", line 468, in main\\n    uresp[\'location\'] = absolute_location(url, uresp[\'location\'])\\n", "url": "http://localhost:10251/healthz", "content": "", "failed": true, "invocation": {"module_args": {"directory_mode": null, "force": false, "remote_src": null, "status_code": [200], "body_format": "raw", "owner": null, "follow": false, "client_key": null, "group": null, "use_proxy": true, "unsafe_writes": null, "setype": null, "content": null, "serole": null, "follow_redirects": "safe", "return_content": false, "method": "GET", "body": null, "timeout": 30, "src": null, "dest": null, "selevel": null, "force_basic_auth": false, "removes": null, "http_agent": "ansible-httpget", "regexp": null, "url_password": null, "url": "http://localhost:10251/healthz", "validate_certs": true, "seuser": null, "client_cert": null, "creates": null, "headers": {}, "delimiter": null, "mode": null, "url_username": null, "attributes": null, "backup": null}}, "redirected": false, "msg": "Status code was not [200]: Request failed: <urlopen error [Errno 111] Connection refused>"}\n', 'OpenSSH_7.2p2 Ubuntu-4ubuntu2.4, OpenSSL 1.0.2g  1 Mar 2016\r\ndebug1: Reading configuration data /home/ivan/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 9513\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 1\r\n')
The full traceback is:
  File "/tmp/ansible_VY3EGx/ansible_module_uri.py", line 468, in main
    uresp['location'] = absolute_location(url, uresp['location'])

fatal: [k8s-m3.me]: FAILED! => {
    "attempts": 60, 
    "changed": false, 
    "content": "", 
    "failed": true, 
    "invocation": {
        "module_args": {
            "attributes": null, 
            "backup": null, 
            "body": null, 
            "body_format": "raw", 
            "client_cert": null, 
            "client_key": null, 
            "content": null, 
            "creates": null, 
            "delimiter": null, 
            "dest": null, 
            "directory_mode": null, 
            "follow": false, 
            "follow_redirects": "safe", 
            "force": false, 
            "force_basic_auth": false, 
            "group": null, 
            "headers": {}, 
            "http_agent": "ansible-httpget", 
            "method": "GET", 
            "mode": null, 
            "owner": null, 
            "regexp": null, 
            "remote_src": null, 
            "removes": null, 
            "return_content": false, 
            "selevel": null, 
            "serole": null, 
            "setype": null, 
            "seuser": null, 
            "src": null, 
            "status_code": [
                200
            ], 
            "timeout": 30, 
            "unsafe_writes": null, 
            "url": "http://localhost:10251/healthz", 
            "url_password": null, 
            "url_username": null, 
            "use_proxy": true, 
            "validate_certs": true
        }
    }, 
    "msg": "Status code was not [200]: Request failed: <urlopen error [Errno 111] Connection refused>", 
    "redirected": false, 
    "status": -1, 
    "url": "http://localhost:10251/healthz"
}
IvanBiv commented 6 years ago

I fix it with add all my nodes in kube-node group (in my_inventory/inventory/hosts) and call: $ ansible-playbook -i my_inventory/inventory remove-node.yml -b -vvv and after again: $ ansible-playbook -i my_inventory/inventory cluster.yml -b -vvv