Initialize master fails with systemctl status docker.service

hkoessler commented 6 years ago

I tried to install a cluster of newly installed pis with raspian lite with releasedate 2018-03-13. After adding "become: yes" to the cluster.yml it works until the Task [master: Initialize master]. This task fails. My assumption is that docker 18.04 is installed but kubernetes only works with max 17.03.

It fails with the following error: fatal: [raspic0]: FAILED! => {"changed": true, "cmd": "kubeadm init --apiserver-advertise-address=192.168.1.104 --token=udy29x.ugyyk3tumg27atmr", "delta": "0:00:02.779513", "end": "2018-04-11 20:46:00.398347", "failed": true, "rc": 2, "start": "2018-04-11 20:45:57.618834", "stderr": "\t[WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 18.04.0-ce. Max validated version: 17.03\n\t[WARNING FileExisting-crictl]: crictl not found in system path\nSuggestion: go get github.com/kubernetes-incubator/cri-tools/cmd/crictl\n[preflight] Some fatal errors occurred:\n\t[ERROR SystemVerification]: missing cgroups: memory\n[preflight] If you know what you are doing, you can make a check non-fatal with --ignore-preflight-errors=...", "stderr_lines": ["\t[WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 18.04.0-ce. Max validated version: 17.03", "\t[WARNING FileExisting-crictl]: crictl not found in system path", "Suggestion: go get github.com/kubernetes-incubator/cri-tools/cmd/crictl", "[preflight] Some fatal errors occurred:", "\t[ERROR SystemVerification]: missing cgroups: memory", "[preflight] If you know what you are doing, you can make a check non-fatal with --ignore-preflight-errors=..."], "stdout": "[init] Using Kubernetes version: v1.10.0\n[init] Using Authorization modes: [Node RBAC]\n[preflight] Running pre-flight checks.\n[preflight] The system verification failed. Printing the output from the verification:\n\u001b[0;37mKERNEL_VERSION\u001b[0m: \u001b[0;32m4.14.30-v7+\u001b[0m\n\u001b[0;37mCONFIG_NAMESPACES\u001b[0m: \u001b[0;32menabled\u001b[0m\n\u001b[0;37mCONFIG_NET_NS\u001b[0m: ...

asachs01 commented 6 years ago

Howdy! This has been fixed in https://github.com/rak8s/rak8s/releases/tag/0.1.4 but I don't know if @chris-short has pushed it to master yet.

hkoessler commented 6 years ago

Hi asachs01: I re-tried with the latest version (which should be tag 0.1.14). Actually the setup worked then but logging into my master node I constantly got "connection refused". Only after copying the /etc/kubernetes/admin.conf to the ~pi of my master pi I got kubectl working. (see https://stackoverflow.com/questions/45536428/why-does-kubectl-have-different-behavior-with-sudo?utm_medium=organic&utm_source=google_rich_qa&utm_campaign=google_rich_qa)

chris-short commented 6 years ago

Hmm... The kube config should be copied locally @hkoessler. Mind sharing the log file from rak8s/.log/ with me? Also, what OS did you run ansible from?

tedsluis commented 6 years ago

I had the same error as hkoessler experienced. I was able to reproduce it and I know what caused it and I am sure it was fixed by #32. Let me explain:

I tried to deploy Kubernetes on a couple raspberry pi's (3B+) with fresh images (raspbian lite 18-04-2018), but it failed on TASK [common : Pass bridged IPv4 traffic to iptables' chains]:

pi@ansible-host ~/git/rak8s $ ansible-playbook cluster.yml 

PLAY [all] *********************************************************************

TASK [setup] *******************************************************************
ok: [node1]
ok: [node2]
ok: [master]

TASK [common : Enabling cgroup options at boot] ********************************
changed: [node1]
changed: [master]
changed: [node2]

TASK [common : Pass bridged IPv4 traffic to iptables' chains] ********************
fatal: [node1]: FAILED! => {"changed": false, "failed": true, "msg": "Failed to reload sysctl: sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory\n"}
fatal: [master]: FAILED! => {"changed": false, "failed": true, "msg": "Failed to reload sysctl: sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory\n"}
fatal: [node2]: FAILED! => {"changed": false, "failed": true, "msg": "Failed to reload sysctl: sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory\n"}

PLAY RECAP *********************************************************************
master                     : ok=2    changed=1    unreachable=0    failed=1   
node1                      : ok=2    changed=1    unreachable=0    failed=1   
node2                      : ok=2    changed=1    unreachable=0    failed=1

It is important to understand that:

This playbook version was a version before May 15 2018. This issue above is fixed now with pull request #32
The task previous to error, TASK [common : Enabling cgroup options at boot], overrides /boot/cmdline.txt with a new version, the first time this playbooks runs (and only then!) and when it does, the variable cmdline.changed is set to true. This variable will later be used in the playbook to trigger a reboot.

I did run the playbook for the second time and this time it exited with the same error as hkoessler experienced:

pi@ansible-host ~/git/rak8s $ ansible-playbook cluster.yml 

PLAY [all] *********************************************************************

TASK [setup] *******************************************************************
ok: [node1]
ok: [node2]
ok: [master]

TASK [common : Enabling cgroup options at boot] ********************************
ok: [node2]
ok: [node1]
ok: [master]

TASK [common : Pass bridged IPv4 traffic to iptables' chains] ******************
ok: [node1]
ok: [master]
ok: [node2]

TASK [common : apt-get update] *************************************************
ok: [node2]
ok: [node1]
ok: [master]

TASK [common : apt-get upgrade] ************************************************
ok: [node2]
ok: [node1]
ok: [master]

TASK [common : Reboot] *********************************************************
skipping: [master]
skipping: [node1]
skipping: [node2]

TASK [common : Wait for Reboot] ************************************************
skipping: [master]
skipping: [node1]
skipping: [node2]

TASK [kubeadm : Disable Swap] **************************************************
changed: [master]
changed: [node1]
changed: [node2]

TASK [kubeadm : Determine if docker is installed] ******************************
ok: [master]
ok: [node2]
ok: [node1]

TASK [kubeadm : Run Docker Install Script] *************************************
changed: [node2]
changed: [node1]
changed: [master]

TASK [kubeadm : Install apt-transport-https] ***********************************
ok: [master]
ok: [node1]
ok: [node2]

TASK [kubeadm : Add Google Cloud Repo Key] *************************************
changed: [master]
 [WARNING]: Consider using get_url or uri module rather than running curl

changed: [node2]
changed: [node1]

TASK [kubeadm : Add Kubernetes to Available apt Sources] ***********************
changed: [node1]
changed: [master]
changed: [node2]

TASK [kubeadm : apt-get update] ************************************************
changed: [node2]
changed: [node1]
changed: [master]

TASK [kubeadm : Install k8s Y'all] *********************************************
changed: [node2] => (item=[u'kubelet', u'kubeadm', u'kubectl'])
changed: [node1] => (item=[u'kubelet', u'kubeadm', u'kubectl'])
changed: [master] => (item=[u'kubelet', u'kubeadm', u'kubectl'])

PLAY [master] ******************************************************************

TASK [master : Reset Kubernetes Master] ****************************************
changed: [master]

TASK [master : Initialize Master] **********************************************
fatal: [master]: FAILED! => {"changed": true, "cmd": "kubeadm init --apiserver-advertise-address=192.168.11.210 --token=udy29x.ugyyk3tumg27atmr", "delta": "0:00:02.406248", "end": "2018-05-11 20:32:27.185350", "failed": true, "rc": 2, "start": "2018-05-11 20:32:24.779102", "stderr": "\t
[WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 18.05.0-ce. Max validated version: 17.03\n\t
[WARNING FileExisting-crictl]: crictl not found in system path\nSuggestion: go get github.com/kubernetes-incubator/cri-tools/cmd/crictl\n[preflight] Some fatal errors occurred:\n\t
[ERROR SystemVerification]: missing cgroups: memory\n[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`", "stdout": "
[init] Using Kubernetes version: v1.10.2\n[init] Using Authorization modes: [Node RBAC]\n[preflight] Running pre-flight checks.\n[preflight] 
The system verification failed. Printing the output from the verification:\n\u001b[0;37mKERNEL_VERSION\u001b[0m: \u001b[0;32m4.14.34-v7+\u001b[0m\n\u001b[0;37mCONFIG_NAMESPACES\u001b[0m: \u001b[0;32menabled\u001b[0m\n\u001b[0;37mCONFIG_NET_NS\u001b[0m: \u001b[0;32menabled\u001b[0m\n\u001b[0;37mCONFIG_PID_NS\u001b[0m: \u001b[0;32menabled\u001b[0m\n\u001b[0;37mCONFIG_IPC_NS\u001b[0m: \u001b[0;32menabled\u001b[0m\n\u001b[0;37mCONFIG_UTS_NS\u001b[0m: \u001b[0;32menabled\u001b[0m\n\u001b[0;37mCONFIG_CGROUPS\u001b[0m: \u001b[0;32menabled\u001b[0m\n\u001b[0;37mCONFIG_CGROUP_CPUACCT\u001b[0m: \u001b[0;32menabled\u001b[0m\n\u001b[0;37mCONFIG_CGROUP_DEVICE\u001b[0m: \u001b[0;32menabled\u001b[0m\n\u001b[0;37mCONFIG_CGROUP_FREEZER\u001b[0m: \u001b[0;32menabled\u001b[0m\n\u001b[0;37mCONFIG_CGROUP_SCHED\u001b[0m: \u001b[0;32menabled\u001b[0m\n\u001b[0;37mCONFIG_CPUSETS\u001b[0m: \u001b[0;32menabled\u001b[0m\n\u001b[0;37mCONFIG_MEMCG\u001b[0m: \u001b[0;32menabled\u001b[0m\n\u001b[0;37mCONFIG_INET\u001b[0m: \u001b[0;32menabled\u001b[0m\n\u001b[0;37mCONFIG_EXT4_FS\u001b[0m: \u001b[0;32menabled\u001b[0m\n\u001b[0;37mCONFIG_PROC_FS\u001b[0m: \u001b[0;32menabled\u001b[0m\n\u001b[0;37mCONFIG_NETFILTER_XT_TARGET_REDIRECT\u001b[0m: \u001b[0;32menabled (as module)\u001b[0m\n\u001b[0;37mCONFIG_NETFILTER_XT_MATCH_COMMENT\u001b[0m: \u001b[0;32menabled (as module)\u001b[0m\n\u001b[0;37mCONFIG_OVERLAY_FS\u001b[0m: \u001b[0;32menabled (as module)\u001b[0m\n\u001b[0;37mCONFIG_AUFS_FS\u001b[0m: \u001b[0;33mnot set - Required for aufs.\u001b[0m\n\u001b[0;37mCONFIG_BLK_DEV_DM\u001b[0m: \u001b[0;32menabled (as module)\u001b[0m\n\u001b[0;37mDOCKER_VERSION\u001b[0m: \u001b[0;32m18.05.0-ce\u001b[0m\n\u001b[0;37mOS\u001b[0m: \u001b[0;32mLinux\u001b[0m\n\u001b[0;37mCGROUPS_CPU\u001b[0m: \u001b[0;32menabled\u001b[0m\n\u001b[0;37mCGROUPS_CPUACCT\u001b[0m: \u001b[0;32menabled\u001b[0m\n\u001b[0;37mCGROUPS_CPUSET\u001b[0m: \u001b[0;32menabled\u001b[0m\n\u001b[0;37mCGROUPS_DEVICES\u001b[0m: \u001b[0;32menabled\u001b[0m\n\u001b[0;37mCGROUPS_FREEZER\u001b[0m: \u001b[0;32menabled\u001b[0m\n\u001b[0;37mCGROUPS_MEMORY\u001b[0m: \u001b[0;31mmissing\u001b[0m", "stdout_lines": ["
[init] Using Kubernetes version: v1.10.2", "[init] Using Authorization modes: [Node RBAC]", "[preflight] Running pre-flight checks.", "
[preflight] The system verification failed. Printing the output from the verification:", "\u001b[0;37mKERNEL_VERSION\u001b[0m: \u001b[0;32m4.14.34-v7+\u001b[0m", "\u001b[0;37mCONFIG_NAMESPACES\u001b[0m: \u001b[0;32menabled\u001b[0m", "\u001b[0;37mCONFIG_NET_NS\u001b[0m: \u001b[0;32menabled\u001b[0m", "\u001b[0;37mCONFIG_PID_NS\u001b[0m: \u001b[0;32menabled\u001b[0m", "\u001b[0;37mCONFIG_IPC_NS\u001b[0m: \u001b[0;32menabled\u001b[0m", "\u001b[0;37mCONFIG_UTS_NS\u001b[0m: \u001b[0;32menabled\u001b[0m", "\u001b[0;37mCONFIG_CGROUPS\u001b[0m: \u001b[0;32menabled\u001b[0m", "\u001b[0;37mCONFIG_CGROUP_CPUACCT\u001b[0m: \u001b[0;32menabled\u001b[0m", "\u001b[0;37mCONFIG_CGROUP_DEVICE\u001b[0m: \u001b[0;32menabled\u001b[0m", "\u001b[0;37mCONFIG_CGROUP_FREEZER\u001b[0m: \u001b[0;32menabled\u001b[0m", "\u001b[0;37mCONFIG_CGROUP_SCHED\u001b[0m: \u001b[0;32menabled\u001b[0m", "\u001b[0;37mCONFIG_CPUSETS\u001b[0m: \u001b[0;32menabled\u001b[0m", "\u001b[0;37mCONFIG_MEMCG\u001b[0m: \u001b[0;32menabled\u001b[0m", "\u001b[0;37mCONFIG_INET\u001b[0m: \u001b[0;32menabled\u001b[0m", "\u001b[0;37mCONFIG_EXT4_FS\u001b[0m: \u001b[0;32menabled\u001b[0m", "\u001b[0;37mCONFIG_PROC_FS\u001b[0m: \u001b[0;32menabled\u001b[0m", "\u001b[0;37mCONFIG_NETFILTER_XT_TARGET_REDIRECT\u001b[0m: \u001b[0;32menabled (as module)\u001b[0m", "\u001b[0;37mCONFIG_NETFILTER_XT_MATCH_COMMENT\u001b[0m: \u001b[0;32menabled (as module)\u001b[0m", "\u001b[0;37mCONFIG_OVERLAY_FS\u001b[0m: \u001b[0;32menabled (as module)\u001b[0m", "\u001b[0;37mCONFIG_AUFS_FS\u001b[0m: \u001b[0;33mnot set - Required for aufs.\u001b[0m", "\u001b[0;37mCONFIG_BLK_DEV_DM\u001b[0m: \u001b[0;32menabled (as module)\u001b[0m", "\u001b[0;37mDOCKER_VERSION\u001b[0m: \u001b[0;32m18.05.0-ce\u001b[0m", "\u001b[0;37mOS\u001b[0m: \u001b[0;32mLinux\u001b[0m", "\u001b[0;37mCGROUPS_CPU\u001b[0m: \u001b[0;32menabled\u001b[0m", "\u001b[0;37mCGROUPS_CPUACCT\u001b[0m: \u001b[0;32menabled\u001b[0m", "\u001b[0;37mCGROUPS_CPUSET\u001b[0m: \u001b[0;32menabled\u001b[0m", "\u001b[0;37mCGROUPS_DEVICES\u001b[0m: \u001b[0;32menabled\u001b[0m", "\u001b[0;37mCGROUPS_FREEZER\u001b[0m: \u001b[0;32menabled\u001b[0m", "\u001b[0;37mCGROUPS_MEMORY\u001b[0m: \u001b[0;31mmissing\u001b[0m"], "warnings": []}

PLAY RECAP *********************************************************************
master                     : ok=14   changed=7    unreachable=0    failed=1   
node1                      : ok=13   changed=6    unreachable=0    failed=0   
node2                      : ok=13   changed=6    unreachable=0    failed=0

It is important to understand that:

This time TASK [common : Enabling cgroup options at boot] did not copy the /boot/cmdline.txt, because it was already copied during the first run of the playbook. So the variable cmdline.changed is set to false.
Because the variable cmdline.changed is set to false, the TASK [common : Reboot] is not triggered! Therefore the settings in the /boot/cmdline.txt (like cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory) won't be effective until you do a manual reboot!
The playbook finally failed on this error [ERROR SystemVerification]: missing cgroups: memory, because the changes in /boot/cmdline were not effective.
The warning about the Docker version, [WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 18.05.0-ce. Max validated version: 17.03 is only a warning and is NOT in any way the cause that kubeadm --init failed.

I have tested this on fresh raspbian images and my conclusion is that this issue is fixed with #32

chris-short / rak8s

Initialize master fails with systemctl status docker.service #21