search

techno-tim / k3s-ansible

The easiest way to bootstrap a self-hosted High Availability Kubernetes cluster. A fully automated HA k3s etcd install with kube-vip, MetalLB, and more. Build. Destroy. Repeat.
https://technotim.live/posts/k3s-etcd-ansible/
Apache License 2.0
2.41k stars 1.05k forks source link

TASK [k3s/master : Init cluster inside the transient k3s-init service] Fails on Ubuntu 22.04 #177

Closed s0undy closed 1 year ago

s0undy commented 1 year ago

Expected Behavior

Playbook should run and setup K3S on the nodes.

Current Behavior

Errors out on the task Init cluster inside the transient k3s-init service

image

The same error message is repeated for all 3 master nodes.

Steps to Reproduce

  1. Setup 5x new Ubuntu 22.04 VM's on Proxmox, install latest updates.
  2. Add ssh key for ansible to get SSH access from control node.
  3. Edit all.yml to match private ip-networks.
  4. Run playbook with: ansible-playbook ./site.yml -i ./inventory/my-cluster/hosts.ini --ask-become-pass

Context (variables)

Operating system: Ubuntu 22.04

Hardware: Proxmox running on Lenovo M910q MFF PC

Variables Used

all.yml

k3s_version: v1.24.8+k3s1
# this is the user that has ssh access to these machines
ansible_user: k3s-admin
systemd_dir: /etc/systemd/system

# Set your timezone
system_timezone: "Europe/Stockholm"

# interface which will be used for flannel
flannel_iface: "eth0"

# apiserver_endpoint is virtual ip-address which will be configured on each master
apiserver_endpoint: "10.10.10.130"

# k3s_token is required  masters can talk together securely
# this token should be alpha numeric only
k3s_token: "3CjpRV4pnDEb9vC2Cw2y3CjpRV4pnDEb9vC2Cw2y"

# The IP on which the node is reachable in the cluster.
# Here, a sensible default is provided, you can still override
# it for each of your hosts, though.
k3s_node_ip: '{{ ansible_facts[flannel_iface]["ipv4"]["address"] }}'

# Disable the taint manually by setting: k3s_master_taint = false
k3s_master_taint: "{{ true if groups['node'] | default([]) | length >= 1 else false }}"

# these arguments are recommended for servers as well as agents:
extra_args: >-
  --flannel-iface={{ flannel_iface }}
  --node-ip={{ k3s_node_ip }}
# change these to your liking, the only required are: --disable servicelb, --tls-san {{ apiserver_endpoint }}
extra_server_args: >-
  {{ extra_args }}
  {{ '--node-taint node-role.kubernetes.io/master=true:NoSchedule' if k3s_master_taint else '' }}
  --tls-san {{ apiserver_endpoint }}
  --disable servicelb
  --disable traefik
extra_agent_args: >-
  {{ extra_args }}
# image tag for kube-vip
kube_vip_tag_version: "v0.5.7"

# image tag for metal lb
metal_lb_speaker_tag_version: "v0.13.7"
metal_lb_controller_tag_version: "v0.13.7"

# metallb ip range for load balancer
metal_lb_ip_range: "10.10.10.210-10.10.10.250"

Hosts

host.ini

[master]
10.10.10.131
10.10.10.132
10.10.10.133

[node]
10.10.10.134
10.10.10.135

[k3s_cluster:children]
master
node

Possible Solution

Full Ansible Playbook Recap

PLAY [k3s_cluster] *****************************************************************************************************************************************************************************************************
TASK [Gathering Facts] *************************************************************************************************************************************************************************************************
Wednesday 07 December 2022  23:34:58 +0100 (0:00:00.007)       0:00:00.007 ****
fatal: [10.10.10.131]: FAILED! =>
  msg: Missing sudo password
fatal: [10.10.10.132]: FAILED! =>
  msg: Missing sudo password
fatal: [10.10.10.134]: FAILED! =>
  msg: Missing sudo password
fatal: [10.10.10.133]: FAILED! =>
  msg: Missing sudo password
fatal: [10.10.10.135]: FAILED! =>
  msg: Missing sudo password

PLAY RECAP *************************************************************************************************************************************************************************************************************
10.10.10.131               : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0
10.10.10.132               : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0
10.10.10.133               : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0
10.10.10.134               : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0
10.10.10.135               : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

Wednesday 07 December 2022  23:35:03 +0100 (0:00:05.025)       0:00:05.032 ****
===============================================================================
Gathering Facts ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 5.03s
s0undy@GIBBSTATION:~/k3s-ansible$
s0undy@GIBBSTATION:~/k3s-ansible$ ansible-playbook ./site.yml -i ./inventory/my-cluster/hosts.ini --ask-become-pass
BECOME password:

PLAY [k3s_cluster] *****************************************************************************************************************************************************************************************************
TASK [Gathering Facts] *************************************************************************************************************************************************************************************************
Wednesday 07 December 2022  23:35:15 +0100 (0:00:00.008)       0:00:00.008 ****
ok: [10.10.10.131]
ok: [10.10.10.132]
ok: [10.10.10.133]
ok: [10.10.10.135]
ok: [10.10.10.134]

TASK [prereq : Set same timezone on every Server] **********************************************************************************************************************************************************************
Wednesday 07 December 2022  23:35:16 +0100 (0:00:01.081)       0:00:01.089 ****
ok: [10.10.10.133]
ok: [10.10.10.132]
ok: [10.10.10.135]
ok: [10.10.10.131]
ok: [10.10.10.134]

TASK [prereq : Set SELinux to disabled state] **************************************************************************************************************************************************************************
Wednesday 07 December 2022  23:35:16 +0100 (0:00:00.371)       0:00:01.461 ****
skipping: [10.10.10.131]
skipping: [10.10.10.132]
skipping: [10.10.10.133]
skipping: [10.10.10.134]
skipping: [10.10.10.135]

TASK [prereq : Enable IPv4 forwarding] *********************************************************************************************************************************************************************************
Wednesday 07 December 2022  23:35:16 +0100 (0:00:00.033)       0:00:01.494 ****
ok: [10.10.10.133]
ok: [10.10.10.135]
ok: [10.10.10.132]
ok: [10.10.10.134]
ok: [10.10.10.131]

TASK [prereq : Enable IPv6 forwarding] *********************************************************************************************************************************************************************************
Wednesday 07 December 2022  23:35:16 +0100 (0:00:00.276)       0:00:01.771 ****
ok: [10.10.10.131]
ok: [10.10.10.132]
ok: [10.10.10.133]
ok: [10.10.10.134]
ok: [10.10.10.135]

TASK [prereq : Enable IPv6 router advertisements] **********************************************************************************************************************************************************************
Wednesday 07 December 2022  23:35:17 +0100 (0:00:00.215)       0:00:01.986 ****
ok: [10.10.10.131]
ok: [10.10.10.132]
ok: [10.10.10.133]
ok: [10.10.10.134]
ok: [10.10.10.135]

TASK [prereq : Add br_netfilter to /etc/modules-load.d/] ***************************************************************************************************************************************************************
Wednesday 07 December 2022  23:35:17 +0100 (0:00:00.229)       0:00:02.216 ****
skipping: [10.10.10.131]
skipping: [10.10.10.132]
skipping: [10.10.10.133]
skipping: [10.10.10.134]
skipping: [10.10.10.135]

TASK [prereq : Load br_netfilter] **************************************************************************************************************************************************************************************
Wednesday 07 December 2022  23:35:17 +0100 (0:00:00.052)       0:00:02.268 ****
skipping: [10.10.10.131]
skipping: [10.10.10.132]
skipping: [10.10.10.133]
skipping: [10.10.10.134]
skipping: [10.10.10.135]

TASK [prereq : Set bridge-nf-call-iptables (just to be sure)] **********************************************************************************************************************************************************
Wednesday 07 December 2022  23:35:17 +0100 (0:00:00.038)       0:00:02.307 ****
skipping: [10.10.10.131] => (item=net.bridge.bridge-nf-call-iptables)
skipping: [10.10.10.131] => (item=net.bridge.bridge-nf-call-ip6tables)
skipping: [10.10.10.132] => (item=net.bridge.bridge-nf-call-iptables)
skipping: [10.10.10.132] => (item=net.bridge.bridge-nf-call-ip6tables)
skipping: [10.10.10.133] => (item=net.bridge.bridge-nf-call-iptables)
skipping: [10.10.10.133] => (item=net.bridge.bridge-nf-call-ip6tables)
skipping: [10.10.10.134] => (item=net.bridge.bridge-nf-call-iptables)
skipping: [10.10.10.134] => (item=net.bridge.bridge-nf-call-ip6tables)
skipping: [10.10.10.135] => (item=net.bridge.bridge-nf-call-iptables)
skipping: [10.10.10.135] => (item=net.bridge.bridge-nf-call-ip6tables)

TASK [prereq : Add /usr/local/bin to sudo secure_path] *****************************************************************************************************************************************************************
Wednesday 07 December 2022  23:35:17 +0100 (0:00:00.045)       0:00:02.353 ****
skipping: [10.10.10.131]
skipping: [10.10.10.132]
skipping: [10.10.10.133]
skipping: [10.10.10.134]
skipping: [10.10.10.135]

TASK [download : Download k3s binary x64] ******************************************************************************************************************************************************************************
Wednesday 07 December 2022  23:35:17 +0100 (0:00:00.051)       0:00:02.405 ****
ok: [10.10.10.135]
ok: [10.10.10.131]
ok: [10.10.10.133]
ok: [10.10.10.134]
ok: [10.10.10.132]

TASK [download : Download k3s binary arm64] ****************************************************************************************************************************************************************************
Wednesday 07 December 2022  23:35:19 +0100 (0:00:01.515)       0:00:03.921 ****
skipping: [10.10.10.131]
skipping: [10.10.10.132]
skipping: [10.10.10.133]
skipping: [10.10.10.134]
skipping: [10.10.10.135]

TASK [download : Download k3s binary armhf] ****************************************************************************************************************************************************************************
Wednesday 07 December 2022  23:35:19 +0100 (0:00:00.039)       0:00:03.960 ****
skipping: [10.10.10.131]
skipping: [10.10.10.132]
skipping: [10.10.10.133]
skipping: [10.10.10.134]
skipping: [10.10.10.135]

TASK [raspberrypi : Test for raspberry pi /proc/cpuinfo] ***************************************************************************************************************************************************************
Wednesday 07 December 2022  23:35:19 +0100 (0:00:00.058)       0:00:04.019 ****
ok: [10.10.10.133]
ok: [10.10.10.132]
ok: [10.10.10.134]
ok: [10.10.10.131]
ok: [10.10.10.135]

TASK [raspberrypi : Test for raspberry pi /proc/device-tree/model] *****************************************************************************************************************************************************
Wednesday 07 December 2022  23:35:19 +0100 (0:00:00.276)       0:00:04.295 ****
ok: [10.10.10.131]
ok: [10.10.10.132]
ok: [10.10.10.133]
ok: [10.10.10.134]
ok: [10.10.10.135]

TASK [raspberrypi : Set raspberry_pi fact to true] *********************************************************************************************************************************************************************
Wednesday 07 December 2022  23:35:19 +0100 (0:00:00.217)       0:00:04.512 ****
skipping: [10.10.10.131]
skipping: [10.10.10.132]
skipping: [10.10.10.133]
skipping: [10.10.10.134]
skipping: [10.10.10.135]

TASK [raspberrypi : Set detected_distribution to Raspbian] *************************************************************************************************************************************************************
Wednesday 07 December 2022  23:35:19 +0100 (0:00:00.037)       0:00:04.549 ****
skipping: [10.10.10.131]
skipping: [10.10.10.132]
skipping: [10.10.10.134]
skipping: [10.10.10.135]
skipping: [10.10.10.133]

TASK [raspberrypi : Set detected_distribution to Raspbian (ARM64 on Debian Buster)] ************************************************************************************************************************************
Wednesday 07 December 2022  23:35:19 +0100 (0:00:00.061)       0:00:04.611 ****
skipping: [10.10.10.131]
skipping: [10.10.10.132]
skipping: [10.10.10.133]
skipping: [10.10.10.134]
skipping: [10.10.10.135]

TASK [raspberrypi : Set detected_distribution_major_version] ***********************************************************************************************************************************************************
Wednesday 07 December 2022  23:35:19 +0100 (0:00:00.043)       0:00:04.655 ****
skipping: [10.10.10.131]
skipping: [10.10.10.132]
skipping: [10.10.10.133]
skipping: [10.10.10.134]
skipping: [10.10.10.135]

TASK [raspberrypi : Set detected_distribution to Raspbian (ARM64 on Debian Bullseye)] **********************************************************************************************************************************
Wednesday 07 December 2022  23:35:19 +0100 (0:00:00.045)       0:00:04.700 ****
skipping: [10.10.10.131]
skipping: [10.10.10.132]
skipping: [10.10.10.133]
skipping: [10.10.10.134]
skipping: [10.10.10.135]

TASK [raspberrypi : execute OS related tasks on the Raspberry Pi - setup] **********************************************************************************************************************************************
Wednesday 07 December 2022  23:35:19 +0100 (0:00:00.065)       0:00:04.766 ****
skipping: [10.10.10.131] => (item=/home/s0undy/k3s-ansible/roles/raspberrypi/tasks/setup/Ubuntu.yml)
skipping: [10.10.10.132] => (item=/home/s0undy/k3s-ansible/roles/raspberrypi/tasks/setup/Ubuntu.yml)
skipping: [10.10.10.133] => (item=/home/s0undy/k3s-ansible/roles/raspberrypi/tasks/setup/Ubuntu.yml)
skipping: [10.10.10.134] => (item=/home/s0undy/k3s-ansible/roles/raspberrypi/tasks/setup/Ubuntu.yml)
skipping: [10.10.10.135] => (item=/home/s0undy/k3s-ansible/roles/raspberrypi/tasks/setup/Ubuntu.yml)

PLAY [master] **********************************************************************************************************************************************************************************************************

TASK [Gathering Facts] *************************************************************************************************************************************************************************************************
Wednesday 07 December 2022  23:35:20 +0100 (0:00:00.084)       0:00:04.851 ****
ok: [10.10.10.132]
ok: [10.10.10.131]
ok: [10.10.10.133]

TASK [k3s/master : Clean previous runs of k3s-init] ********************************************************************************************************************************************************************
Wednesday 07 December 2022  23:35:20 +0100 (0:00:00.889)       0:00:05.740 ****
ok: [10.10.10.132]
ok: [10.10.10.131]
ok: [10.10.10.133]

TASK [k3s/master : Clean previous runs of k3s-init] ********************************************************************************************************************************************************************
Wednesday 07 December 2022  23:35:21 +0100 (0:00:00.452)       0:00:06.193 ****
ok: [10.10.10.131]
ok: [10.10.10.132]
ok: [10.10.10.133]

TASK [k3s/master : Create manifests directory on first master] *********************************************************************************************************************************************************
Wednesday 07 December 2022  23:35:21 +0100 (0:00:00.208)       0:00:06.402 ****
skipping: [10.10.10.132]
skipping: [10.10.10.133]
ok: [10.10.10.131]

TASK [k3s/master : Copy vip rbac manifest to first master] *************************************************************************************************************************************************************
Wednesday 07 December 2022  23:35:21 +0100 (0:00:00.269)       0:00:06.671 ****
skipping: [10.10.10.132]
skipping: [10.10.10.133]
ok: [10.10.10.131]

TASK [k3s/master : Copy vip manifest to first master] ******************************************************************************************************************************************************************
Wednesday 07 December 2022  23:35:22 +0100 (0:00:00.470)       0:00:07.141 ****
skipping: [10.10.10.132]
skipping: [10.10.10.133]
ok: [10.10.10.131]

TASK [k3s/master : Copy metallb namespace to first master] *************************************************************************************************************************************************************
Wednesday 07 December 2022  23:35:22 +0100 (0:00:00.421)       0:00:07.563 ****
skipping: [10.10.10.132]
skipping: [10.10.10.133]
ok: [10.10.10.131]

TASK [k3s/master : Copy metallb namespace to first master] *************************************************************************************************************************************************************
Wednesday 07 December 2022  23:35:23 +0100 (0:00:00.395)       0:00:07.959 ****
skipping: [10.10.10.132]
skipping: [10.10.10.133]
ok: [10.10.10.131]

TASK [k3s/master : Init cluster inside the transient k3s-init service] *************************************************************************************************************************************************
Wednesday 07 December 2022  23:35:23 +0100 (0:00:00.397)       0:00:08.356 ****
fatal: [10.10.10.131]: FAILED! =>
  msg: |-
    The task includes an option with an undefined variable. The error was: {% if groups['master'] | length > 1 %}
      {% if ansible_hostname == hostvars[groups['master'][0]]['ansible_hostname'] %}
        --cluster-init
      {% else %}
        --server https://{{ hostvars[groups['master'][0]].k3s_node_ip }}:6443
      {% endif %}
      --token {{ k3s_token }}
    {% endif %} {{ extra_server_args | default('') }}: {{ extra_args }} {{ '--node-taint node-role.kubernetes.io/master=true:NoSchedule' if k3s_master_taint else '' }} --tls-san {{ apiserver_endpoint }} --disable servicelb --disable traefik: --flannel-iface={{ flannel_iface }} --node-ip={{ k3s_node_ip }}: {{ ansible_facts[flannel_iface]["ipv4"]["address"] }}: 'dict object' has no attribute 'eth0'

    The error appears to be in '/home/s0undy/k3s-ansible/roles/k3s/master/tasks/main.yml': line 62, column 3, but may
    be elsewhere in the file depending on the exact syntax problem.

    The offending line appears to be:

    - name: Init cluster inside the transient k3s-init service
      ^ here
fatal: [10.10.10.132]: FAILED! =>
  msg: |-
    The task includes an option with an undefined variable. The error was: {% if groups['master'] | length > 1 %}
      {% if ansible_hostname == hostvars[groups['master'][0]]['ansible_hostname'] %}
        --cluster-init
      {% else %}
        --server https://{{ hostvars[groups['master'][0]].k3s_node_ip }}:6443
      {% endif %}
      --token {{ k3s_token }}
    {% endif %} {{ extra_server_args | default('') }}: {{ extra_args }} {{ '--node-taint node-role.kubernetes.io/master=true:NoSchedule' if k3s_master_taint else '' }} --tls-san {{ apiserver_endpoint }} --disable servicelb --disable traefik: --flannel-iface={{ flannel_iface }} --node-ip={{ k3s_node_ip }}: {{ ansible_facts[flannel_iface]["ipv4"]["address"] }}: 'dict object' has no attribute 'eth0'

    The error appears to be in '/home/s0undy/k3s-ansible/roles/k3s/master/tasks/main.yml': line 62, column 3, but may
    be elsewhere in the file depending on the exact syntax problem.

    The offending line appears to be:

    - name: Init cluster inside the transient k3s-init service
      ^ here
fatal: [10.10.10.133]: FAILED! =>
  msg: |-
    The task includes an option with an undefined variable. The error was: {% if groups['master'] | length > 1 %}
      {% if ansible_hostname == hostvars[groups['master'][0]]['ansible_hostname'] %}
        --cluster-init
      {% else %}
        --server https://{{ hostvars[groups['master'][0]].k3s_node_ip }}:6443
      {% endif %}
      --token {{ k3s_token }}
    {% endif %} {{ extra_server_args | default('') }}: {{ extra_args }} {{ '--node-taint node-role.kubernetes.io/master=true:NoSchedule' if k3s_master_taint else '' }} --tls-san {{ apiserver_endpoint }} --disable servicelb --disable traefik: --flannel-iface={{ flannel_iface }} --node-ip={{ k3s_node_ip }}: {{ ansible_facts[flannel_iface]["ipv4"]["address"] }}: 'dict object' has no attribute 'eth0'

    The error appears to be in '/home/s0undy/k3s-ansible/roles/k3s/master/tasks/main.yml': line 62, column 3, but may
    be elsewhere in the file depending on the exact syntax problem.

    The offending line appears to be:

    - name: Init cluster inside the transient k3s-init service
      ^ here

PLAY RECAP *************************************************************************************************************************************************************************************************************
10.10.10.131               : ok=16   changed=0    unreachable=0    failed=1    skipped=13   rescued=0    ignored=0
10.10.10.132               : ok=11   changed=0    unreachable=0    failed=1    skipped=18   rescued=0    ignored=0
10.10.10.133               : ok=11   changed=0    unreachable=0    failed=1    skipped=18   rescued=0    ignored=0
10.10.10.134               : ok=8    changed=0    unreachable=0    failed=0    skipped=13   rescued=0    ignored=0
10.10.10.135               : ok=8    changed=0    unreachable=0    failed=0    skipped=13   rescued=0    ignored=0

Wednesday 07 December 2022  23:35:23 +0100 (0:00:00.030)       0:00:08.387 ****
===============================================================================
download : Download k3s binary x64 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 1.52s
Gathering Facts ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.08s
Gathering Facts ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.89s
k3s/master : Copy vip rbac manifest to first master ------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.47s
k3s/master : Clean previous runs of k3s-init -------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.45s
k3s/master : Copy vip manifest to first master ------------------------------------------------------------------------------------------------------------------------------------------------------------------ 0.42s
k3s/master : Copy metallb namespace to first master ------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.40s
k3s/master : Copy metallb namespace to first master ------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.40s
prereq : Set same timezone on every Server ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.37s
prereq : Enable IPv4 forwarding --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.28s
raspberrypi : Test for raspberry pi /proc/cpuinfo --------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.28s
k3s/master : Create manifests directory on first master --------------------------------------------------------------------------------------------------------------------------------------------------------- 0.27s
prereq : Enable IPv6 router advertisements ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.23s
raspberrypi : Test for raspberry pi /proc/device-tree/model ----------------------------------------------------------------------------------------------------------------------------------------------------- 0.22s
prereq : Enable IPv6 forwarding --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.22s
k3s/master : Clean previous runs of k3s-init -------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.21s
raspberrypi : execute OS related tasks on the Raspberry Pi - setup ---------------------------------------------------------------------------------------------------------------------------------------------- 0.08s
raspberrypi : Set detected_distribution to Raspbian (ARM64 on Debian Bullseye) ---------------------------------------------------------------------------------------------------------------------------------- 0.07s
raspberrypi : Set detected_distribution to Raspbian ------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.06s
download : Download k3s binary armhf ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.06s