search

kubernetes-sigs / kubespray

Deploy a Production Ready Kubernetes Cluster
Apache License 2.0
15.86k stars 6.41k forks source link

Failed to add control-plane node #10973

Closed nvalembois closed 1 week ago

nvalembois commented 6 months ago

What happened?

Playbook cluster.yml fails when adding a new control plane node to an existing cluster.

What did you expect to happen?

Playbook cluster.yml succeeds.

How can we reproduce it (as minimally and precisely as possible)?

In a cluster deployed with Kubespray :

  1. remove a control plane node (not the first in the inventory) with remove-node playbook,
  2. reset system of removed control plane node
  3. try to add node in the cluster with cluster playbook --limit=kube_control_plane

OS

Linux 6.1.0-18-amd64 x86_64 PRETTY_NAME="Debian GNU/Linux 12 (bookworm)" NAME="Debian GNU/Linux" VERSION_ID="12" VERSION="12 (bookworm)" VERSION_CODENAME=bookworm ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/"

Version of Ansible

ansible [core 2.16.2] config file = /home/nicolas/git/kub/k8s-leno/kubespray/ansible.cfg configured module search path = ['/home/nicolas/git/kub/k8s-leno/kubespray/library'] ansible python module location = /home/nicolas/ansible/lib/python3.11/site-packages/ansible ansible collection location = /home/nicolas/.ansible/collections:/usr/share/ansible/collections executable location = /home/nicolas/ansible/bin/ansible python version = 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0] (/home/nicolas/ansible/bin/python3) jinja version = 3.1.3 libyaml = True

Version of Python

Python 3.11.2

Version of Kubespray (commit)

8760abf4b

Network plugin used

calico

Full inventory with variables

localhost | SUCCESS => 

{
    "hostvars[inventory_hostname]": {
        "access_ip": "{{ ansible_host }}",
        "additional_sysctl": [
            {
                "name": "fs.inotify.max_user_instances",
                "value": 512
            }
        ],
        "ansible_check_mode": false,
        "ansible_config_file": "/home/nicolas/git/kub/k8s-leno/ansible.cfg",
        "ansible_connection": "local",
        "ansible_diff_mode": false,
        "ansible_facts": {},
        "ansible_forks": 10,
        "ansible_inventory_sources": [
            "/home/nicolas/git/kub/k8s-leno/inventory/leno"
        ],
        "ansible_playbook_python": "/home/nicolas/ansible/bin/python3",
        "ansible_python_interpreter": "/home/nicolas/ansible/bin/python3",
        "ansible_user": "core",
        "ansible_verbosity": 0,
        "ansible_version": {
            "full": "2.16.2",
            "major": 2,
            "minor": 16,
            "revision": 2,
            "string": "2.16.2"
        },
        "coreos_zincati_disable": true,
        "download_container": false,
        "download_localhost": true,
        "download_run_once": true,
        "group_names": [
            "ungrouped"
        ],
        "groups": {
            "all": [
                "localhost",
                "lenop",
                "leno1",
                "leno2",
                "leno3",
                "leno4"
            ],
            "calico_rr": [],
            "etcd": [
                "lenop",
                "leno3",
                "leno4"
            ],
            "k8s_cluster": [
                "lenop",
                "leno3",
                "leno4",
                "leno1",
                "leno2"
            ],
            "kube_control_plane": [
                "lenop",
                "leno3",
                "leno4"
            ],
            "kube_node": [
                "leno1",
                "leno2",
                "leno3",
                "leno4",
                "lenop"
            ],
            "kube_storage_nodes": [
                "leno1",
                "leno2",
                "leno3",
                "leno4"
            ],
            "metadata": [],
            "ungrouped": [
                "localhost"
            ]
        },
        "if_name": "eth0",
        "inventory_dir": "/home/nicolas/git/kub/k8s-leno/inventory/leno",
        "inventory_file": "/home/nicolas/git/kub/k8s-leno/inventory/leno/hosts.yml",
        "inventory_hostname": "localhost",
        "inventory_hostname_short": "localhost",
        "ip": "{{ ansible_host }}",
        "is_fedora_coreos": true,
        "kube_vip_address": "192.168.1.250",
        "kube_vip_services": "{{ kube_lb_addresses }}",
        "ntp_manage_config": true,
        "ntp_servers": [
            "0.fr.pool.ntp.org iburst",
            "1.fr.pool.ntp.org iburst",
            "2.fr.pool.ntp.org iburst",
            "3.fr.pool.ntp.org iburst"
        ],
        "omit": "__omit_place_holder__6d1590d3cd4c0ff5954bce6dd24b6053512da6ee",
        "ping_access_ip": false,
        "playbook_dir": "/home/nicolas/git/kub/k8s-leno",
        "service_pools": {
            "default": [
                "{{ kube_lb_addresses }}"
            ]
        },
        "sysctl_file_path": "/etc/sysctl.d/99-kubernetes.conf",
        "upstream_dns_servers": [
            "192.168.1.1"
        ]
    }
}

leno1 | SUCCESS => 

{
    "hostvars[inventory_hostname]": {
        "access_ip": "192.168.1.251",
        "additional_sysctl": [
            {
                "name": "vm.max_map_count",
                "value": 262144
            }
        ],
        "ansible_check_mode": false,
        "ansible_config_file": "/home/nicolas/git/kub/k8s-leno/ansible.cfg",
        "ansible_diff_mode": false,
        "ansible_facts": {},
        "ansible_forks": 10,
        "ansible_host": "192.168.1.251",
        "ansible_inventory_sources": [
            "/home/nicolas/git/kub/k8s-leno/inventory/leno"
        ],
        "ansible_playbook_python": "/home/nicolas/ansible/bin/python3",
        "ansible_user": "core",
        "ansible_verbosity": 0,
        "ansible_version": {
            "full": "2.16.2",
            "major": 2,
            "minor": 16,
            "revision": 2,
            "string": "2.16.2"
        },
        "audit_log_maxage": 7,
        "audit_log_maxbackups": 10,
        "audit_log_maxsize": 100,
        "audit_log_path": "/var/log/audit/kube-apiserver-audit.json",
        "authorization_modes": [
            "Node",
            "RBAC"
        ],
        "auto_renew_certificates": true,
        "coreos_zincati_disable": true,
        "dashboard_enabled": false,
        "dns_mode": "manual",
        "download_container": false,
        "download_localhost": true,
        "download_run_once": true,
        "drain_grace_period": 300,
        "drain_retries": 3,
        "drain_timeout": "600s",
        "enable_nodelocaldns": false,
        "etcd_deployment_type": "kubeadm",
        "etcd_metrics_port": 2381,
        "etcd_metrics_service_labels": {
            "app": "kube-prometheus-stack-kube-etcd",
            "app.kubernetes.io/managed-by": "Kubespray",
            "k8s-app": "etcd",
            "release": "prometheus-stack"
        },
        "group_names": [
            "k8s_cluster",
            "kube_node",
            "kube_storage_nodes"
        ],
        "groups": {
            "all": [
                "localhost",
                "lenop",
                "leno1",
                "leno2",
                "leno3",
                "leno4"
            ],
            "calico_rr": [],
            "etcd": [
                "lenop",
                "leno3",
                "leno4"
            ],
            "k8s_cluster": [
                "lenop",
                "leno3",
                "leno4",
                "leno1",
                "leno2"
            ],
            "kube_control_plane": [
                "lenop",
                "leno3",
                "leno4"
            ],
            "kube_node": [
                "leno1",
                "leno2",
                "leno3",
                "leno4",
                "lenop"
            ],
            "kube_storage_nodes": [
                "leno1",
                "leno2",
                "leno3",
                "leno4"
            ],
            "metadata": [],
            "ungrouped": [
                "localhost"
            ]
        },
        "if_name": "eth0",
        "inventory_dir": "/home/nicolas/git/kub/k8s-leno/inventory/leno",
        "inventory_file": "/home/nicolas/git/kub/k8s-leno/inventory/leno/hosts.yml",
        "inventory_hostname": "leno1",
        "inventory_hostname_short": "leno1",
        "ip": "192.168.1.251",
        "is_fedora_coreos": true,
        "is_storage_node": true,
        "kube_apiserver_admission_control_config_file": true,
        "kube_apiserver_admission_event_rate_limits": {
            "limit_1": {
                "burst": 1000,
                "cache_size": 4000,
                "qps": 100,
                "type": "Namespace"
            },
            "limit_2": {
                "burst": 1000,
                "qps": 100,
                "type": "User"
            }
        },
        "kube_apiserver_enable_admission_plugins": [
            "EventRateLimit",
            "AlwaysPullImages",
            "ServiceAccount",
            "NamespaceLifecycle",
            "NodeRestriction",
            "LimitRanger",
            "ResourceQuota",
            "MutatingAdmissionWebhook",
            "ValidatingAdmissionWebhook",
            "PodNodeSelector",
            "PodSecurity"
        ],
        "kube_apiserver_request_timeout": "120s",
        "kube_apiserver_service_account_lookup": true,
        "kube_cert_group": "root",
        "kube_controller_feature_gates": [
            "RotateKubeletServerCertificate=true"
        ],
        "kube_controller_manager_bind_address": "127.0.0.1",
        "kube_controller_terminated_pod_gc_threshold": 50,
        "kube_encrypt_secret_data": true,
        "kube_encryption_algorithm": "secretbox",
        "kube_encryption_resources": [
            "secrets"
        ],
        "kube_lb_addresses": "192.168.1.160/27",
        "kube_master_cpu_reserved": "200m",
        "kube_master_memory_reserved": "128Mi",
        "kube_network_node_prefix": 24,
        "kube_network_plugin": "cni",
        "kube_network_plugin_multus": false,
        "kube_owner": "root",
        "kube_pod_security_default_enforce": "restricted",
        "kube_pod_security_use_default": true,
        "kube_pods_subnet": "10.20.64.0/18",
        "kube_profiling": false,
        "kube_proxy_remove": true,
        "kube_proxy_strict_arp": true,
        "kube_read_only_port": 0,
        "kube_scheduler_bind_address": "127.0.0.1",
        "kube_service_addresses": "10.20.0.0/18",
        "kube_vip_address": "192.168.1.250",
        "kube_vip_arp_enabled": true,
        "kube_vip_controlplane_enabled": true,
        "kube_vip_enabled": true,
        "kube_vip_services": "192.168.1.160/27",
        "kube_vip_services_enabled": false,
        "kubeadm_feature_gates": [
            "PublicKeysECDSA=true",
            "EtcdLearnerMode=true"
        ],
        "kubeadm_scale_down_coredns_enabled": false,
        "kubeconfig_localhost": true,
        "kubectl_localhost": true,
        "kubelet_authentication_token_webhook": true,
        "kubelet_authorization_mode_webhook": true,
        "kubelet_csr_approver_enabled": false,
        "kubelet_event_record_qps": 1,
        "kubelet_feature_gates": [
            "RotateKubeletServerCertificate=true",
            "SeccompDefault=true"
        ],
        "kubelet_image_gc_high_threshold": 90,
        "kubelet_image_gc_low_threshold": 60,
        "kubelet_make_iptables_util_chains": false,
        "kubelet_max_pods": 250,
        "kubelet_protect_kernel_defaults": true,
        "kubelet_rotate_certificates": true,
        "kubelet_rotate_server_certificates": true,
        "kubelet_streaming_connection_idle_timeout": "5m",
        "kubernetes_audit": false,
        "loadbalancer_apiserver": {
            "address": "192.168.1.250",
            "port": 6443
        },
        "loadbalancer_apiserver_localhost": false,
        "manual_dns_server": "10.20.0.10",
        "node_labels": {
            "node-role.kubernetes.io/storage": "true",
            "node.longhorn.io/create-default-disk": "true"
        },
        "ntp_manage_config": true,
        "ntp_servers": [
            "0.fr.pool.ntp.org iburst",
            "1.fr.pool.ntp.org iburst",
            "2.fr.pool.ntp.org iburst",
            "3.fr.pool.ntp.org iburst"
        ],
        "omit": "__omit_place_holder__6d1590d3cd4c0ff5954bce6dd24b6053512da6ee",
        "ping_access_ip": false,
        "playbook_dir": "/home/nicolas/git/kub/k8s-leno",
        "resolvconf_mode": "none",
        "resolved_conf_customization": [
            "LLMNR=no",
            "MulticastDNS=no",
            "ReadEtcHosts=yes"
        ],
        "service_pools": {
            "default": [
                "192.168.1.160/27"
            ]
        },
        "sysctl_file_path": "/etc/sysctl.d/99-kubernetes.conf",
        "system_cpu_reserved": "200m",
        "system_memory_reserved": "128Mi",
        "tls_cipher_suites": [
            "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256",
            "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
            "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305"
        ],
        "tls_min_version": "VersionTLS12",
        "upstream_dns_servers": [
            "192.168.1.1"
        ]
    }
}

lenop | SUCCESS => 

{
    "hostvars[inventory_hostname]": {
        "access_ip": "192.168.1.254",
        "additional_sysctl": [
            {
                "name": "vm.max_map_count",
                "value": 262144
            }
        ],
        "ansible_check_mode": false,
        "ansible_config_file": "/home/nicolas/git/kub/k8s-leno/ansible.cfg",
        "ansible_diff_mode": false,
        "ansible_facts": {},
        "ansible_forks": 10,
        "ansible_host": "192.168.1.254",
        "ansible_inventory_sources": [
            "/home/nicolas/git/kub/k8s-leno/inventory/leno"
        ],
        "ansible_playbook_python": "/home/nicolas/ansible/bin/python3",
        "ansible_user": "core",
        "ansible_verbosity": 0,
        "ansible_version": {
            "full": "2.16.2",
            "major": 2,
            "minor": 16,
            "revision": 2,
            "string": "2.16.2"
        },
        "audit_log_maxage": 7,
        "audit_log_maxbackups": 10,
        "audit_log_maxsize": 100,
        "audit_log_path": "/var/log/audit/kube-apiserver-audit.json",
        "authorization_modes": [
            "Node",
            "RBAC"
        ],
        "auto_renew_certificates": true,
        "coreos_zincati_disable": true,
        "dashboard_enabled": false,
        "dns_etchosts": "{% for item in (groups['k8s_cluster'] + groups['etcd']|default([]) + groups['calico_rr']|default([]))|unique -%}\n{% if 'access_ip' in hostvars[item] or 'ip' in hostvars[item] or 'ansible_default_ipv4' in hostvars[item] -%}\n{{ hostvars[item]['access_ip'] | default(hostvars[item]['ip'] | default(hostvars[item]['ansible_default_ipv4']['address'])) }}\n{%- if ('ansible_hostname' in hostvars[item] and item != hostvars[item]['ansible_hostname']) %} {{ hostvars[item]['ansible_hostname'] }}.{{ dns_domain }} {{ hostvars[item]['ansible_hostname'] }} {% else %} {{ item }}.{{ dns_domain }} {{ item }} {% endif %}\n\n{% endif %}\n{% endfor %}",
        "dns_mode": "manual",
        "download_container": false,
        "download_localhost": true,
        "download_run_once": true,
        "drain_grace_period": 300,
        "drain_retries": 3,
        "drain_timeout": "600s",
        "enable_nodelocaldns": false,
        "etcd_deployment_type": "kubeadm",
        "etcd_metrics_port": 2381,
        "etcd_metrics_service_labels": {
            "app": "kube-prometheus-stack-kube-etcd",
            "app.kubernetes.io/managed-by": "Kubespray",
            "k8s-app": "etcd",
            "release": "prometheus-stack"
        },
        "group_names": [
            "etcd",
            "k8s_cluster",
            "kube_control_plane",
            "kube_node"
        ],
        "groups": {
            "all": [
                "localhost",
                "lenop",
                "leno1",
                "leno2",
                "leno3",
                "leno4"
            ],
            "calico_rr": [],
            "etcd": [
                "lenop",
                "leno3",
                "leno4"
            ],
            "k8s_cluster": [
                "lenop",
                "leno3",
                "leno4",
                "leno1",
                "leno2"
            ],
            "kube_control_plane": [
                "lenop",
                "leno3",
                "leno4"
            ],
            "kube_node": [
                "leno1",
                "leno2",
                "leno3",
                "leno4",
                "lenop"
            ],
            "kube_storage_nodes": [
                "leno1",
                "leno2",
                "leno3",
                "leno4"
            ],
            "metadata": [],
            "ungrouped": [
                "localhost"
            ]
        },
        "if_name": "eth0",
        "inventory_dir": "/home/nicolas/git/kub/k8s-leno/inventory/leno",
        "inventory_file": "/home/nicolas/git/kub/k8s-leno/inventory/leno/hosts.yml",
        "inventory_hostname": "lenop",
        "inventory_hostname_short": "lenop",
        "ip": "192.168.1.254",
        "is_fedora_coreos": true,
        "kube_apiserver_admission_control_config_file": true,
        "kube_apiserver_admission_event_rate_limits": {
            "limit_1": {
                "burst": 1000,
                "cache_size": 4000,
                "qps": 100,
                "type": "Namespace"
            },
            "limit_2": {
                "burst": 1000,
                "qps": 100,
                "type": "User"
            }
        },
        "kube_apiserver_enable_admission_plugins": [
            "EventRateLimit",
            "AlwaysPullImages",
            "ServiceAccount",
            "NamespaceLifecycle",
            "NodeRestriction",
            "LimitRanger",
            "ResourceQuota",
            "MutatingAdmissionWebhook",
            "ValidatingAdmissionWebhook",
            "PodNodeSelector",
            "PodSecurity"
        ],
        "kube_apiserver_request_timeout": "120s",
        "kube_apiserver_service_account_lookup": true,
        "kube_cert_group": "root",
        "kube_controller_feature_gates": [
            "RotateKubeletServerCertificate=true"
        ],
        "kube_controller_manager_bind_address": "127.0.0.1",
        "kube_controller_terminated_pod_gc_threshold": 50,
        "kube_encrypt_secret_data": true,
        "kube_encryption_algorithm": "secretbox",
        "kube_encryption_resources": [
            "secrets"
        ],
        "kube_lb_addresses": "192.168.1.160/27",
        "kube_master_cpu_reserved": "200m",
        "kube_master_memory_reserved": "128Mi",
        "kube_network_node_prefix": 24,
        "kube_network_plugin": "cni",
        "kube_network_plugin_multus": false,
        "kube_owner": "root",
        "kube_pod_security_default_enforce": "restricted",
        "kube_pod_security_use_default": true,
        "kube_pods_subnet": "10.20.64.0/18",
        "kube_profiling": false,
        "kube_proxy_remove": true,
        "kube_proxy_strict_arp": true,
        "kube_read_only_port": 0,
        "kube_scheduler_bind_address": "127.0.0.1",
        "kube_service_addresses": "10.20.0.0/18",
        "kube_vip_address": "192.168.1.250",
        "kube_vip_arp_enabled": true,
        "kube_vip_controlplane_enabled": true,
        "kube_vip_enabled": true,
        "kube_vip_services": "192.168.1.160/27",
        "kube_vip_services_enabled": false,
        "kubeadm_feature_gates": [
            "PublicKeysECDSA=true",
            "EtcdLearnerMode=true"
        ],
        "kubeadm_scale_down_coredns_enabled": false,
        "kubeconfig_localhost": true,
        "kubectl_localhost": true,
        "kubelet_authentication_token_webhook": true,
        "kubelet_authorization_mode_webhook": true,
        "kubelet_csr_approver_enabled": false,
        "kubelet_event_record_qps": 1,
        "kubelet_feature_gates": [
            "RotateKubeletServerCertificate=true",
            "SeccompDefault=true"
        ],
        "kubelet_image_gc_high_threshold": 90,
        "kubelet_image_gc_low_threshold": 60,
        "kubelet_make_iptables_util_chains": false,
        "kubelet_max_pods": 250,
        "kubelet_protect_kernel_defaults": true,
        "kubelet_rotate_certificates": true,
        "kubelet_rotate_server_certificates": true,
        "kubelet_streaming_connection_idle_timeout": "5m",
        "kubernetes_audit": false,
        "loadbalancer_apiserver": {
            "address": "192.168.1.250",
            "port": 6443
        },
        "loadbalancer_apiserver_localhost": false,
        "manual_dns_server": "10.20.0.10",
        "ntp_manage_config": true,
        "ntp_servers": [
            "0.fr.pool.ntp.org iburst",
            "1.fr.pool.ntp.org iburst",
            "2.fr.pool.ntp.org iburst",
            "3.fr.pool.ntp.org iburst"
        ],
        "omit": "__omit_place_holder__6d1590d3cd4c0ff5954bce6dd24b6053512da6ee",
        "ping_access_ip": false,
        "playbook_dir": "/home/nicolas/git/kub/k8s-leno",
        "resolvconf_mode": "none",
        "resolved_conf_customization": [
            "LLMNR=no",
            "MulticastDNS=no",
            "ReadEtcHosts=yes"
        ],
        "service_pools": {
            "default": [
                "192.168.1.160/27"
            ]
        },
        "sysctl_file_path": "/etc/sysctl.d/99-kubernetes.conf",
        "system_cpu_reserved": "200m",
        "system_memory_reserved": "128Mi",
        "tls_cipher_suites": [
            "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256",
            "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
            "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305"
        ],
        "tls_min_version": "VersionTLS12",
        "upstream_dns_servers": [
            "192.168.1.1"
        ]
    }
}

leno2 | SUCCESS => 

{
    "hostvars[inventory_hostname]": {
        "access_ip": "192.168.1.252",
        "additional_sysctl": [
            {
                "name": "vm.max_map_count",
                "value": 262144
            }
        ],
        "ansible_check_mode": false,
        "ansible_config_file": "/home/nicolas/git/kub/k8s-leno/ansible.cfg",
        "ansible_diff_mode": false,
        "ansible_facts": {},
        "ansible_forks": 10,
        "ansible_host": "192.168.1.252",
        "ansible_inventory_sources": [
            "/home/nicolas/git/kub/k8s-leno/inventory/leno"
        ],
        "ansible_playbook_python": "/home/nicolas/ansible/bin/python3",
        "ansible_user": "core",
        "ansible_verbosity": 0,
        "ansible_version": {
            "full": "2.16.2",
            "major": 2,
            "minor": 16,
            "revision": 2,
            "string": "2.16.2"
        },
        "audit_log_maxage": 7,
        "audit_log_maxbackups": 10,
        "audit_log_maxsize": 100,
        "audit_log_path": "/var/log/audit/kube-apiserver-audit.json",
        "authorization_modes": [
            "Node",
            "RBAC"
        ],
        "auto_renew_certificates": true,
        "coreos_zincati_disable": true,
        "dashboard_enabled": false,
        "dns_mode": "manual",
        "download_container": false,
        "download_localhost": true,
        "download_run_once": true,
        "drain_grace_period": 300,
        "drain_retries": 3,
        "drain_timeout": "600s",
        "enable_nodelocaldns": false,
        "etcd_deployment_type": "kubeadm",
        "etcd_metrics_port": 2381,
        "etcd_metrics_service_labels": {
            "app": "kube-prometheus-stack-kube-etcd",
            "app.kubernetes.io/managed-by": "Kubespray",
            "k8s-app": "etcd",
            "release": "prometheus-stack"
        },
        "group_names": [
            "k8s_cluster",
            "kube_node",
            "kube_storage_nodes"
        ],
        "groups": {
            "all": [
                "localhost",
                "lenop",
                "leno1",
                "leno2",
                "leno3",
                "leno4"
            ],
            "calico_rr": [],
            "etcd": [
                "lenop",
                "leno3",
                "leno4"
            ],
            "k8s_cluster": [
                "lenop",
                "leno3",
                "leno4",
                "leno1",
                "leno2"
            ],
            "kube_control_plane": [
                "lenop",
                "leno3",
                "leno4"
            ],
            "kube_node": [
                "leno1",
                "leno2",
                "leno3",
                "leno4",
                "lenop"
            ],
            "kube_storage_nodes": [
                "leno1",
                "leno2",
                "leno3",
                "leno4"
            ],
            "metadata": [],
            "ungrouped": [
                "localhost"
            ]
        },
        "if_name": "eth0",
        "inventory_dir": "/home/nicolas/git/kub/k8s-leno/inventory/leno",
        "inventory_file": "/home/nicolas/git/kub/k8s-leno/inventory/leno/hosts.yml",
        "inventory_hostname": "leno2",
        "inventory_hostname_short": "leno2",
        "ip": "192.168.1.252",
        "is_fedora_coreos": true,
        "is_storage_node": true,
        "kube_apiserver_admission_control_config_file": true,
        "kube_apiserver_admission_event_rate_limits": {
            "limit_1": {
                "burst": 1000,
                "cache_size": 4000,
                "qps": 100,
                "type": "Namespace"
            },
            "limit_2": {
                "burst": 1000,
                "qps": 100,
                "type": "User"
            }
        },
        "kube_apiserver_enable_admission_plugins": [
            "EventRateLimit",
            "AlwaysPullImages",
            "ServiceAccount",
            "NamespaceLifecycle",
            "NodeRestriction",
            "LimitRanger",
            "ResourceQuota",
            "MutatingAdmissionWebhook",
            "ValidatingAdmissionWebhook",
            "PodNodeSelector",
            "PodSecurity"
        ],
        "kube_apiserver_request_timeout": "120s",
        "kube_apiserver_service_account_lookup": true,
        "kube_cert_group": "root",
        "kube_controller_feature_gates": [
            "RotateKubeletServerCertificate=true"
        ],
        "kube_controller_manager_bind_address": "127.0.0.1",
        "kube_controller_terminated_pod_gc_threshold": 50,
        "kube_encrypt_secret_data": true,
        "kube_encryption_algorithm": "secretbox",
        "kube_encryption_resources": [
            "secrets"
        ],
        "kube_lb_addresses": "192.168.1.160/27",
        "kube_master_cpu_reserved": "200m",
        "kube_master_memory_reserved": "128Mi",
        "kube_network_node_prefix": 24,
        "kube_network_plugin": "cni",
        "kube_network_plugin_multus": false,
        "kube_owner": "root",
        "kube_pod_security_default_enforce": "restricted",
        "kube_pod_security_use_default": true,
        "kube_pods_subnet": "10.20.64.0/18",
        "kube_profiling": false,
        "kube_proxy_remove": true,
        "kube_proxy_strict_arp": true,
        "kube_read_only_port": 0,
        "kube_scheduler_bind_address": "127.0.0.1",
        "kube_service_addresses": "10.20.0.0/18",
        "kube_vip_address": "192.168.1.250",
        "kube_vip_arp_enabled": true,
        "kube_vip_controlplane_enabled": true,
        "kube_vip_enabled": true,
        "kube_vip_services": "192.168.1.160/27",
        "kube_vip_services_enabled": false,
        "kubeadm_feature_gates": [
            "PublicKeysECDSA=true",
            "EtcdLearnerMode=true"
        ],
        "kubeadm_scale_down_coredns_enabled": false,
        "kubeconfig_localhost": true,
        "kubectl_localhost": true,
        "kubelet_authentication_token_webhook": true,
        "kubelet_authorization_mode_webhook": true,
        "kubelet_csr_approver_enabled": false,
        "kubelet_event_record_qps": 1,
        "kubelet_feature_gates": [
            "RotateKubeletServerCertificate=true",
            "SeccompDefault=true"
        ],
        "kubelet_image_gc_high_threshold": 90,
        "kubelet_image_gc_low_threshold": 60,
        "kubelet_make_iptables_util_chains": false,
        "kubelet_max_pods": 250,
        "kubelet_protect_kernel_defaults": true,
        "kubelet_rotate_certificates": true,
        "kubelet_rotate_server_certificates": true,
        "kubelet_streaming_connection_idle_timeout": "5m",
        "kubernetes_audit": false,
        "loadbalancer_apiserver": {
            "address": "192.168.1.250",
            "port": 6443
        },
        "loadbalancer_apiserver_localhost": false,
        "manual_dns_server": "10.20.0.10",
        "node_labels": {
            "node-role.kubernetes.io/storage": "true",
            "node.longhorn.io/create-default-disk": "true"
        },
        "ntp_manage_config": true,
        "ntp_servers": [
            "0.fr.pool.ntp.org iburst",
            "1.fr.pool.ntp.org iburst",
            "2.fr.pool.ntp.org iburst",
            "3.fr.pool.ntp.org iburst"
        ],
        "omit": "__omit_place_holder__6d1590d3cd4c0ff5954bce6dd24b6053512da6ee",
        "ping_access_ip": false,
        "playbook_dir": "/home/nicolas/git/kub/k8s-leno",
        "resolvconf_mode": "none",
        "resolved_conf_customization": [
            "LLMNR=no",
            "MulticastDNS=no",
            "ReadEtcHosts=yes"
        ],
        "service_pools": {
            "default": [
                "192.168.1.160/27"
            ]
        },
        "sysctl_file_path": "/etc/sysctl.d/99-kubernetes.conf",
        "system_cpu_reserved": "200m",
        "system_memory_reserved": "128Mi",
        "tls_cipher_suites": [
            "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256",
            "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
            "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305"
        ],
        "tls_min_version": "VersionTLS12",
        "upstream_dns_servers": [
            "192.168.1.1"
        ]
    }
}

leno4 | SUCCESS => 

{
    "hostvars[inventory_hostname]": {
        "access_ip": "192.168.1.247",
        "additional_sysctl": [
            {
                "name": "vm.max_map_count",
                "value": 262144
            }
        ],
        "ansible_check_mode": false,
        "ansible_config_file": "/home/nicolas/git/kub/k8s-leno/ansible.cfg",
        "ansible_diff_mode": false,
        "ansible_facts": {},
        "ansible_forks": 10,
        "ansible_host": "192.168.1.247",
        "ansible_inventory_sources": [
            "/home/nicolas/git/kub/k8s-leno/inventory/leno"
        ],
        "ansible_playbook_python": "/home/nicolas/ansible/bin/python3",
        "ansible_user": "core",
        "ansible_verbosity": 0,
        "ansible_version": {
            "full": "2.16.2",
            "major": 2,
            "minor": 16,
            "revision": 2,
            "string": "2.16.2"
        },
        "audit_log_maxage": 7,
        "audit_log_maxbackups": 10,
        "audit_log_maxsize": 100,
        "audit_log_path": "/var/log/audit/kube-apiserver-audit.json",
        "authorization_modes": [
            "Node",
            "RBAC"
        ],
        "auto_renew_certificates": true,
        "coreos_zincati_disable": true,
        "dashboard_enabled": false,
        "dns_etchosts": "{% for item in (groups['k8s_cluster'] + groups['etcd']|default([]) + groups['calico_rr']|default([]))|unique -%}\n{% if 'access_ip' in hostvars[item] or 'ip' in hostvars[item] or 'ansible_default_ipv4' in hostvars[item] -%}\n{{ hostvars[item]['access_ip'] | default(hostvars[item]['ip'] | default(hostvars[item]['ansible_default_ipv4']['address'])) }}\n{%- if ('ansible_hostname' in hostvars[item] and item != hostvars[item]['ansible_hostname']) %} {{ hostvars[item]['ansible_hostname'] }}.{{ dns_domain }} {{ hostvars[item]['ansible_hostname'] }} {% else %} {{ item }}.{{ dns_domain }} {{ item }} {% endif %}\n\n{% endif %}\n{% endfor %}",
        "dns_mode": "manual",
        "download_container": false,
        "download_localhost": true,
        "download_run_once": true,
        "drain_grace_period": 300,
        "drain_retries": 3,
        "drain_timeout": "600s",
        "enable_nodelocaldns": false,
        "etcd_deployment_type": "kubeadm",
        "etcd_metrics_port": 2381,
        "etcd_metrics_service_labels": {
            "app": "kube-prometheus-stack-kube-etcd",
            "app.kubernetes.io/managed-by": "Kubespray",
            "k8s-app": "etcd",
            "release": "prometheus-stack"
        },
        "group_names": [
            "etcd",
            "k8s_cluster",
            "kube_control_plane",
            "kube_node",
            "kube_storage_nodes"
        ],
        "groups": {
            "all": [
                "localhost",
                "lenop",
                "leno1",
                "leno2",
                "leno3",
                "leno4"
            ],
            "calico_rr": [],
            "etcd": [
                "lenop",
                "leno3",
                "leno4"
            ],
            "k8s_cluster": [
                "lenop",
                "leno3",
                "leno4",
                "leno1",
                "leno2"
            ],
            "kube_control_plane": [
                "lenop",
                "leno3",
                "leno4"
            ],
            "kube_node": [
                "leno1",
                "leno2",
                "leno3",
                "leno4",
                "lenop"
            ],
            "kube_storage_nodes": [
                "leno1",
                "leno2",
                "leno3",
                "leno4"
            ],
            "metadata": [],
            "ungrouped": [
                "localhost"
            ]
        },
        "if_name": "eth0",
        "inventory_dir": "/home/nicolas/git/kub/k8s-leno/inventory/leno",
        "inventory_file": "/home/nicolas/git/kub/k8s-leno/inventory/leno/hosts.yml",
        "inventory_hostname": "leno4",
        "inventory_hostname_short": "leno4",
        "ip": "192.168.1.247",
        "is_fedora_coreos": true,
        "is_storage_node": true,
        "kube_apiserver_admission_control_config_file": true,
        "kube_apiserver_admission_event_rate_limits": {
            "limit_1": {
                "burst": 1000,
                "cache_size": 4000,
                "qps": 100,
                "type": "Namespace"
            },
            "limit_2": {
                "burst": 1000,
                "qps": 100,
                "type": "User"
            }
        },
        "kube_apiserver_enable_admission_plugins": [
            "EventRateLimit",
            "AlwaysPullImages",
            "ServiceAccount",
            "NamespaceLifecycle",
            "NodeRestriction",
            "LimitRanger",
            "ResourceQuota",
            "MutatingAdmissionWebhook",
            "ValidatingAdmissionWebhook",
            "PodNodeSelector",
            "PodSecurity"
        ],
        "kube_apiserver_request_timeout": "120s",
        "kube_apiserver_service_account_lookup": true,
        "kube_cert_group": "root",
        "kube_controller_feature_gates": [
            "RotateKubeletServerCertificate=true"
        ],
        "kube_controller_manager_bind_address": "127.0.0.1",
        "kube_controller_terminated_pod_gc_threshold": 50,
        "kube_encrypt_secret_data": true,
        "kube_encryption_algorithm": "secretbox",
        "kube_encryption_resources": [
            "secrets"
        ],
        "kube_lb_addresses": "192.168.1.160/27",
        "kube_master_cpu_reserved": "200m",
        "kube_master_memory_reserved": "128Mi",
        "kube_network_node_prefix": 24,
        "kube_network_plugin": "cni",
        "kube_network_plugin_multus": false,
        "kube_owner": "root",
        "kube_pod_security_default_enforce": "restricted",
        "kube_pod_security_use_default": true,
        "kube_pods_subnet": "10.20.64.0/18",
        "kube_profiling": false,
        "kube_proxy_remove": true,
        "kube_proxy_strict_arp": true,
        "kube_read_only_port": 0,
        "kube_scheduler_bind_address": "127.0.0.1",
        "kube_service_addresses": "10.20.0.0/18",
        "kube_vip_address": "192.168.1.250",
        "kube_vip_arp_enabled": true,
        "kube_vip_controlplane_enabled": true,
        "kube_vip_enabled": true,
        "kube_vip_services": "192.168.1.160/27",
        "kube_vip_services_enabled": false,
        "kubeadm_feature_gates": [
            "PublicKeysECDSA=true",
            "EtcdLearnerMode=true"
        ],
        "kubeadm_scale_down_coredns_enabled": false,
        "kubeconfig_localhost": true,
        "kubectl_localhost": true,
        "kubelet_authentication_token_webhook": true,
        "kubelet_authorization_mode_webhook": true,
        "kubelet_csr_approver_enabled": false,
        "kubelet_event_record_qps": 1,
        "kubelet_feature_gates": [
            "RotateKubeletServerCertificate=true",
            "SeccompDefault=true"
        ],
        "kubelet_image_gc_high_threshold": 90,
        "kubelet_image_gc_low_threshold": 60,
        "kubelet_make_iptables_util_chains": false,
        "kubelet_max_pods": 250,
        "kubelet_protect_kernel_defaults": true,
        "kubelet_rotate_certificates": true,
        "kubelet_rotate_server_certificates": true,
        "kubelet_streaming_connection_idle_timeout": "5m",
        "kubernetes_audit": false,
        "loadbalancer_apiserver": {
            "address": "192.168.1.250",
            "port": 6443
        },
        "loadbalancer_apiserver_localhost": false,
        "manual_dns_server": "10.20.0.10",
        "node_labels": {
            "node-role.kubernetes.io/storage": "true",
            "node.longhorn.io/create-default-disk": "true"
        },
        "ntp_manage_config": true,
        "ntp_servers": [
            "0.fr.pool.ntp.org iburst",
            "1.fr.pool.ntp.org iburst",
            "2.fr.pool.ntp.org iburst",
            "3.fr.pool.ntp.org iburst"
        ],
        "omit": "__omit_place_holder__6d1590d3cd4c0ff5954bce6dd24b6053512da6ee",
        "ping_access_ip": false,
        "playbook_dir": "/home/nicolas/git/kub/k8s-leno",
        "resolvconf_mode": "none",
        "resolved_conf_customization": [
            "LLMNR=no",
            "MulticastDNS=no",
            "ReadEtcHosts=yes"
        ],
        "service_pools": {
            "default": [
                "192.168.1.160/27"
            ]
        },
        "sysctl_file_path": "/etc/sysctl.d/99-kubernetes.conf",
        "system_cpu_reserved": "200m",
        "system_memory_reserved": "128Mi",
        "tls_cipher_suites": [
            "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256",
            "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
            "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305"
        ],
        "tls_min_version": "VersionTLS12",
        "upstream_dns_servers": [
            "192.168.1.1"
        ]
    }
}

leno3 | SUCCESS => 

{
    "hostvars[inventory_hostname]": {
        "access_ip": "192.168.1.253",
        "additional_sysctl": [
            {
                "name": "vm.max_map_count",
                "value": 262144
            }
        ],
        "ansible_check_mode": false,
        "ansible_config_file": "/home/nicolas/git/kub/k8s-leno/ansible.cfg",
        "ansible_diff_mode": false,
        "ansible_facts": {},
        "ansible_forks": 10,
        "ansible_host": "192.168.1.253",
        "ansible_inventory_sources": [
            "/home/nicolas/git/kub/k8s-leno/inventory/leno"
        ],
        "ansible_playbook_python": "/home/nicolas/ansible/bin/python3",
        "ansible_user": "core",
        "ansible_verbosity": 0,
        "ansible_version": {
            "full": "2.16.2",
            "major": 2,
            "minor": 16,
            "revision": 2,
            "string": "2.16.2"
        },
        "audit_log_maxage": 7,
        "audit_log_maxbackups": 10,
        "audit_log_maxsize": 100,
        "audit_log_path": "/var/log/audit/kube-apiserver-audit.json",
        "authorization_modes": [
            "Node",
            "RBAC"
        ],
        "auto_renew_certificates": true,
        "coreos_zincati_disable": true,
        "dashboard_enabled": false,
        "dns_etchosts": "{% for item in (groups['k8s_cluster'] + groups['etcd']|default([]) + groups['calico_rr']|default([]))|unique -%}\n{% if 'access_ip' in hostvars[item] or 'ip' in hostvars[item] or 'ansible_default_ipv4' in hostvars[item] -%}\n{{ hostvars[item]['access_ip'] | default(hostvars[item]['ip'] | default(hostvars[item]['ansible_default_ipv4']['address'])) }}\n{%- if ('ansible_hostname' in hostvars[item] and item != hostvars[item]['ansible_hostname']) %} {{ hostvars[item]['ansible_hostname'] }}.{{ dns_domain }} {{ hostvars[item]['ansible_hostname'] }} {% else %} {{ item }}.{{ dns_domain }} {{ item }} {% endif %}\n\n{% endif %}\n{% endfor %}",
        "dns_mode": "manual",
        "download_container": false,
        "download_localhost": true,
        "download_run_once": true,
        "drain_grace_period": 300,
        "drain_retries": 3,
        "drain_timeout": "600s",
        "enable_nodelocaldns": false,
        "etcd_deployment_type": "kubeadm",
        "etcd_metrics_port": 2381,
        "etcd_metrics_service_labels": {
            "app": "kube-prometheus-stack-kube-etcd",
            "app.kubernetes.io/managed-by": "Kubespray",
            "k8s-app": "etcd",
            "release": "prometheus-stack"
        },
        "group_names": [
            "etcd",
            "k8s_cluster",
            "kube_control_plane",
            "kube_node",
            "kube_storage_nodes"
        ],
        "groups": {
            "all": [
                "localhost",
                "lenop",
                "leno1",
                "leno2",
                "leno3",
                "leno4"
            ],
            "calico_rr": [],
            "etcd": [
                "lenop",
                "leno3",
                "leno4"
            ],
            "k8s_cluster": [
                "lenop",
                "leno3",
                "leno4",
                "leno1",
                "leno2"
            ],
            "kube_control_plane": [
                "lenop",
                "leno3",
                "leno4"
            ],
            "kube_node": [
                "leno1",
                "leno2",
                "leno3",
                "leno4",
                "lenop"
            ],
            "kube_storage_nodes": [
                "leno1",
                "leno2",
                "leno3",
                "leno4"
            ],
            "metadata": [],
            "ungrouped": [
                "localhost"
            ]
        },
        "if_name": "eth0",
        "inventory_dir": "/home/nicolas/git/kub/k8s-leno/inventory/leno",
        "inventory_file": "/home/nicolas/git/kub/k8s-leno/inventory/leno/hosts.yml",
        "inventory_hostname": "leno3",
        "inventory_hostname_short": "leno3",
        "ip": "192.168.1.253",
        "is_fedora_coreos": true,
        "is_storage_node": true,
        "kube_apiserver_admission_control_config_file": true,
        "kube_apiserver_admission_event_rate_limits": {
            "limit_1": {
                "burst": 1000,
                "cache_size": 4000,
                "qps": 100,
                "type": "Namespace"
            },
            "limit_2": {
                "burst": 1000,
                "qps": 100,
                "type": "User"
            }
        },
        "kube_apiserver_enable_admission_plugins": [
            "EventRateLimit",
            "AlwaysPullImages",
            "ServiceAccount",
            "NamespaceLifecycle",
            "NodeRestriction",
            "LimitRanger",
            "ResourceQuota",
            "MutatingAdmissionWebhook",
            "ValidatingAdmissionWebhook",
            "PodNodeSelector",
            "PodSecurity"
        ],
        "kube_apiserver_request_timeout": "120s",
        "kube_apiserver_service_account_lookup": true,
        "kube_cert_group": "root",
        "kube_controller_feature_gates": [
            "RotateKubeletServerCertificate=true"
        ],
        "kube_controller_manager_bind_address": "127.0.0.1",
        "kube_controller_terminated_pod_gc_threshold": 50,
        "kube_encrypt_secret_data": true,
        "kube_encryption_algorithm": "secretbox",
        "kube_encryption_resources": [
            "secrets"
        ],
        "kube_lb_addresses": "192.168.1.160/27",
        "kube_master_cpu_reserved": "200m",
        "kube_master_memory_reserved": "128Mi",
        "kube_network_node_prefix": 24,
        "kube_network_plugin": "cni",
        "kube_network_plugin_multus": false,
        "kube_owner": "root",
        "kube_pod_security_default_enforce": "restricted",
        "kube_pod_security_use_default": true,
        "kube_pods_subnet": "10.20.64.0/18",
        "kube_profiling": false,
        "kube_proxy_remove": true,
        "kube_proxy_strict_arp": true,
        "kube_read_only_port": 0,
        "kube_scheduler_bind_address": "127.0.0.1",
        "kube_service_addresses": "10.20.0.0/18",
        "kube_vip_address": "192.168.1.250",
        "kube_vip_arp_enabled": true,
        "kube_vip_controlplane_enabled": true,
        "kube_vip_enabled": true,
        "kube_vip_services": "192.168.1.160/27",
        "kube_vip_services_enabled": false,
        "kubeadm_feature_gates": [
            "PublicKeysECDSA=true",
            "EtcdLearnerMode=true"
        ],
        "kubeadm_scale_down_coredns_enabled": false,
        "kubeconfig_localhost": true,
        "kubectl_localhost": true,
        "kubelet_authentication_token_webhook": true,
        "kubelet_authorization_mode_webhook": true,
        "kubelet_csr_approver_enabled": false,
        "kubelet_event_record_qps": 1,
        "kubelet_feature_gates": [
            "RotateKubeletServerCertificate=true",
            "SeccompDefault=true"
        ],
        "kubelet_image_gc_high_threshold": 90,
        "kubelet_image_gc_low_threshold": 60,
        "kubelet_make_iptables_util_chains": false,
        "kubelet_max_pods": 250,
        "kubelet_protect_kernel_defaults": true,
        "kubelet_rotate_certificates": true,
        "kubelet_rotate_server_certificates": true,
        "kubelet_streaming_connection_idle_timeout": "5m",
        "kubernetes_audit": false,
        "loadbalancer_apiserver": {
            "address": "192.168.1.250",
            "port": 6443
        },
        "loadbalancer_apiserver_localhost": false,
        "manual_dns_server": "10.20.0.10",
        "node_labels": {
            "node-role.kubernetes.io/storage": "true",
            "node.longhorn.io/create-default-disk": "true"
        },
        "ntp_manage_config": true,
        "ntp_servers": [
            "0.fr.pool.ntp.org iburst",
            "1.fr.pool.ntp.org iburst",
            "2.fr.pool.ntp.org iburst",
            "3.fr.pool.ntp.org iburst"
        ],
        "omit": "__omit_place_holder__6d1590d3cd4c0ff5954bce6dd24b6053512da6ee",
        "ping_access_ip": false,
        "playbook_dir": "/home/nicolas/git/kub/k8s-leno",
        "resolvconf_mode": "none",
        "resolved_conf_customization": [
            "LLMNR=no",
            "MulticastDNS=no",
            "ReadEtcHosts=yes"
        ],
        "service_pools": {
            "default": [
                "192.168.1.160/27"
            ]
        },
        "sysctl_file_path": "/etc/sysctl.d/99-kubernetes.conf",
        "system_cpu_reserved": "200m",
        "system_memory_reserved": "128Mi",
        "tls_cipher_suites": [
            "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256",
            "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
            "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305"
        ],
        "tls_min_version": "VersionTLS12",
        "upstream_dns_servers": [
            "192.168.1.1"
        ]
    }
}

Command used to invoke ansible

ansible-playbook playbooks/cluster.yml --limit=kube_control_plane --become

Output of ansible run

Only put then end, full log is too long.

TASK [kubernetes/control-plane : Create kubeadm token for joining nodes with 24h expiration (default)] ***************************************************************************************************************************
ok: [lenop -> leno4(192.168.1.247)]
ok: [leno3 -> leno4(192.168.1.247)]
ok: [leno4]
Saturday 02 March 2024  12:31:24 -0600 (0:00:01.743)       0:08:43.787 ******** 

TASK [kubernetes/control-plane : Set kubeadm_token] ******************************************************************************************************************************************************************************
ok: [lenop]
ok: [leno3]
ok: [leno4]
Saturday 02 March 2024  12:31:24 -0600 (0:00:00.113)       0:08:43.901 ******** 

TASK [kubernetes/control-plane : Kubeadm | Join other masters] *******************************************************************************************************************************************************************
included: /home/nicolas/git/kub/k8s-leno/kubespray/roles/kubernetes/control-plane/tasks/kubeadm-secondary.yml for lenop, leno3, leno4
Saturday 02 March 2024  12:31:24 -0600 (0:00:00.101)       0:08:44.002 ******** 

TASK [kubernetes/control-plane : Set kubeadm_discovery_address] ******************************************************************************************************************************************************************
ok: [lenop]
ok: [leno4]
ok: [leno3]
Saturday 02 March 2024  12:31:24 -0600 (0:00:00.184)       0:08:44.186 ******** 

TASK [kubernetes/control-plane : Upload certificates so they are fresh and not expired] ******************************************************************************************************************************************
changed: [leno4]
Saturday 02 March 2024  12:31:26 -0600 (0:00:01.115)       0:08:45.302 ******** 
Saturday 02 March 2024  12:31:26 -0600 (0:00:00.065)       0:08:45.368 ******** 

TASK [kubernetes/control-plane : Create kubeadm ControlPlane config] *************************************************************************************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ansible.errors.AnsibleUndefinedVariable: 'kubeadm_certificate_key' is undefined. 'kubeadm_certificate_key' is undefined
fatal: [leno3]: FAILED! => {"changed": false, "msg": "AnsibleUndefinedVariable: 'kubeadm_certificate_key' is undefined. 'kubeadm_certificate_key' is undefined"}

NO MORE HOSTS LEFT ***************************************************************************************************************************************************************************************************************

PLAY RECAP ***********************************************************************************************************************************************************************************************************************
leno3                      : ok=276  changed=79   unreachable=0    failed=1    skipped=385  rescued=0    ignored=3   
leno4                      : ok=278  changed=36   unreachable=0    failed=0    skipped=371  rescued=0    ignored=2   
lenop                      : ok=351  changed=45   unreachable=0    failed=0    skipped=410  rescued=0    ignored=3   

Saturday 02 March 2024  12:31:26 -0600 (0:00:00.189)       0:08:45.557 ******** 
=============================================================================== 
download : Download_file | Download item --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 37.07s
download : Download_file | Download item --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 19.99s
container-engine/containerd : Download_file | Copy file from cache to nodes, if it is available -------------------------------------------------------------------------------------------------------------------------- 13.63s
download : Download_file | Copy file from cache to nodes, if it is available --------------------------------------------------------------------------------------------------------------------------------------------- 12.43s
kubespray-defaults : Gather ansible_default_ipv4 from all hosts ---------------------------------------------------------------------------------------------------------------------------------------------------------- 11.72s
kubernetes/control-plane : Kubeadm | Check apiserver.crt SAN hosts -------------------------------------------------------------------------------------------------------------------------------------------------------- 9.73s
container-engine/containerd : Download_file | Download item --------------------------------------------------------------------------------------------------------------------------------------------------------------- 8.55s
download : Download_file | Download item ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 8.29s
container-engine/crictl : Download_file | Download item ------------------------------------------------------------------------------------------------------------------------------------------------------------------- 7.57s
container-engine/crictl : Download_file | Copy file from cache to nodes, if it is available ------------------------------------------------------------------------------------------------------------------------------- 7.56s
download : Download_file | Copy file from cache to nodes, if it is available ---------------------------------------------------------------------------------------------------------------------------------------------- 7.54s
container-engine/crictl : Extract_file | Unpacking archive ---------------------------------------------------------------------------------------------------------------------------------------------------------------- 7.16s
container-engine/containerd : Containerd | Unpack containerd archive ------------------------------------------------------------------------------------------------------------------------------------------------------ 7.02s
container-engine/validate-container-engine : Populate service facts ------------------------------------------------------------------------------------------------------------------------------------------------------- 6.78s
download : Download_file | Download item ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 6.27s
kubernetes/control-plane : Backup old certs and keys ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- 5.95s
kubernetes/preinstall : Ensure kube-bench parameters are set -------------------------------------------------------------------------------------------------------------------------------------------------------------- 5.67s
kubernetes/node : Install | Copy kubelet binary from download dir --------------------------------------------------------------------------------------------------------------------------------------------------------- 5.52s
container-engine/nerdctl : Extract_file | Unpacking archive --------------------------------------------------------------------------------------------------------------------------------------------------------------- 5.40s
kubernetes/control-plane : Kubeadm | Check apiserver.crt SAN IPs ---------------------------------------------------------------------------------------------------------------------------------------------------------- 5.26s

Anything else we need to know

When adding a new control-plane, the task kubernetes/control-plane : Create kubeadm token for joining nodes with 24h expiration (default) is delegated to first_kube_control_plane (leno4 in the run). The task Parse certificate key if not set in kubeadm-secondary sets facts from groups['kube_control_plane'][0] which is not the same node (lenop in the run). For this reason kubeadm_certificate_key is not valid and task kubernetes/control-plane : Create kubeadm ControlPlane config fails.

This patch resolves the bug in my situation :

diff --git a/roles/kubernetes/control-plane/tasks/kubeadm-secondary.yml b/roles/kubernetes/control-plane/tasks/kubeadm-secondary.yml
index f3fd207c4..743a86b9e 100644
--- a/roles/kubernetes/control-plane/tasks/kubeadm-secondary.yml
+++ b/roles/kubernetes/control-plane/tasks/kubeadm-secondary.yml
@@ -24,11 +24,11 @@

 - name: Parse certificate key if not set
   set_fact:
-    kubeadm_certificate_key: "{{ hostvars[groups['kube_control_plane'][0]]['kubeadm_upload_cert'].stdout_lines[-1] | trim }}"
+    kubeadm_certificate_key: "{{ hostvars[first_kube_control_plane]['kubeadm_upload_cert'].stdout_lines[-1] | trim }}"
   run_once: yes
   when:
-    - hostvars[groups['kube_control_plane'][0]]['kubeadm_upload_cert'] is defined
-    - hostvars[groups['kube_control_plane'][0]]['kubeadm_upload_cert'] is not skipped
+    - hostvars[first_kube_control_plane]['kubeadm_upload_cert'] is defined
+    - hostvars[first_kube_control_plane]['kubeadm_upload_cert'] is not skipped

 - name: Create kubeadm ControlPlane config
   template:
diff --git a/roles/kubernetes/control-plane/tasks/kubeadm-setup.yml b/roles/kubernetes/control-plane/tasks/kubeadm-setup.yml
index 86926cb2d..bbeb630c1 100644
--- a/roles/kubernetes/control-plane/tasks/kubeadm-setup.yml
+++ b/roles/kubernetes/control-plane/tasks/kubeadm-setup.yml
@@ -187,7 +187,7 @@
 - name: Set kubeadm certificate key
   set_fact:
     kubeadm_certificate_key: "{{ item | regex_search('--certificate-key ([^ ]+)', '\\1') | first }}"
-  with_items: "{{ hostvars[groups['kube_control_plane'][0]]['kubeadm_init'].stdout_lines | default([]) }}"
+  with_items: "{{ hostvars[first_kube_control_plane]['kubeadm_init'].stdout_lines | default([]) }}"
   when:
     - kubeadm_certificate_key is not defined
     - (item | trim) is match('.*--certificate-key.*')
joshelb commented 5 months ago

I have the same problem when trying to replace a first-controlplane node. The docs https://github.com/kubernetes-sigs/kubespray/blob/master/docs/nodes.md just dont work.

k8s-triage-robot commented 2 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 1 month ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 1 week ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-ci-robot commented 1 week ago

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to [this](https://github.com/kubernetes-sigs/kubespray/issues/10973#issuecomment-2317594783): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue with `/reopen` >- Mark this issue as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close not-planned > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.