search

kubernetes-sigs / kubespray

Deploy a Production Ready Kubernetes Cluster
Apache License 2.0
15.93k stars 6.42k forks source link

Kubespray re-adding CentOS 8 node preinstall curiously looking for /etc/resolvconf/resolv.conf.d/ directory and files #11317

Open sinaowolabi opened 3 months ago

sinaowolabi commented 3 months ago

What happened?

Im trying to restore a removed node, and yes CentOS 8. I have tried with multiple releases (2.14.0-master) and the failure still occurs. All nodes are CentOS8, cluster is a few years old Im attempting the oldest version I have yet, which was about the time I created the cluster:

~/kubespray$ git branch -v

Relevant logs:

TASK [kubernetes/preinstall : Remove search/domain/nameserver options before block] **** ok: [srv0node05.prod.isec.internal] => (item=['/etc/resolv.conf', 'search ']) ok: [srv0node05.prod.isec.internal] => (item=['/etc/resolv.conf', 'nameserver ']) ok: [srv0node05.prod.isec.internal] => (item=['/etc/resolv.conf', 'domain ']) ok: [srv0node05.prod.isec.internal] => (item=['/etc/resolv.conf', 'options ']) failed: [srv0node05.prod.isec.internal] (item=['/etc/resolvconf/resolv.conf.d/base', 'search ']) => {"ansible_loop_var": "item", "changed": false, "item": ["/etc/resolvconf/resolv.conf.d/base", "search "], "msg": "Path /etc/resolvconf/resolv.conf.d/base does not exist !", "rc": 257} failed: [srv0node05.prod.isec.internal] (item=['/etc/resolvconf/resolv.conf.d/base', 'nameserver ']) => {"ansible_loop_var": "item", "changed": false, "item": ["/etc/resolvconf/resolv.conf.d/base", "nameserver "], "msg": "Path /etc/resolvconf/resolv.conf.d/base does not exist !", "rc": 257} failed: [srv0node05.prod.isec.internal] (item=['/etc/resolvconf/resolv.conf.d/base', 'domain ']) => {"ansible_loop_var": "item", "changed": false, "item": ["/etc/resolvconf/resolv.conf.d/base", "domain "], "msg": "Path /etc/resolvconf/resolv.conf.d/base does not exist !", "rc": 257} failed: [srv0node05.prod.isec.internal] (item=['/etc/resolvconf/resolv.conf.d/base', 'options ']) => {"ansible_loop_var": "item", "changed": false, "item": ["/etc/resolvconf/resolv.conf.d/base", "options "], "msg": "Path /etc/resolvconf/resolv.conf.d/base does not exist !", "rc": 257} failed: [srv0node05.prod.isec.internal] (item=['/etc/resolvconf/resolv.conf.d/head', 'search ']) => {"ansible_loop_var": "item", "changed": false, "item": ["/etc/resolvconf/resolv.conf.d/head", "search "], "msg": "Path /etc/resolvconf/resolv.conf.d/head does not exist !", "rc": 257} failed: [srv0node05.prod.isec.internal] (item=['/etc/resolvconf/resolv.conf.d/head', 'nameserver ']) => {"ansible_loop_var": "item", "changed": false, "item": ["/etc/resolvconf/resolv.conf.d/head", "nameserver "], "msg": "Path /etc/resolvconf/resolv.conf.d/head does not exist !", "rc": 257} failed: [srv0node05.prod.isec.internal] (item=['/etc/resolvconf/resolv.conf.d/head', 'domain ']) => {"ansible_loop_var": "item", "changed": false, "item": ["/etc/resolvconf/resolv.conf.d/head", "domain "], "msg": "Path /etc/resolvconf/resolv.conf.d/head does not exist !", "rc": 257} failed: [srv0node05.prod.isec.internal] (item=['/etc/resolvconf/resolv.conf.d/head', 'options ']) => {"ansible_loop_var": "item", "changed": false, "item": ["/etc/resolvconf/resolv.conf.d/head", "options "], "msg": "Path /etc/resolvconf/resolv.conf.d/head does not exist !", "rc": 257}

What did you expect to happen?

For the node to join the cluster again.

How can we reproduce it (as minimally and precisely as possible)?

docker pull quay.io/kubespray/kubespray:v2.(14|15|16).0 git checkout release-2.(14|15|16) docker run --rm -it --mount type=bind,source="$(pwd)"/inventory,dst=/inventory --mount type=bind,source="${HOME}"/.ssh/id_rsa,dst=/root/.ssh/id_rsa quay.io/kubespray/kubespray:v2.14.0 bash ansible-playbook -i /inventory/prod1/hosts.yaml facts.yml -u ansible-playbook -i /inventory/prod1/hosts.yaml cluster.yml -u -b -l

OS

printf "$(uname -srm)\n$(cat /etc/os-release)\n" Linux 4.18.0-553.6.1.el8.x86_64 x86_64 NAME="CentOS Stream" VERSION="8" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="8" PLATFORM_ID="platform:el8" PRETTY_NAME="CentOS Stream 8" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:8" HOME_URL="https://centos.org/" BUG_REPORT_URL="https://bugzilla.redhat.com/" REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"

Version of Ansible

ansible --version ansible 2.9.6 config file = /kubespray/ansible.cfg configured module search path = ['/kubespray/library'] ansible python module location = /usr/local/lib/python3.6/dist-packages/ansible executable location = /usr/local/bin/ansible python version = 3.6.9 (default, Apr 18 2020, 01:56:04) [GCC 8.4.0] root@8e73fffbd6bb:/kubespray#

Version of Python

Python 3.6.9

Version of Kubespray (commit)

1f47d5b74

Network plugin used

cilium

Full inventory with variables

srv0node00.prod.isec.internal | SUCCESS => { "hostvars[inventory_hostname]": { "access_ip": "10.100.14.41", "ansible_check_mode": false, "ansible_diff_mode": false, "ansible_facts": {}, "ansible_forks": 5, "ansible_host": "10.100.14.41", "ansible_inventory_sources": [ "/inventory/prod1/hosts.yaml" ], "ansible_playbook_python": "/usr/bin/python3", "ansible_user": "root", "ansible_verbosity": 0, "ansible_version": { "full": "2.9.6", "major": 2, "minor": 9, "revision": 6, "string": "2.9.6" }, "argocd_enabled": false, "auto_renew_certificates": false, "bin_dir": "/usr/bin", "cephfs_provisioner_enabled": false, "cert_manager_enabled": false, "cluster_name": "prod.isec.local", "container_manager": "containerd", "coredns_k8s_external_zone": "k8s_external.local", "credentials_dir": "/inventory/prod1/credentials", "default_kubelet_config_dir": "/etc/kubernetes/dynamic_kubelet_dir", "deploy_netchecker": false, "dns_domain": "prod.isec.local", "dns_mode": "coredns", "docker_bin_dir": "/usr/bin", "docker_container_storage_setup": false, "docker_daemon_graph": "/var/lib/docker", "docker_dns_servers_strict": false, "docker_iptables_enabled": "false", "docker_log_opts": "--log-opt max-size=50m --log-opt max-file=5", "docker_rpm_keepcache": 1, "dynamic_kubelet_configuration": false, "dynamic_kubelet_configuration_dir": "/etc/kubernetes/dynamic_kubelet_dir", "enable_coredns_k8s_endpoint_pod_names": false, "enable_coredns_k8s_external": false, "enable_dual_stack_networks": false, "enable_nat_default_gateway": true, "enable_nodelocaldns": true, "enable_nodelocaldns_secondary": false, "etcd_data_dir": "/var/lib/etcd", "etcd_deployment_type": "host", "etcd_kubeadm_enabled": false, "event_ttl_duration": "1h0m0s", "group_names": [ "etcd", "k8s-cluster", "k8s_cluster", "kube-master", "kube-node", "kube_control_plane", "kube_node" ], "groups": { "all": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal", "srv0node02.prod.isec.internal", "srv0node03.prod.isec.internal", "srv0node04.prod.isec.internal", "srv0node05.prod.isec.internal" ], "calico-rr": [], "calico_rr": [], "etcd": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal", "srv0node02.prod.isec.internal" ], "k8s-cluster": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal", "srv0node02.prod.isec.internal", "srv0node03.prod.isec.internal", "srv0node04.prod.isec.internal", "srv0node05.prod.isec.internal" ], "k8s_cluster": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal", "srv0node02.prod.isec.internal", "srv0node03.prod.isec.internal", "srv0node04.prod.isec.internal", "srv0node05.prod.isec.internal" ], "kube-master": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal" ], "kube-node": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal", "srv0node02.prod.isec.internal", "srv0node03.prod.isec.internal", "srv0node04.prod.isec.internal", "srv0node05.prod.isec.internal" ], "kube_control_plane": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal" ], "kube_node": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal", "srv0node02.prod.isec.internal", "srv0node03.prod.isec.internal", "srv0node04.prod.isec.internal", "srv0node05.prod.isec.internal" ], "ungrouped": [] }, "helm_enabled": false, "ingress_alb_enabled": false, "ingress_nginx_enabled": false, "ingress_publish_status_address": "", "inventory_dir": "/inventory/prod1", "inventory_file": "/inventory/prod1/hosts.yaml", "inventory_hostname": "srv0node00.prod.isec.internal", "inventory_hostname_short": "srv0node00", "ip": "10.100.14.41", "k8s_image_pull_policy": "IfNotPresent", "kata_containers_enabled": false, "krew_enabled": false, "krew_root_dir": "/usr/local/krew", "kube_api_anonymous_auth": true, "kube_apiserver_insecure_port": 0, "kube_apiserver_ip": "172.32.0.1", "kube_apiserver_port": 6443, "kube_cert_dir": "/etc/kubernetes/ssl", "kube_cert_group": "kube-cert", "kube_config_dir": "/etc/kubernetes", "kube_encrypt_secret_data": false, "kube_log_level": 2, "kube_manifest_dir": "/etc/kubernetes/manifests", "kube_network_node_prefix": 24, "kube_network_node_prefix_ipv6": 120, "kube_network_plugin": "cilium", "kube_network_plugin_multus": false, "kube_pods_subnet": "172.32.64.0/18", "kube_pods_subnet_ipv6": "fd85:ee78:d8a6:8607::1:0000/112", "kube_proxy_mode": "ipvs", "kube_proxy_nodeport_addresses": [], "kube_proxy_strict_arp": false, "kube_script_dir": "/usr/bin/kubernetes-scripts", "kube_service_addresses": "172.32.0.0/18", "kube_service_addresses_ipv6": "fd85:ee78:d8a6:8607::1000/116", "kube_token_dir": "/etc/kubernetes/tokens", "kube_version": "v1.23.2", "kubeadm_certificate_key": "bfa3bfa9011f9e1d8c137905c222a51fcab34a68c0be5beacdca3ef2d93b901b", "kubernetes_audit": false, "loadbalancer_apiserver_healthcheck_port": 8081, "loadbalancer_apiserver_port": 6443, "local_path_provisioner_enabled": false, "local_release_dir": "/tmp/releases", "local_volume_provisioner_enabled": false, "macvlan_interface": "eth1", "metallb_enabled": false, "metallb_speaker_enabled": true, "metrics_server_enabled": false, "ndots": 2, "no_proxy_exclude_workers": false, "nodelocaldns_bind_metrics_host_ip": false, "nodelocaldns_health_port": 9254, "nodelocaldns_ip": "169.254.31.10", "nodelocaldns_second_health_port": 9256, "nodelocaldns_secondary_skew_seconds": 5, "omit": "omit_place_holder03dfbbf9527bee169985cf26e292210131ded120", "persistent_volumes_enabled": false, "playbook_dir": "/kubespray", "podsecuritypolicy_enabled": false, "rbd_provisioner_enabled": false, "registry_enabled": false, "resolvconf_mode": "host_resolvconf", "retry_stagger": 5, "skydns_server": "172.32.0.3", "skydns_server_secondary": "172.32.0.4", "upstream_dns_servers": [ "10.100.10.3", "10.100.10.4" ], "volume_cross_zone_attachment": false } } srv0node03.prod.isec.internal | SUCCESS => { "hostvars[inventory_hostname]": { "access_ip": "10.100.14.44", "ansible_check_mode": false, "ansible_diff_mode": false, "ansible_facts": {}, "ansible_forks": 5, "ansible_host": "10.100.14.44", "ansible_inventory_sources": [ "/inventory/prod1/hosts.yaml" ], "ansible_playbook_python": "/usr/bin/python3", "ansible_user": "root", "ansible_verbosity": 0, "ansible_version": { "full": "2.9.6", "major": 2, "minor": 9, "revision": 6, "string": "2.9.6" }, "argocd_enabled": false, "auto_renew_certificates": false, "bin_dir": "/usr/bin", "cephfs_provisioner_enabled": false, "cert_manager_enabled": false, "cluster_name": "prod.isec.local", "container_manager": "containerd", "coredns_k8s_external_zone": "k8s_external.local", "credentials_dir": "/inventory/prod1/credentials", "default_kubelet_config_dir": "/etc/kubernetes/dynamic_kubelet_dir", "deploy_netchecker": false, "dns_domain": "prod.isec.local", "dns_mode": "coredns", "docker_bin_dir": "/usr/bin", "docker_container_storage_setup": false, "docker_daemon_graph": "/var/lib/docker", "docker_dns_servers_strict": false, "docker_iptables_enabled": "false", "docker_log_opts": "--log-opt max-size=50m --log-opt max-file=5", "docker_rpm_keepcache": 1, "dynamic_kubelet_configuration": false, "dynamic_kubelet_configuration_dir": "/etc/kubernetes/dynamic_kubelet_dir", "enable_coredns_k8s_endpoint_pod_names": false, "enable_coredns_k8s_external": false, "enable_dual_stack_networks": false, "enable_nat_default_gateway": true, "enable_nodelocaldns": true, "enable_nodelocaldns_secondary": false, "etcd_data_dir": "/var/lib/etcd", "etcd_kubeadm_enabled": false, "event_ttl_duration": "1h0m0s", "group_names": [ "k8s-cluster", "k8s_cluster", "kube-node", "kube_node" ], "groups": { "all": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal", "srv0node02.prod.isec.internal", "srv0node03.prod.isec.internal", "srv0node04.prod.isec.internal", "srv0node05.prod.isec.internal" ], "calico-rr": [], "calico_rr": [], "etcd": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal", "srv0node02.prod.isec.internal" ], "k8s-cluster": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal", "srv0node02.prod.isec.internal", "srv0node03.prod.isec.internal", "srv0node04.prod.isec.internal", "srv0node05.prod.isec.internal" ], "k8s_cluster": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal", "srv0node02.prod.isec.internal", "srv0node03.prod.isec.internal", "srv0node04.prod.isec.internal", "srv0node05.prod.isec.internal" ], "kube-master": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal" ], "kube-node": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal", "srv0node02.prod.isec.internal", "srv0node03.prod.isec.internal", "srv0node04.prod.isec.internal", "srv0node05.prod.isec.internal" ], "kube_control_plane": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal" ], "kube_node": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal", "srv0node02.prod.isec.internal", "srv0node03.prod.isec.internal", "srv0node04.prod.isec.internal", "srv0node05.prod.isec.internal" ], "ungrouped": [] }, "helm_enabled": false, "ingress_alb_enabled": false, "ingress_nginx_enabled": false, "ingress_publish_status_address": "", "inventory_dir": "/inventory/prod1", "inventory_file": "/inventory/prod1/hosts.yaml", "inventory_hostname": "srv0node03.prod.isec.internal", "inventory_hostname_short": "srv0node03", "ip": "10.100.14.44", "k8s_image_pull_policy": "IfNotPresent", "kata_containers_enabled": false, "krew_enabled": false, "krew_root_dir": "/usr/local/krew", "kube_api_anonymous_auth": true, "kube_apiserver_insecure_port": 0, "kube_apiserver_ip": "172.32.0.1", "kube_apiserver_port": 6443, "kube_cert_dir": "/etc/kubernetes/ssl", "kube_cert_group": "kube-cert", "kube_config_dir": "/etc/kubernetes", "kube_encrypt_secret_data": false, "kube_log_level": 2, "kube_manifest_dir": "/etc/kubernetes/manifests", "kube_network_node_prefix": 24, "kube_network_node_prefix_ipv6": 120, "kube_network_plugin": "cilium", "kube_network_plugin_multus": false, "kube_pods_subnet": "172.32.64.0/18", "kube_pods_subnet_ipv6": "fd85:ee78:d8a6:8607::1:0000/112", "kube_proxy_mode": "ipvs", "kube_proxy_nodeport_addresses": [], "kube_proxy_strict_arp": false, "kube_script_dir": "/usr/bin/kubernetes-scripts", "kube_service_addresses": "172.32.0.0/18", "kube_service_addresses_ipv6": "fd85:ee78:d8a6:8607::1000/116", "kube_token_dir": "/etc/kubernetes/tokens", "kube_version": "v1.23.2", "kubeadm_certificate_key": "bfa3bfa9011f9e1d8c137905c222a51fcab34a68c0be5beacdca3ef2d93b901b", "kubernetes_audit": false, "loadbalancer_apiserver_healthcheck_port": 8081, "loadbalancer_apiserver_port": 6443, "local_path_provisioner_enabled": false, "local_release_dir": "/tmp/releases", "local_volume_provisioner_enabled": false, "macvlan_interface": "eth1", "metallb_enabled": false, "metallb_speaker_enabled": true, "metrics_server_enabled": false, "ndots": 2, "no_proxy_exclude_workers": false, "nodelocaldns_bind_metrics_host_ip": false, "nodelocaldns_health_port": 9254, "nodelocaldns_ip": "169.254.31.10", "nodelocaldns_second_health_port": 9256, "nodelocaldns_secondary_skew_seconds": 5, "omit": "omit_place_holder03dfbbf9527bee169985cf26e292210131ded120", "persistent_volumes_enabled": false, "playbook_dir": "/kubespray", "podsecuritypolicy_enabled": false, "rbd_provisioner_enabled": false, "registry_enabled": false, "resolvconf_mode": "host_resolvconf", "retry_stagger": 5, "skydns_server": "172.32.0.3", "skydns_server_secondary": "172.32.0.4", "upstream_dns_servers": [ "10.100.10.3", "10.100.10.4" ], "volume_cross_zone_attachment": false } } srv0node02.prod.isec.internal | SUCCESS => { "hostvars[inventory_hostname]": { "access_ip": "10.100.14.43", "ansible_check_mode": false, "ansible_diff_mode": false, "ansible_facts": {}, "ansible_forks": 5, "ansible_host": "10.100.14.43", "ansible_inventory_sources": [ "/inventory/prod1/hosts.yaml" ], "ansible_playbook_python": "/usr/bin/python3", "ansible_user": "root", "ansible_verbosity": 0, "ansible_version": { "full": "2.9.6", "major": 2, "minor": 9, "revision": 6, "string": "2.9.6" }, "argocd_enabled": false, "auto_renew_certificates": false, "bin_dir": "/usr/bin", "cephfs_provisioner_enabled": false, "cert_manager_enabled": false, "cluster_name": "prod.isec.local", "container_manager": "containerd", "coredns_k8s_external_zone": "k8s_external.local", "credentials_dir": "/inventory/prod1/credentials", "default_kubelet_config_dir": "/etc/kubernetes/dynamic_kubelet_dir", "deploy_netchecker": false, "dns_domain": "prod.isec.local", "dns_mode": "coredns", "docker_bin_dir": "/usr/bin", "docker_container_storage_setup": false, "docker_daemon_graph": "/var/lib/docker", "docker_dns_servers_strict": false, "docker_iptables_enabled": "false", "docker_log_opts": "--log-opt max-size=50m --log-opt max-file=5", "docker_rpm_keepcache": 1, "dynamic_kubelet_configuration": false, "dynamic_kubelet_configuration_dir": "/etc/kubernetes/dynamic_kubelet_dir", "enable_coredns_k8s_endpoint_pod_names": false, "enable_coredns_k8s_external": false, "enable_dual_stack_networks": false, "enable_nat_default_gateway": true, "enable_nodelocaldns": true, "enable_nodelocaldns_secondary": false, "etcd_data_dir": "/var/lib/etcd", "etcd_deployment_type": "host", "etcd_kubeadm_enabled": false, "event_ttl_duration": "1h0m0s", "group_names": [ "etcd", "k8s-cluster", "k8s_cluster", "kube-node", "kube_node" ], "groups": { "all": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal", "srv0node02.prod.isec.internal", "srv0node03.prod.isec.internal", "srv0node04.prod.isec.internal", "srv0node05.prod.isec.internal" ], "calico-rr": [], "calico_rr": [], "etcd": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal", "srv0node02.prod.isec.internal" ], "k8s-cluster": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal", "srv0node02.prod.isec.internal", "srv0node03.prod.isec.internal", "srv0node04.prod.isec.internal", "srv0node05.prod.isec.internal" ], "k8s_cluster": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal", "srv0node02.prod.isec.internal", "srv0node03.prod.isec.internal", "srv0node04.prod.isec.internal", "srv0node05.prod.isec.internal" ], "kube-master": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal" ], "kube-node": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal", "srv0node02.prod.isec.internal", "srv0node03.prod.isec.internal", "srv0node04.prod.isec.internal", "srv0node05.prod.isec.internal" ], "kube_control_plane": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal" ], "kube_node": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal", "srv0node02.prod.isec.internal", "srv0node03.prod.isec.internal", "srv0node04.prod.isec.internal", "srv0node05.prod.isec.internal" ], "ungrouped": [] }, "helm_enabled": false, "ingress_alb_enabled": false, "ingress_nginx_enabled": false, "ingress_publish_status_address": "", "inventory_dir": "/inventory/prod1", "inventory_file": "/inventory/prod1/hosts.yaml", "inventory_hostname": "srv0node02.prod.isec.internal", "inventory_hostname_short": "srv0node02", "ip": "10.100.14.43", "k8s_image_pull_policy": "IfNotPresent", "kata_containers_enabled": false, "krew_enabled": false, "krew_root_dir": "/usr/local/krew", "kube_api_anonymous_auth": true, "kube_apiserver_insecure_port": 0, "kube_apiserver_ip": "172.32.0.1", "kube_apiserver_port": 6443, "kube_cert_dir": "/etc/kubernetes/ssl", "kube_cert_group": "kube-cert", "kube_config_dir": "/etc/kubernetes", "kube_encrypt_secret_data": false, "kube_log_level": 2, "kube_manifest_dir": "/etc/kubernetes/manifests", "kube_network_node_prefix": 24, "kube_network_node_prefix_ipv6": 120, "kube_network_plugin": "cilium", "kube_network_plugin_multus": false, "kube_pods_subnet": "172.32.64.0/18", "kube_pods_subnet_ipv6": "fd85:ee78:d8a6:8607::1:0000/112", "kube_proxy_mode": "ipvs", "kube_proxy_nodeport_addresses": [], "kube_proxy_strict_arp": false, "kube_script_dir": "/usr/bin/kubernetes-scripts", "kube_service_addresses": "172.32.0.0/18", "kube_service_addresses_ipv6": "fd85:ee78:d8a6:8607::1000/116", "kube_token_dir": "/etc/kubernetes/tokens", "kube_version": "v1.23.2", "kubeadm_certificate_key": "bfa3bfa9011f9e1d8c137905c222a51fcab34a68c0be5beacdca3ef2d93b901b", "kubernetes_audit": false, "loadbalancer_apiserver_healthcheck_port": 8081, "loadbalancer_apiserver_port": 6443, "local_path_provisioner_enabled": false, "local_release_dir": "/tmp/releases", "local_volume_provisioner_enabled": false, "macvlan_interface": "eth1", "metallb_enabled": false, "metallb_speaker_enabled": true, "metrics_server_enabled": false, "ndots": 2, "no_proxy_exclude_workers": false, "nodelocaldns_bind_metrics_host_ip": false, "nodelocaldns_health_port": 9254, "nodelocaldns_ip": "169.254.31.10", "nodelocaldns_second_health_port": 9256, "nodelocaldns_secondary_skew_seconds": 5, "omit": "omit_place_holder03dfbbf9527bee169985cf26e292210131ded120", "persistent_volumes_enabled": false, "playbook_dir": "/kubespray", "podsecuritypolicy_enabled": false, "rbd_provisioner_enabled": false, "registry_enabled": false, "resolvconf_mode": "host_resolvconf", "retry_stagger": 5, "skydns_server": "172.32.0.3", "skydns_server_secondary": "172.32.0.4", "upstream_dns_servers": [ "10.100.10.3", "10.100.10.4" ], "volume_cross_zone_attachment": false } } srv0node04.prod.isec.internal | SUCCESS => { "hostvars[inventory_hostname]": { "access_ip": "10.100.14.45", "ansible_check_mode": false, "ansible_diff_mode": false, "ansible_facts": {}, "ansible_forks": 5, "ansible_host": "10.100.14.45", "ansible_inventory_sources": [ "/inventory/prod1/hosts.yaml" ], "ansible_playbook_python": "/usr/bin/python3", "ansible_user": "root", "ansible_verbosity": 0, "ansible_version": { "full": "2.9.6", "major": 2, "minor": 9, "revision": 6, "string": "2.9.6" }, "argocd_enabled": false, "auto_renew_certificates": false, "bin_dir": "/usr/bin", "cephfs_provisioner_enabled": false, "cert_manager_enabled": false, "cluster_name": "prod.isec.local", "container_manager": "containerd", "coredns_k8s_external_zone": "k8s_external.local", "credentials_dir": "/inventory/prod1/credentials", "default_kubelet_config_dir": "/etc/kubernetes/dynamic_kubelet_dir", "deploy_netchecker": false, "dns_domain": "prod.isec.local", "dns_mode": "coredns", "docker_bin_dir": "/usr/bin", "docker_container_storage_setup": false, "docker_daemon_graph": "/var/lib/docker", "docker_dns_servers_strict": false, "docker_iptables_enabled": "false", "docker_log_opts": "--log-opt max-size=50m --log-opt max-file=5", "docker_rpm_keepcache": 1, "dynamic_kubelet_configuration": false, "dynamic_kubelet_configuration_dir": "/etc/kubernetes/dynamic_kubelet_dir", "enable_coredns_k8s_endpoint_pod_names": false, "enable_coredns_k8s_external": false, "enable_dual_stack_networks": false, "enable_nat_default_gateway": true, "enable_nodelocaldns": true, "enable_nodelocaldns_secondary": false, "etcd_data_dir": "/var/lib/etcd", "etcd_kubeadm_enabled": false, "event_ttl_duration": "1h0m0s", "group_names": [ "k8s-cluster", "k8s_cluster", "kube-node", "kube_node" ], "groups": { "all": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal", "srv0node02.prod.isec.internal", "srv0node03.prod.isec.internal", "srv0node04.prod.isec.internal", "srv0node05.prod.isec.internal" ], "calico-rr": [], "calico_rr": [], "etcd": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal", "srv0node02.prod.isec.internal" ], "k8s-cluster": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal", "srv0node02.prod.isec.internal", "srv0node03.prod.isec.internal", "srv0node04.prod.isec.internal", "srv0node05.prod.isec.internal" ], "k8s_cluster": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal", "srv0node02.prod.isec.internal", "srv0node03.prod.isec.internal", "srv0node04.prod.isec.internal", "srv0node05.prod.isec.internal" ], "kube-master": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal" ], "kube-node": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal", "srv0node02.prod.isec.internal", "srv0node03.prod.isec.internal", "srv0node04.prod.isec.internal", "srv0node05.prod.isec.internal" ], "kube_control_plane": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal" ], "kube_node": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal", "srv0node02.prod.isec.internal", "srv0node03.prod.isec.internal", "srv0node04.prod.isec.internal", "srv0node05.prod.isec.internal" ], "ungrouped": [] }, "helm_enabled": false, "ingress_alb_enabled": false, "ingress_nginx_enabled": false, "ingress_publish_status_address": "", "inventory_dir": "/inventory/prod1", "inventory_file": "/inventory/prod1/hosts.yaml", "inventory_hostname": "srv0node04.prod.isec.internal", "inventory_hostname_short": "srv0node04", "ip": "10.100.14.45", "k8s_image_pull_policy": "IfNotPresent", "kata_containers_enabled": false, "krew_enabled": false, "krew_root_dir": "/usr/local/krew", "kube_api_anonymous_auth": true, "kube_apiserver_insecure_port": 0, "kube_apiserver_ip": "172.32.0.1", "kube_apiserver_port": 6443, "kube_cert_dir": "/etc/kubernetes/ssl", "kube_cert_group": "kube-cert", "kube_config_dir": "/etc/kubernetes", "kube_encrypt_secret_data": false, "kube_log_level": 2, "kube_manifest_dir": "/etc/kubernetes/manifests", "kube_network_node_prefix": 24, "kube_network_node_prefix_ipv6": 120, "kube_network_plugin": "cilium", "kube_network_plugin_multus": false, "kube_pods_subnet": "172.32.64.0/18", "kube_pods_subnet_ipv6": "fd85:ee78:d8a6:8607::1:0000/112", "kube_proxy_mode": "ipvs", "kube_proxy_nodeport_addresses": [], "kube_proxy_strict_arp": false, "kube_script_dir": "/usr/bin/kubernetes-scripts", "kube_service_addresses": "172.32.0.0/18", "kube_service_addresses_ipv6": "fd85:ee78:d8a6:8607::1000/116", "kube_token_dir": "/etc/kubernetes/tokens", "kube_version": "v1.23.2", "kubeadm_certificate_key": "bfa3bfa9011f9e1d8c137905c222a51fcab34a68c0be5beacdca3ef2d93b901b", "kubernetes_audit": false, "loadbalancer_apiserver_healthcheck_port": 8081, "loadbalancer_apiserver_port": 6443, "local_path_provisioner_enabled": false, "local_release_dir": "/tmp/releases", "local_volume_provisioner_enabled": false, "macvlan_interface": "eth1", "metallb_enabled": false, "metallb_speaker_enabled": true, "metrics_server_enabled": false, "ndots": 2, "no_proxy_exclude_workers": false, "nodelocaldns_bind_metrics_host_ip": false, "nodelocaldns_health_port": 9254, "nodelocaldns_ip": "169.254.31.10", "nodelocaldns_second_health_port": 9256, "nodelocaldns_secondary_skew_seconds": 5, "omit": "omit_place_holder03dfbbf9527bee169985cf26e292210131ded120", "persistent_volumes_enabled": false, "playbook_dir": "/kubespray", "podsecuritypolicy_enabled": false, "rbd_provisioner_enabled": false, "registry_enabled": false, "resolvconf_mode": "host_resolvconf", "retry_stagger": 5, "skydns_server": "172.32.0.3", "skydns_server_secondary": "172.32.0.4", "upstream_dns_servers": [ "10.100.10.3", "10.100.10.4" ], "volume_cross_zone_attachment": false } } srv0node01.prod.isec.internal | SUCCESS => { "hostvars[inventory_hostname]": { "access_ip": "10.100.14.42", "ansible_check_mode": false, "ansible_diff_mode": false, "ansible_facts": {}, "ansible_forks": 5, "ansible_host": "10.100.14.42", "ansible_inventory_sources": [ "/inventory/prod1/hosts.yaml" ], "ansible_playbook_python": "/usr/bin/python3", "ansible_user": "root", "ansible_verbosity": 0, "ansible_version": { "full": "2.9.6", "major": 2, "minor": 9, "revision": 6, "string": "2.9.6" }, "argocd_enabled": false, "auto_renew_certificates": false, "bin_dir": "/usr/bin", "cephfs_provisioner_enabled": false, "cert_manager_enabled": false, "cluster_name": "prod.isec.local", "container_manager": "containerd", "coredns_k8s_external_zone": "k8s_external.local", "credentials_dir": "/inventory/prod1/credentials", "default_kubelet_config_dir": "/etc/kubernetes/dynamic_kubelet_dir", "deploy_netchecker": false, "dns_domain": "prod.isec.local", "dns_mode": "coredns", "docker_bin_dir": "/usr/bin", "docker_container_storage_setup": false, "docker_daemon_graph": "/var/lib/docker", "docker_dns_servers_strict": false, "docker_iptables_enabled": "false", "docker_log_opts": "--log-opt max-size=50m --log-opt max-file=5", "docker_rpm_keepcache": 1, "dynamic_kubelet_configuration": false, "dynamic_kubelet_configuration_dir": "/etc/kubernetes/dynamic_kubelet_dir", "enable_coredns_k8s_endpoint_pod_names": false, "enable_coredns_k8s_external": false, "enable_dual_stack_networks": false, "enable_nat_default_gateway": true, "enable_nodelocaldns": true, "enable_nodelocaldns_secondary": false, "etcd_data_dir": "/var/lib/etcd", "etcd_deployment_type": "host", "etcd_kubeadm_enabled": false, "event_ttl_duration": "1h0m0s", "group_names": [ "etcd", "k8s-cluster", "k8s_cluster", "kube-master", "kube-node", "kube_control_plane", "kube_node" ], "groups": { "all": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal", "srv0node02.prod.isec.internal", "srv0node03.prod.isec.internal", "srv0node04.prod.isec.internal", "srv0node05.prod.isec.internal" ], "calico-rr": [], "calico_rr": [], "etcd": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal", "srv0node02.prod.isec.internal" ], "k8s-cluster": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal", "srv0node02.prod.isec.internal", "srv0node03.prod.isec.internal", "srv0node04.prod.isec.internal", "srv0node05.prod.isec.internal" ], "k8s_cluster": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal", "srv0node02.prod.isec.internal", "srv0node03.prod.isec.internal", "srv0node04.prod.isec.internal", "srv0node05.prod.isec.internal" ], "kube-master": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal" ], "kube-node": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal", "srv0node02.prod.isec.internal", "srv0node03.prod.isec.internal", "srv0node04.prod.isec.internal", "srv0node05.prod.isec.internal" ], "kube_control_plane": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal" ], "kube_node": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal", "srv0node02.prod.isec.internal", "srv0node03.prod.isec.internal", "srv0node04.prod.isec.internal", "srv0node05.prod.isec.internal" ], "ungrouped": [] }, "helm_enabled": false, "ingress_alb_enabled": false, "ingress_nginx_enabled": false, "ingress_publish_status_address": "", "inventory_dir": "/inventory/prod1", "inventory_file": "/inventory/prod1/hosts.yaml", "inventory_hostname": "srv0node01.prod.isec.internal", "inventory_hostname_short": "srv0node01", "ip": "10.100.14.42", "k8s_image_pull_policy": "IfNotPresent", "kata_containers_enabled": false, "krew_enabled": false, "krew_root_dir": "/usr/local/krew", "kube_api_anonymous_auth": true, "kube_apiserver_insecure_port": 0, "kube_apiserver_ip": "172.32.0.1", "kube_apiserver_port": 6443, "kube_cert_dir": "/etc/kubernetes/ssl", "kube_cert_group": "kube-cert", "kube_config_dir": "/etc/kubernetes", "kube_encrypt_secret_data": false, "kube_log_level": 2, "kube_manifest_dir": "/etc/kubernetes/manifests", "kube_network_node_prefix": 24, "kube_network_node_prefix_ipv6": 120, "kube_network_plugin": "cilium", "kube_network_plugin_multus": false, "kube_pods_subnet": "172.32.64.0/18", "kube_pods_subnet_ipv6": "fd85:ee78:d8a6:8607::1:0000/112", "kube_proxy_mode": "ipvs", "kube_proxy_nodeport_addresses": [], "kube_proxy_strict_arp": false, "kube_script_dir": "/usr/bin/kubernetes-scripts", "kube_service_addresses": "172.32.0.0/18", "kube_service_addresses_ipv6": "fd85:ee78:d8a6:8607::1000/116", "kube_token_dir": "/etc/kubernetes/tokens", "kube_version": "v1.23.2", "kubeadm_certificate_key": "bfa3bfa9011f9e1d8c137905c222a51fcab34a68c0be5beacdca3ef2d93b901b", "kubernetes_audit": false, "loadbalancer_apiserver_healthcheck_port": 8081, "loadbalancer_apiserver_port": 6443, "local_path_provisioner_enabled": false, "local_release_dir": "/tmp/releases", "local_volume_provisioner_enabled": false, "macvlan_interface": "eth1", "metallb_enabled": false, "metallb_speaker_enabled": true, "metrics_server_enabled": false, "ndots": 2, "no_proxy_exclude_workers": false, "nodelocaldns_bind_metrics_host_ip": false, "nodelocaldns_health_port": 9254, "nodelocaldns_ip": "169.254.31.10", "nodelocaldns_second_health_port": 9256, "nodelocaldns_secondary_skew_seconds": 5, "omit": "omit_place_holder03dfbbf9527bee169985cf26e292210131ded120", "persistent_volumes_enabled": false, "playbook_dir": "/kubespray", "podsecuritypolicy_enabled": false, "rbd_provisioner_enabled": false, "registry_enabled": false, "resolvconf_mode": "host_resolvconf", "retry_stagger": 5, "skydns_server": "172.32.0.3", "skydns_server_secondary": "172.32.0.4", "upstream_dns_servers": [ "10.100.10.3", "10.100.10.4" ], "volume_cross_zone_attachment": false } } srv0node05.prod.isec.internal | SUCCESS => { "hostvars[inventory_hostname]": { "access_ip": "10.100.14.46", "ansible_check_mode": false, "ansible_diff_mode": false, "ansible_facts": {}, "ansible_forks": 5, "ansible_host": "10.100.14.46", "ansible_inventory_sources": [ "/inventory/prod1/hosts.yaml" ], "ansible_playbook_python": "/usr/bin/python3", "ansible_user": "root", "ansible_verbosity": 0, "ansible_version": { "full": "2.9.6", "major": 2, "minor": 9, "revision": 6, "string": "2.9.6" }, "argocd_enabled": false, "auto_renew_certificates": false, "bin_dir": "/usr/bin", "cephfs_provisioner_enabled": false, "cert_manager_enabled": false, "cluster_name": "prod.isec.local", "container_manager": "containerd", "coredns_k8s_external_zone": "k8s_external.local", "credentials_dir": "/inventory/prod1/credentials", "default_kubelet_config_dir": "/etc/kubernetes/dynamic_kubelet_dir", "deploy_netchecker": false, "dns_domain": "prod.isec.local", "dns_mode": "coredns", "docker_bin_dir": "/usr/bin", "docker_container_storage_setup": false, "docker_daemon_graph": "/var/lib/docker", "docker_dns_servers_strict": false, "docker_iptables_enabled": "false", "docker_log_opts": "--log-opt max-size=50m --log-opt max-file=5", "docker_rpm_keepcache": 1, "dynamic_kubelet_configuration": false, "dynamic_kubelet_configuration_dir": "/etc/kubernetes/dynamic_kubelet_dir", "enable_coredns_k8s_endpoint_pod_names": false, "enable_coredns_k8s_external": false, "enable_dual_stack_networks": false, "enable_nat_default_gateway": true, "enable_nodelocaldns": true, "enable_nodelocaldns_secondary": false, "etcd_data_dir": "/var/lib/etcd", "etcd_kubeadm_enabled": false, "event_ttl_duration": "1h0m0s", "group_names": [ "k8s-cluster", "k8s_cluster", "kube-node", "kube_node" ], "groups": { "all": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal", "srv0node02.prod.isec.internal", "srv0node03.prod.isec.internal", "srv0node04.prod.isec.internal", "srv0node05.prod.isec.internal" ], "calico-rr": [], "calico_rr": [], "etcd": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal", "srv0node02.prod.isec.internal" ], "k8s-cluster": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal", "srv0node02.prod.isec.internal", "srv0node03.prod.isec.internal", "srv0node04.prod.isec.internal", "srv0node05.prod.isec.internal" ], "k8s_cluster": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal", "srv0node02.prod.isec.internal", "srv0node03.prod.isec.internal", "srv0node04.prod.isec.internal", "srv0node05.prod.isec.internal" ], "kube-master": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal" ], "kube-node": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal", "srv0node02.prod.isec.internal", "srv0node03.prod.isec.internal", "srv0node04.prod.isec.internal", "srv0node05.prod.isec.internal" ], "kube_control_plane": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal" ], "kube_node": [ "srv0node00.prod.isec.internal", "srv0node01.prod.isec.internal", "srv0node02.prod.isec.internal", "srv0node03.prod.isec.internal", "srv0node04.prod.isec.internal", "srv0node05.prod.isec.internal" ], "ungrouped": [] }, "helm_enabled": false, "ingress_alb_enabled": false, "ingress_nginx_enabled": false, "ingress_publish_status_address": "", "inventory_dir": "/inventory/prod1", "inventory_file": "/inventory/prod1/hosts.yaml", "inventory_hostname": "srv0node05.prod.isec.internal", "inventory_hostname_short": "srv0node05", "ip": "10.100.14.46", "k8s_image_pull_policy": "IfNotPresent", "kata_containers_enabled": false, "krew_enabled": false, "krew_root_dir": "/usr/local/krew", "kube_api_anonymous_auth": true, "kube_apiserver_insecure_port": 0, "kube_apiserver_ip": "172.32.0.1", "kube_apiserver_port": 6443, "kube_cert_dir": "/etc/kubernetes/ssl", "kube_cert_group": "kube-cert", "kube_config_dir": "/etc/kubernetes", "kube_encrypt_secret_data": false, "kube_log_level": 2, "kube_manifest_dir": "/etc/kubernetes/manifests", "kube_network_node_prefix": 24, "kube_network_node_prefix_ipv6": 120, "kube_network_plugin": "cilium", "kube_network_plugin_multus": false, "kube_pods_subnet": "172.32.64.0/18", "kube_pods_subnet_ipv6": "fd85:ee78:d8a6:8607::1:0000/112", "kube_proxy_mode": "ipvs", "kube_proxy_nodeport_addresses": [], "kube_proxy_strict_arp": false, "kube_script_dir": "/usr/bin/kubernetes-scripts", "kube_service_addresses": "172.32.0.0/18", "kube_service_addresses_ipv6": "fd85:ee78:d8a6:8607::1000/116", "kube_token_dir": "/etc/kubernetes/tokens", "kube_version": "v1.23.2", "kubeadm_certificate_key": "bfa3bfa9011f9e1d8c137905c222a51fcab34a68c0be5beacdca3ef2d93b901b", "kubernetes_audit": false, "loadbalancer_apiserver_healthcheck_port": 8081, "loadbalancer_apiserver_port": 6443, "local_path_provisioner_enabled": false, "local_release_dir": "/tmp/releases", "local_volume_provisioner_enabled": false, "macvlan_interface": "eth1", "metallb_enabled": false, "metallb_speaker_enabled": true, "metrics_server_enabled": false, "ndots": 2, "no_proxy_exclude_workers": false, "nodelocaldns_bind_metrics_host_ip": false, "nodelocaldns_health_port": 9254, "nodelocaldns_ip": "169.254.31.10", "nodelocaldns_second_health_port": 9256, "nodelocaldns_secondary_skew_seconds": 5, "omit": "omit_place_holder03dfbbf9527bee169985cf26e292210131ded120", "persistent_volumes_enabled": false, "playbook_dir": "/kubespray", "podsecuritypolicy_enabled": false, "rbd_provisioner_enabled": false, "registry_enabled": false, "resolvconf_mode": "host_resolvconf", "retry_stagger": 5, "skydns_server": "172.32.0.3", "skydns_server_secondary": "172.32.0.4", "upstream_dns_servers": [ "10.100.10.3", "10.100.10.4" ], "volume_cross_zone_attachment": false } }

Command used to invoke ansible

ansible?

Output of ansible run

https://gist.github.com/sinaowolabi/22cadf30b89587c9f401197b98288341

Anything else we need to know

Nothing else

k8s-triage-robot commented 16 hours ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale