takamori-tech
commented
3 years ago I met the issue like this on my Raspberry Pi 4B Cluster.
Caused by Calico registry definition in Kubespray doesn't check architecture. quay.io hasn't multiarch functionality, so image tag requires suffix -arm64.
To fix this issue in temporally, I just change YAML definition to below: kubespray/roles/download/defaults/main.yml :
calico_node_image_tag: "{{ calico_version }}-arm64"
calico_cni_image_tag: "{{ calico_cni_version }}-arm64"
calico_policy_image_tag: "{{ calico_policy_version }}-arm64"
calico_typha_image_tag: "{{ calico_typha_version }}-arm64"
ghost
dong8650
Hi there and thanks so far for this great script (kubespray).
The script runs through well, so far.
Except one thing.
It fails at the point:
TASK [network_plugin/calico : Wait for calico kubeconfig to be created] **** fatal: [node3]: FAILED! => {"changed": false, "elapsed": 300, "msg": "Timeout when waiting for file /etc/cni/net.d/calico-kubeconfig"} fatal: [node4]: FAILED! => {"changed": false, "elapsed": 300, "msg": "Timeout when waiting for file /etc/cni/net.d/calico-kubeconfig"}
By checking the path manually afterwards, on the non-master-nodes (node3 and node4), gives me:
ubuntu@node3:~$ ls /etc/cni/net.d/ calico.conflist.template
ubuntu@node4:~$ ls /etc/cni/net.d/ calico.conflist.template
..so the file calico-kubeconfig doesnt exist at all!
Should be weird if this would be a permission issue, due to i run the script as root [user].
There might already have been a related issue:
https://github.com/kubernetes-sigs/kubespray/issues/5683
...but it was not really solved, yet.
I appreciate your help and I would like to contribute to this awesome project (kubespray) in the future as well!
Cheers
Marco
P.s. please excuse if this bug report is formally not absolutely correct - I'm new to GitHub! ;)
Environment:
printf "$(uname -srm)\n$(cat /etc/os-release)\n"): Linux 5.4.0-1026-raspi aarch64 NAME="Ubuntu" VERSION="20.04.1 LTS (Focal Fossa)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 20.04.1 LTS" VERSION_ID="20.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=focal UBUNTU_CODENAME=focalansible --version): ansible 2.9.6 config file = None configured module search path = ['/home/marco/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules'] ansible python module location = /home/marco/.pyenv/versions/3.9.1/lib/python3.9/site-packages/ansible executable location = /home/marco/.pyenv/versions/3.9.1/bin/ansible python version = 3.9.1 (default, Jan 2 2021, 20:21:39) [GCC 9.3.0]python --version): Python 3.9.1Kubespray version (commit) (
git rev-parse --short HEAD): ff952924Network plugin used:
Full inventory with variables (
ansible -i inventory/sample/inventory.ini all -m debug -a "var=hostvars[inventory_hostname]"):node1 | SUCCESS => { "hostvars[inventory_hostname]": { "access_ip": "192.168.2.115", "ansible_check_mode": false, "ansible_diff_mode": false, "ansible_facts": {}, "ansible_forks": 5, "ansible_host": "192.168.2.115", "ansible_inventory_sources": [ "/home/marco/kubespray/inventory/mycluster/hosts.yaml" ], "ansible_playbook_python": "/home/marco/.pyenv/versions/3.9.1/bin/python3.9", "ansible_verbosity": 0, "ansible_version": { "full": "2.9.6", "major": 2, "minor": 9, "revision": 6, "string": "2.9.6" }, "bin_dir": "/usr/local/bin", "cephfs_provisioner_enabled": false, "cert_manager_enabled": false, "cluster_name": "cluster.local", "container_manager": "docker", "coredns_k8s_external_zone": "k8s_external.local", "credentials_dir": "/home/marco/kubespray/inventory/mycluster/credentials", "default_kubelet_config_dir": "/etc/kubernetes/dynamic_kubelet_dir", "deploy_netchecker": false, "dns_domain": "cluster.local", "dns_mode": "coredns", "docker_bin_dir": "/usr/bin", "docker_container_storage_setup": false, "docker_daemon_graph": "/var/lib/docker", "docker_dns_servers_strict": false, "docker_iptables_enabled": "false", "docker_log_opts": "--log-opt max-size=50m --log-opt max-file=5", "docker_rpm_keepcache": 0, "dynamic_kubelet_configuration": false, "dynamic_kubelet_configuration_dir": "/etc/kubernetes/dynamic_kubelet_dir", "enable_coredns_k8s_endpoint_pod_names": false, "enable_coredns_k8s_external": false, "enable_nat_default_gateway": true, "enable_nodelocaldns": true, "etcd_data_dir": "/var/lib/etcd", "etcd_deployment_type": "docker", "etcd_kubeadm_enabled": false, "event_ttl_duration": "1h0m0s", "force_certificate_regeneration": false, "group_names": [ "etcd", "k8s-cluster", "kube-master", "kube-node" ], "groups": { "all": [ "node1", "node2", "node3", "node4" ], "calico-rr": [], "etcd": [ "node1", "node2", "node3" ], "k8s-cluster": [ "node1", "node2", "node3", "node4" ], "kube-master": [ "node1", "node2" ], "kube-node": [ "node1", "node2", "node3", "node4" ], "ungrouped": [] }, "helm_enabled": false, "ingress_alb_enabled": false, "ingress_ambassador_enabled": false, "ingress_nginx_enabled": false, "ingress_publish_status_address": "", "inventory_dir": "/home/marco/kubespray/inventory/mycluster", "inventory_file": "/home/marco/kubespray/inventory/mycluster/hosts.yaml", "inventory_hostname": "node1", "inventory_hostname_short": "node1", "ip": "192.168.2.115", "k8s_image_pull_policy": "IfNotPresent", "kata_containers_enabled": false, "kube_api_anonymous_auth": true, "kube_apiserver_insecure_port": 0, "kube_apiserver_ip": "10.233.0.1", "kube_apiserver_port": 6443, "kube_cert_dir": "/etc/kubernetes/ssl", "kube_cert_group": "kube-cert", "kube_config_dir": "/etc/kubernetes", "kube_encrypt_secret_data": false, "kube_log_level": 2, "kube_manifest_dir": "/etc/kubernetes/manifests", "kube_network_node_prefix": 24, "kube_network_plugin": "calico", "kube_network_plugin_multus": false, "kube_pods_subnet": "10.233.64.0/18", "kube_proxy_mode": "ipvs", "kube_proxy_nodeport_addresses": [], "kube_proxy_strict_arp": false, "kube_script_dir": "/usr/local/bin/kubernetes-scripts", "kube_service_addresses": "10.233.0.0/18", "kube_token_dir": "/etc/kubernetes/tokens", "kube_version": "v1.19.6", "kubeadm_certificate_key": "eacdef11bd212e5fa6bf513c15fa607ebeff3698aaade46eb5e6baa600ee1e9a", "kubernetes_audit": false, "loadbalancer_apiserver_healthcheck_port": 8081, "loadbalancer_apiserver_port": 6443, "local_path_provisioner_enabled": false, "local_release_dir": "/tmp/releases", "local_volume_provisioner_enabled": false, "macvlan_interface": "eth1", "metallb_enabled": false, "metrics_server_enabled": false, "ndots": 2, "no_proxy_exclude_workers": false, "nodelocaldns_health_port": 9254, "nodelocaldns_ip": "169.254.25.10", "omit": "omit_place_holdera1f9366412dbba2c0c20b38f192197f2042bb27a", "persistent_volumes_enabled": false, "playbook_dir": "/home/marco/kubespray", "podsecuritypolicy_enabled": false, "rbd_provisioner_enabled": false, "registry_enabled": false, "resolvconf_mode": "docker_dns", "retry_stagger": 5, "skydns_server": "10.233.0.3", "skydns_server_secondary": "10.233.0.4", "volume_cross_zone_attachment": false } } node2 | SUCCESS => { "hostvars[inventory_hostname]": { "access_ip": "192.168.2.117", "ansible_check_mode": false, "ansible_diff_mode": false, "ansible_facts": {}, "ansible_forks": 5, "ansible_host": "192.168.2.117", "ansible_inventory_sources": [ "/home/marco/kubespray/inventory/mycluster/hosts.yaml" ], "ansible_playbook_python": "/home/marco/.pyenv/versions/3.9.1/bin/python3.9", "ansible_verbosity": 0, "ansible_version": { "full": "2.9.6", "major": 2, "minor": 9, "revision": 6, "string": "2.9.6" }, "bin_dir": "/usr/local/bin", "cephfs_provisioner_enabled": false, "cert_manager_enabled": false, "cluster_name": "cluster.local", "container_manager": "docker", "coredns_k8s_external_zone": "k8s_external.local", "credentials_dir": "/home/marco/kubespray/inventory/mycluster/credentials", "default_kubelet_config_dir": "/etc/kubernetes/dynamic_kubelet_dir", "deploy_netchecker": false, "dns_domain": "cluster.local", "dns_mode": "coredns", "docker_bin_dir": "/usr/bin", "docker_container_storage_setup": false, "docker_daemon_graph": "/var/lib/docker", "docker_dns_servers_strict": false, "docker_iptables_enabled": "false", "docker_log_opts": "--log-opt max-size=50m --log-opt max-file=5", "docker_rpm_keepcache": 0, "dynamic_kubelet_configuration": false, "dynamic_kubelet_configuration_dir": "/etc/kubernetes/dynamic_kubelet_dir", "enable_coredns_k8s_endpoint_pod_names": false, "enable_coredns_k8s_external": false, "enable_nat_default_gateway": true, "enable_nodelocaldns": true, "etcd_data_dir": "/var/lib/etcd", "etcd_deployment_type": "docker", "etcd_kubeadm_enabled": false, "event_ttl_duration": "1h0m0s", "force_certificate_regeneration": false, "group_names": [ "etcd", "k8s-cluster", "kube-master", "kube-node" ], "groups": { "all": [ "node1", "node2", "node3", "node4" ], "calico-rr": [], "etcd": [ "node1", "node2", "node3" ], "k8s-cluster": [ "node1", "node2", "node3", "node4" ], "kube-master": [ "node1", "node2" ], "kube-node": [ "node1", "node2", "node3", "node4" ], "ungrouped": [] }, "helm_enabled": false, "ingress_alb_enabled": false, "ingress_ambassador_enabled": false, "ingress_nginx_enabled": false, "ingress_publish_status_address": "", "inventory_dir": "/home/marco/kubespray/inventory/mycluster", "inventory_file": "/home/marco/kubespray/inventory/mycluster/hosts.yaml", "inventory_hostname": "node2", "inventory_hostname_short": "node2", "ip": "192.168.2.117", "k8s_image_pull_policy": "IfNotPresent", "kata_containers_enabled": false, "kube_api_anonymous_auth": true, "kube_apiserver_insecure_port": 0, "kube_apiserver_ip": "10.233.0.1", "kube_apiserver_port": 6443, "kube_cert_dir": "/etc/kubernetes/ssl", "kube_cert_group": "kube-cert", "kube_config_dir": "/etc/kubernetes", "kube_encrypt_secret_data": false, "kube_log_level": 2, "kube_manifest_dir": "/etc/kubernetes/manifests", "kube_network_node_prefix": 24, "kube_network_plugin": "calico", "kube_network_plugin_multus": false, "kube_pods_subnet": "10.233.64.0/18", "kube_proxy_mode": "ipvs", "kube_proxy_nodeport_addresses": [], "kube_proxy_strict_arp": false, "kube_script_dir": "/usr/local/bin/kubernetes-scripts", "kube_service_addresses": "10.233.0.0/18", "kube_token_dir": "/etc/kubernetes/tokens", "kube_version": "v1.19.6", "kubeadm_certificate_key": "eacdef11bd212e5fa6bf513c15fa607ebeff3698aaade46eb5e6baa600ee1e9a", "kubernetes_audit": false, "loadbalancer_apiserver_healthcheck_port": 8081, "loadbalancer_apiserver_port": 6443, "local_path_provisioner_enabled": false, "local_release_dir": "/tmp/releases", "local_volume_provisioner_enabled": false, "macvlan_interface": "eth1", "metallb_enabled": false, "metrics_server_enabled": false, "ndots": 2, "no_proxy_exclude_workers": false, "nodelocaldns_health_port": 9254, "nodelocaldns_ip": "169.254.25.10", "omit": "omit_place_holdera1f9366412dbba2c0c20b38f192197f2042bb27a", "persistent_volumes_enabled": false, "playbook_dir": "/home/marco/kubespray", "podsecuritypolicy_enabled": false, "rbd_provisioner_enabled": false, "registry_enabled": false, "resolvconf_mode": "docker_dns", "retry_stagger": 5, "skydns_server": "10.233.0.3", "skydns_server_secondary": "10.233.0.4", "volume_cross_zone_attachment": false } } node3 | SUCCESS => { "hostvars[inventory_hostname]": { "access_ip": "192.168.2.116", "ansible_check_mode": false, "ansible_diff_mode": false, "ansible_facts": {}, "ansible_forks": 5, "ansible_host": "192.168.2.116", "ansible_inventory_sources": [ "/home/marco/kubespray/inventory/mycluster/hosts.yaml" ], "ansible_playbook_python": "/home/marco/.pyenv/versions/3.9.1/bin/python3.9", "ansible_verbosity": 0, "ansible_version": { "full": "2.9.6", "major": 2, "minor": 9, "revision": 6, "string": "2.9.6" }, "bin_dir": "/usr/local/bin", "cephfs_provisioner_enabled": false, "cert_manager_enabled": false, "cluster_name": "cluster.local", "container_manager": "docker", "coredns_k8s_external_zone": "k8s_external.local", "credentials_dir": "/home/marco/kubespray/inventory/mycluster/credentials", "default_kubelet_config_dir": "/etc/kubernetes/dynamic_kubelet_dir", "deploy_netchecker": false, "dns_domain": "cluster.local", "dns_mode": "coredns", "docker_bin_dir": "/usr/bin", "docker_container_storage_setup": false, "docker_daemon_graph": "/var/lib/docker", "docker_dns_servers_strict": false, "docker_iptables_enabled": "false", "docker_log_opts": "--log-opt max-size=50m --log-opt max-file=5", "docker_rpm_keepcache": 0, "dynamic_kubelet_configuration": false, "dynamic_kubelet_configuration_dir": "/etc/kubernetes/dynamic_kubelet_dir", "enable_coredns_k8s_endpoint_pod_names": false, "enable_coredns_k8s_external": false, "enable_nat_default_gateway": true, "enable_nodelocaldns": true, "etcd_data_dir": "/var/lib/etcd", "etcd_deployment_type": "docker", "etcd_kubeadm_enabled": false, "event_ttl_duration": "1h0m0s", "force_certificate_regeneration": false, "group_names": [ "etcd", "k8s-cluster", "kube-node" ], "groups": { "all": [ "node1", "node2", "node3", "node4" ], "calico-rr": [], "etcd": [ "node1", "node2", "node3" ], "k8s-cluster": [ "node1", "node2", "node3", "node4" ], "kube-master": [ "node1", "node2" ], "kube-node": [ "node1", "node2", "node3", "node4" ], "ungrouped": [] }, "helm_enabled": false, "ingress_alb_enabled": false, "ingress_ambassador_enabled": false, "ingress_nginx_enabled": false, "ingress_publish_status_address": "", "inventory_dir": "/home/marco/kubespray/inventory/mycluster", "inventory_file": "/home/marco/kubespray/inventory/mycluster/hosts.yaml", "inventory_hostname": "node3", "inventory_hostname_short": "node3", "ip": "192.168.2.116", "k8s_image_pull_policy": "IfNotPresent", "kata_containers_enabled": false, "kube_api_anonymous_auth": true, "kube_apiserver_insecure_port": 0, "kube_apiserver_ip": "10.233.0.1", "kube_apiserver_port": 6443, "kube_cert_dir": "/etc/kubernetes/ssl", "kube_cert_group": "kube-cert", "kube_config_dir": "/etc/kubernetes", "kube_encrypt_secret_data": false, "kube_log_level": 2, "kube_manifest_dir": "/etc/kubernetes/manifests", "kube_network_node_prefix": 24, "kube_network_plugin": "calico", "kube_network_plugin_multus": false, "kube_pods_subnet": "10.233.64.0/18", "kube_proxy_mode": "ipvs", "kube_proxy_nodeport_addresses": [], "kube_proxy_strict_arp": false, "kube_script_dir": "/usr/local/bin/kubernetes-scripts", "kube_service_addresses": "10.233.0.0/18", "kube_token_dir": "/etc/kubernetes/tokens", "kube_version": "v1.19.6", "kubeadm_certificate_key": "eacdef11bd212e5fa6bf513c15fa607ebeff3698aaade46eb5e6baa600ee1e9a", "kubernetes_audit": false, "loadbalancer_apiserver_healthcheck_port": 8081, "loadbalancer_apiserver_port": 6443, "local_path_provisioner_enabled": false, "local_release_dir": "/tmp/releases", "local_volume_provisioner_enabled": false, "macvlan_interface": "eth1", "metallb_enabled": false, "metrics_server_enabled": false, "ndots": 2, "no_proxy_exclude_workers": false, "nodelocaldns_health_port": 9254, "nodelocaldns_ip": "169.254.25.10", "omit": "omit_place_holdera1f9366412dbba2c0c20b38f192197f2042bb27a", "persistent_volumes_enabled": false, "playbook_dir": "/home/marco/kubespray", "podsecuritypolicy_enabled": false, "rbd_provisioner_enabled": false, "registry_enabled": false, "resolvconf_mode": "docker_dns", "retry_stagger": 5, "skydns_server": "10.233.0.3", "skydns_server_secondary": "10.233.0.4", "volume_cross_zone_attachment": false } } node4 | SUCCESS => { "hostvars[inventory_hostname]": { "access_ip": "192.168.2.118", "ansible_check_mode": false, "ansible_diff_mode": false, "ansible_facts": {}, "ansible_forks": 5, "ansible_host": "192.168.2.118", "ansible_inventory_sources": [ "/home/marco/kubespray/inventory/mycluster/hosts.yaml" ], "ansible_playbook_python": "/home/marco/.pyenv/versions/3.9.1/bin/python3.9", "ansible_verbosity": 0, "ansible_version": { "full": "2.9.6", "major": 2, "minor": 9, "revision": 6, "string": "2.9.6" }, "bin_dir": "/usr/local/bin", "cephfs_provisioner_enabled": false, "cert_manager_enabled": false, "cluster_name": "cluster.local", "container_manager": "docker", "coredns_k8s_external_zone": "k8s_external.local", "credentials_dir": "/home/marco/kubespray/inventory/mycluster/credentials", "default_kubelet_config_dir": "/etc/kubernetes/dynamic_kubelet_dir", "deploy_netchecker": false, "dns_domain": "cluster.local", "dns_mode": "coredns", "docker_bin_dir": "/usr/bin", "docker_container_storage_setup": false, "docker_daemon_graph": "/var/lib/docker", "docker_dns_servers_strict": false, "docker_iptables_enabled": "false", "docker_log_opts": "--log-opt max-size=50m --log-opt max-file=5", "docker_rpm_keepcache": 0, "dynamic_kubelet_configuration": false, "dynamic_kubelet_configuration_dir": "/etc/kubernetes/dynamic_kubelet_dir", "enable_coredns_k8s_endpoint_pod_names": false, "enable_coredns_k8s_external": false, "enable_nat_default_gateway": true, "enable_nodelocaldns": true, "etcd_data_dir": "/var/lib/etcd", "etcd_kubeadm_enabled": false, "event_ttl_duration": "1h0m0s", "force_certificate_regeneration": false, "group_names": [ "k8s-cluster", "kube-node" ], "groups": { "all": [ "node1", "node2", "node3", "node4" ], "calico-rr": [], "etcd": [ "node1", "node2", "node3" ], "k8s-cluster": [ "node1", "node2", "node3", "node4" ], "kube-master": [ "node1", "node2" ], "kube-node": [ "node1", "node2", "node3", "node4" ], "ungrouped": [] }, "helm_enabled": false, "ingress_alb_enabled": false, "ingress_ambassador_enabled": false, "ingress_nginx_enabled": false, "ingress_publish_status_address": "", "inventory_dir": "/home/marco/kubespray/inventory/mycluster", "inventory_file": "/home/marco/kubespray/inventory/mycluster/hosts.yaml", "inventory_hostname": "node4", "inventory_hostname_short": "node4", "ip": "192.168.2.118", "k8s_image_pull_policy": "IfNotPresent", "kata_containers_enabled": false, "kube_api_anonymous_auth": true, "kube_apiserver_insecure_port": 0, "kube_apiserver_ip": "10.233.0.1", "kube_apiserver_port": 6443, "kube_cert_dir": "/etc/kubernetes/ssl", "kube_cert_group": "kube-cert", "kube_config_dir": "/etc/kubernetes", "kube_encrypt_secret_data": false, "kube_log_level": 2, "kube_manifest_dir": "/etc/kubernetes/manifests", "kube_network_node_prefix": 24, "kube_network_plugin": "calico", "kube_network_plugin_multus": false, "kube_pods_subnet": "10.233.64.0/18", "kube_proxy_mode": "ipvs", "kube_proxy_nodeport_addresses": [], "kube_proxy_strict_arp": false, "kube_script_dir": "/usr/local/bin/kubernetes-scripts", "kube_service_addresses": "10.233.0.0/18", "kube_token_dir": "/etc/kubernetes/tokens", "kube_version": "v1.19.6", "kubeadm_certificate_key": "eacdef11bd212e5fa6bf513c15fa607ebeff3698aaade46eb5e6baa600ee1e9a", "kubernetes_audit": false, "loadbalancer_apiserver_healthcheck_port": 8081, "loadbalancer_apiserver_port": 6443, "local_path_provisioner_enabled": false, "local_release_dir": "/tmp/releases", "local_volume_provisioner_enabled": false, "macvlan_interface": "eth1", "metallb_enabled": false, "metrics_server_enabled": false, "ndots": 2, "no_proxy_exclude_workers": false, "nodelocaldns_health_port": 9254, "nodelocaldns_ip": "169.254.25.10", "omit": "omit_place_holdera1f9366412dbba2c0c20b38f192197f2042bb27a", "persistent_volumes_enabled": false, "playbook_dir": "/home/marco/kubespray", "podsecuritypolicy_enabled": false, "rbd_provisioner_enabled": false, "registry_enabled": false, "resolvconf_mode": "docker_dns", "retry_stagger": 5, "skydns_server": "10.233.0.3", "skydns_server_secondary": "10.233.0.4", "volume_cross_zone_attachment": false } } Command used to invoke ansible: ansible-playbook -i inventory/mycluster/hosts.yaml --user=root cluster.yml
Output of ansible run:
Anything else do we need to know: