search

kubernetes-sigs / kubespray

Deploy a Production Ready Kubernetes Cluster
Apache License 2.0
16.06k stars 6.45k forks source link

Pods not able to ping each other on Multus interface - ARP oui unknown error #8614

Closed yockgen closed 2 years ago

yockgen commented 2 years ago

I'm following this to setup Multus second network interface in my cluster: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/multus.md

Pods not able to ping each others via MacVlan overlay, tcpdump the receiving interface seeing following error: 06:18:42.470987 ARP, Request who-has 192.168.15.227 tell 192.168.15.226, length 46 06:18:42.471229 ARP, Reply 192.168.15.227 is-at 76:10:85:ea:e4:5c (oui Unknown), length 28

Macvlan configuration as below: cat <<EOF | kubectl create -f - apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: yockgen-network spec: config: '{ "cniVersion": "0.4.0", "name": "yockgen-network", "type": "macvlan", "master": "enp0s8", "mode": "bridge",
"ipam": { "type": "whereabouts", "range": "192.168.15.225/28" } }' EOF

However, IPVlan configuration as below is working to ping each other:

cat <<EOF | kubectl create -f - apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: yockgen-network spec: config: '{ "cniVersion": "0.3.1", "name": "yockgen-network", "type": "ipvlan", "master": "enp0s8",
"capabilities": {"ips":true}, "ipam": { "type": "whereabouts", "range": "192.168.15.225/28" } }' EOF

=================================================================================== Environment:

Kubespray version (commit) (git rev-parse --short HEAD): 471585dc

Network plugin used: Multus

Full inventory with variables (ansible -i inventory/sample/inventory.ini all -m debug -a "var=hostvars[inventory_hostname]"):

node3 | SUCCESS => { "hostvars[inventory_hostname]": { "access_ip": "192.168.0.19", "ansible_check_mode": false, "ansible_config_file": "/home/yockgenm/kubespray/ansible.cfg", "ansible_diff_mode": false, "ansible_facts": {}, "ansible_forks": 5, "ansible_host": "192.168.0.19", "ansible_inventory_sources": [ "/home/yockgenm/kubespray/inventory/mycluster/hosts.yml" ], "ansible_playbook_python": "/usr/bin/python3", "ansible_verbosity": 0, "ansible_version": { "full": "2.10.15", "major": 2, "minor": 10, "revision": 15, "string": "2.10.15" }, "argocd_enabled": false, "auto_renew_certificates": false, "bin_dir": "/usr/local/bin", "cephfs_provisioner_enabled": false, "cert_manager_enabled": false, "cluster_name": "cluster.local", "container_manager": "containerd", "coredns_k8s_external_zone": "k8s_external.local", "credentials_dir": "/home/yockgenm/kubespray/inventory/mycluster/credentials", "default_kubelet_config_dir": "/etc/kubernetes/dynamic_kubelet_dir", "deploy_netchecker": false, "dns_domain": "cluster.local", "dns_mode": "coredns", "docker_bin_dir": "/usr/bin", "docker_container_storage_setup": false, "docker_daemon_graph": "/var/lib/docker", "docker_dns_servers_strict": false, "docker_iptables_enabled": "false", "docker_log_opts": "--log-opt max-size=50m --log-opt max-file=5", "docker_rpm_keepcache": 1, "dynamic_kubelet_configuration": false, "dynamic_kubelet_configuration_dir": "/etc/kubernetes/dynamic_kubelet_dir", "enable_coredns_k8s_endpoint_pod_names": false, "enable_coredns_k8s_external": false, "enable_dual_stack_networks": false, "enable_nat_default_gateway": true, "enable_nodelocaldns": true, "enable_nodelocaldns_secondary": false, "etcd_data_dir": "/var/lib/etcd", "etcd_deployment_type": "host", "event_ttl_duration": "1h0m0s", "group_names": [ "etcd", "k8s_cluster", "kube_node" ], "groups": { "all": [ "node1", "node2", "node3" ], "calico_rr": [], "etcd": [ "node1", "node2", "node3" ], "k8s_cluster": [ "node1", "node2", "node3" ], "kube_control_plane": [ "node1" ], "kube_node": [ "node1", "node2", "node3" ], "ungrouped": [] }, "helm_enabled": false, "ingress_alb_enabled": false, "ingress_nginx_enabled": false, "ingress_publish_status_address": "", "inventory_dir": "/home/yockgenm/kubespray/inventory/mycluster", "inventory_file": "/home/yockgenm/kubespray/inventory/mycluster/hosts.yml", "inventory_hostname": "node3", "inventory_hostname_short": "node3", "ip": "192.168.0.19", "k8s_image_pull_policy": "IfNotPresent", "kata_containers_enabled": false, "krew_enabled": false, "krew_root_dir": "/usr/local/krew", "kube_api_anonymous_auth": true, "kube_apiserver_insecure_port": 0, "kube_apiserver_ip": "10.233.0.1", "kube_apiserver_port": 6443, "kube_cert_dir": "/etc/kubernetes/ssl", "kube_cert_group": "kube-cert", "kube_config_dir": "/etc/kubernetes", "kube_encrypt_secret_data": false, "kube_log_level": 2, "kube_manifest_dir": "/etc/kubernetes/manifests", "kube_network_node_prefix": 24, "kube_network_node_prefix_ipv6": 120, "kube_network_plugin": "flannel", "kube_network_plugin_multus": true, "kube_pods_subnet": "10.233.64.0/18", "kube_pods_subnet_ipv6": "fd85:ee78:d8a6:8607::1:0000/112", "kube_proxy_mode": "ipvs", "kube_proxy_nodeport_addresses": [], "kube_proxy_strict_arp": false, "kube_script_dir": "/usr/local/bin/kubernetes-scripts", "kube_service_addresses": "10.233.0.0/18", "kube_service_addresses_ipv6": "fd85:ee78:d8a6:8607::1000/116", "kube_token_dir": "/etc/kubernetes/tokens", "kube_version": "v1.23.4", "kubeadm_certificate_key": "f0e7a53b907e5efeb049c515bafebe4faf37b63fe78cd96b52d6df0de9ee8bee", "kubernetes_audit": false, "loadbalancer_apiserver_healthcheck_port": 8081, "loadbalancer_apiserver_port": 6443, "local_path_provisioner_enabled": false, "local_release_dir": "/tmp/releases", "local_volume_provisioner_enabled": false, "macvlan_interface": "eth1", "metallb_enabled": false, "metallb_speaker_enabled": true, "metrics_server_enabled": false, "ndots": 2, "no_proxy_exclude_workers": false, "nodelocaldns_bind_metrics_host_ip": false, "nodelocaldns_health_port": 9254, "nodelocaldns_ip": "169.254.25.10", "nodelocaldns_second_health_port": 9256, "nodelocaldns_secondary_skew_seconds": 5, "omit": "omit_place_holder69aed336041c5bddb6abe2520ea66522475af758", "persistent_volumes_enabled": false, "playbook_dir": "/home/yockgenm/kubespray", "podsecuritypolicy_enabled": false, "rbd_provisioner_enabled": false, "registry_enabled": false, "resolvconf_mode": "host_resolvconf", "retry_stagger": 5, "skydns_server": "10.233.0.3", "skydns_server_secondary": "10.233.0.4", "volume_cross_zone_attachment": false } } node2 | SUCCESS => { "hostvars[inventory_hostname]": { "access_ip": "192.168.0.18", "ansible_check_mode": false, "ansible_config_file": "/home/yockgenm/kubespray/ansible.cfg", "ansible_diff_mode": false, "ansible_facts": {}, "ansible_forks": 5, "ansible_host": "192.168.0.18", "ansible_inventory_sources": [ "/home/yockgenm/kubespray/inventory/mycluster/hosts.yml" ], "ansible_playbook_python": "/usr/bin/python3", "ansible_verbosity": 0, "ansible_version": { "full": "2.10.15", "major": 2, "minor": 10, "revision": 15, "string": "2.10.15" }, "argocd_enabled": false, "auto_renew_certificates": false, "bin_dir": "/usr/local/bin", "cephfs_provisioner_enabled": false, "cert_manager_enabled": false, "cluster_name": "cluster.local", "container_manager": "containerd", "coredns_k8s_external_zone": "k8s_external.local", "credentials_dir": "/home/yockgenm/kubespray/inventory/mycluster/credentials", "default_kubelet_config_dir": "/etc/kubernetes/dynamic_kubelet_dir", "deploy_netchecker": false, "dns_domain": "cluster.local", "dns_mode": "coredns", "docker_bin_dir": "/usr/bin", "docker_container_storage_setup": false, "docker_daemon_graph": "/var/lib/docker", "docker_dns_servers_strict": false, "docker_iptables_enabled": "false", "docker_log_opts": "--log-opt max-size=50m --log-opt max-file=5", "docker_rpm_keepcache": 1, "dynamic_kubelet_configuration": false, "dynamic_kubelet_configuration_dir": "/etc/kubernetes/dynamic_kubelet_dir", "enable_coredns_k8s_endpoint_pod_names": false, "enable_coredns_k8s_external": false, "enable_dual_stack_networks": false, "enable_nat_default_gateway": true, "enable_nodelocaldns": true, "enable_nodelocaldns_secondary": false, "etcd_data_dir": "/var/lib/etcd", "etcd_deployment_type": "host", "event_ttl_duration": "1h0m0s", "group_names": [ "etcd", "k8s_cluster", "kube_node" ], "groups": { "all": [ "node1", "node2", "node3" ], "calico_rr": [], "etcd": [ "node1", "node2", "node3" ], "k8s_cluster": [ "node1", "node2", "node3" ], "kube_control_plane": [ "node1" ], "kube_node": [ "node1", "node2", "node3" ], "ungrouped": [] }, "helm_enabled": false, "ingress_alb_enabled": false, "ingress_nginx_enabled": false, "ingress_publish_status_address": "", "inventory_dir": "/home/yockgenm/kubespray/inventory/mycluster", "inventory_file": "/home/yockgenm/kubespray/inventory/mycluster/hosts.yml", "inventory_hostname": "node2", "inventory_hostname_short": "node2", "ip": "192.168.0.18", "k8s_image_pull_policy": "IfNotPresent", "kata_containers_enabled": false, "krew_enabled": false, "krew_root_dir": "/usr/local/krew", "kube_api_anonymous_auth": true, "kube_apiserver_insecure_port": 0, "kube_apiserver_ip": "10.233.0.1", "kube_apiserver_port": 6443, "kube_cert_dir": "/etc/kubernetes/ssl", "kube_cert_group": "kube-cert", "kube_config_dir": "/etc/kubernetes", "kube_encrypt_secret_data": false, "kube_log_level": 2, "kube_manifest_dir": "/etc/kubernetes/manifests", "kube_network_node_prefix": 24, "kube_network_node_prefix_ipv6": 120, "kube_network_plugin": "flannel", "kube_network_plugin_multus": true, "kube_pods_subnet": "10.233.64.0/18", "kube_pods_subnet_ipv6": "fd85:ee78:d8a6:8607::1:0000/112", "kube_proxy_mode": "ipvs", "kube_proxy_nodeport_addresses": [], "kube_proxy_strict_arp": false, "kube_script_dir": "/usr/local/bin/kubernetes-scripts", "kube_service_addresses": "10.233.0.0/18", "kube_service_addresses_ipv6": "fd85:ee78:d8a6:8607::1000/116", "kube_token_dir": "/etc/kubernetes/tokens", "kube_version": "v1.23.4", "kubeadm_certificate_key": "f0e7a53b907e5efeb049c515bafebe4faf37b63fe78cd96b52d6df0de9ee8bee", "kubernetes_audit": false, "loadbalancer_apiserver_healthcheck_port": 8081, "loadbalancer_apiserver_port": 6443, "local_path_provisioner_enabled": false, "local_release_dir": "/tmp/releases", "local_volume_provisioner_enabled": false, "macvlan_interface": "eth1", "metallb_enabled": false, "metallb_speaker_enabled": true, "metrics_server_enabled": false, "ndots": 2, "no_proxy_exclude_workers": false, "nodelocaldns_bind_metrics_host_ip": false, "nodelocaldns_health_port": 9254, "nodelocaldns_ip": "169.254.25.10", "nodelocaldns_second_health_port": 9256, "nodelocaldns_secondary_skew_seconds": 5, "omit": "omit_place_holder69aed336041c5bddb6abe2520ea66522475af758", "persistent_volumes_enabled": false, "playbook_dir": "/home/yockgenm/kubespray", "podsecuritypolicy_enabled": false, "rbd_provisioner_enabled": false, "registry_enabled": false, "resolvconf_mode": "host_resolvconf", "retry_stagger": 5, "skydns_server": "10.233.0.3", "skydns_server_secondary": "10.233.0.4", "volume_cross_zone_attachment": false } } node1 | SUCCESS => { "hostvars[inventory_hostname]": { "access_ip": "192.168.0.17", "ansible_check_mode": false, "ansible_config_file": "/home/yockgenm/kubespray/ansible.cfg", "ansible_diff_mode": false, "ansible_facts": {}, "ansible_forks": 5, "ansible_host": "192.168.0.17", "ansible_inventory_sources": [ "/home/yockgenm/kubespray/inventory/mycluster/hosts.yml" ], "ansible_playbook_python": "/usr/bin/python3", "ansible_verbosity": 0, "ansible_version": { "full": "2.10.15", "major": 2, "minor": 10, "revision": 15, "string": "2.10.15" }, "argocd_enabled": false, "auto_renew_certificates": false, "bin_dir": "/usr/local/bin", "cephfs_provisioner_enabled": false, "cert_manager_enabled": false, "cluster_name": "cluster.local", "container_manager": "containerd", "coredns_k8s_external_zone": "k8s_external.local", "credentials_dir": "/home/yockgenm/kubespray/inventory/mycluster/credentials", "default_kubelet_config_dir": "/etc/kubernetes/dynamic_kubelet_dir", "deploy_netchecker": false, "dns_domain": "cluster.local", "dns_mode": "coredns", "docker_bin_dir": "/usr/bin", "docker_container_storage_setup": false, "docker_daemon_graph": "/var/lib/docker", "docker_dns_servers_strict": false, "docker_iptables_enabled": "false", "docker_log_opts": "--log-opt max-size=50m --log-opt max-file=5", "docker_rpm_keepcache": 1, "dynamic_kubelet_configuration": false, "dynamic_kubelet_configuration_dir": "/etc/kubernetes/dynamic_kubelet_dir", "enable_coredns_k8s_endpoint_pod_names": false, "enable_coredns_k8s_external": false, "enable_dual_stack_networks": false, "enable_nat_default_gateway": true, "enable_nodelocaldns": true, "enable_nodelocaldns_secondary": false, "etcd_data_dir": "/var/lib/etcd", "etcd_deployment_type": "host", "event_ttl_duration": "1h0m0s", "group_names": [ "etcd", "k8s_cluster", "kube_control_plane", "kube_node" ], "groups": { "all": [ "node1", "node2", "node3" ], "calico_rr": [], "etcd": [ "node1", "node2", "node3" ], "k8s_cluster": [ "node1", "node2", "node3" ], "kube_control_plane": [ "node1" ], "kube_node": [ "node1", "node2", "node3" ], "ungrouped": [] }, "helm_enabled": false, "ingress_alb_enabled": false, "ingress_nginx_enabled": false, "ingress_publish_status_address": "", "inventory_dir": "/home/yockgenm/kubespray/inventory/mycluster", "inventory_file": "/home/yockgenm/kubespray/inventory/mycluster/hosts.yml", "inventory_hostname": "node1", "inventory_hostname_short": "node1", "ip": "192.168.0.17", "k8s_image_pull_policy": "IfNotPresent", "kata_containers_enabled": false, "krew_enabled": false, "krew_root_dir": "/usr/local/krew", "kube_api_anonymous_auth": true, "kube_apiserver_insecure_port": 0, "kube_apiserver_ip": "10.233.0.1", "kube_apiserver_port": 6443, "kube_cert_dir": "/etc/kubernetes/ssl", "kube_cert_group": "kube-cert", "kube_config_dir": "/etc/kubernetes", "kube_encrypt_secret_data": false, "kube_log_level": 2, "kube_manifest_dir": "/etc/kubernetes/manifests", "kube_network_node_prefix": 24, "kube_network_node_prefix_ipv6": 120, "kube_network_plugin": "flannel", "kube_network_plugin_multus": true, "kube_pods_subnet": "10.233.64.0/18", "kube_pods_subnet_ipv6": "fd85:ee78:d8a6:8607::1:0000/112", "kube_proxy_mode": "ipvs", "kube_proxy_nodeport_addresses": [], "kube_proxy_strict_arp": false, "kube_script_dir": "/usr/local/bin/kubernetes-scripts", "kube_service_addresses": "10.233.0.0/18", "kube_service_addresses_ipv6": "fd85:ee78:d8a6:8607::1000/116", "kube_token_dir": "/etc/kubernetes/tokens", "kube_version": "v1.23.4", "kubeadm_certificate_key": "f0e7a53b907e5efeb049c515bafebe4faf37b63fe78cd96b52d6df0de9ee8bee", "kubernetes_audit": false, "loadbalancer_apiserver_healthcheck_port": 8081, "loadbalancer_apiserver_port": 6443, "local_path_provisioner_enabled": false, "local_release_dir": "/tmp/releases", "local_volume_provisioner_enabled": false, "macvlan_interface": "eth1", "metallb_enabled": false, "metallb_speaker_enabled": true, "metrics_server_enabled": false, "ndots": 2, "no_proxy_exclude_workers": false, "nodelocaldns_bind_metrics_host_ip": false, "nodelocaldns_health_port": 9254, "nodelocaldns_ip": "169.254.25.10", "nodelocaldns_second_health_port": 9256, "nodelocaldns_secondary_skew_seconds": 5, "omit": "omit_place_holder69aed336041c5bddb6abe2520ea66522475af758", "persistent_volumes_enabled": false, "playbook_dir": "/home/yockgenm/kubespray", "podsecuritypolicy_enabled": false, "rbd_provisioner_enabled": false, "registry_enabled": false, "resolvconf_mode": "host_resolvconf", "retry_stagger": 5, "skydns_server": "10.233.0.3", "skydns_server_secondary": "10.233.0.4", "volume_cross_zone_attachment": false } }

Command used to invoke ansible:

Output of ansible run:

Anything else do we need to know: cat <<EOF | kubectl create -f - apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: yockgen-network spec: config: '{ "cniVersion": "0.4.0", "name": "yockgen-network", "type": "macvlan", "master": "enp0s8", "mode": "bridge",
"ipam": { "type": "whereabouts", "range": "192.168.15.225/28" } }' EOF

root@node1:~# kubectl exec -it test01-deployment-6f5b9549f9-fjj7t -- tcpdump -i net1 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on net1, link-type EN10MB (Ethernet), capture size 262144 bytes 06:18:42.470987 ARP, Request who-has 192.168.15.227 tell 192.168.15.226, length 46 06:18:42.471229 ARP, Reply 192.168.15.227 is-at 76:10:85:ea:e4:5c (oui Unknown), length 28 06:18:43.519617 ARP, Request who-has 192.168.15.227 tell 192.168.15.226, length 46 06:18:43.556713 ARP, Reply 192.168.15.227 is-at 76:10:85:ea:e4:5c (oui Unknown), length 28 06:18:44.511899 ARP, Request who-has 192.168.15.227 tell 192.168.15.226, length 46 06:18:44.512195 ARP, Reply 192.168.15.227 is-at 76:10:85:ea:e4:5c (oui Unknown), length 28 06:18:45.559174 ARP, Request who-has 192.168.15.227 tell 192.168.15.226, length 46 06:18:45.559847 ARP, Reply 192.168.15.227 is-at 76:10:85:ea:e4:5c (oui Unknown), length 28 06:18:46.560188 ARP, Request who-has 192.168.15.227 tell 192.168.15.226, length 46

cristicalin commented 2 years ago

@yockgen this is probably best answered by the multus project themselves, if there is an issue in the way kubespray sets up multus, we are happy to accept fixes.

yockgen commented 2 years ago

Alright, will check with Multus