search

kubeovn / kube-ovn

A Bridge between SDN and Cloud Native (Project under CNCF)
https://kubeovn.github.io/docs/stable/en/
Apache License 2.0
1.96k stars 448 forks source link

resolve MAC address of XXX timeout #2646

Closed willzhang closed 1 year ago

willzhang commented 1 year ago

Expected Behavior

after install kubeovn coredns in running status

Actual Behavior

after install kubeovn coredns in ContainerCreatingstatus

use NETWORK_TYPE="vlan" in vmware vsphere vm environment.

image

Steps to Reproduce the Problem

1、change install.sh config

#!/usr/bin/env bash
set -euo pipefail

IPV6=${IPV6:-false}
DUAL_STACK=${DUAL_STACK:-false}
ENABLE_SSL=${ENABLE_SSL:-false}
ENABLE_VLAN=${ENABLE_VLAN:-false}
CHECK_GATEWAY=${CHECK_GATEWAY:-true}
LOGICAL_GATEWAY=${LOGICAL_GATEWAY:-false}
U2O_INTERCONNECTION=${U2O_INTERCONNECTION:-false}
ENABLE_MIRROR=${ENABLE_MIRROR:-false}
VLAN_NIC=${VLAN_NIC:-}
HW_OFFLOAD=${HW_OFFLOAD:-false}
ENABLE_LB=${ENABLE_LB:-true}
ENABLE_NP=${ENABLE_NP:-true}
ENABLE_EIP_SNAT=${ENABLE_EIP_SNAT:-true}
LS_DNAT_MOD_DL_DST=${LS_DNAT_MOD_DL_DST:-true}
ENABLE_EXTERNAL_VPC=${ENABLE_EXTERNAL_VPC:-true}
CNI_CONFIG_PRIORITY=${CNI_CONFIG_PRIORITY:-01}
ENABLE_LB_SVC=${ENABLE_LB_SVC:-false}
ENABLE_KEEP_VM_IP=${ENABLE_KEEP_VM_IP:-true}
# exchange link names of OVS bridge and the provider nic
# in the default provider-network
EXCHANGE_LINK_NAME=true
EXCHANGE_LINK_NAME=${EXCHANGE_LINK_NAME:-false}
# The nic to support container network can be a nic name or a group of regex
# separated by comma, if empty will use the nic that the default route use
IFACE=ens160
IFACE=${IFACE:-}
# Specifies the name of the dpdk tunnel iface.
# Note that the dpdk tunnel iface and tunnel ip cidr should be diffierent with Kubernetes api cidr,otherwise the route will be a problem.
DPDK_TUNNEL_IFACE=${DPDK_TUNNEL_IFACE:-br-phy}
ENABLE_BIND_LOCAL_IP=${ENABLE_BIND_LOCAL_IP:-true}

CNI_CONF_DIR="/etc/cni/net.d"
CNI_BIN_DIR="/opt/cni/bin"

REGISTRY="kubeovn"
VERSION="v1.11.1"
IMAGE_PULL_POLICY="IfNotPresent"
POD_CIDR="192.168.72.0/24"                # Do NOT overlap with NODE/SVC/JOIN CIDR
POD_GATEWAY="192.168.72.8"
SVC_CIDR="10.96.0.0/22"                # Do NOT overlap with NODE/POD/JOIN CIDR
JOIN_CIDR="20.65.0.0/16"              # Do NOT overlap with NODE/POD/SVC CIDR
PINGER_EXTERNAL_ADDRESS="114.114.114.114"  # Pinger check external ip probe
PINGER_EXTERNAL_DOMAIN="alauda.cn"         # Pinger check external domain probe
SVC_YAML_IPFAMILYPOLICY=""
if [ "$IPV6" = "true" ]; then
  POD_CIDR="fd00:10:16::/64"                # Do NOT overlap with NODE/SVC/JOIN CIDR
  POD_GATEWAY="fd00:10:16::1"
  SVC_CIDR="fd00:10:96::/112"               # Do NOT overlap with NODE/POD/JOIN CIDR
  JOIN_CIDR="fd00:100:64::/64"              # Do NOT overlap with NODE/POD/SVC CIDR
  PINGER_EXTERNAL_ADDRESS="2400:3200::1"
  PINGER_EXTERNAL_DOMAIN="google.com"
fi
if [ "$DUAL_STACK" = "true" ]; then
  POD_CIDR="10.16.0.0/16,fd00:10:16::/64"                # Do NOT overlap with NODE/SVC/JOIN CIDR
  POD_GATEWAY="10.16.0.1,fd00:10:16::1"
  SVC_CIDR="10.96.0.0/12,fd00:10:96::/112"               # Do NOT overlap with NODE/POD/JOIN CIDR
  JOIN_CIDR="100.64.0.0/16,fd00:100:64::/64"             # Do NOT overlap with NODE/POD/SVC CIDR
  PINGER_EXTERNAL_ADDRESS="114.114.114.114,2400:3200::1"
  PINGER_EXTERNAL_DOMAIN="google.com"
  SVC_YAML_IPFAMILYPOLICY="ipFamilyPolicy: PreferDualStack"
fi

EXCLUDE_IPS="192.168.72.1..192.168.72.200"                                    # EXCLUDE_IPS for default subnet
LABEL="node-role.kubernetes.io/control-plane"     # The node label to deploy OVN DB
DEPRECATED_LABEL="node-role.kubernetes.io/master" # The node label to deploy OVN DB in earlier versions
NETWORK_TYPE="vlan"                             # geneve or vlan
TUNNEL_TYPE="geneve"                              # geneve, vxlan or stt. ATTENTION: some networkpolicy cannot take effect when using vxlan and stt need custom compile ovs kernel mod
ule
POD_NIC_TYPE="veth-pair"                          # veth-pair or internal-port
POD_DEFAULT_FIP_TYPE=""                           # iptables, pod can set iptables fip automatically by enable fip annotation

# VLAN Config only take effect when NETWORK_TYPE is vlan
PROVIDER_NAME="provider"
VLAN_INTERFACE_NAME="ens160"
VLAN_NAME="ovn-vlan"
VLAN_ID="0"

if [ "$ENABLE_VLAN" = "true" ]; then
  NETWORK_TYPE="vlan"
  if [ "$VLAN_NIC" != "" ]; then
    VLAN_INTERFACE_NAME="$VLAN_NIC"
  fi
fi

2、run install.bash

bash install.sh

3、install logs

root@ubuntu:~/kube-ovn# bash install.sh 
-------------------------------
Kube-OVN Version:     v1.11.1
Default Network Mode: vlan
Default Vlan Nic:     ens160
Default Vlan ID:      0
Default Subnet CIDR:  192.168.72.0/24
Join Subnet CIDR:     20.65.0.0/16
Enable SVC LB:        true
Enable Networkpolicy: true
Enable EIP and SNAT:  true
Enable Mirror:        false
-------------------------------
[Step 1/6] Label kube-ovn-master node and label datapath type
node/ubuntu labeled
-------------------------------

[Step 2/6] Install OVN components
Install OVN DB in 192.168.72.34,
customresourcedefinition.apiextensions.k8s.io/vpc-dnses.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/switch-lb-rules.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/vpc-nat-gateways.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/iptables-eips.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/iptables-fip-rules.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/iptables-dnat-rules.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/iptables-snat-rules.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/ovn-eips.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/ovn-fips.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/ovn-snat-rules.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/vpcs.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/ips.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/vips.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/subnets.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/vlans.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/provider-networks.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/security-groups.kubeovn.io created
serviceaccount/ovn created
clusterrole.rbac.authorization.k8s.io/system:ovn created
clusterrolebinding.rbac.authorization.k8s.io/ovn created
service/ovn-nb created
service/ovn-sb created
service/ovn-northd created
deployment.apps/ovn-central created
daemonset.apps/ovs-ovn created
Waiting for deployment "ovn-central" rollout to finish: 0 out of 1 new replicas have been updated...
Waiting for deployment "ovn-central" rollout to finish: 0 of 1 updated replicas are available...
deployment "ovn-central" successfully rolled out
-------------------------------

[Step 3/6] Install Kube-OVN
deployment.apps/kube-ovn-controller created
daemonset.apps/kube-ovn-cni created
daemonset.apps/kube-ovn-pinger created
deployment.apps/kube-ovn-monitor created
service/kube-ovn-monitor created
service/kube-ovn-pinger created
service/kube-ovn-controller created
service/kube-ovn-cni created
Waiting for deployment "kube-ovn-controller" rollout to finish: 0 of 1 updated replicas are available...
deployment "kube-ovn-controller" successfully rolled out
Waiting for daemon set "kube-ovn-cni" rollout to finish: 0 of 1 updated pods are available...
daemon set "kube-ovn-cni" successfully rolled out
-------------------------------

[Step 4/6] Delete pod that not in host network mode
pod "coredns-565d847f94-gb695" deleted
pod "coredns-565d847f94-pdzqz" deleted
daemon set "kube-ovn-pinger" successfully rolled out
Waiting for deployment "coredns" rollout to finish: 0 of 2 updated replicas are available...
error: deployment "coredns" exceeded its progress deadline
root@ubuntu:~/kube-ovn# kubectl get pods 
No resources found in default namespace.
root@ubuntu:~/kube-ovn# kubectl get pods  -A
NAMESPACE     NAME                                   READY   STATUS              RESTARTS      AGE
kube-system   coredns-565d847f94-96fl4               0/1     ContainerCreating   0             17m
kube-system   coredns-565d847f94-j78sv               0/1     ContainerCreating   0             21m
kube-system   etcd-ubuntu                            1/1     Running             2 (24m ago)   51m
kube-system   kube-apiserver-ubuntu                  1/1     Running             2 (24m ago)   51m
kube-system   kube-controller-manager-ubuntu         1/1     Running             2 (24m ago)   51m
kube-system   kube-ovn-cni-lggm7                     1/1     Running             0             21m
kube-system   kube-ovn-controller-5966b7f576-8knjm   1/1     Running             0             21m
kube-system   kube-ovn-monitor-9754c455c-88bl5       1/1     Running             0             21m
kube-system   kube-ovn-pinger-m6x9l                  0/1     ContainerCreating   0             12m
kube-system   kube-proxy-gwdkn                       1/1     Running             2 (24m ago)   51m
kube-system   kube-scheduler-ubuntu                  1/1     Running             2 (24m ago)   51m
kube-system   ovn-central-6877744858-l5mqk           1/1     Running             0             22m
kube-system   ovs-ovn-747kg                          1/1     Running             0             22m

4、coredns logs

root@ubuntu:~/kube-ovn# kubectl -n kube-system describe pods coredns-565d847f94-96fl4 
Name:                 coredns-565d847f94-96fl4
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Service Account:      coredns
Node:                 ubuntu/192.168.72.34
Start Time:           Thu, 13 Apr 2023 00:42:26 +0800
Labels:               k8s-app=kube-dns
                      pod-template-hash=565d847f94
Annotations:          ovn.kubernetes.io/allocated: true
                      ovn.kubernetes.io/cidr: 192.168.72.0/24
                      ovn.kubernetes.io/gateway: 192.168.72.8
                      ovn.kubernetes.io/ip_address: 192.168.72.204
                      ovn.kubernetes.io/logical_switch: ovn-default
                      ovn.kubernetes.io/mac_address: 00:00:00:39:7E:CC
                      ovn.kubernetes.io/pod_nic_type: veth-pair
                      ovn.kubernetes.io/provider_network: provider
                      ovn.kubernetes.io/routed: true
                      ovn.kubernetes.io/vlan_id: 0
Status:               Pending
IP:                   
IPs:                  <none>
Controlled By:        ReplicaSet/coredns-565d847f94
Containers:
  coredns:
    Container ID:  
    Image:         registry.k8s.io/coredns/coredns:v1.9.3
    Image ID:      
    Ports:         53/UDP, 53/TCP, 9153/TCP
    Host Ports:    0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
    Readiness:    http-get http://:8181/ready delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-t7prz (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns
    Optional:  false
  kube-api-access-t7prz:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 CriticalAddonsOnly op=Exists
                             node-role.kubernetes.io/control-plane:NoSchedule
                             node-role.kubernetes.io/master:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age    From               Message
  ----     ------                  ----   ----               -------
  Normal   Scheduled               21m    default-scheduler  Successfully assigned kube-system/coredns-565d847f94-96fl4 to ubuntu
  Warning  FailedCreatePodSandBox  18m    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d9f5b6d3cb9bd121403632c3c42b5a3600b1084b60f1e840a63dfcb10df84b47": plugin type="kube-ovn" failed (add): RPC failed; request ip return 500 configure nic failed network 192.168.72.204/24 with gateway 192.168.72.8 is not ready for interface eth0 after 200 checks: resolve MAC address of 192.168.72.8 timeout: read packet 00:00:00:39:7e:cc: i/o timeout
  Warning  FailedCreatePodSandBox  14m    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "720457a14038737419b34a2dc92ae677b68270c9d3d91dfe472c00f8923f4565": plugin type="kube-ovn" failed (add): RPC failed; request ip return 500 configure nic failed network 192.168.72.204/24 with gateway 192.168.72.8 is not ready for interface eth0 after 200 checks: resolve MAC address of 192.168.72.8 timeout: read packet 00:00:00:39:7e:cc: i/o timeout
  Warning  FailedCreatePodSandBox  11m    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "31cf59eb23ab5e44e538226f1ac954a9216753ca9ff5f832bbbf6f590f68e9bc": plugin type="kube-ovn" failed (add): RPC failed; request ip return 500 configure nic failed network 192.168.72.204/24 with gateway 192.168.72.8 is not ready for interface eth0 after 200 checks: resolve MAC address of 192.168.72.8 timeout: read packet 00:00:00:39:7e:cc: i/o timeout
  Warning  FailedCreatePodSandBox  7m51s  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e9c3a87a801a2d7337c015942ef3ef695a4093bddd87e748ca63f40fcacb2112": plugin type="kube-ovn" failed (add): RPC failed; request ip return 500 configure nic failed network 192.168.72.204/24 with gateway 192.168.72.8 is not ready for interface eth0 after 200 checks: resolve MAC address of 192.168.72.8 timeout: read packet 00:00:00:39:7e:cc: i/o timeout
  Warning  FailedCreatePodSandBox  4m26s  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "742bf1b4f5420ab94716e7dee2a45115e8edcfb4258e07cb665c74de9cab2764": plugin type="kube-ovn" failed (add): RPC failed; request ip return 500 configure nic failed network 192.168.72.204/24 with gateway 192.168.72.8 is not ready for interface eth0 after 200 checks: resolve MAC address of 192.168.72.8 timeout: read packet 00:00:00:39:7e:cc: i/o timeout
  Warning  FailedCreatePodSandBox  60s    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "7d4ddd9d4c9fa9482e6df170a82cb83ef1ecb5417cf41f472baee441fafd70a7": plugin type="kube-ovn" failed (add): RPC failed; request ip return 500 configure nic failed network 192.168.72.204/24 with gateway 192.168.72.8 is not ready for interface eth0 after 200 checks: resolve MAC address of 192.168.72.8 timeout: read packet 00:00:00:39:7e:cc: i/o timeout

4、kubelet logs

Apr 13 00:48:13 ubuntu kubelet[1180]: I0413 00:48:13.002578    1180 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-config-openvswitch\" (UniqueName: \"kubernetes.io/host-path/32362817-8066-4b67-8c75-c2a59d2f0379-host-config-openvswitch\") pod \"kube-ovn-pinger-m6x9l\" (UID: \"32362817-8066-4b67-8c75-c2a59d2f0379\") " pod="kube-system/kube-ovn-pinger-m6x9l"
Apr 13 00:48:13 ubuntu kubelet[1180]: I0413 00:48:13.002633    1180 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-ovn-log\" (UniqueName: \"kubernetes.io/host-path/32362817-8066-4b67-8c75-c2a59d2f0379-kube-ovn-log\") pod \"kube-ovn-pinger-m6x9l\" (UID: \"32362817-8066-4b67-8c75-c2a59d2f0379\") " pod="kube-system/kube-ovn-pinger-m6x9l"
Apr 13 00:48:13 ubuntu kubelet[1180]: I0413 00:48:13.002683    1180 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-modules\" (UniqueName: \"kubernetes.io/host-path/32362817-8066-4b67-8c75-c2a59d2f0379-host-modules\") pod \"kube-ovn-pinger-m6x9l\" (UID: \"32362817-8066-4b67-8c75-c2a59d2f0379\") " pod="kube-system/kube-ovn-pinger-m6x9l"
Apr 13 00:49:18 ubuntu kubelet[1180]: E0413 00:49:18.180994    1180 remote_runtime.go:222] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"720457a14038737419b34a2dc92ae677b68270c9d3d91dfe472c00f8923f4565\": plugin type=\"kube-ovn\" failed (add): RPC failed; request ip return 500 configure nic failed network 192.168.72.204/24 with gateway 192.168.72.8 is not ready for interface eth0 after 200 checks: resolve MAC address of 192.168.72.8 timeout: read packet 00:00:00:39:7e:cc: i/o timeout"
Apr 13 00:49:18 ubuntu kubelet[1180]: E0413 00:49:18.181044    1180 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"720457a14038737419b34a2dc92ae677b68270c9d3d91dfe472c00f8923f4565\": plugin type=\"kube-ovn\" failed (add): RPC failed; request ip return 500 configure nic failed network 192.168.72.204/24 with gateway 192.168.72.8 is not ready for interface eth0 after 200 checks: resolve MAC address of 192.168.72.8 timeout: read packet 00:00:00:39:7e:cc: i/o timeout" pod="kube-system/coredns-565d847f94-96fl4"
Apr 13 00:49:18 ubuntu kubelet[1180]: E0413 00:49:18.181071    1180 kuberuntime_manager.go:772] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"720457a14038737419b34a2dc92ae677b68270c9d3d91dfe472c00f8923f4565\": plugin type=\"kube-ovn\" failed (add): RPC failed; request ip return 500 configure nic failed network 192.168.72.204/24 with gateway 192.168.72.8 is not ready for interface eth0 after 200 checks: resolve MAC address of 192.168.72.8 timeout: read packet 00:00:00:39:7e:cc: i/o timeout" pod="kube-system/coredns-565d847f94-96fl4"
Apr 13 00:49:18 ubuntu kubelet[1180]: E0413 00:49:18.181126    1180 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"coredns-565d847f94-96fl4_kube-system(e6432e5f-c196-4695-ac2c-8f3802fd8cd0)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"coredns-565d847f94-96fl4_kube-system(e6432e5f-c196-4695-ac2c-8f3802fd8cd0)\\\": rpc error: code = Unknown desc = failed to setup network for sandbox \\\"720457a14038737419b34a2dc92ae677b68270c9d3d91dfe472c00f8923f4565\\\": plugin type=\\\"kube-ovn\\\" failed (add): RPC failed; request ip return 500 configure nic failed network 192.168.72.204/24 with gateway 192.168.72.8 is not ready for interface eth0 after 200 checks: resolve MAC address of 192.168.72.8 timeout: read packet 00:00:00:39:7e:cc: i/o timeout\"" pod="kube-system/coredns-565d847f94-96fl4" podUID=e6432e5f-c196-4695-ac2c-8f3802fd8cd0
Apr 13 00:49:34 ubuntu kubelet[1180]: E0413 00:49:34.730508    1180 remote_runtime.go:222] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"7148a21b2f843ac0ad3d9204df237780dc3080dd4cad3c418671393517331e26\": plugin type=\"kube-ovn\" failed (add): RPC failed; request ip return 500 configure nic failed network 192.168.72.203/24 with gateway 192.168.72.8 is not ready for interface eth0 after 200 checks: resolve MAC address of 192.168.72.8 timeout: read packet 00:00:00:67:95:54: i/o timeout"
Apr 13 00:49:34 ubuntu kubelet[1180]: E0413 00:49:34.730580    1180 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"7148a21b2f843ac0ad3d9204df237780dc3080dd4cad3c418671393517331e26\": plugin type=\"kube-ovn\" failed (add): RPC failed; request ip return 500 configure nic failed network 192.168.72.203/24 with gateway 192.168.72.8 is not ready for interface eth0 after 200 checks: resolve MAC address of 192.168.72.8 timeout: read packet 00:00:00:67:95:54: i/o timeout" pod="kube-system/coredns-565d847f94-j78sv"
Apr 13 00:49:34 ubuntu kubelet[1180]: E0413 00:49:34.730607    1180 kuberuntime_manager.go:772] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"7148a21b2f843ac0ad3d9204df237780dc3080dd4cad3c418671393517331e26\": plugin type=\"kube-ovn\" failed (add): RPC failed; request ip return 500 configure nic failed network 192.168.72.203/24 with gateway 192.168.72.8 is not ready for interface eth0 after 200 checks: resolve MAC address of 192.168.72.8 timeout: read packet 00:00:00:67:95:54: i/o timeout" pod="kube-system/coredns-565d847f94-j78sv"
Apr 13 00:49:34 ubuntu kubelet[1180]: E0413 00:49:34.730675    1180 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"coredns-565d847f94-j78sv_kube-system(6732bb83-9823-4a89-bfd8-e3c93ee5d0db)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"coredns-565d847f94-j78sv_kube-system(6732bb83-9823-4a89-bfd8-e3c93ee5d0db)\\\": rpc error: code = Unknown desc = failed to setup network for sandbox \\\"7148a21b2f843ac0ad3d9204df237780dc3080dd4cad3c418671393517331e26\\\": plugin type=\\\"kube-ovn\\\" failed (add): RPC failed; request ip return 500 configure nic failed network 192.168.72.203/24 with gateway 192.168.72.8 is not ready for interface eth0 after 200 checks: resolve MAC address of 192.168.72.8 timeout: read packet 00:00:00:67:95:54: i/o timeout\"" pod="kube-system/coredns-565d847f94-j78sv" podUID=6732bb83-9823-4a89-bfd8-e3c93ee5d0db
Apr 13 00:50:06 ubuntu kubelet[1180]: W0413 00:50:06.141833    1180 watcher.go:93] Error while processing event ("/sys/fs/cgroup/system.slice/systemd-tmpfiles-clean.service": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/system.slice/systemd-tmpfiles-clean.service: no such file or directory
Apr 13 00:51:39 ubuntu kubelet[1180]: E0413 00:51:39.534548    1180 remote_runtime.go:222] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"421aba4d0b6838b1cff940180e445ebe08835d7fc169add53941181eb9194f1b\": plugin type=\"kube-ovn\" failed (add): RPC failed; request ip return 500 configure nic failed network 192.168.72.205/24 with gateway 192.168.72.8 is not ready for interface eth0 after 200 checks: resolve MAC address of 192.168.72.8 timeout: read packet 00:00:00:a5:96:7e: i/o timeout"
Apr 13 00:51:39 ubuntu kubelet[1180]: E0413 00:51:39.534683    1180 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"421aba4d0b6838b1cff940180e445ebe08835d7fc169add53941181eb9194f1b\": plugin type=\"kube-ovn\" failed (add): RPC failed; request ip return 500 configure nic failed network 192.168.72.205/24 with gateway 192.168.72.8 is not ready for interface eth0 after 200 checks: resolve MAC address of 192.168.72.8 timeout: read packet 00:00:00:a5:96:7e: i/o timeout" pod="kube-system/kube-ovn-pinger-m6x9l"
Apr 13 00:51:39 ubuntu kubelet[1180]: E0413 00:51:39.534764    1180 kuberuntime_manager.go:772] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"421aba4d0b6838b1cff940180e445ebe08835d7fc169add53941181eb9194f1b\": plugin type=\"kube-ovn\" failed (add): RPC failed; request ip return 500 configure nic failed network 192.168.72.205/24 with gateway 192.168.72.8 is not ready for interface eth0 after 200 checks: resolve MAC address of 192.168.72.8 timeout: read packet 00:00:00:a5:96:7e: i/o timeout" pod="kube-system/kube-ovn-pinger-m6x9l"
Apr 13 00:51:39 ubuntu kubelet[1180]: E0413 00:51:39.534894    1180 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"kube-ovn-pinger-m6x9l_kube-system(32362817-8066-4b67-8c75-c2a59d2f0379)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"kube-ovn-pinger-m6x9l_kube-system(32362817-8066-4b67-8c75-c2a59d2f0379)\\\": rpc error: code = Unknown desc = failed to setup network for sandbox \\\"421aba4d0b6838b1cff940180e445ebe08835d7fc169add53941181eb9194f1b\\\": plugin type=\\\"kube-ovn\\\" failed (add): RPC failed; request ip return 500 configure nic failed network 192.168.72.205/24 with gateway 192.168.72.8 is not ready for interface eth0 after 200 checks: resolve MAC address of 192.168.72.8 timeout: read packet 00:00:00:a5:96:7e: i/o timeout\"" pod="kube-system/kube-ovn-pinger-m6x9l" podUID=32362817-8066-4b67-8c75-c2a59d2f0379
Apr 13 00:52:44 ubuntu kubelet[1180]: E0413 00:52:44.930636    1180 remote_runtime.go:222] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"31cf59eb23ab5e44e538226f1ac954a9216753ca9ff5f832bbbf6f590f68e9bc\": plugin type=\"kube-ovn\" failed (add): RPC failed; request ip return 500 configure nic failed network 192.168.72.204/24 with gateway 192.168.72.8 is not ready for interface eth0 after 200 checks: resolve MAC address of 192.168.72.8 timeout: read packet 00:00:00:39:7e:cc: i/o timeout"
Apr 13 00:52:44 ubuntu kubelet[1180]: E0413 00:52:44.930694    1180 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"31cf59eb23ab5e44e538226f1ac954a9216753ca9ff5f832bbbf6f590f68e9bc\": plugin type=\"kube-ovn\" failed (add): RPC failed; request ip return 500 configure nic failed network 192.168.72.204/24 with gateway 192.168.72.8 is not ready for interface eth0 after 200 checks: resolve MAC address of 192.168.72.8 timeout: read packet 00:00:00:39:7e:cc: i/o timeout" pod="kube-system/coredns-565d847f94-96fl4"
Apr 13 00:52:44 ubuntu kubelet[1180]: E0413 00:52:44.930727    1180 kuberuntime_manager.go:772] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"31cf59eb23ab5e44e538226f1ac954a9216753ca9ff5f832bbbf6f590f68e9bc\": plugin type=\"kube-ovn\" failed (add): RPC failed; request ip return 500 configure nic failed network 192.168.72.204/24 with gateway 192.168.72.8 is not ready for interface eth0 after 200 checks: resolve MAC address of 192.168.72.8 timeout: read packet 00:00:00:39:7e:cc: i/o timeout" pod="kube-system/coredns-565d847f94-96fl4"
Apr 13 00:52:44 ubuntu kubelet[1180]: E0413 00:52:44.930805    1180 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"coredns-565d847f94-96fl4_kube-system(e6432e5f-c196-4695-ac2c-8f3802fd8cd0)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"coredns-565d847f94-96fl4_kube-system(e6432e5f-c196-4695-ac2c-8f3802fd8cd0)\\\": rpc error: code = Unknown desc = failed to setup network for sandbox \\\"31cf59eb23ab5e44e538226f1ac954a9216753ca9ff5f832bbbf6f590f68e9bc\\\": plugin type=\\\"kube-ovn\\\" failed (add): RPC failed; request ip return 500 configure nic failed network 192.168.72.204/24 with gateway 192.168.72.8 is not ready for interface eth0 after 200 checks: resolve MAC address of 192.168.72.8 timeout: read packet 00:00:00:39:7e:cc: i/o timeout\"" pod="kube-system/coredns-565d847f94-96fl4" podUID=e6432e5f-c196-4695-ac2c-8f3802fd8cd0
Apr 13 00:53:01 ubuntu kubelet[1180]: E0413 00:53:01.706025    1180 remote_runtime.go:222] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"c521a39f522965686dc7b97bd8bfa51af325ea0beade31bed001065964e72099\": plugin type=\"kube-ovn\" failed (add): RPC failed; request ip return 500 configure nic failed network 192.168.72.203/24 with gateway 192.168.72.8 is not ready for interface eth0 after 200 checks: resolve MAC address of 192.168.72.8 timeout: read packet 00:00:00:67:95:54: i/o timeout"
Apr 13 00:53:01 ubuntu kubelet[1180]: E0413 00:53:01.706094    1180 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"c521a39f522965686dc7b97bd8bfa51af325ea0beade31bed001065964e72099\": plugin type=\"kube-ovn\" failed (add): RPC failed; request ip return 500 configure nic failed network 192.168.72.203/24 with gateway 192.168.72.8 is not ready for interface eth0 after 200 checks: resolve MAC address of 192.168.72.8 timeout: read packet 00:00:00:67:95:54: i/o timeout" pod="kube-system/coredns-565d847f94-j78sv"
Apr 13 00:53:01 ubuntu kubelet[1180]: E0413 00:53:01.706130    1180 kuberuntime_manager.go:772] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"c521a39f522965686dc7b97bd8bfa51af325ea0beade31bed001065964e72099\": plugin type=\"kube-ovn\" failed (add): RPC failed; request ip return 500 configure nic failed network 192.168.72.203/24 with gateway 192.168.72.8 is not ready for interface eth0 after 200 checks: resolve MAC address of 192.168.72.8 timeout: read packet 00:00:00:67:95:54: i/o timeout" pod="kube-system/coredns-565d847f94-j78sv"
Apr 13 00:53:01 ubuntu kubelet[1180]: E0413 00:53:01.706506    1180 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"coredns-565d847f94-j78sv_kube-system(6732bb83-9823-4a89-bfd8-e3c93ee5d0db)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"coredns-565d847f94-j78sv_kube-system(6732bb83-9823-4a89-bfd8-e3c93ee5d0db)\\\": rpc error: code = Unknown desc = failed to setup network for sandbox \\\"c521a39f522965686dc7b97bd8bfa51af325ea0beade31bed001065964e72099\\\": plugin type=\\\"kube-ovn\\\" failed (add): RPC failed; request ip return 500 configure nic failed network 192.168.72.203/24 with gateway 192.168.72.8 is not ready for interface eth0 after 200 checks: resolve MAC address of 192.168.72.8 timeout: read packet 00:00:00:67:95:54: i/o timeout\"" pod="kube-system/coredns-565d847f94-j78sv" podUID=6732bb83-9823-4a89-bfd8-e3c93ee5d0db
Apr 13 00:55:07 ubuntu kubelet[1180]: E0413 00:55:07.314743    1180 remote_runtime.go:222] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"b2079d584775504f2a878c1786063a95167076e37eeb3b1c4d0557e6c83bab76\": plugin type=\"kube-ovn\" failed (add): RPC failed; request ip return 500 configure nic failed network 192.168.72.205/24 with gateway 192.168.72.8 is not ready for interface eth0 after 200 checks: resolve MAC address of 192.168.72.8 timeout: read packet 00:00:00:a5:96:7e: i/o timeout"
Apr 13 00:55:07 ubuntu kubelet[1180]: E0413 00:55:07.314820    1180 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"b2079d584775504f2a878c1786063a95167076e37eeb3b1c4d0557e6c83bab76\": plugin type=\"kube-ovn\" failed (add): RPC failed; request ip return 500 configure nic failed network 192.168.72.205/24 with gateway 192.168.72.8 is not ready for interface eth0 after 200 checks: resolve MAC address of 192.168.72.8 timeout: read packet 00:00:00:a5:96:7e: i/o timeout" pod="kube-system/kube-ovn-pinger-m6x9l"
Apr 13 00:55:07 ubuntu kubelet[1180]: E0413 00:55:07.314857    1180 kuberuntime_manager.go:772] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"b2079d584775504f2a878c1786063a95167076e37eeb3b1c4d0557e6c83bab76\": plugin type=\"kube-ovn\" failed (add): RPC failed; request ip return 500 configure nic failed network 192.168.72.205/24 with gateway 192.168.72.8 is not ready for interface eth0 after 200 checks: resolve MAC address of 192.168.72.8 timeout: read packet 00:00:00:a5:96:7e: i/o timeout" pod="kube-system/kube-ovn-pinger-m6x9l"
Apr 13 00:55:07 ubuntu kubelet[1180]: E0413 00:55:07.314937    1180 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"kube-ovn-pinger-m6x9l_kube-system(32362817-8066-4b67-8c75-c2a59d2f0379)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"kube-ovn-pinger-m6x9l_kube-system(32362817-8066-4b67-8c75-c2a59d2f0379)\\\": rpc error: code = Unknown desc = failed to setup network for sandbox \\\"b2079d584775504f2a878c1786063a95167076e37eeb3b1c4d0557e6c83bab76\\\": plugin type=\\\"kube-ovn\\\" failed (add): RPC failed; request ip return 500 configure nic failed network 192.168.72.205/24 with gateway 192.168.72.8 is not ready for interface eth0 after 200 checks: resolve MAC address of 192.168.72.8 timeout: read packet 00:00:00:a5:96:7e: i/o timeout\"" pod="kube-system/kube-ovn-pinger-m6x9l" podUID=32362817-8066-4b67-8c75-c2a59d2f0379
Apr 13 00:56:12 ubuntu kubelet[1180]: E0413 00:56:12.015033    1180 remote_runtime.go:222] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"e9c3a87a801a2d7337c015942ef3ef695a4093bddd87e748ca63f40fcacb2112\": plugin type=\"kube-ovn\" failed (add): RPC failed; request ip return 500 configure nic failed network 192.168.72.204/24 with gateway 192.168.72.8 is not ready for interface eth0 after 200 checks: resolve MAC address of 192.168.72.8 timeout: read packet 00:00:00:39:7e:cc: i/o timeout"
Apr 13 00:56:12 ubuntu kubelet[1180]: E0413 00:56:12.015125    1180 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"e9c3a87a801a2d7337c015942ef3ef695a4093bddd87e748ca63f40fcacb2112\": plugin type=\"kube-ovn\" failed (add): RPC failed; request ip return 500 configure nic failed network 192.168.72.204/24 with gateway 192.168.72.8 is not ready for interface eth0 after 200 checks: resolve MAC address of 192.168.72.8 timeout: read packet 00:00:00:39:7e:cc: i/o timeout" pod="kube-system/coredns-565d847f94-96fl4"
Apr 13 00:56:12 ubuntu kubelet[1180]: E0413 00:56:12.015166    1180 kuberuntime_manager.go:772] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"e9c3a87a801a2d7337c015942ef3ef695a4093bddd87e748ca63f40fcacb2112\": plugin type=\"kube-ovn\" failed (add): RPC failed; request ip return 500 configure nic failed network 192.168.72.204/24 with gateway 192.168.72.8 is not ready for interface eth0 after 200 checks: resolve MAC address of 192.168.72.8 timeout: read packet 00:00:00:39:7e:cc: i/o timeout" pod="kube-system/coredns-565d847f94-96fl4"
Apr 13 00:56:12 ubuntu kubelet[1180]: E0413 00:56:12.015237    1180 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"coredns-565d847f94-96fl4_kube-system(e6432e5f-c196-4695-ac2c-8f3802fd8cd0)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"coredns-565d847f94-96fl4_kube-system(e6432e5f-c196-4695-ac2c-8f3802fd8cd0)\\\": rpc error: code = Unknown desc = failed to setup network for sandbox \\\"e9c3a87a801a2d7337c015942ef3ef695a4093bddd87e748ca63f40fcacb2112\\\": plugin type=\\\"kube-ovn\\\" failed (add): RPC failed; request ip return 500 configure nic failed network 192.168.72.204/24 with gateway 192.168.72.8 is not ready for interface eth0 after 200 checks: resolve MAC address of 192.168.72.8 timeout: read packet 00:00:00:39:7e:cc: i/o timeout\"" pod="kube-system/coredns-565d847f94-96fl4" podUID=e6432e5f-c196-4695-ac2c-8f3802fd8cd0
Apr 13 00:56:26 ubuntu kubelet[1180]: E0413 00:56:26.796226    1180 remote_runtime.go:222] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"446bf1a35997c40dcd7ebec2f57421dd66ee55445f14e957ebe7cc825aad20c9\": plugin type=\"kube-ovn\" failed (add): RPC failed; request ip return 500 configure nic failed network 192.168.72.203/24 with gateway 192.168.72.8 is not ready for interface eth0 after 200 checks: resolve MAC address of 192.168.72.8 timeout: read packet 00:00:00:67:95:54: i/o timeout"
Apr 13 00:56:26 ubuntu kubelet[1180]: E0413 00:56:26.796298    1180 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"446bf1a35997c40dcd7ebec2f57421dd66ee55445f14e957ebe7cc825aad20c9\": plugin type=\"kube-ovn\" failed (add): RPC failed; request ip return 500 configure nic failed network 192.168.72.203/24 with gateway 192.168.72.8 is not ready for interface eth0 after 200 checks: resolve MAC address of 192.168.72.8 timeout: read packet 00:00:00:67:95:54: i/o timeout" pod="kube-system/coredns-565d847f94-j78sv"
Apr 13 00:56:26 ubuntu kubelet[1180]: E0413 00:56:26.796334    1180 kuberuntime_manager.go:772] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"446bf1a35997c40dcd7ebec2f57421dd66ee55445f14e957ebe7cc825aad20c9\": plugin type=\"kube-ovn\" failed (add): RPC failed; request ip return 500 configure nic failed network 192.168.72.203/24 with gateway 192.168.72.8 is not ready for interface eth0 after 200 checks: resolve MAC address of 192.168.72.8 timeout: read packet 00:00:00:67:95:54: i/o timeout" pod="kube-system/coredns-565d847f94-j78sv"
Apr 13 00:56:26 ubuntu kubelet[1180]: E0413 00:56:26.796407    1180 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"coredns-565d847f94-j78sv_kube-system(6732bb83-9823-4a89-bfd8-e3c93ee5d0db)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"coredns-565d847f94-j78sv_kube-system(6732bb83-9823-4a89-bfd8-e3c93ee5d0db)\\\": rpc error: code = Unknown desc = failed to setup network for sandbox \\\"446bf1a35997c40dcd7ebec2f57421dd66ee55445f14e957ebe7cc825aad20c9\\\": plugin type=\\\"kube-ovn\\\" failed (add): RPC failed; request ip return 500 configure nic failed network 192.168.72.203/24 with gateway 192.168.72.8 is not ready for interface eth0 after 200 checks: resolve MAC address of 192.168.72.8 timeout: read packet 00:00:00:67:95:54: i/o timeout\"" pod="kube-system/coredns-565d847f94-j78sv" podUID=6732bb83-9823-4a89-bfd8-e3c93ee5d0db
root@ubuntu:~#

Additional Info

zhangzujian commented 1 year ago

Please check your network configuration.

You can also check access to the external network using macvlan.