Failed to create pod sandbox: rpc error: code = Unknown desc

withlin commented 3 years ago

What happend:

  Type     Reason                  Age                     From               Message
  ----     ------                  ----                    ----               -------
  Normal   Scheduled               9m54s                   default-scheduler  Successfully assigned kube-system/samplepod to zhc-netmis-compass-node01
  Normal   AddedInterface          9m53s                   multus             Add eth0 [10.16.0.40/16]
  Warning  FailedCreatePodSandBox  9m52s                   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "e1a7c40a9b0b4b038a81c0d00548b3890632494a686f4351ffb5f277fe65c105" network for pod "samplepod": networkPlugin cni failed to set up pod "samplepod_kube-system" network: [kube-system/samplepod:macvlan-conf]: error adding container to network "macvlan-conf": Link not found, failed to clean up sandbox container "e1a7c40a9b0b4b038a81c0d00548b3890632494a686f4351ffb5f277fe65c105" network for pod "samplepod": networkPlugin cni failed to teardown pod "samplepod_kube-system" network: delegateDel: error invoking DelegateDel - "macvlan": error in getting result from DelNetwork: Link not found]
  Normal   SandboxChanged          4m52s (x25 over 9m52s)  kubelet            Pod sandbox changed, it will be killed and re-created.
[root@zhc-netmis-compass-master01 dpdk]#

What you expected to happen: It should create po How to reproduce it (as minimally and precisely as possible):

quickstart

Anything else we need to know?:

Environment:

Multus version

docker.io/nfvpe/multus:stable

Kubernetes version (use kubectl version):

[root@zhc-netmis-compass-master01 dpdk]# kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:51:04Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

Primary CNI for Kubernetes cluster:

alauda/kube-ovn
OS (e.g. from /etc/os-release):

   [root@zhc-netmis-compass-master01 dpdk]# cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

File of '/etc/cni/net.d/'

[root@zhc-netmis-compass-master01 dpdk]# ls /etc/cni/net.d/
00-multus.conf  01-kube-ovn.conflist  multus.d

File of '/etc/cni/multus/net.d'

[root@zhc-netmis-compass-master01 dpdk]# ls /etc/cni/net.d/multus.d/
multus.kubeconfig

NetworkAttachment info (use kubectl get net-attach-def -o yaml)

   [root@zhc-netmis-compass-master01 dpdk]# kubectl get net-attach-def -o yaml
apiVersion: v1
items:
- apiVersion: k8s.cni.cncf.io/v1
  kind: NetworkAttachmentDefinition
  metadata:
    creationTimestamp: "2020-11-08T11:14:56Z"
    generation: 1
    managedFields:
    - apiVersion: k8s.cni.cncf.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        f:spec:
          .: {}
          f:config: {}
      manager: kubectl-create
      operation: Update
      time: "2020-11-08T11:14:56Z"
    name: macvlan-conf
    namespace: kube-system
    resourceVersion: "45837"
    selfLink: /apis/k8s.cni.cncf.io/v1/namespaces/kube-system/network-attachment-definitions/macvlan-conf
    uid: 546f6430-d02d-4fe3-8316-6cbe8c916546
  spec:
    config: '{ "cniVersion": "0.3.0", "type": "macvlan", "master": "eth0", "mode":
      "bridge", "ipam": { "type": "host-local", "subnet": "192.168.1.0/24", "rangeStart":
      "192.168.1.200", "rangeEnd": "192.168.1.216", "routes": [ { "dst": "0.0.0.0/0"
      } ], "gateway": "192.168.1.1" } }'
- apiVersion: k8s.cni.cncf.io/v1
  kind: NetworkAttachmentDefinition
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"k8s.cni.cncf.io/v1","kind":"NetworkAttachmentDefinition","metadata":{"annotations":{},"name":"macvlan-conf1","namespace":"kube-system"},"spec":{"config":"{ \"cniVersion\": \"0.3.0\", \"type\": \"macvlan\", \"master\": \"eth0\", \"mode\": \"bridge\", \"isGateway\": true, \"ipMasq\": false, \"ipam\": { \"type\": \"host-local\", \"subnet\": \"10.200.108.0/24\", \"rangeStart\": \"10.200.108.237\", \"rangeEnd\": \"10.200.108.240\", \"routes\": [ { \"dst\": \"0.0.0.0/0\" } ], \"gateway\": \"10.200.108.254\" } }"}}
    creationTimestamp: "2020-11-08T11:09:13Z"
    generation: 2
    managedFields:
    - apiVersion: k8s.cni.cncf.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:kubectl.kubernetes.io/last-applied-configuration: {}
        f:spec:
          .: {}
          f:config: {}
      manager: kubectl-client-side-apply
      operation: Update
      time: "2020-11-08T11:10:45Z"
    name: macvlan-conf1
    namespace: kube-system
    resourceVersion: "44703"
    selfLink: /apis/k8s.cni.cncf.io/v1/namespaces/kube-system/network-attachment-definitions/macvlan-conf1
    uid: b37ed784-19fe-4831-b66a-1275e813d9db
  spec:
    config: '{ "cniVersion": "0.3.0", "type": "macvlan", "master": "eth0", "mode":
      "bridge", "isGateway": true, "ipMasq": false, "ipam": { "type": "host-local",
      "subnet": "10.200.108.0/24", "rangeStart": "10.200.108.237", "rangeEnd": "10.200.108.240",
      "routes": [ { "dst": "0.0.0.0/0" } ], "gateway": "10.200.108.254" } }'
- apiVersion: k8s.cni.cncf.io/v1
  kind: NetworkAttachmentDefinition
  metadata:
    creationTimestamp: "2020-11-08T09:32:09Z"
    generation: 1
    managedFields:
    - apiVersion: k8s.cni.cncf.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        f:spec:
          .: {}
          f:config: {}
      manager: kubectl-create
      operation: Update
      time: "2020-11-08T09:32:09Z"
    name: ovs-dpdk-br0
    namespace: kube-system
    resourceVersion: "20975"
    selfLink: /apis/k8s.cni.cncf.io/v1/namespaces/kube-system/network-attachment-definitions/ovs-dpdk-br0
    uid: 0dbc5098-dc09-449c-84ff-30c199471771
  spec:
    config: |
      {
        "cniVersion": "0.3.1",
        "type": "userspace",
        "name": "ovs-dpdk-br0",
        "kubeconfig": "/etc/cni/net.d/multus.d/multus.kubeconfig",
        "logFile": "/var/log/userspace-ovs-dpdk-br0.log",
        "logLevel": "debug",
        "host": {
          "engine": "ovs-dpdk",
          "iftype": "vhostuser",
          "netType": "bridge",
          "vhost": {
            "mode": "server"
          },
          "bridge": {
            "bridgeName": "br0"
          }
        },
        "container": {
          "engine": "ovs-dpdk",
          "iftype": "vhostuser",
          "netType": "interface",
          "vhost": {
            "mode": "client"
          }
        }
      }
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Target pod yaml info (with annotation, use kubectl get pod <podname> -o yaml)

[root@zhc-netmis-compass-master01 dpdk]# kg po samplepod -o yaml
apiVersion: v1
kind: Pod
metadata:
annotations:
k8s.v1.cni.cncf.io/networks: macvlan-conf
ovn.kubernetes.io/allocated: "true"
ovn.kubernetes.io/cidr: 10.16.0.0/16
ovn.kubernetes.io/gateway: 10.16.0.1
ovn.kubernetes.io/ip_address: 10.16.0.40
ovn.kubernetes.io/logical_switch: ovn-default
ovn.kubernetes.io/mac_address: 00:00:00:2C:E1:82
ovn.kubernetes.io/network_types: geneve
ovn.kubernetes.io/routed: "true"
creationTimestamp: "2020-11-08T11:17:40Z"
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
  f:metadata:
    f:annotations:
      f:ovn.kubernetes.io/allocated: {}
      f:ovn.kubernetes.io/cidr: {}
      f:ovn.kubernetes.io/gateway: {}
      f:ovn.kubernetes.io/ip_address: {}
      f:ovn.kubernetes.io/logical_switch: {}
      f:ovn.kubernetes.io/mac_address: {}
      f:ovn.kubernetes.io/network_types: {}
      f:ovn.kubernetes.io/routed: {}
manager: kube-ovn-controller
operation: Update
time: "2020-11-08T11:17:40Z"
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
  f:metadata:
    f:annotations:
      .: {}
      f:k8s.v1.cni.cncf.io/networks: {}
  f:spec:
    f:containers:
      k:{"name":"samplepod"}:
        .: {}
        f:command: {}
        f:image: {}
        f:imagePullPolicy: {}
        f:name: {}
        f:resources: {}
        f:terminationMessagePath: {}
        f:terminationMessagePolicy: {}
    f:dnsPolicy: {}
    f:enableServiceLinks: {}
    f:restartPolicy: {}
    f:schedulerName: {}
    f:securityContext: {}
    f:terminationGracePeriodSeconds: {}
manager: kubectl-create
operation: Update
time: "2020-11-08T11:17:40Z"
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
  f:status:
    f:conditions:
      k:{"type":"ContainersReady"}:
        .: {}
        f:lastProbeTime: {}
        f:lastTransitionTime: {}
        f:message: {}
        f:reason: {}
        f:status: {}
        f:type: {}
      k:{"type":"Initialized"}:
        .: {}
        f:lastProbeTime: {}
        f:lastTransitionTime: {}
        f:status: {}
        f:type: {}
      k:{"type":"Ready"}:
        .: {}
        f:lastProbeTime: {}
        f:lastTransitionTime: {}
        f:message: {}
        f:reason: {}
        f:status: {}
        f:type: {}
    f:containerStatuses: {}
    f:hostIP: {}
    f:startTime: {}
manager: kubelet
operation: Update
time: "2020-11-08T11:17:40Z"
name: samplepod
namespace: kube-system
resourceVersion: "46580"
selfLink: /api/v1/namespaces/kube-system/pods/samplepod
uid: 56688f3f-0587-4e35-99ed-ec3da34dd2bc
spec:
containers:
- command:
- /bin/ash
- -c
- 'trap : TERM INT; sleep infinity & wait'
image: alpine
imagePullPolicy: Always
name: samplepod
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
  name: default-token-t2fzg
  readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
nodeName: zhc-netmis-compass-node01
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: default
serviceAccountName: default
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- name: default-token-t2fzg
secret:
  defaultMode: 420
  secretName: default-token-t2fzg
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2020-11-08T11:17:40Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2020-11-08T11:17:40Z"
message: 'containers with unready status: [samplepod]'
reason: ContainersNotReady
status: "False"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2020-11-08T11:17:40Z"
message: 'containers with unready status: [samplepod]'
reason: ContainersNotReady
status: "False"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2020-11-08T11:17:40Z"
status: "True"
type: PodScheduled
containerStatuses:
- image: alpine
imageID: ""
lastState: {}
name: samplepod
ready: false
restartCount: 0
started: false
state:
  waiting:
    reason: ContainerCreating
hostIP: 10.200.108.13
phase: Pending
qosClass: BestEffort
startTime: "2020-11-08T11:17:40Z"

Other log outputs (if you use multus logging)

2020-11-08T09:58:15.147748043Z 2020-11-08T09:58:15+0000 Generating Multus configuration file using files in /host/etc/cni/net.d...
2020-11-08T09:58:15.244760286Z 2020-11-08T09:58:15+0000 Attemping to find master plugin configuration, attempt 0
2020-11-08T09:58:20.642841380Z 2020-11-08T09:58:20+0000 Attemping to find master plugin configuration, attempt 5
2020-11-08T09:58:26.043785217Z 2020-11-08T09:58:26+0000 Attemping to find master plugin configuration, attempt 10
2020-11-08T09:58:31.442863156Z 2020-11-08T09:58:31+0000 Attemping to find master plugin configuration, attempt 15
2020-11-08T09:58:36.940894139Z 2020-11-08T09:58:36+0000 Attemping to find master plugin configuration, attempt 20
2020-11-08T09:58:42.342939193Z 2020-11-08T09:58:42+0000 Attemping to find master plugin configuration, attempt 25
2020-11-08T09:58:43.445072492Z 2020-11-08T09:58:43+0000 Using /host/etc/cni/net.d/01-kube-ovn.conflist as a source to generate the Multus configuration
2020-11-08T09:58:43.543692295Z 2020-11-08T09:58:43+0000 Config file created @ /host/etc/cni/net.d/00-multus.conf
2020-11-08T09:58:43.543764818Z { "cniVersion": "0.3.1", "name": "multus-cni-network", "type": "multus", "kubeconfig": "/etc/cni/net.d/multus.d/multus.kubeconfig", "delegates": [ { "name":"kube-ovn", "cniVersion":"0.3.1", "plugins":[ { "type":"kube-ovn", "server_socket":"/run/openvswitch/kube-ovn-daemon.sock" }, { "type":"portmap", "capabilities":{ "portMappings":true } } ] } ] }
2020-11-08T09:58:43.546082800Z 2020-11-08T09:58:43+0000 Entering sleep (success)...

withlin commented 3 years ago

I create another pod,which not annotations k8s.v1.cni.cncf.io/networks: mac-vlan-configthat pod is create successfully.

withlin commented 3 years ago

Link not found

cyclinder commented 2 years ago

hi @withlin

I have the same problem as you, Can you tell me if this problem is solved? thanks

withlin commented 2 years ago

hi @cyclinder It’s been a long time, I probably remember that the network card name is configured incorrectly.

cyclinder commented 2 years ago

I found some issues that reported the same problem, It seems to the network interface eth0 does not exist in the node

cat <<EOF | kubectl create -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: macvlan-conf
spec:
  config: '{
      "cniVersion": "0.3.0",
      "type": "macvlan",
      "master": "eth0",  // there
      "mode": "bridge",
      "ipam": {
        "type": "host-local",
        "subnet": "192.168.1.0/24",
        "rangeStart": "192.168.1.200",
        "rangeEnd": "192.168.1.216",
        "routes": [
          { "dst": "0.0.0.0/0" }
        ],
        "gateway": "192.168.1.1"
      }
    }'
EOF

but I edit master filed in the macvlan-conf and change the value eth0 to ens192, it works😄

withlin commented 2 years ago

the network card name is configured incorrectly.

@cyclinder Isn't this what i said

a-c-dream commented 3 months ago

@withlin @cyclinder What should master be set to? I have multiple nodes. Can I just set up any network card?

diegodurrod commented 4 weeks ago

@withlin @cyclinder What should master be set to? I have multiple nodes. Can I just set up any network card?

You can set a virtual interface with common name between all nodes or rename the current interface matching with MAC address.

k8snetworkplumbingwg / multus-cni

Failed to create pod sandbox: rpc error: code = Unknown desc #581