search

k8snetworkplumbingwg / multus-cni

A CNI meta-plugin for multi-homed pods in Kubernetes
Apache License 2.0
2.34k stars 583 forks source link

cannot set "host-device" interface name to "net1": validateIfName: interface name net1 already exists #1089

Closed cucosion closed 6 months ago

cucosion commented 1 year ago

What happend:

Warning FailedCreatePodSandBox 20s (x16 over 35s) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "ee66bc0cfe368c3512d5d0504cf9d625d465cddf9d5d290a61bceb785f87fc59": plugin type="multus" name="multus-cni-network" failed (add): [default/teravm-testmodule-client-1-0/5f30eb60-e2e9-4545-be53-1e02095d4e92:sriovnetens1f0]: error adding container to network "sriovnetens1f0": DelegateAdd: cannot set "host-device" interface name to "net1": validateIfName: interface name net1 already exists

What you expected to happen:

pod to be created with two interfaces, the second one with the host-device CNI plugin

How to reproduce it (as minimally and precisely as possible): After i created the nad and created the pod with that nad the first time it worked. After a cluster restart, I tried one more time to deploy same pod and I got the above error

[root@kubemaster ~]# cat multus-cni/tmp/nad.yaml apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: sriovnetens1f0 spec: config: '{ "cniVersion": "0.3.0", "type": "host-device", "device": "ens1f0", "ipam": { "type": "static", "addresses": [ { "address": "192.168.1.23/24" } ] } }'

apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: sriovnetens1f1 spec: config: '{ "cniVersion": "0.3.0", "type": "host-device", "device": "ens1f1", "ipam": { "type": "static", "addresses": [ { "address": "192.168.1.24/24" } ] } }' [root@kubemaster ~]# cat multus-cni/tmp/ex.yaml apiVersion: v1 kind: Pod metadata: name: samplepod-left annotations: k8s.v1.cni.cncf.io/networks: sriovnetens1f0@net2 spec: containers:

Anything else we need to know?: Kubernetes 1.27.1 with Flannel latest version Environment:

ALMALINUX_MANTISBT_PROJECT="AlmaLinux-9" ALMALINUX_MANTISBT_PROJECT_VERSION="9.1" REDHAT_SUPPORT_PRODUCT="AlmaLinux" REDHAT_SUPPORT_PRODUCT_VERSION="9.1"

cucosion commented 1 year ago

[root@kubemaster log]# tail -n 200 multus.log "interface": "net2", "ips": [ "192.168.1.23" ], "mac": "24:1c:04:f3:d7:fd", "dns": {} }] k8s.v1.cni.cncf.io/networks:sriovnetens1f0@net2 kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{"k8s.v1.cni.cncf.io/networks":"sriovnetens1f0@net2"},"name":"samplepod-left","namespace":"default"},"spec":{"containers":[{"command":["/bin/bash","-c","trap : TERM INT; sleep infinity \u0026 wait"],"image":"nicolaka/netshoot","name":"samplepod-left"}]}} ] [] [] [{kubectl-client-side-apply Update v1 2023-05-08 20:53:06 +0100 IST FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:k8s.v1.cni.cncf.io/networks":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"samplepod-left\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}} } {kubelet Update v1 2023-05-08 20:53:06 +0100 IST FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:startTime":{}}} status} {multus Update v1 2023-05-08 20:56:28 +0100 IST FieldsV1 {"f:metadata":{"f:annotations":{"f:k8s.v1.cni.cncf.io/network-status":{}}}} status}]},Spec:PodSpec{Volumes:[]Volume{Volume{Name:kube-api-access-shv9t,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:420,},StorageOS:nil,CSI:nil,Ephemeral:nil,},},},Containers:[]Container{Container{Name:samplepod-left,Image:nicolaka/netshoot,Command:[/bin/bash -c trap : TERM INT; sleep infinity & wait],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-shv9t,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:Always,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Always,TerminationGracePeriodSeconds:30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:kubemaster,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,},ImagePullSecrets:[]LocalObjectReference{LocalObjectReference{Name:docker-registry-credentials,},},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:true,PreemptionPolicy:PreemptLowerPriority,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},SetHostnameAsFQDN:nil,},Status:PodStatus{Phase:Pending,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-05-08 20:53:06 +0100 IST,Reason:,Message:,},PodCondition{Type:Ready,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-05-08 20:53:06 +0100 IST,Reason:ContainersNotReady,Message:containers with unready status: [samplepod-left],},PodCondition{Type:ContainersReady,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-05-08 20:53:06 +0100 IST,Reason:ContainersNotReady,Message:containers with unready status: [samplepod-left],},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-05-08 20:53:06 +0100 IST,Reason:,Message:,},},Message:,Reason:,HostIP:10.160.160.168,PodIP:,StartTime:2023-05-08 20:53:06 +0100 IST,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:samplepod-left,State:ContainerState{Waiting:&ContainerStateWaiting{Reason:ContainerCreating,Message:,},Running:nil,Terminated:nil,},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:false,RestartCount:0,Image:nicolaka/netshoot,ImageID:,ContainerID:,Started:*false,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{},EphemeralContainerStatuses:[]ContainerStatus{},},}, &{fa70a1a44372892e78299ed55a94850d20f22e9a14a88bd9ac488a86ae2fdf99 /var/run/netns/cni-a6b3368e-d2d5-8977-9e04-406936c2cf5b eth0 K8S_POD_UID=05fd6a50-8679-44a5-92ab-61c432560445;IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NAME=samplepod-left;K8S_POD_INFRA_CONTAINER_ID=fa70a1a44372892e78299ed55a94850d20f22e9a14a88bd9ac488a86ae2fdf99 /opt/cni/bin [123 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 123 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 116 114 117 101 125 44 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 100 101 108 101 103 97 116 101 115 34 58 91 123 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 123 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 116 114 117 101 125 44 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 100 101 108 101 103 97 116 101 115 34 58 91 123 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 110 97 109 101 34 58 34 99 98 114 48 34 44 34 112 108 117 103 105 110 115 34 58 91 123 34 100 101 108 101 103 97 116 101 34 58 123 34 104 97 105 114 112 105 110 77 111 100 101 34 58 116 114 117 101 44 34 105 115 68 101 102 97 117 108 116 71 97 116 101 119 97 121 34 58 116 114 117 101 125 44 34 116 121 112 101 34 58 34 102 108 97 110 110 101 108 34 125 44 123 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 123 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 116 114 117 101 125 44 34 116 121 112 101 34 58 34 112 111 114 116 109 97 112 34 125 93 125 93 44 34 107 117 98 101 99 111 110 102 105 103 34 58 34 47 101 116 99 47 99 110 105 47 110 101 116 46 100 47 109 117 108 116 117 115 46 100 47 109 117 108 116 117 115 46 107 117 98 101 99 111 110 102 105 103 34 44 34 110 97 109 101 34 58 34 109 117 108 116 117 115 45 99 110 105 45 110 101 116 119 111 114 107 34 44 34 116 121 112 101 34 58 34 109 117 108 116 117 115 34 125 93 44 34 107 117 98 101 99 111 110 102 105 103 34 58 34 47 101 116 99 47 99 110 105 47 110 101 116 46 100 47 109 117 108 116 117 115 46 100 47 109 117 108 116 117 115 46 107 117 98 101 99 111 110 102 105 103 34 44 34 108 111 103 70 105 108 101 34 58 34 47 118 97 114 47 108 111 103 47 109 117 108 116 117 115 46 108 111 103 34 44 34 108 111 103 76 101 118 101 108 34 58 34 100 101 98 117 103 34 44 34 110 97 109 101 34 58 34 109 117 108 116 117 115 45 99 110 105 45 110 101 116 119 111 114 107 34 44 34 116 121 112 101 34 58 34 109 117 108 116 117 115 34 125]}, &{{true} samplepod-left default fa70a1a44372892e78299ed55a94850d20f22e9a14a88bd9ac488a86ae2fdf99 05fd6a50-8679-44a5-92ab-61c432560445}, [0xc00026d340 0xc00026c1c0], 1, 2023-05-08T20:56:28+01:00 [debug] getIfname: &{{0.3.0 sriovnetens1f0 host-device map[] {static} {[] [] []} map[] } { false []} default/sriovnetens1f0 net2 [] [] false false false false [123 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 48 34 44 34 100 101 118 105 99 101 34 58 34 101 110 115 49 102 48 34 44 34 105 112 97 109 34 58 123 34 97 100 100 114 101 115 115 101 115 34 58 91 123 34 97 100 100 114 101 115 115 34 58 34 49 57 50 46 49 54 56 46 49 46 50 51 47 50 52 34 125 93 44 34 116 121 112 101 34 58 34 115 116 97 116 105 99 34 125 44 34 110 97 109 101 34 58 34 115 114 105 111 118 110 101 116 101 110 115 49 102 48 34 44 34 116 121 112 101 34 58 34 104 111 115 116 45 100 101 118 105 99 101 34 125]}, eth0, 1 2023-05-08T20:56:28+01:00 [debug] LoadCNIRuntimeConf: net2, &{{0.3.0 sriovnetens1f0 host-device map[] {static} {[] [] []} map[] } { false []} default/sriovnetens1f0 net2 [] [] false false false false [123 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 48 34 44 34 100 101 118 105 99 101 34 58 34 101 110 115 49 102 48 34 44 34 105 112 97 109 34 58 123 34 97 100 100 114 101 115 115 101 115 34 58 91 123 34 97 100 100 114 101 115 115 34 58 34 49 57 50 46 49 54 56 46 49 46 50 51 47 50 52 34 125 93 44 34 116 121 112 101 34 58 34 115 116 97 116 105 99 34 125 44 34 110 97 109 101 34 58 34 115 114 105 111 118 110 101 116 101 110 115 49 102 48 34 44 34 116 121 112 101 34 58 34 104 111 115 116 45 100 101 118 105 99 101 34 125]} 2023-05-08T20:56:28+01:00 [debug] mergeCNIRuntimeConfig: &{{0.3.0 sriovnetens1f0 host-device map[] {static} {[] [] []} map[] } { false []} default/sriovnetens1f0 net2 [] [] false false false false [123 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 48 34 44 34 100 101 118 105 99 101 34 58 34 101 110 115 49 102 48 34 44 34 105 112 97 109 34 58 123 34 97 100 100 114 101 115 115 101 115 34 58 91 123 34 97 100 100 114 101 115 115 34 58 34 49 57 50 46 49 54 56 46 49 46 50 51 47 50 52 34 125 93 44 34 116 121 112 101 34 58 34 115 116 97 116 105 99 34 125 44 34 110 97 109 101 34 58 34 115 114 105 111 118 110 101 116 101 110 115 49 102 48 34 44 34 116 121 112 101 34 58 34 104 111 115 116 45 100 101 118 105 99 101 34 125]} 2023-05-08T20:56:28+01:00 [debug] mergeCNIRuntimeConfig: add runtimeConfig for net-attach-def: {[] [] } 2023-05-08T20:56:28+01:00 [debug] mergeCNIRuntimeConfig: add runtimeConfig for net-attach-def: {[] [] } 2023-05-08T20:56:28+01:00 [debug] ARGS: K8S_POD_UID=05fd6a50-8679-44a5-92ab-61c432560445;IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NAME=samplepod-left;K8S_POD_INFRA_CONTAINER_ID=fa70a1a44372892e78299ed55a94850d20f22e9a14a88bd9ac488a86ae2fdf99 2023-05-08T20:56:28+01:00 [debug] DelegateDel: , &Pod{ObjectMeta:{samplepod-left default 05fd6a50-8679-44a5-92ab-61c432560445 26888 0 2023-05-08 20:53:06 +0100 IST map[] map[k8s.v1.cni.cncf.io/network-status:[{ "name": "cbr0", "interface": "eth0", "ips": [ "10.244.0.171" ], "mac": "76:9e:23:07:d7:fa", "default": true, "dns": {}, "gateway": [ "10.244.0.1" ] },{ "name": "default/sriovnetens1f0", "interface": "net2", "ips": [ "192.168.1.23" ], "mac": "24:1c:04:f3:d7:fd", "dns": {} }] k8s.v1.cni.cncf.io/networks:sriovnetens1f0@net2 kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{"k8s.v1.cni.cncf.io/networks":"sriovnetens1f0@net2"},"name":"samplepod-left","namespace":"default"},"spec":{"containers":[{"command":["/bin/bash","-c","trap : TERM INT; sleep infinity \u0026 wait"],"image":"nicolaka/netshoot","name":"samplepod-left"}]}} ] [] [] [{kubectl-client-side-apply Update v1 2023-05-08 20:53:06 +0100 IST FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:k8s.v1.cni.cncf.io/networks":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"samplepod-left\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}} } {kubelet Update v1 2023-05-08 20:53:06 +0100 IST FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:startTime":{}}} status} {multus Update v1 2023-05-08 20:56:28 +0100 IST FieldsV1 {"f:metadata":{"f:annotations":{"f:k8s.v1.cni.cncf.io/network-status":{}}}} status}]},Spec:PodSpec{Volumes:[]Volume{Volume{Name:kube-api-access-shv9t,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:420,},StorageOS:nil,CSI:nil,Ephemeral:nil,},},},Containers:[]Container{Container{Name:samplepod-left,Image:nicolaka/netshoot,Command:[/bin/bash -c trap : TERM INT; sleep infinity & wait],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-shv9t,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:Always,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Always,TerminationGracePeriodSeconds:30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:kubemaster,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,},ImagePullSecrets:[]LocalObjectReference{LocalObjectReference{Name:docker-registry-credentials,},},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:true,PreemptionPolicy:PreemptLowerPriority,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},SetHostnameAsFQDN:nil,},Status:PodStatus{Phase:Pending,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-05-08 20:53:06 +0100 IST,Reason:,Message:,},PodCondition{Type:Ready,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-05-08 20:53:06 +0100 IST,Reason:ContainersNotReady,Message:containers with unready status: [samplepod-left],},PodCondition{Type:ContainersReady,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-05-08 20:53:06 +0100 IST,Reason:ContainersNotReady,Message:containers with unready status: [samplepod-left],},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-05-08 20:53:06 +0100 IST,Reason:,Message:,},},Message:,Reason:,HostIP:10.160.160.168,PodIP:,StartTime:2023-05-08 20:53:06 +0100 IST,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:samplepod-left,State:ContainerState{Waiting:&ContainerStateWaiting{Reason:ContainerCreating,Message:,},Running:nil,Terminated:nil,},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:false,RestartCount:0,Image:nicolaka/netshoot,ImageID:,ContainerID:,Started:*false,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{},EphemeralContainerStatuses:[]ContainerStatus{},},}, &{{0.3.0 sriovnetens1f0 host-device map[] {static} {[] [] []} map[] } { false []} default/sriovnetens1f0 net2 [] [] false false false false [123 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 48 34 44 34 100 101 118 105 99 101 34 58 34 101 110 115 49 102 48 34 44 34 105 112 97 109 34 58 123 34 97 100 100 114 101 115 115 101 115 34 58 91 123 34 97 100 100 114 101 115 115 34 58 34 49 57 50 46 49 54 56 46 49 46 50 51 47 50 52 34 125 93 44 34 116 121 112 101 34 58 34 115 116 97 116 105 99 34 125 44 34 110 97 109 101 34 58 34 115 114 105 111 118 110 101 116 101 110 115 49 102 48 34 44 34 116 121 112 101 34 58 34 104 111 115 116 45 100 101 118 105 99 101 34 125]}, &{fa70a1a44372892e78299ed55a94850d20f22e9a14a88bd9ac488a86ae2fdf99 /var/run/netns/cni-a6b3368e-d2d5-8977-9e04-406936c2cf5b net2 [[IgnoreUnknown true] [K8S_POD_NAMESPACE default] [K8S_POD_NAME samplepod-left] [K8S_POD_INFRA_CONTAINER_ID fa70a1a44372892e78299ed55a94850d20f22e9a14a88bd9ac488a86ae2fdf99] [K8S_POD_UID 05fd6a50-8679-44a5-92ab-61c432560445] [K8S_POD_UID 05fd6a50-8679-44a5-92ab-61c432560445] [IgnoreUnknown 1] [K8S_POD_NAMESPACE default] [K8S_POD_NAME samplepod-left] [K8S_POD_INFRA_CONTAINER_ID fa70a1a44372892e78299ed55a94850d20f22e9a14a88bd9ac488a86ae2fdf99]] map[] } 2023-05-08T20:56:28+01:00 [verbose] Del: default:samplepod-left:05fd6a50-8679-44a5-92ab-61c432560445:sriovnetens1f0:net2 {"cniVersion":"0.3.0","device":"ens1f0","ipam":{"addresses":[{"address":"192.168.1.23/24"}],"type":"static"},"name":"sriovnetens1f0","type":"host-device"} 2023-05-08T20:56:28+01:00 [debug] confDel: &{fa70a1a44372892e78299ed55a94850d20f22e9a14a88bd9ac488a86ae2fdf99 /var/run/netns/cni-a6b3368e-d2d5-8977-9e04-406936c2cf5b net2 [[IgnoreUnknown true] [K8S_POD_NAMESPACE default] [K8S_POD_NAME samplepod-left] [K8S_POD_INFRA_CONTAINER_ID fa70a1a44372892e78299ed55a94850d20f22e9a14a88bd9ac488a86ae2fdf99] [K8S_POD_UID 05fd6a50-8679-44a5-92ab-61c432560445] [K8S_POD_UID 05fd6a50-8679-44a5-92ab-61c432560445] [IgnoreUnknown 1] [K8S_POD_NAMESPACE default] [K8S_POD_NAME samplepod-left] [K8S_POD_INFRA_CONTAINER_ID fa70a1a44372892e78299ed55a94850d20f22e9a14a88bd9ac488a86ae2fdf99]] map[] }, {"cniVersion":"0.3.0","device":"ens1f0","ipam":{"addresses":[{"address":"192.168.1.23/24"}],"type":"static"},"name":"sriovnetens1f0","type":"host-device"} 2023-05-08T20:56:28+01:00 [error] error in getting result from DelNetwork: failed to find "net2": Link not found 2023-05-08T20:56:28+01:00 [error] DelegateDel: error invoking DelegateDel - "host-device": error in getting result from DelNetwork: failed to find "net2": Link not found 2023-05-08T20:56:28+01:00 [debug] getIfname: &{{0.3.1 multus-cni-network multus map[portMappings:true] {} {[] [] []} map[] } { false []} multus-cni-network [] [] false false true false [123 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 123 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 116 114 117 101 125 44 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 100 101 108 101 103 97 116 101 115 34 58 91 123 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 110 97 109 101 34 58 34 99 98 114 48 34 44 34 112 108 117 103 105 110 115 34 58 91 123 34 100 101 108 101 103 97 116 101 34 58 123 34 104 97 105 114 112 105 110 77 111 100 101 34 58 116 114 117 101 44 34 105 115 68 101 102 97 117 108 116 71 97 116 101 119 97 121 34 58 116 114 117 101 125 44 34 116 121 112 101 34 58 34 102 108 97 110 110 101 108 34 125 44 123 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 123 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 116 114 117 101 125 44 34 116 121 112 101 34 58 34 112 111 114 116 109 97 112 34 125 93 125 93 44 34 107 117 98 101 99 111 110 102 105 103 34 58 34 47 101 116 99 47 99 110 105 47 110 101 116 46 100 47 109 117 108 116 117 115 46 100 47 109 117 108 116 117 115 46 107 117 98 101 99 111 110 102 105 103 34 44 34 110 97 109 101 34 58 34 109 117 108 116 117 115 45 99 110 105 45 110 101 116 119 111 114 107 34 44 34 116 121 112 101 34 58 34 109 117 108 116 117 115 34 125]}, eth0, 0 2023-05-08T20:56:28+01:00 [debug] LoadCNIRuntimeConf: eth0, &{{0.3.1 multus-cni-network multus map[portMappings:true] {} {[] [] []} map[] } { false []} multus-cni-network [] [] false false true false [123 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 123 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 116 114 117 101 125 44 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 100 101 108 101 103 97 116 101 115 34 58 91 123 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 110 97 109 101 34 58 34 99 98 114 48 34 44 34 112 108 117 103 105 110 115 34 58 91 123 34 100 101 108 101 103 97 116 101 34 58 123 34 104 97 105 114 112 105 110 77 111 100 101 34 58 116 114 117 101 44 34 105 115 68 101 102 97 117 108 116 71 97 116 101 119 97 121 34 58 116 114 117 101 125 44 34 116 121 112 101 34 58 34 102 108 97 110 110 101 108 34 125 44 123 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 123 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 116 114 117 101 125 44 34 116 121 112 101 34 58 34 112 111 114 116 109 97 112 34 125 93 125 93 44 34 107 117 98 101 99 111 110 102 105 103 34 58 34 47 101 116 99 47 99 110 105 47 110 101 116 46 100 47 109 117 108 116 117 115 46 100 47 109 117 108 116 117 115 46 107 117 98 101 99 111 110 102 105 103 34 44 34 110 97 109 101 34 58 34 109 117 108 116 117 115 45 99 110 105 45 110 101 116 119 111 114 107 34 44 34 116 121 112 101 34 58 34 109 117 108 116 117 115 34 125]} 2023-05-08T20:56:28+01:00 [debug] mergeCNIRuntimeConfig: &{{0.3.1 multus-cni-network multus map[portMappings:true] {} {[] [] []} map[] } { false []} multus-cni-network [] [] false false true false [123 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 123 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 116 114 117 101 125 44 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 100 101 108 101 103 97 116 101 115 34 58 91 123 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 110 97 109 101 34 58 34 99 98 114 48 34 44 34 112 108 117 103 105 110 115 34 58 91 123 34 100 101 108 101 103 97 116 101 34 58 123 34 104 97 105 114 112 105 110 77 111 100 101 34 58 116 114 117 101 44 34 105 115 68 101 102 97 117 108 116 71 97 116 101 119 97 121 34 58 116 114 117 101 125 44 34 116 121 112 101 34 58 34 102 108 97 110 110 101 108 34 125 44 123 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 123 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 116 114 117 101 125 44 34 116 121 112 101 34 58 34 112 111 114 116 109 97 112 34 125 93 125 93 44 34 107 117 98 101 99 111 110 102 105 103 34 58 34 47 101 116 99 47 99 110 105 47 110 101 116 46 100 47 109 117 108 116 117 115 46 100 47 109 117 108 116 117 115 46 107 117 98 101 99 111 110 102 105 103 34 44 34 110 97 109 101 34 58 34 109 117 108 116 117 115 45 99 110 105 45 110 101 116 119 111 114 107 34 44 34 116 121 112 101 34 58 34 109 117 108 116 117 115 34 125]} 2023-05-08T20:56:28+01:00 [debug] ARGS: K8S_POD_UID=05fd6a50-8679-44a5-92ab-61c432560445;IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NAME=samplepod-left;K8S_POD_INFRA_CONTAINER_ID=fa70a1a44372892e78299ed55a94850d20f22e9a14a88bd9ac488a86ae2fdf99 2023-05-08T20:56:28+01:00 [debug] DelegateDel: , &Pod{ObjectMeta:{samplepod-left default 05fd6a50-8679-44a5-92ab-61c432560445 26888 0 2023-05-08 20:53:06 +0100 IST map[] map[k8s.v1.cni.cncf.io/network-status:[{ "name": "cbr0", "interface": "eth0", "ips": [ "10.244.0.171" ], "mac": "76:9e:23:07:d7:fa", "default": true, "dns": {}, "gateway": [ "10.244.0.1" ] },{ "name": "default/sriovnetens1f0", "interface": "net2", "ips": [ "192.168.1.23" ], "mac": "24:1c:04:f3:d7:fd", "dns": {} }] k8s.v1.cni.cncf.io/networks:sriovnetens1f0@net2 kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{"k8s.v1.cni.cncf.io/networks":"sriovnetens1f0@net2"},"name":"samplepod-left","namespace":"default"},"spec":{"containers":[{"command":["/bin/bash","-c","trap : TERM INT; sleep infinity \u0026 wait"],"image":"nicolaka/netshoot","name":"samplepod-left"}]}} ] [] [] [{kubectl-client-side-apply Update v1 2023-05-08 20:53:06 +0100 IST FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:k8s.v1.cni.cncf.io/networks":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"samplepod-left\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}} } {kubelet Update v1 2023-05-08 20:53:06 +0100 IST FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:startTime":{}}} status} {multus Update v1 2023-05-08 20:56:28 +0100 IST FieldsV1 {"f:metadata":{"f:annotations":{"f:k8s.v1.cni.cncf.io/network-status":{}}}} status}]},Spec:PodSpec{Volumes:[]Volume{Volume{Name:kube-api-access-shv9t,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:420,},StorageOS:nil,CSI:nil,Ephemeral:nil,},},},Containers:[]Container{Container{Name:samplepod-left,Image:nicolaka/netshoot,Command:[/bin/bash -c trap : TERM INT; sleep infinity & wait],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-shv9t,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:Always,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Always,TerminationGracePeriodSeconds:30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:kubemaster,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,},ImagePullSecrets:[]LocalObjectReference{LocalObjectReference{Name:docker-registry-credentials,},},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:true,PreemptionPolicy:PreemptLowerPriority,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},SetHostnameAsFQDN:nil,},Status:PodStatus{Phase:Pending,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-05-08 20:53:06 +0100 IST,Reason:,Message:,},PodCondition{Type:Ready,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-05-08 20:53:06 +0100 IST,Reason:ContainersNotReady,Message:containers with unready status: [samplepod-left],},PodCondition{Type:ContainersReady,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-05-08 20:53:06 +0100 IST,Reason:ContainersNotReady,Message:containers with unready status: [samplepod-left],},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-05-08 20:53:06 +0100 IST,Reason:,Message:,},},Message:,Reason:,HostIP:10.160.160.168,PodIP:,StartTime:2023-05-08 20:53:06 +0100 IST,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:samplepod-left,State:ContainerState{Waiting:&ContainerStateWaiting{Reason:ContainerCreating,Message:,},Running:nil,Terminated:nil,},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:false,RestartCount:0,Image:nicolaka/netshoot,ImageID:,ContainerID:,Started:*false,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{},EphemeralContainerStatuses:[]ContainerStatus{},},}, &{{0.3.1 multus-cni-network multus map[portMappings:true] {} {[] [] []} map[] } { false []} multus-cni-network [] [] false false true false [123 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 123 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 116 114 117 101 125 44 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 100 101 108 101 103 97 116 101 115 34 58 91 123 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 110 97 109 101 34 58 34 99 98 114 48 34 44 34 112 108 117 103 105 110 115 34 58 91 123 34 100 101 108 101 103 97 116 101 34 58 123 34 104 97 105 114 112 105 110 77 111 100 101 34 58 116 114 117 101 44 34 105 115 68 101 102 97 117 108 116 71 97 116 101 119 97 121 34 58 116 114 117 101 125 44 34 116 121 112 101 34 58 34 102 108 97 110 110 101 108 34 125 44 123 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 123 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 116 114 117 101 125 44 34 116 121 112 101 34 58 34 112 111 114 116 109 97 112 34 125 93 125 93 44 34 107 117 98 101 99 111 110 102 105 103 34 58 34 47 101 116 99 47 99 110 105 47 110 101 116 46 100 47 109 117 108 116 117 115 46 100 47 109 117 108 116 117 115 46 107 117 98 101 99 111 110 102 105 103 34 44 34 110 97 109 101 34 58 34 109 117 108 116 117 115 45 99 110 105 45 110 101 116 119 111 114 107 34 44 34 116 121 112 101 34 58 34 109 117 108 116 117 115 34 125]}, &{fa70a1a44372892e78299ed55a94850d20f22e9a14a88bd9ac488a86ae2fdf99 /var/run/netns/cni-a6b3368e-d2d5-8977-9e04-406936c2cf5b eth0 [[IgnoreUnknown true] [K8S_POD_NAMESPACE default] [K8S_POD_NAME samplepod-left] [K8S_POD_INFRA_CONTAINER_ID fa70a1a44372892e78299ed55a94850d20f22e9a14a88bd9ac488a86ae2fdf99] [K8S_POD_UID 05fd6a50-8679-44a5-92ab-61c432560445] [K8S_POD_UID 05fd6a50-8679-44a5-92ab-61c432560445] [IgnoreUnknown 1] [K8S_POD_NAMESPACE default] [K8S_POD_NAME samplepod-left] [K8S_POD_INFRA_CONTAINER_ID fa70a1a44372892e78299ed55a94850d20f22e9a14a88bd9ac488a86ae2fdf99]] map[] } 2023-05-08T20:56:28+01:00 [verbose] Del: default:samplepod-left:05fd6a50-8679-44a5-92ab-61c432560445:multus-cni-network:eth0 {"capabilities":{"portMappings":true},"cniVersion":"0.3.1","delegates":[{"cniVersion":"0.3.1","name":"cbr0","plugins":[{"delegate":{"hairpinMode":true,"isDefaultGateway":true},"type":"flannel"},{"capabilities":{"portMappings":true},"type":"portmap"}]}],"kubeconfig":"/etc/cni/net.d/multus.d/multus.kubeconfig","name":"multus-cni-network","type":"multus"} 2023-05-08T20:56:28+01:00 [debug] confDel: &{fa70a1a44372892e78299ed55a94850d20f22e9a14a88bd9ac488a86ae2fdf99 /var/run/netns/cni-a6b3368e-d2d5-8977-9e04-406936c2cf5b eth0 [[IgnoreUnknown true] [K8S_POD_NAMESPACE default] [K8S_POD_NAME samplepod-left] [K8S_POD_INFRA_CONTAINER_ID fa70a1a44372892e78299ed55a94850d20f22e9a14a88bd9ac488a86ae2fdf99] [K8S_POD_UID 05fd6a50-8679-44a5-92ab-61c432560445] [K8S_POD_UID 05fd6a50-8679-44a5-92ab-61c432560445] [IgnoreUnknown 1] [K8S_POD_NAMESPACE default] [K8S_POD_NAME samplepod-left] [K8S_POD_INFRA_CONTAINER_ID fa70a1a44372892e78299ed55a94850d20f22e9a14a88bd9ac488a86ae2fdf99]] map[] }, {"capabilities":{"portMappings":true},"cniVersion":"0.3.1","delegates":[{"cniVersion":"0.3.1","name":"cbr0","plugins":[{"delegate":{"hairpinMode":true,"isDefaultGateway":true},"type":"flannel"},{"capabilities":{"portMappings":true},"type":"portmap"}]}],"kubeconfig":"/etc/cni/net.d/multus.d/multus.kubeconfig","name":"multus-cni-network","type":"multus"} 2023-05-08T20:56:28+01:00 [error] error in getting result from DelNetwork: DelegateDel: error invoking DelegateDel - "host-device": error in getting result from DelNetwork: failed to find "net2": Link not found 2023-05-08T20:56:28+01:00 [error] DelegateDel: error invoking DelegateDel - "multus": error in getting result from DelNetwork: DelegateDel: error invoking DelegateDel - "host-device": error in getting result from DelNetwork: failed to find "net2": Link not found 2023-05-08T20:56:29+01:00 [debug] LoadDelegateNetConf: {"capabilities":{"portMappings":true},"cniVersion":"0.3.1","delegates":[{"cniVersion":"0.3.1","name":"cbr0","plugins":[{"delegate":{"hairpinMode":true,"isDefaultGateway":true},"type":"flannel"},{"capabilities":{"portMappings":true},"type":"portmap"}]}],"kubeconfig":"/etc/cni/net.d/multus.d/multus.kubeconfig","name":"multus-cni-network","type":"multus"}, , 2023-05-08T20:56:29+01:00 [debug] CmdAdd: &{726ccd2d0f61f772fe3fc6f0d9444f3940e8166030142b176f04b564337fbcf8 /var/run/netns/cni-323606e5-17b7-6860-8cdf-6aa24d2d5e63 eth0 K8S_POD_NAMESPACE=default;K8S_POD_NAME=samplepod-left;K8S_POD_INFRA_CONTAINER_ID=726ccd2d0f61f772fe3fc6f0d9444f3940e8166030142b176f04b564337fbcf8;K8S_POD_UID=05fd6a50-8679-44a5-92ab-61c432560445;IgnoreUnknown=1 /opt/cni/bin [123 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 123 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 116 114 117 101 125 44 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 100 101 108 101 103 97 116 101 115 34 58 91 123 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 123 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 116 114 117 101 125 44 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 100 101 108 101 103 97 116 101 115 34 58 91 123 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 110 97 109 101 34 58 34 99 98 114 48 34 44 34 112 108 117 103 105 110 115 34 58 91 123 34 100 101 108 101 103 97 116 101 34 58 123 34 104 97 105 114 112 105 110 77 111 100 101 34 58 116 114 117 101 44 34 105 115 68 101 102 97 117 108 116 71 97 116 101 119 97 121 34 58 116 114 117 101 125 44 34 116 121 112 101 34 58 34 102 108 97 110 110 101 108 34 125 44 123 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 123 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 116 114 117 101 125 44 34 116 121 112 101 34 58 34 112 111 114 116 109 97 112 34 125 93 125 93 44 34 107 117 98 101 99 111 110 102 105 103 34 58 34 47 101 116 99 47 99 110 105 47 110 101 116 46 100 47 109 117 108 116 117 115 46 100 47 109 117 108 116 117 115 46 107 117 98 101 99 111 110 102 105 103 34 44 34 110 97 109 101 34 58 34 109 117 108 116 117 115 45 99 110 105 45 110 101 116 119 111 114 107 34 44 34 116 121 112 101 34 58 34 109 117 108 116 117 115 34 125 93 44 34 107 117 98 101 99 111 110 102 105 103 34 58 34 47 101 116 99 47 99 110 105 47 110 101 116 46 100 47 109 117 108 116 117 115 46 100 47 109 117 108 116 117 115 46 107 117 98 101 99 111 110 102 105 103 34 44 34 108 111 103 70 105 108 101 34 58 34 47 118 97 114 47 108 111 103 47 109 117 108 116 117 115 46 108 111 103 34 44 34 108 111 103 76 101 118 101 108 34 58 34 100 101 98 117 103 34 44 34 110 97 109 101 34 58 34 109 117 108 116 117 115 45 99 110 105 45 110 101 116 119 111 114 107 34 44 34 116 121 112 101 34 58 34 109 117 108 116 117 115 34 125]}, , 2023-05-08T20:56:29+01:00 [debug] GetK8sClient: /etc/cni/net.d/multus.d/multus.kubeconfig, 2023-05-08T20:56:29+01:00 [debug] GetK8sArgs: &{726ccd2d0f61f772fe3fc6f0d9444f3940e8166030142b176f04b564337fbcf8 /var/run/netns/cni-323606e5-17b7-6860-8cdf-6aa24d2d5e63 eth0 K8S_POD_NAMESPACE=default;K8S_POD_NAME=samplepod-left;K8S_POD_INFRA_CONTAINER_ID=726ccd2d0f61f772fe3fc6f0d9444f3940e8166030142b176f04b564337fbcf8;K8S_POD_UID=05fd6a50-8679-44a5-92ab-61c432560445;IgnoreUnknown=1 /opt/cni/bin [123 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 123 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 116 114 117 101 125 44 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 100 101 108 101 103 97 116 101 115 34 58 91 123 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 123 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 116 114 117 101 125 44 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 100 101 108 101 103 97 116 101 115 34 58 91 123 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 110 97 109 101 34 58 34 99 98 114 48 34 44 34 112 108 117 103 105 110 115 34 58 91 123 34 100 101 108 101 103 97 116 101 34 58 123 34 104 97 105 114 112 105 110 77 111 100 101 34 58 116 114 117 101 44 34 105 115 68 101 102 97 117 108 116 71 97 116 101 119 97 121 34 58 116 114 117 101 125 44 34 116 121 112 101 34 58 34 102 108 97 110 110 101 108 34 125 44 123 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 123 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 116 114 117 101 125 44 34 116 121 112 101 34 58 34 112 111 114 116 109 97 112 34 125 93 125 93 44 34 107 117 98 101 99 111 110 102 105 103 34 58 34 47 101 116 99 47 99 110 105 47 110 101 116 46 100 47 109 117 108 116 117 115 46 100 47 109 117 108 116 117 115 46 107 117 98 101 99 111 110 102 105 103 34 44 34 110 97 109 101 34 58 34 109 117 108 116 117 115 45 99 110 105 45 110 101 116 119 111 114 107 34 44 34 116 121 112 101 34 58 34 109 117 108 116 117 115 34 125 93 44 34 107 117 98 101 99 111 110 102 105 103 34 58 34 47 101 116 99 47 99 110 105 47 110 101 116 46 100 47 109 117 108 116 117 115 46 100 47 109 117 108 116 117 115 46 107 117 98 101 99 111 110 102 105 103 34 44 34 108 111 103 70 105 108 101 34 58 34 47 118 97 114 47 108 111 103 47 109 117 108 116 117 115 46 108 111 103 34 44 34 108 111 103 76 101 118 101 108 34 58 34 100 101 98 117 103 34 44 34 110 97 109 101 34 58 34 109 117 108 116 117 115 45 99 110 105 45 110 101 116 119 111 114 107 34 44 34 116 121 112 101 34 58 34 109 117 108 116 117 115 34 125]} 2023-05-08T20:56:29+01:00 [debug] TryLoadPodDelegates: &Pod{ObjectMeta:{samplepod-left default 05fd6a50-8679-44a5-92ab-61c432560445 26888 0 2023-05-08 20:53:06 +0100 IST map[] map[k8s.v1.cni.cncf.io/network-status:[{ "name": "cbr0", "interface": "eth0", "ips": [ "10.244.0.171" ], "mac": "76:9e:23:07:d7:fa", "default": true, "dns": {}, "gateway": [ "10.244.0.1" ] },{ "name": "default/sriovnetens1f0", "interface": "net2", "ips": [ "192.168.1.23" ], "mac": "24:1c:04:f3:d7:fd", "dns": {} }] k8s.v1.cni.cncf.io/networks:sriovnetens1f0@net2 kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{"k8s.v1.cni.cncf.io/networks":"sriovnetens1f0@net2"},"name":"samplepod-left","namespace":"default"},"spec":{"containers":[{"command":["/bin/bash","-c","trap : TERM INT; sleep infinity \u0026 wait"],"image":"nicolaka/netshoot","name":"samplepod-left"}]}} ] [] [] [{kubectl-client-side-apply Update v1 2023-05-08 20:53:06 +0100 IST FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:k8s.v1.cni.cncf.io/networks":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"samplepod-left\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}} } {kubelet Update v1 2023-05-08 20:53:06 +0100 IST FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:startTime":{}}} status} {multus Update v1 2023-05-08 20:56:28 +0100 IST FieldsV1 {"f:metadata":{"f:annotations":{"f:k8s.v1.cni.cncf.io/network-status":{}}}} status}]},Spec:PodSpec{Volumes:[]Volume{Volume{Name:kube-api-access-shv9t,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:420,},StorageOS:nil,CSI:nil,Ephemeral:nil,},},},Containers:[]Container{Container{Name:samplepod-left,Image:nicolaka/netshoot,Command:[/bin/bash -c trap : TERM INT; sleep infinity & wait],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-shv9t,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:Always,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Always,TerminationGracePeriodSeconds:30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:kubemaster,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,},ImagePullSecrets:[]LocalObjectReference{LocalObjectReference{Name:docker-registry-credentials,},},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:true,PreemptionPolicy:PreemptLowerPriority,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},SetHostnameAsFQDN:nil,},Status:PodStatus{Phase:Pending,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-05-08 20:53:06 +0100 IST,Reason:,Message:,},PodCondition{Type:Ready,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-05-08 20:53:06 +0100 IST,Reason:ContainersNotReady,Message:containers with unready status: [samplepod-left],},PodCondition{Type:ContainersReady,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-05-08 20:53:06 +0100 IST,Reason:ContainersNotReady,Message:containers with unready status: [samplepod-left],},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-05-08 20:53:06 +0100 IST,Reason:,Message:,},},Message:,Reason:,HostIP:10.160.160.168,PodIP:,StartTime:2023-05-08 20:53:06 +0100 IST,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:samplepod-left,State:ContainerState{Waiting:&ContainerStateWaiting{Reason:ContainerCreating,Message:,},Running:nil,Terminated:nil,},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:false,RestartCount:0,Image:nicolaka/netshoot,ImageID:,ContainerID:,Started:false,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{},EphemeralContainerStatuses:[]ContainerStatus{},},}, &{{0.3.1 multus-cni-network multus map[portMappings:true] {} {[] [] []} map[] } /etc/cni/multus/net.d /var/lib/cni/multus /opt/cni/bin [] [0xc0002cf340] [] /etc/cni/net.d/multus.d/multus.kubeconfig /var/log/multus.log debug true false [default] [kube-system] kube-system false}, &{0xc0003a4580 0xc00040a620 0xc0000ce240 0xc0000da380} 2023-05-08T20:56:29+01:00 [debug] GetK8sClient: /etc/cni/net.d/multus.d/multus.kubeconfig, &{0xc0003a4580 0xc00040a620 0xc0000ce240 0xc0000da380} 2023-05-08T20:56:29+01:00 [debug] tryLoadK8sPodDefaultNetwork: &{0xc0003a4580 0xc00040a620 0xc0000ce240 0xc0000da380}, &Pod{ObjectMeta:{samplepod-left default 05fd6a50-8679-44a5-92ab-61c432560445 26888 0 2023-05-08 20:53:06 +0100 IST map[] map[k8s.v1.cni.cncf.io/network-status:[{ "name": "cbr0", "interface": "eth0", "ips": [ "10.244.0.171" ], "mac": "76:9e:23:07:d7:fa", "default": true, "dns": {}, "gateway": [ "10.244.0.1" ] },{ "name": "default/sriovnetens1f0", "interface": "net2", "ips": [ "192.168.1.23" ], "mac": "24:1c:04:f3:d7:fd", "dns": {} }] k8s.v1.cni.cncf.io/networks:sriovnetens1f0@net2 kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{"k8s.v1.cni.cncf.io/networks":"sriovnetens1f0@net2"},"name":"samplepod-left","namespace":"default"},"spec":{"containers":[{"command":["/bin/bash","-c","trap : TERM INT; sleep infinity \u0026 wait"],"image":"nicolaka/netshoot","name":"samplepod-left"}]}} ] [] [] [{kubectl-client-side-apply Update v1 2023-05-08 20:53:06 +0100 IST FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:k8s.v1.cni.cncf.io/networks":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"samplepod-left\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}} } {kubelet Update v1 2023-05-08 20:53:06 +0100 IST FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:startTime":{}}} status} {multus Update v1 2023-05-08 20:56:28 +0100 IST FieldsV1 {"f:metadata":{"f:annotations":{"f:k8s.v1.cni.cncf.io/network-status":{}}}} status}]},Spec:PodSpec{Volumes:[]Volume{Volume{Name:kube-api-access-shv9t,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:420,},StorageOS:nil,CSI:nil,Ephemeral:nil,},},},Containers:[]Container{Container{Name:samplepod-left,Image:nicolaka/netshoot,Command:[/bin/bash -c trap : TERM INT; sleep infinity & wait],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-shv9t,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:Always,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Always,TerminationGracePeriodSeconds:30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:kubemaster,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,},ImagePullSecrets:[]LocalObjectReference{LocalObjectReference{Name:docker-registry-credentials,},},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:true,PreemptionPolicy:PreemptLowerPriority,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},SetHostnameAsFQDN:nil,},Status:PodStatus{Phase:Pending,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-05-08 20:53:06 +0100 IST,Reason:,Message:,},PodCondition{Type:Ready,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-05-08 20:53:06 +0100 IST,Reason:ContainersNotReady,Message:containers with unready status: [samplepod-left],},PodCondition{Type:ContainersReady,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-05-08 20:53:06 +0100 IST,Reason:ContainersNotReady,Message:containers with unready status: [samplepod-left],},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-05-08 20:53:06 +0100 IST,Reason:,Message:,},},Message:,Reason:,HostIP:10.160.160.168,PodIP:,StartTime:2023-05-08 20:53:06 +0100 IST,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:samplepod-left,State:ContainerState{Waiting:&ContainerStateWaiting{Reason:ContainerCreating,Message:,},Running:nil,Terminated:nil,},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:false,RestartCount:0,Image:nicolaka/netshoot,ImageID:,ContainerID:,Started:false,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{},EphemeralContainerStatuses:[]ContainerStatus{},},}, &{{0.3.1 multus-cni-network multus map[portMappings:true] {} {[] [] []} map[] } /etc/cni/multus/net.d /var/lib/cni/multus /opt/cni/bin [] [0xc0002cf340] [] /etc/cni/net.d/multus.d/multus.kubeconfig /var/log/multus.log debug true false [default] [kube-system] kube-system false} 2023-05-08T20:56:29+01:00 [debug] tryLoadK8sPodDefaultNetwork: Pod default network annotation is not defined 2023-05-08T20:56:29+01:00 [debug] GetPodNetwork: &Pod{ObjectMeta:{samplepod-left default 05fd6a50-8679-44a5-92ab-61c432560445 26888 0 2023-05-08 20:53:06 +0100 IST map[] map[k8s.v1.cni.cncf.io/network-status:[{ "name": "cbr0", "interface": "eth0", "ips": [ "10.244.0.171" ], "mac": "76:9e:23:07:d7:fa", "default": true, "dns": {}, "gateway": [ "10.244.0.1" ] },{ "name": "default/sriovnetens1f0", "interface": "net2", "ips": [ "192.168.1.23" ], "mac": "24:1c:04:f3:d7:fd", "dns": {} }] k8s.v1.cni.cncf.io/networks:sriovnetens1f0@net2 kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{"k8s.v1.cni.cncf.io/networks":"sriovnetens1f0@net2"},"name":"samplepod-left","namespace":"default"},"spec":{"containers":[{"command":["/bin/bash","-c","trap : TERM INT; sleep infinity \u0026 wait"],"image":"nicolaka/netshoot","name":"samplepod-left"}]}} ] [] [] [{kubectl-client-side-apply Update v1 2023-05-08 20:53:06 +0100 IST FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:k8s.v1.cni.cncf.io/networks":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"samplepod-left\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}} } {kubelet Update v1 2023-05-08 20:53:06 +0100 IST FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:startTime":{}}} status} {multus Update v1 2023-05-08 20:56:28 +0100 IST FieldsV1 {"f:metadata":{"f:annotations":{"f:k8s.v1.cni.cncf.io/network-status":{}}}} status}]},Spec:PodSpec{Volumes:[]Volume{Volume{Name:kube-api-access-shv9t,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:420,},StorageOS:nil,CSI:nil,Ephemeral:nil,},},},Containers:[]Container{Container{Name:samplepod-left,Image:nicolaka/netshoot,Command:[/bin/bash -c trap : TERM INT; sleep infinity & wait],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-shv9t,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:Always,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Always,TerminationGracePeriodSeconds:30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:kubemaster,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,},ImagePullSecrets:[]LocalObjectReference{LocalObjectReference{Name:docker-registry-credentials,},},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:true,PreemptionPolicy:PreemptLowerPriority,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},SetHostnameAsFQDN:nil,},Status:PodStatus{Phase:Pending,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-05-08 20:53:06 +0100 IST,Reason:,Message:,},PodCondition{Type:Ready,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-05-08 20:53:06 +0100 IST,Reason:ContainersNotReady,Message:containers with unready status: [samplepod-left],},PodCondition{Type:ContainersReady,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-05-08 20:53:06 +0100 IST,Reason:ContainersNotReady,Message:containers with unready status: [samplepod-left],},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-05-08 20:53:06 +0100 IST,Reason:,Message:,},},Message:,Reason:,HostIP:10.160.160.168,PodIP:,StartTime:2023-05-08 20:53:06 +0100 IST,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:samplepod-left,State:ContainerState{Waiting:&ContainerStateWaiting{Reason:ContainerCreating,Message:,},Running:nil,Terminated:nil,},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:false,RestartCount:0,Image:nicolaka/netshoot,ImageID:,ContainerID:,Started:false,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{},EphemeralContainerStatuses:[]ContainerStatus{},},} 2023-05-08T20:56:29+01:00 [debug] parsePodNetworkAnnotation: sriovnetens1f0@net2, default 2023-05-08T20:56:29+01:00 [debug] parsePodNetworkObjectName: sriovnetens1f0@net2 2023-05-08T20:56:29+01:00 [debug] parsePodNetworkObjectName: parsed: , sriovnetens1f0, net2 2023-05-08T20:56:29+01:00 [debug] GetNetworkDelegates: &{0xc0003a4580 0xc00040a620 0xc0000ce240 0xc0000da380}, &Pod{ObjectMeta:{samplepod-left default 05fd6a50-8679-44a5-92ab-61c432560445 26888 0 2023-05-08 20:53:06 +0100 IST map[] map[k8s.v1.cni.cncf.io/network-status:[{ "name": "cbr0", "interface": "eth0", "ips": [ "10.244.0.171" ], "mac": "76:9e:23:07:d7:fa", "default": true, "dns": {}, "gateway": [ "10.244.0.1" ] },{ "name": "default/sriovnetens1f0", "interface": "net2", "ips": [ "192.168.1.23" ], "mac": "24:1c:04:f3:d7:fd", "dns": {} }] k8s.v1.cni.cncf.io/networks:sriovnetens1f0@net2 kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{"k8s.v1.cni.cncf.io/networks":"sriovnetens1f0@net2"},"name":"samplepod-left","namespace":"default"},"spec":{"containers":[{"command":["/bin/bash","-c","trap : TERM INT; sleep infinity \u0026 wait"],"image":"nicolaka/netshoot","name":"samplepod-left"}]}} ] [] [] [{kubectl-client-side-apply Update v1 2023-05-08 20:53:06 +0100 IST FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:k8s.v1.cni.cncf.io/networks":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"samplepod-left\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}} } {kubelet Update v1 2023-05-08 20:53:06 +0100 IST FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:startTime":{}}} status} {multus Update v1 2023-05-08 20:56:28 +0100 IST FieldsV1 {"f:metadata":{"f:annotations":{"f:k8s.v1.cni.cncf.io/network-status":{}}}} status}]},Spec:PodSpec{Volumes:[]Volume{Volume{Name:kube-api-access-shv9t,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:420,},StorageOS:nil,CSI:nil,Ephemeral:nil,},},},Containers:[]Container{Container{Name:samplepod-left,Image:nicolaka/netshoot,Command:[/bin/bash -c trap : TERM INT; sleep infinity & wait],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-shv9t,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:Always,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Always,TerminationGracePeriodSeconds:30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:kubemaster,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,},ImagePullSecrets:[]LocalObjectReference{LocalObjectReference{Name:docker-registry-credentials,},},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:true,PreemptionPolicy:PreemptLowerPriority,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},SetHostnameAsFQDN:nil,},Status:PodStatus{Phase:Pending,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-05-08 20:53:06 +0100 IST,Reason:,Message:,},PodCondition{Type:Ready,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-05-08 20:53:06 +0100 IST,Reason:ContainersNotReady,Message:containers with unready status: [samplepod-left],},PodCondition{Type:ContainersReady,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-05-08 20:53:06 +0100 IST,Reason:ContainersNotReady,Message:containers with unready status: [samplepod-left],},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-05-08 20:53:06 +0100 IST,Reason:,Message:,},},Message:,Reason:,HostIP:10.160.160.168,PodIP:,StartTime:2023-05-08 20:53:06 +0100 IST,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:samplepod-left,State:ContainerState{Waiting:&ContainerStateWaiting{Reason:ContainerCreating,Message:,},Running:nil,Terminated:nil,},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:false,RestartCount:0,Image:nicolaka/netshoot,ImageID:,ContainerID:,Started:false,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{},EphemeralContainerStatuses:[]ContainerStatus{},},}, [0xc000224480], &{{0.3.1 multus-cni-network multus map[portMappings:true] {} {[] [] []} map[] } /etc/cni/multus/net.d /var/lib/cni/multus /opt/cni/bin [] [0xc0002cf340] [] /etc/cni/net.d/multus.d/multus.kubeconfig /var/log/multus.log debug true false [default] [kube-system] kube-system false}, map[] 2023-05-08T20:56:29+01:00 [debug] getKubernetesDelegate: &{0xc0003a4580 0xc00040a620 0xc0000ce240 0xc0000da380}, &{sriovnetens1f0 default [] net2 [] }, /etc/cni/multus/net.d, &Pod{ObjectMeta:{samplepod-left default 05fd6a50-8679-44a5-92ab-61c432560445 26888 0 2023-05-08 20:53:06 +0100 IST map[] map[k8s.v1.cni.cncf.io/network-status:[{ "name": "cbr0", "interface": "eth0", "ips": [ "10.244.0.171" ], "mac": "76:9e:23:07:d7:fa", "default": true, "dns": {}, "gateway": [ "10.244.0.1" ] },{ "name": "default/sriovnetens1f0", "interface": "net2", "ips": [ "192.168.1.23" ], "mac": "24:1c:04:f3:d7:fd", "dns": {} }] k8s.v1.cni.cncf.io/networks:sriovnetens1f0@net2 kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{"k8s.v1.cni.cncf.io/networks":"sriovnetens1f0@net2"},"name":"samplepod-left","namespace":"default"},"spec":{"containers":[{"command":["/bin/bash","-c","trap : TERM INT; sleep infinity \u0026 wait"],"image":"nicolaka/netshoot","name":"samplepod-left"}]}} ] [] [] [{kubectl-client-side-apply Update v1 2023-05-08 20:53:06 +0100 IST FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:k8s.v1.cni.cncf.io/networks":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"samplepod-left\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}} } {kubelet Update v1 2023-05-08 20:53:06 +0100 IST FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:startTime":{}}} status} {multus Update v1 2023-05-08 20:56:28 +0100 IST FieldsV1 {"f:metadata":{"f:annotations":{"f:k8s.v1.cni.cncf.io/network-status":{}}}} status}]},Spec:PodSpec{Volumes:[]Volume{Volume{Name:kube-api-access-shv9t,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:420,},StorageOS:nil,CSI:nil,Ephemeral:nil,},},},Containers:[]Container{Container{Name:samplepod-left,Image:nicolaka/netshoot,Command:[/bin/bash -c trap : TERM INT; sleep infinity & wait],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-shv9t,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:Always,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Always,TerminationGracePeriodSeconds:30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:kubemaster,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,},ImagePullSecrets:[]LocalObjectReference{LocalObjectReference{Name:docker-registry-credentials,},},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:true,PreemptionPolicy:PreemptLowerPriority,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},SetHostnameAsFQDN:nil,},Status:PodStatus{Phase:Pending,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-05-08 20:53:06 +0100 IST,Reason:,Message:,},PodCondition{Type:Ready,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-05-08 20:53:06 +0100 IST,Reason:ContainersNotReady,Message:containers with unready status: [samplepod-left],},PodCondition{Type:ContainersReady,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-05-08 20:53:06 +0100 IST,Reason:ContainersNotReady,Message:containers with unready status: [samplepod-left],},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-05-08 20:53:06 +0100 IST,Reason:,Message:,},},Message:,Reason:,HostIP:10.160.160.168,PodIP:,StartTime:2023-05-08 20:53:06 +0100 IST,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:samplepod-left,State:ContainerState{Waiting:&ContainerStateWaiting{Reason:ContainerCreating,Message:,},Running:nil,Terminated:nil,},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:false,RestartCount:0,Image:nicolaka/netshoot,ImageID:,ContainerID:,Started:*false,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{},EphemeralContainerStatuses:[]ContainerStatus{},},}, map[] 2023-05-08T20:56:29+01:00 [debug] LoadDelegateNetConf: {"cniVersion":"0.3.0","device":"ens1f0","ipam":{"addresses":[{"address":"192.168.1.23/24"}],"type":"static"},"name":"sriovnetens1f0","type":"host-device"}, &{sriovnetens1f0 default [] net2 [] }, 2023-05-08T20:56:29+01:00 [debug] AddDelegates: [0xc0002ce1c0] 2023-05-08T20:56:29+01:00 [debug] saveDelegates: 726ccd2d0f61f772fe3fc6f0d9444f3940e8166030142b176f04b564337fbcf8, /var/lib/cni/multus, [0xc0002cf340 0xc0002ce1c0] 2023-05-08T20:56:29+01:00 [debug] saveScratchNetConf: 726ccd2d0f61f772fe3fc6f0d9444f3940e8166030142b176f04b564337fbcf8, /var/lib/cni/multus, [{"Conf":{"cniVersion":"0.3.1","name":"multus-cni-network","type":"multus","capabilities":{"portMappings":true},"ipam":{},"dns":{}},"ConfList":{},"Name":"multus-cni-network","IsFilterV4Gateway":false,"IsFilterV6Gateway":false,"Bytes":"eyJjYXBhYmlsaXRpZXMiOnsicG9ydE1hcHBpbmdzIjp0cnVlfSwiY25pVmVyc2lvbiI6IjAuMy4xIiwiZGVsZWdhdGVzIjpbeyJjbmlWZXJzaW9uIjoiMC4zLjEiLCJuYW1lIjoiY2JyMCIsInBsdWdpbnMiOlt7ImRlbGVnYXRlIjp7ImhhaXJwaW5Nb2RlIjp0cnVlLCJpc0RlZmF1bHRHYXRld2F5Ijp0cnVlfSwidHlwZSI6ImZsYW5uZWwifSx7ImNhcGFiaWxpdGllcyI6eyJwb3J0TWFwcGluZ3MiOnRydWV9LCJ0eXBlIjoicG9ydG1hcCJ9XX1dLCJrdWJlY29uZmlnIjoiL2V0Yy9jbmkvbmV0LmQvbXVsdHVzLmQvbXVsdHVzLmt1YmVjb25maWciLCJuYW1lIjoibXVsdHVzLWNuaS1uZXR3b3JrIiwidHlwZSI6Im11bHR1cyJ9"},{"Conf":{"cniVersion":"0.3.0","name":"sriovnetens1f0","type":"host-device","ipam":{"type":"static"},"dns":{}},"ConfList":{},"Name":"default/sriovnetens1f0","ifnameRequest":"net2","IsFilterV4Gateway":false,"IsFilterV6Gateway":false,"Bytes":"eyJjbmlWZXJzaW9uIjoiMC4zLjAiLCJkZXZpY2UiOiJlbnMxZjAiLCJpcGFtIjp7ImFkZHJlc3NlcyI6W3siYWRkcmVzcyI6IjE5Mi4xNjguMS4yMy8yNCJ9XSwidHlwZSI6InN0YXRpYyJ9LCJuYW1lIjoic3Jpb3ZuZXRlbnMxZjAiLCJ0eXBlIjoiaG9zdC1kZXZpY2UifQ=="}] 2023-05-08T20:56:29+01:00 [debug] getIfname: &{{0.3.1 multus-cni-network multus map[portMappings:true] {} {[] [] []} map[] } { false []} multus-cni-network [] [] false false true false [123 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 123 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 116 114 117 101 125 44 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 100 101 108 101 103 97 116 101 115 34 58 91 123 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 110 97 109 101 34 58 34 99 98 114 48 34 44 34 112 108 117 103 105 110 115 34 58 91 123 34 100 101 108 101 103 97 116 101 34 58 123 34 104 97 105 114 112 105 110 77 111 100 101 34 58 116 114 117 101 44 34 105 115 68 101 102 97 117 108 116 71 97 116 101 119 97 121 34 58 116 114 117 101 125 44 34 116 121 112 101 34 58 34 102 108 97 110 110 101 108 34 125 44 123 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 123 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 116 114 117 101 125 44 34 116 121 112 101 34 58 34 112 111 114 116 109 97 112 34 125 93 125 93 44 34 107 117 98 101 99 111 110 102 105 103 34 58 34 47 101 116 99 47 99 110 105 47 110 101 116 46 100 47 109 117 108 116 117 115 46 100 47 109 117 108 116 117 115 46 107 117 98 101 99 111 110 102 105 103 34 44 34 110 97 109 101 34 58 34 109 117 108 116 117 115 45 99 110 105 45 110 101 116 119 111 114 107 34 44 34 116 121 112 101 34 58 34 109 117 108 116 117 115 34 125]}, eth0, 0 2023-05-08T20:56:29+01:00 [debug] LoadCNIRuntimeConf: eth0, &{{0.3.1 multus-cni-network multus map[portMappings:true] {} {[] [] []} map[] } { false []} multus-cni-network [] [] false false true false [123 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 123 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 116 114 117 101 125 44 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 100 101 108 101 103 97 116 101 115 34 58 91 123 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 110 97 109 101 34 58 34 99 98 114 48 34 44 34 112 108 117 103 105 110 115 34 58 91 123 34 100 101 108 101 103 97 116 101 34 58 123 34 104 97 105 114 112 105 110 77 111 100 101 34 58 116 114 117 101 44 34 105 115 68 101 102 97 117 108 116 71 97 116 101 119 97 121 34 58 116 114 117 101 125 44 34 116 121 112 101 34 58 34 102 108 97 110 110 101 108 34 125 44 123 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 123 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 116 114 117 101 125 44 34 116 121 112 101 34 58 34 112 111 114 116 109 97 112 34 125 93 125 93 44 34 107 117 98 101 99 111 110 102 105 103 34 58 34 47 101 116 99 47 99 110 105 47 110 101 116 46 100 47 109 117 108 116 117 115 46 100 47 109 117 108 116 117 115 46 107 117 98 101 99 111 110 102 105 103 34 44 34 110 97 109 101 34 58 34 109 117 108 116 117 115 45 99 110 105 45 110 101 116 119 111 114 107 34 44 34 116 121 112 101 34 58 34 109 117 108 116 117 115 34 125]} 2023-05-08T20:56:29+01:00 [debug] mergeCNIRuntimeConfig: &{{0.3.1 multus-cni-network multus map[portMappings:true] {} {[] [] []} map[] } { false []} multus-cni-network [] [] false false true false [123 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 123 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 116 114 117 101 125 44 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 100 101 108 101 103 97 116 101 115 34 58 91 123 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 110 97 109 101 34 58 34 99 98 114 48 34 44 34 112 108 117 103 105 110 115 34 58 91 123 34 100 101 108 101 103 97 116 101 34 58 123 34 104 97 105 114 112 105 110 77 111 100 101 34 58 116 114 117 101 44 34 105 115 68 101 102 97 117 108 116 71 97 116 101 119 97 121 34 58 116 114 117 101 125 44 34 116 121 112 101 34 58 34 102 108 97 110 110 101 108 34 125 44 123 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 123 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 116 114 117 101 125 44 34 116 121 112 101 34 58 34 112 111 114 116 109 97 112 34 125 93 125 93 44 34 107 117 98 101 99 111 110 102 105 103 34 58 34 47 101 116 99 47 99 110 105 47 110 101 116 46 100 47 109 117 108 116 117 115 46 100 47 109 117 108 116 117 115 46 107 117 98 101 99 111 110 102 105 103 34 44 34 110 97 109 101 34 58 34 109 117 108 116 117 115 45 99 110 105 45 110 101 116 119 111 114 107 34 44 34 116 121 112 101 34 58 34 109 117 108 116 117 115 34 125]} 2023-05-08T20:56:29+01:00 [debug] ARGS: K8S_POD_NAMESPACE=default;K8S_POD_NAME=samplepod-left;K8S_POD_INFRA_CONTAINER_ID=726ccd2d0f61f772fe3fc6f0d9444f3940e8166030142b176f04b564337fbcf8;K8S_POD_UID=05fd6a50-8679-44a5-92ab-61c432560445;IgnoreUnknown=1 2023-05-08T20:56:29+01:00 [debug] DelegateAdd: , &{{0.3.1 multus-cni-network multus map[portMappings:true] {} {[] [] []} map[] } { false []} multus-cni-network [] [] false false true false [123 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 123 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 116 114 117 101 125 44 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 100 101 108 101 103 97 116 101 115 34 58 91 123 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 110 97 109 101 34 58 34 99 98 114 48 34 44 34 112 108 117 103 105 110 115 34 58 91 123 34 100 101 108 101 103 97 116 101 34 58 123 34 104 97 105 114 112 105 110 77 111 100 101 34 58 116 114 117 101 44 34 105 115 68 101 102 97 117 108 116 71 97 116 101 119 97 121 34 58 116 114 117 101 125 44 34 116 121 112 101 34 58 34 102 108 97 110 110 101 108 34 125 44 123 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 123 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 116 114 117 101 125 44 34 116 121 112 101 34 58 34 112 111 114 116 109 97 112 34 125 93 125 93 44 34 107 117 98 101 99 111 110 102 105 103 34 58 34 47 101 116 99 47 99 110 105 47 110 101 116 46 100 47 109 117 108 116 117 115 46 100 47 109 117 108 116 117 115 46 107 117 98 101 99 111 110 102 105 103 34 44 34 110 97 109 101 34 58 34 109 117 108 116 117 115 45 99 110 105 45 110 101 116 119 111 114 107 34 44 34 116 121 112 101 34 58 34 109 117 108 116 117 115 34 125]}, &{726ccd2d0f61f772fe3fc6f0d9444f3940e8166030142b176f04b564337fbcf8 /var/run/netns/cni-323606e5-17b7-6860-8cdf-6aa24d2d5e63 eth0 [[IgnoreUnknown true] [K8S_POD_NAMESPACE default] [K8S_POD_NAME samplepod-left] [K8S_POD_INFRA_CONTAINER_ID 726ccd2d0f61f772fe3fc6f0d9444f3940e8166030142b176f04b564337fbcf8] [K8S_POD_UID 05fd6a50-8679-44a5-92ab-61c432560445] [K8S_POD_NAMESPACE default] [K8S_POD_NAME samplepod-left] [K8S_POD_INFRA_CONTAINER_ID 726ccd2d0f61f772fe3fc6f0d9444f3940e8166030142b176f04b564337fbcf8] [K8S_POD_UID 05fd6a50-8679-44a5-92ab-61c432560445] [IgnoreUnknown 1]] map[] } 2023-05-08T20:56:29+01:00 [debug] validateIfName: /var/run/netns/cni-323606e5-17b7-6860-8cdf-6aa24d2d5e63, eth0 2023-05-08T20:56:29+01:00 [debug] confAdd: &{726ccd2d0f61f772fe3fc6f0d9444f3940e8166030142b176f04b564337fbcf8 /var/run/netns/cni-323606e5-17b7-6860-8cdf-6aa24d2d5e63 eth0 [[IgnoreUnknown true] [K8S_POD_NAMESPACE default] [K8S_POD_NAME samplepod-left] [K8S_POD_INFRA_CONTAINER_ID 726ccd2d0f61f772fe3fc6f0d9444f3940e8166030142b176f04b564337fbcf8] [K8S_POD_UID 05fd6a50-8679-44a5-92ab-61c432560445] [K8S_POD_NAMESPACE default] [K8S_POD_NAME samplepod-left] [K8S_POD_INFRA_CONTAINER_ID 726ccd2d0f61f772fe3fc6f0d9444f3940e8166030142b176f04b564337fbcf8] [K8S_POD_UID 05fd6a50-8679-44a5-92ab-61c432560445] [IgnoreUnknown 1]] map[] }, {"capabilities":{"portMappings":true},"cniVersion":"0.3.1","delegates":[{"cniVersion":"0.3.1","name":"cbr0","plugins":[{"delegate":{"hairpinMode":true,"isDefaultGateway":true},"type":"flannel"},{"capabilities":{"portMappings":true},"type":"portmap"}]}],"kubeconfig":"/etc/cni/net.d/multus.d/multus.kubeconfig","name":"multus-cni-network","type":"multus"}

cucosion commented 1 year ago

https://github.com/k8snetworkplumbingwg/multus-cni/commit/f0794119ba29b0282583612c77cb3bfe0f3cf127 @s1061123 can you please take a look

cucosion commented 1 year ago

[root@kubemaster ~]# kubectl get pods NAME READY STATUS RESTARTS AGE samplepod-left 0/1 ContainerCreating 0 79s [root@kubemaster ~]# ip netns list cni-0b4588e1-0f51-5e79-b2ba-d2c244ee9943 (id: 1) cni-71351945-f2ed-da37-3ee1-fa26dd374bef (id: 0) [root@kubemaster ~]# ip netns exec cni-0b4588e1-0f51-5e79-b2ba-d2c244ee9943 ip link 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: eth0@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default link/ether 5a:bb:a7:72:22:d7 brd ff:ff:ff:ff:ff:ff link-netnsid 0 [root@kubemaster ~]# ip netns exec cni-71351945-f2ed-da37-3ee1-fa26dd374bef ip link 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: eth0@if15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default link/ether 6a:5e:d7:54:4e:c2 brd ff:ff:ff:ff:ff:ff link-netnsid 0 [root@kubemaster ~]# kubectl describe pod samplepod-left | grep event Warning FailedCreatePodSandBox 119s (x16 over 2m15s) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b0a5baae64a8f44f008871faba0e39531de9cb640fdc2f66305818d2eea08372": plugin type="multus" name="multus-cni-network" failed (add): [default/samplepod-left/2536b549-9930-40dc-9dd4-b483e91759ba:multus-cni-network]: error adding container to network "multus-cni-network": [default/samplepod-left/2536b549-9930-40dc-9dd4-b483e91759ba:sriovnetens1f0]: error adding container to network "sriovnetens1f0": DelegateAdd: cannot set "host-device" interface name to "net2": validateIfName: interface name net2 already exists

cucosion commented 1 year ago

[root@kubemaster ~]# ip link show; ip -all netns exec ip link show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 link/ether 78:ac:44:57:8e:98 brd ff:ff:ff:ff:ff:ff altname enp24s0f0 4: eno2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000 link/ether 78:ac:44:57:8e:99 brd ff:ff:ff:ff:ff:ff altname enp24s0f1 5: ens1f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 link/ether 24:1c:04:f3:d7:fe brd ff:ff:ff:ff:ff:ff altname enp59s0f1 6: eno3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000 link/ether 78:ac:44:57:8e:9a brd ff:ff:ff:ff:ff:ff altname enp24s0f2 7: eno4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000 link/ether 78:ac:44:57:8e:9b brd ff:ff:ff:ff:ff:ff altname enp24s0f3 8: ens2f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 link/ether b4:96:91:88:0c:b4 brd ff:ff:ff:ff:ff:ff altname enp94s0f0 9: ens2f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 link/ether b4:96:91:88:0c:b5 brd ff:ff:ff:ff:ff:ff altname enp94s0f1 10: ens3f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 link/ether b4:96:91:88:12:e8 brd ff:ff:ff:ff:ff:ff altname enp216s0f0 11: ens3f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 link/ether b4:96:91:88:12:e9 brd ff:ff:ff:ff:ff:ff altname enp216s0f1 12: idrac: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN mode DEFAULT group default qlen 1000 link/ether ec:2a:72:1d:a5:43 brd ff:ff:ff:ff:ff:ff altname enp0s20f0u14u3 13: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group default link/ether 1a:4a:ac:19:cf:a5 brd ff:ff:ff:ff:ff:ff 14: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether fe:bb:87:20:df:51 brd ff:ff:ff:ff:ff:ff 15: veth7aa6d375@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP mode DEFAULT group default link/ether ea:93:64:84:06:89 brd ff:ff:ff:ff:ff:ff link-netns cni-71351945-f2ed-da37-3ee1-fa26dd374bef 16: veth0fc0b358@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP mode DEFAULT group default link/ether 92:4b:89:85:46:68 brd ff:ff:ff:ff:ff:ff link-netns cni-0b4588e1-0f51-5e79-b2ba-d2c244ee9943 66487: vethc7c1249f@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP mode DEFAULT group default link/ether 22:dc:fc:9b:c2:29 brd ff:ff:ff:ff:ff:ff link-netns cni-ff6ab8ee-65a5-b106-f6ea-1364039d5e81

netns: cni-ff6ab8ee-65a5-b106-f6ea-1364039d5e81 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: eth0@if66487: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default link/ether 0a:df:4c:1b:a4:a0 brd ff:ff:ff:ff:ff:ff link-netnsid 0 3: net2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 link/ether 24:1c:04:f3:d7:fd brd ff:ff:ff:ff:ff:ff alias ens1f0 altname enp59s0f0

netns: cni-0b4588e1-0f51-5e79-b2ba-d2c244ee9943 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: eth0@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default link/ether 5a:bb:a7:72:22:d7 brd ff:ff:ff:ff:ff:ff link-netnsid 0

netns: cni-71351945-f2ed-da37-3ee1-fa26dd374bef 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: eth0@if15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default link/ether 6a:5e:d7:54:4e:c2 brd ff:ff:ff:ff:ff:ff link-netnsid 0

cucosion commented 1 year ago

[root@kubemaster tmp]# kubectl describe pods samplepod-left Name: samplepod-left Namespace: default Priority: 0 Service Account: default Node: kubemaster/10.160.160.168 Start Time: Tue, 09 May 2023 21:22:40 +0100 Labels: Annotations: k8s.v1.cni.cncf.io/network-status: [{ "name": "cbr0", "interface": "eth0", "ips": [ "10.244.0.148" ], "mac": "82:74:72:57:4b:bd", "default": true, "dns": {}, "gateway": [ "10.244.0.1" ] },{ "name": "default/macvlan-conf-1", "interface": "net1", "ips": [ "10.10.1.33" ], "mac": "a6:70:19:1e:c9:01", "dns": {} }] k8s.v1.cni.cncf.io/networks: macvlan-conf-1 Status: Pending IP: IPs: Containers: samplepod-left: Container ID: Image: nicolaka/netshoot Image ID: Port: Host Port: Command: /bin/bash -c trap : TERM INT; sleep infinity & wait State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Environment: Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zvsd4 (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: kube-api-access-zvsd4: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: BestEffort Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message

Normal Scheduled 14s default-scheduler Successfully assigned default/samplepod-left to kubemaster Normal AddedInterface 13s multus Add eth0 [10.244.0.135/24] from cbr0 Normal AddedInterface 13s multus Add net1 [10.10.1.20/16] from default/macvlan-conf-1 Normal AddedInterface 13s multus Add eth0 [10.244.0.135/24] from multus-cni-network Warning FailedCreatePodSandBox 13s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "23aff24517f0ed9bf6da45d74b6551d2b686f8b56f86399386b81ae65c0709ba": plugin type="multus" name="multus-cni-network" failed (add): [default/samplepod-left/e09fe03c-0ada-4a8c-9e18-1003cb6d8269:multus-cni-network]: error adding container to network "multus-cni-network": [default/samplepod-left/e09fe03c-0ada-4a8c-9e18-1003cb6d8269:multus-cni-network]: error adding container to network "multus-cni-network": [default/samplepod-left/e09fe03c-0ada-4a8c-9e18-1003cb6d8269:multus-cni-network]: error adding container to network "multus-cni-network": [default/samplepod-left/e09fe03c-0ada-4a8c-9e18-1003cb6d8269:macvlan-conf-1]: error adding container to network "macvlan-conf-1": DelegateAdd: cannot set "macvlan" interface name to "net1": validateIfName: interface name net1 already exists Normal AddedInterface 12s multus Add eth0 [10.244.0.136/24] from cbr0 Normal AddedInterface 12s multus Add net1 [10.10.1.21/16] from default/macvlan-conf-1 Normal AddedInterface 12s multus Add eth0 [10.244.0.136/24] from multus-cni-network Warning FailedCreatePodSandBox 11s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "67d78fdb21775c1441bbccbc423ff5ad09fd0f91e6a415ff08d7b550921330f6": plugin type="multus" name="multus-cni-network" failed (add): [default/samplepod-left/e09fe03c-0ada-4a8c-9e18-1003cb6d8269:multus-cni-network]: error adding container to network "multus-cni-network": [default/samplepod-left/e09fe03c-0ada-4a8c-9e18-1003cb6d8269:multus-cni-network]: error adding container to network "multus-cni-network": [default/samplepod-left/e09fe03c-0ada-4a8c-9e18-1003cb6d8269:multus-cni-network]: error adding container to network "multus-cni-network": [default/samplepod-left/e09fe03c-0ada-4a8c-9e18-1003cb6d8269:macvlan-conf-1]: error adding container to network "macvlan-conf-1": DelegateAdd: cannot set "macvlan" interface name to "net1": validateIfName: interface name net1 already exists Normal AddedInterface 11s multus Add eth0 [10.244.0.137/24] from cbr0 Normal AddedInterface 11s multus Add net1 [10.10.1.22/16] from default/macvlan-conf-1 Normal AddedInterface 11s multus Add eth0 [10.244.0.137/24] from multus-cni-network Warning FailedCreatePodSandBox 10s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "227d01cf93d64393104defa329bc069c2b17383a24b346cb69e7c3207030cdc1": plugin type="multus" name="multus-cni-network" failed (add): [default/samplepod-left/e09fe03c-0ada-4a8c-9e18-1003cb6d8269:multus-cni-network]: error adding container to network "multus-cni-network": [default/samplepod-left/e09fe03c-0ada-4a8c-9e18-1003cb6d8269:multus-cni-network]: error adding container to network "multus-cni-network": [default/samplepod-left/e09fe03c-0ada-4a8c-9e18-1003cb6d8269:multus-cni-network]: error adding container to network "multus-cni-network": [default/samplepod-left/e09fe03c-0ada-4a8c-9e18-1003cb6d8269:macvlan-conf-1]: error adding container to network "macvlan-conf-1": DelegateAdd: cannot set "macvlan" interface name to "net1": validateIfName: interface name net1 already exists Normal AddedInterface 10s multus Add eth0 [10.244.0.138/24] from cbr0 Normal AddedInterface 10s multus Add net1 [10.10.1.23/16] from default/macvlan-conf-1 Normal AddedInterface 10s multus Add eth0 [10.244.0.138/24] from multus-cni-network Warning FailedCreatePodSandBox 9s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a4d66b1f8e165bb2930d9b266329ea8d17f5ca48de627d3f45a05be89a48cb93": plugin type="multus" name="multus-cni-network" failed (add): [default/samplepod-left/e09fe03c-0ada-4a8c-9e18-1003cb6d8269:multus-cni-network]: error adding container to network "multus-cni-network": [default/samplepod-left/e09fe03c-0ada-4a8c-9e18-1003cb6d8269:multus-cni-network]: error adding container to network "multus-cni-network": [default/samplepod-left/e09fe03c-0ada-4a8c-9e18-1003cb6d8269:multus-cni-network]: error adding container to network "multus-cni-network": [default/samplepod-left/e09fe03c-0ada-4a8c-9e18-1003cb6d8269:macvlan-conf-1]: error adding container to network "macvlan-conf-1": DelegateAdd: cannot set "macvlan" interface name to "net1": validateIfName: interface name net1 already exists Normal AddedInterface 9s multus Add eth0 [10.244.0.139/24] from cbr0 Normal AddedInterface 9s multus Add net1 [10.10.1.24/16] from default/macvlan-conf-1 Normal AddedInterface 9s multus Add eth0 [10.244.0.139/24] from multus-cni-network Warning FailedCreatePodSandBox 8s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b75d26b96d4030f3b705d72b69e266a72cf6571c16b6af67e282e65bc9d7a166": plugin type="multus" name="multus-cni-network" failed (add): [default/samplepod-left/e09fe03c-0ada-4a8c-9e18-1003cb6d8269:multus-cni-network]: error adding container to network "multus-cni-network": [default/samplepod-left/e09fe03c-0ada-4a8c-9e18-1003cb6d8269:multus-cni-network]: error adding container to network "multus-cni-network": [default/samplepod-left/e09fe03c-0ada-4a8c-9e18-1003cb6d8269:multus-cni-network]: error adding container to network "multus-cni-network": [default/samplepod-left/e09fe03c-0ada-4a8c-9e18-1003cb6d8269:macvlan-conf-1]: error adding container to network "macvlan-conf-1": DelegateAdd: cannot set "macvlan" interface name to "net1": validateIfName: interface name net1 already exists Normal AddedInterface 8s multus Add eth0 [10.244.0.140/24] from cbr0 Normal AddedInterface 8s multus Add net1 [10.10.1.25/16] from default/macvlan-conf-1 Normal AddedInterface 8s multus Add eth0 [10.244.0.140/24] from multus-cni-network Warning FailedCreatePodSandBox 7s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "8daf2664750443914d15826abc37acd251e6de876d69296a26abe53f3af7466c": plugin type="multus" name="multus-cni-network" failed (add): [default/samplepod-left/e09fe03c-0ada-4a8c-9e18-1003cb6d8269:multus-cni-network]: error adding container to network "multus-cni-network": [default/samplepod-left/e09fe03c-0ada-4a8c-9e18-1003cb6d8269:multus-cni-network]: error adding container to network "multus-cni-network": [default/samplepod-left/e09fe03c-0ada-4a8c-9e18-1003cb6d8269:multus-cni-network]: error adding container to network "multus-cni-network": [default/samplepod-left/e09fe03c-0ada-4a8c-9e18-1003cb6d8269:macvlan-conf-1]: error adding container to network "macvlan-conf-1": DelegateAdd: cannot set "macvlan" interface name to "net1": validateIfName: interface name net1 already exists Normal AddedInterface 7s multus Add eth0 [10.244.0.141/24] from cbr0 Normal AddedInterface 7s multus Add net1 [10.10.1.26/16] from default/macvlan-conf-1 Normal AddedInterface 7s multus Add eth0 [10.244.0.141/24] from multus-cni-network Warning FailedCreatePodSandBox 6s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "236b8776b20c68484e003e8f88dffac4c7c70d16d928c14c0dc2d071897fceb6": plugin type="multus" name="multus-cni-network" failed (add): [default/samplepod-left/e09fe03c-0ada-4a8c-9e18-1003cb6d8269:multus-cni-network]: error adding container to network "multus-cni-network": [default/samplepod-left/e09fe03c-0ada-4a8c-9e18-1003cb6d8269:multus-cni-network]: error adding container to network "multus-cni-network": [default/samplepod-left/e09fe03c-0ada-4a8c-9e18-1003cb6d8269:multus-cni-network]: error adding container to network "multus-cni-network": [default/samplepod-left/e09fe03c-0ada-4a8c-9e18-1003cb6d8269:macvlan-conf-1]: error adding container to network "macvlan-conf-1": DelegateAdd: cannot set "macvlan" interface name to "net1": validateIfName: interface name net1 already exists Normal AddedInterface 6s multus Add eth0 [10.244.0.142/24] from cbr0 Normal AddedInterface 6s multus Add net1 [10.10.1.27/16] from default/macvlan-conf-1 Normal AddedInterface 6s multus Add eth0 [10.244.0.142/24] from multus-cni-network Warning FailedCreatePodSandBox 5s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "3ed2f9df082b074dd65ca80a64a4994ec0f601793b1e57ba9a8e6262623fade0": plugin type="multus" name="multus-cni-network" failed (add): [default/samplepod-left/e09fe03c-0ada-4a8c-9e18-1003cb6d8269:multus-cni-network]: error adding container to network "multus-cni-network": [default/samplepod-left/e09fe03c-0ada-4a8c-9e18-1003cb6d8269:multus-cni-network]: error adding container to network "multus-cni-network": [default/samplepod-left/e09fe03c-0ada-4a8c-9e18-1003cb6d8269:multus-cni-network]: error adding container to network "multus-cni-network": [default/samplepod-left/e09fe03c-0ada-4a8c-9e18-1003cb6d8269:macvlan-conf-1]: error adding container to network "macvlan-conf-1": DelegateAdd: cannot set "macvlan" interface name to "net1": validateIfName: interface name net1 already exists Normal AddedInterface 5s multus Add eth0 [10.244.0.143/24] from cbr0 Normal AddedInterface 5s multus Add net1 [10.10.1.28/16] from default/macvlan-conf-1 Normal AddedInterface 5s multus Add eth0 [10.244.0.143/24] from multus-cni-network Warning FailedCreatePodSandBox 4s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "5808a28806a3e036c97bcafa005007df08f63fcfd2af2db6b44f04a32b80a12b": plugin type="multus" name="multus-cni-network" failed (add): [default/samplepod-left/e09fe03c-0ada-4a8c-9e18-1003cb6d8269:multus-cni-network]: error adding container to network "multus-cni-network": [default/samplepod-left/e09fe03c-0ada-4a8c-9e18-1003cb6d8269:multus-cni-network]: error adding container to network "multus-cni-network": [default/samplepod-left/e09fe03c-0ada-4a8c-9e18-1003cb6d8269:multus-cni-network]: error adding container to network "multus-cni-network": [default/samplepod-left/e09fe03c-0ada-4a8c-9e18-1003cb6d8269:macvlan-conf-1]: error adding container to network "macvlan-conf-1": DelegateAdd: cannot set "macvlan" interface name to "net1": validateIfName: interface name net1 already exists Normal AddedInterface 4s multus Add eth0 [10.244.0.144/24] from cbr0 Normal AddedInterface 4s multus Add net1 [10.10.1.29/16] from default/macvlan-conf-1 Normal AddedInterface 4s multus Add eth0 [10.244.0.144/24] from multus-cni-network Normal AddedInterface 3s multus Add eth0 [10.244.0.145/24] from cbr0 Normal AddedInterface 3s multus Add net1 [10.10.1.30/16] from default/macvlan-conf-1 Normal AddedInterface 3s multus Add eth0 [10.244.0.145/24] from multus-cni-network Normal AddedInterface 2s multus Add eth0 [10.244.0.146/24] from cbr0 Normal AddedInterface 2s multus Add net1 [10.10.1.31/16] from default/macvlan-conf-1 Normal AddedInterface 2s multus Add eth0 [10.244.0.146/24] from multus-cni-network Normal AddedInterface 1s multus Add eth0 [10.244.0.147/24] from cbr0 Normal AddedInterface 1s multus Add net1 [10.10.1.32/16] from default/macvlan-conf-1 Normal AddedInterface 1s multus Add eth0 [10.244.0.147/24] from multus-cni-network Warning FailedCreatePodSandBox 0s (x4 over 3s) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "c608120ff19f57b91f70eac5daf2d2a29bb781e1642b00cb6618f17074e2a9b6": plugin type="multus" name="multus-cni-network" failed (add): [default/samplepod-left/e09fe03c-0ada-4a8c-9e18-1003cb6d8269:multus-cni-network]: error adding container to network "multus-cni-network": [default/samplepod-left/e09fe03c-0ada-4a8c-9e18-1003cb6d8269:multus-cni-network]: error adding container to network "multus-cni-network": [default/samplepod-left/e09fe03c-0ada-4a8c-9e18-1003cb6d8269:multus-cni-network]: error adding container to network "multus-cni-network": [default/samplepod-left/e09fe03c-0ada-4a8c-9e18-1003cb6d8269:macvlan-conf-1]: error adding container to network "macvlan-conf-1": DelegateAdd: cannot set "macvlan" interface name to "net1": validateIfName: interface name net1 already exists Normal AddedInterface 0s multus Add eth0 [10.244.0.148/24] from cbr0 Normal AddedInterface 0s multus Add net1 [10.10.1.33/16] from default/macvlan-conf-1 Normal AddedInterface 0s multus Add eth0 [10.244.0.148/24] from multus-cni-network [root@kubemaster tmp]# cat nad2.yaml apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: macvlan-conf-1 spec: config: '{ "cniVersion": "0.3.0", "type": "macvlan", "master": "ens1f0", "mode": "bridge", "ipam": { "type": "host-local", "ranges": [ [ { "subnet": "10.10.0.0/16", "rangeStart": "10.10.1.20", "rangeEnd": "10.10.3.50", "gateway": "10.10.0.254" } ] ] } }' [root@kubemaster tmp]# cat ex2.yaml apiVersion: v1 kind: Pod metadata: name: samplepod-left annotations: k8s.v1.cni.cncf.io/networks: macvlan-conf-1 spec: containers:

cucosion commented 1 year ago

I did a fresh install and i can reproduce the issue: On a fresh install with Alma Linux 9.1, latest version of flannel and multus and kubernetes version 1.27.1.

If I use the below nad and pod the first time it works ok and adds an additional interface with the host device as the second interface. After a simple restart, it does not work when i try to start the same pod

[root@kubemaster tmp]# cat nad.yaml apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: nad-host spec: config: '{ "cniVersion": "0.3.0", "type": "host-device", "device": "ens192", "ipam": { "type": "static", "addresses": [ { "address": "192.168.1.23/24" } ] } }' [root@kubemaster tmp]# cat ex.yaml apiVersion: v1 kind: Pod metadata: name: samplepod-left annotations: k8s.v1.cni.cncf.io/networks: nad-host spec: containers:

cucosion commented 1 year ago

I removed the existing multus deployment, cleaned up the /etc/cni/net.d/ and /opt/cni/bin/multus

I switched to an older branch

git checkout release-3.7

I edited the deployment files to add instead of the latest and actual version

[root@kubemaster tmp]# git diff diff --git a/images/multus-daemonset.yml b/images/multus-daemonset.yml index 20c962c0..42482d21 100644 --- a/images/multus-daemonset.yml +++ b/images/multus-daemonset.yml @@ -171,7 +171,7 @@ spec: serviceAccountName: multus containers:

@s1061123 @dougbtv seems that there is an issue with latest version of the multus

handoyo-lyrid commented 1 year ago

I have the same issues on a k3s cluster that I am testing with multus v4.0.1. My generated multus config is at: /var/lib/rancher/k3s/agent/etc/cni/net.d/00-multus.conflist

It seems like the cni is loading the configuration that already has multus configuration when it is on multus-conf-file=auto. And it does recursively. There's probably some race condition somewhere in the code. Here's my multus config in the beginning (the correct one):

{
    "cniVersion": "1.0.0",
    "name": "multus-cni-network",
    "plugins": [ {
        "type": "multus",
        "capabilities": {"portMappings":true},
        "logLevel": "verbose",
        "kubeconfig": "/var/lib/rancher/k3s/agent/etc/cni/net.d/multus.d/multus.kubeconfig",
        "delegates": [
            {"cniVersion":"1.0.0","name":"cbr0","plugins":[{"delegate":{"forceAddress":true,"hairpinMode":true,"isDefaultGateway":true},"type":"flannel"},{"capabilities":{"portMappings":true},"type":"portmap"}]}
        ]
    }]
}

then after a few reboot (wrong one):

{
    "cniVersion": "1.0.0",
    "name": "multus-cni-network",
    "plugins": [
        {
            "capabilities": {
                "portMappings": true
            },
            "delegates": [
                {
                    "cniVersion": "1.0.0",
                    "name": "multus-cni-network",
                    "plugins": [
                        {
                            "capabilities": {
                                "portMappings": true
                            },
                            "delegates": [
                                {
                                    "cniVersion": "1.0.0",
                                    "name": "multus-cni-network",
                                    "plugins": [
                                        {
                                            "capabilities": {
                                                "portMappings": true
                                            },
                                            "delegates": [
                                                {
                                                    "cniVersion": "1.0.0",
                                                    "name": "cbr0",
                                                    "plugins": [
                                                        {
                                                            "delegate": {
                                                                "forceAddress": true,
                                                                "hairpinMode": true,
                                                                "isDefaultGateway": true
                                                            },
                                                            "type": "flannel"
                                                        },
                                                        {
                                                            "capabilities": {
                                                                "portMappings": true
                                                            },
                                                            "type": "portmap"
                                                        }
                                                    ]
                                                }
                                            ],
                                            "kubeconfig": "/var/lib/rancher/k3s/agent/etc/cni/net.d/multus.d/multus.kubeconfig",
                                            "logLevel": "verbose",
                                            "type": "multus"
                                        }
                                    ]
                                }
                            ],
                            "kubeconfig": "/var/lib/rancher/k3s/agent/etc/cni/net.d/multus.d/multus.kubeconfig",
                            "logLevel": "verbose",
                            "type": "multus"
                        }
                    ]
                }
            ],
            "kubeconfig": "/var/lib/rancher/k3s/agent/etc/cni/net.d/multus.d/multus.kubeconfig",
            "logLevel": "verbose",
            "type": "multus"
        }
    ]
}

So my workaround is to delete the multus config:

rm /var/lib/rancher/k3s/agent/etc/cni/net.d/00-multus.conflist

and let it regenerate. I also created a simple systemd startup script to delete that file, and so far, it has been working, but it may have other implications that I didn't know of. Of course, I would like to see this fixed properly, not by my hack.

Hope this helps.

beefcheeks commented 1 year ago

I found this issue after discovering the same nested delegate misconfiguration and wanted to post here as well. I fixed this issue by adding an initContainer that deletes the multus cni configuration on startup.

My patch command looks like:

kubectl patch daemonset -n kube-system kube-multus-ds --type='json' -p='[
  {
    "op": "add",
    "path": "/spec/template/spec/initContainers/-",
    "value": {
      "name": "delete-multus-conf-workaround",
      "image": "alpine",
      "command": ["rm"],
      "args": ["/host/etc/cni/net.d/00-multus.conf"],
      "securityContext": {
        "privileged": true
      },
      "volumeMounts": [
        {
          "name": "cni",
          "mountPath": "/host/etc/cni/net.d"
        }
      ]
    }
  }
]'

@handoyo-lyrid thanks for posting your multus configuration details - I was able to find this issue as a result.

~I also noticed you're using k3s as well, although it appears you're using the default flannel configuration that ships with it (based on the kubeconfig path in your multus conf file). I ended up installing k3s without the flannel backend and installed the vanilla flannel DaemonSet to solve the missing flannel CNI configuration issue (see my github repo multus README for additional details). Also, when you attach a macvlan/secondary host interface to your pods, are you able to route to the coredns/kubedns service? I had to make some specific modifications to the default flannel configuration to get this to work for me.~

~EDIT: I see you're also using CNI version 1.0.0 - would you recommend updating from 0.3.1? I was considering upgrading my multus configuration to use the latest CNI version. Additionally, I'm using the thin plugin, but considering upgrading the thick plugin since the configuration abstraction seems better. If you're using the thick plugin, is it working well for you? Thanks in advance for help!~

EDIT 2: Symlinking the k3s cni configuration and binary locations to the ones multus expects prior to running multus seems to have fixed all of my issues. I wrote a DaemonSet to do this for me - see here. I also updated to v4.0.2 of the thin plugin (no arm64 image for the thick plugin) and updated to CNI version 1.0.0.

hectorgpuy commented 1 year ago

Any updates on this issue? Or any workaround anyone might has discovered? I have run into this issue every noe and then. Any help will be very appreciated!! Thanks!!

timmy59100 commented 1 year ago

Same here with rk2 installed multus

quadeare commented 11 months ago

Same here with RKE2 :cry:

Do you plan to fix it ? Maybe just put workaround by default on multus daemonset (Still ugly but it works !)

Thank you !!

timmy59100 commented 11 months ago

Same here with RKE2 😢

Do you plan to fix it ? Maybe just put workaround by default on multus daemonset (Still ugly but it works !)

Thank you !!

You can revert to rke2 v1.26.6 it is working.

Next version should also get fixed: https://github.com/rancher/rke2/issues/4568

beefcheeks commented 9 months ago

I'm running multus v4.0.2 now (thin plugin) with CNI version 1.0.0 on k3s and have fixed most of my issues by deploying a DaemonSet prior to multus that symlinks /etc/cni/net.d and /opt/cni/bin to the corresponding k3s directories. DaemonSet details can be found here.

This also means a custom flannel deployment is not required and the standard multus install can be used without modifications 🎉.

EDIT: formatting

github-actions[bot] commented 6 months ago

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 7 days.

micw commented 5 months ago

This is a serious issue in the current release and should not be auto closed :-(