CNI doesn't find link after reboot

aeliusrs commented 1 year ago

After rebooting my kubernetes nodes with the AF_XDP CNI, the pod doesn't work anymore.

During multus adding network process, post-reboot, I've got this error :

 [...] error adding container to network "afxdp-network": cmdAdd(): failed to find device: Link not found

This node was running fine with AF_XDP before the reboot, for 5 days.

my configuration is as following. (note that for the moment I'm just doing some test, this is not a production node)

daemon.set

apiVersion: v1
kind: ConfigMap
metadata:
  name: afxdp-dp-config
  namespace: kube-system
data:
  config.json: |
    {
       "logLevel":"debug",
       "logFile":"afxdp-dp.log",
       "pools":[
          {
             "name":"myPool",
             "UdsTimeout":-1,
             "mode":"primary",
             "devices":[
               {
                   "name":"enp2s0"
               }
             ],
             "drivers":[
                {
                 "name":"virtio_net",
                 "ExcludeDevices":[
                    {
                       "name":"enp1s0"
                    }
                  ]
                }
             ]
          }
       ]
    }
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: afxdp-device-plugin
  namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
[...etc, this is the classic one]

My network attach definition

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: afxdp-network
  annotations:
    k8s.v1.cni.cncf.io/resourceName: afxdp/myPool
spec:
  config: '{
      "cniVersion": "0.3.0",
      "type": "afxdp",
      "mode": "primary",
      "logFile": "afxdp-cni.log",
      "logLevel": "debug",
      "ipam": {
        "type": "whereabouts",
        "range": "10.99.99.0/24"
      }
  }'

the deployment of the pod

apiVersion: apps/v1
kind: Deployment
metadata:
  name: afxdp-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: afxdp
  template:
    metadata:
      labels:
        app: afxdp
      annotations:
        k8s.v1.cni.cncf.io/networks: kube-system/afxdp-network
    spec:
      containers:
      - name: afxdptest
        image: travelping/nettools
        imagePullPolicy: IfNotPresent
        command: ["tail", "-f", "/dev/null"]
        resources:
          requests:
            afxdp/myPool: '1'
          limits:
            afxdp/myPool: '1'
        securityContext:
          privileged: true

I've tried to remove multus, whereabout and all the files concerned by AF_XDP. It still doesn't work.

note that libbpf is installed in my machine, and sysctl is configure permanently with the following:

sysctl kernel.unprivileged_bpf_disabled=0
sysctl net.core.bpf_jit_enable=1

Am I missing a point in the use of this CNI ?

thank for you help !

garyloug commented 1 year ago

Hi @aeliusrs,

This is a strange one. The usual responsibilities and timings of the two plugins are as follows:

Device Plugin (DP) discovers suitable netdevs on the node. Provides a list of these allocatable resources to K8s
On pod create, K8s calls the DP very early (pod doesn't yet exist) and informs the DP "netdev X is about to be allocated to a pod"
The DP loads the BPF app onto the allocated netdev and spins up the little UDS server, waiting for the pod to start.
Slightly later (pod is not running, but is starting to be created) K8s will call the CNI. At this stage, other than IPAM, the CNIs only job is to "attach" the netdev to the pod. "Attach" here means to move the netdev from the host network namespace into the pod network namespace.

Your error failed to find device: Link not found is the CNI saying "I cannot find this netdev on the host" and so it is impossible for the CNI to move the netdev to the pod.

What I can't understand is how it got this far. If the netdev is not present on the host, then the DP should have produced an error first.

If you have logging enabled, your log files will be under /var/log/afxdp-k8s-plugins/. A look at the cni log around this error should tell you which netdev it was attempting to attach to the pod. You could then check if this netdev is actually present on your host.

If it's not present, then the only place it can be is in a pod.. Are there any pods left running?

The device plugin logs from around the time of this error would also be interesting. It would be nice to know if the device plugin (re)discovered this netdev after the reboot. The DP should log a list of devices it discovered on startup. It would also be interesting to see if the DP was called to allocate this missing netdev before the pod was created. I'm guessing it was not called.

I'm not sure exactly the behavior of all the various components after a reboot, but to me it seems like there is likely an afxdp pod still running and the missing netdev must still be attached. Then kubernetes/multus/something is attempting to put the node back to the state it was in pre-reboot, and it seems to be calling the cni without first calling on the DP. If the netdev is not on the host then the DP should have thrown an error long before.

If it's possible to attach some of those logs it would be very useful too.

KR, Gary

aeliusrs commented 1 year ago

Hi @garyloug

Thank you for your quick answer and nice explanation, I understand better now, how the full CNI is working.

I do have activated the log files.

It's as follow:

afxdp-dp.log:

ESC[36mINFOESC[0m[2023-04-27 07:31:30] Setting log level: debug
ESC[36mINFOESC[0m[2023-04-27 07:31:30] Switching to debug log format
ESC[36mINFOESC[0m[2023-04-27 07:31:30] [main.go:75] [main] Starting AF_XDP Device Plugin
ESC[36mINFOESC[0m[2023-04-27 07:31:30] [main.go:78] [main] Checking if host meets requirements
ESC[37mDEBUESC[0m[2023-04-27 07:31:30] [main.go:171] [checkHost] Checking kernel version
ESC[37mDEBUESC[0m[2023-04-27 07:31:30] [main.go:197] [checkHost] Kernel version: 6.2.12-300.fc38.x86_64 meets minimum requirements
ESC[37mDEBUESC[0m[2023-04-27 07:31:30] [main.go:200] [checkHost] Checking host for Libbpf
ESC[37mDEBUESC[0m[2023-04-27 07:31:30] [host.go:85] [HasLibbpf] Directory /usr/lib64/ does not exist
ESC[37mDEBUESC[0m[2023-04-27 07:31:30] [main.go:207] [checkHost] Libbpf found on host:
ESC[37mDEBUESC[0m[2023-04-27 07:31:30] [main.go:209] [checkHost]        /usr/lib/libbpf.so.0
ESC[37mDEBUESC[0m[2023-04-27 07:31:30] [main.go:209] [checkHost]        /usr/lib/libbpf.so.0.5.0
ESC[36mINFOESC[0m[2023-04-27 07:31:30] [main.go:88] [main] Host meets requirements
ESC[36mINFOESC[0m[2023-04-27 07:31:30] [main.go:91] [main] Getting device pools
ESC[37mDEBUESC[0m[2023-04-27 07:31:30] [config.go:111] [GetPoolConfigs] Unprivileged BPF is allowed on this host
ESC[37mDEBUESC[0m[2023-04-27 07:31:30] [config.go:130] [GetPoolConfigs] caliafa15bd5e4c is not a physical device, removing from list of host devices
ESC[37mDEBUESC[0m[2023-04-27 07:31:30] [config.go:130] [GetPoolConfigs] calia0f9e9ece0a is not a physical device, removing from list of host devices
ESC[37mDEBUESC[0m[2023-04-27 07:31:30] [config.go:130] [GetPoolConfigs] caliee83c91241b is not a physical device, removing from list of host devices
ESC[37mDEBUESC[0m[2023-04-27 07:31:30] [config.go:130] [GetPoolConfigs] calic812c63d2df is not a physical device, removing from list of host devices
ESC[37mDEBUESC[0m[2023-04-27 07:31:30] [config.go:130] [GetPoolConfigs] tunl0 is not a physical device, removing from list of host devices
ESC[37mDEBUESC[0m[2023-04-27 07:31:30] [config.go:130] [GetPoolConfigs] cali017ff5624a8 is not a physical device, removing from list of host devices
ESC[37mDEBUESC[0m[2023-04-27 07:31:30] [config.go:130] [GetPoolConfigs] calia70de7e4bb0 is not a physical device, removing from list of host devices
ESC[37mDEBUESC[0m[2023-04-27 07:31:30] [config.go:130] [GetPoolConfigs] lo is not a physical device, removing from list of host devices
ESC[37mDEBUESC[0m[2023-04-27 07:31:30] [config.go:145] [GetPoolConfigs] Host devices:
{
  "enp1s0": {},
  "enp2s0": {}
}
ESC[36mINFOESC[0m[2023-04-27 07:31:30] [config.go:149] [GetPoolConfigs] Processing Pool: myPool
ESC[37mDEBUESC[0m[2023-04-27 07:31:30] [config.go:160] [GetPoolConfigs] UDS timeout is disabled: 0 seconds
ESC[36mINFOESC[0m[2023-04-27 07:31:30] [config.go:264] [getDeviceListOfDriverType] enp1s0 added to pool
ESC[37mDEBUESC[0m[2023-04-27 07:31:30] [config.go:328] [validateDevice] enp2s0 is already in this pool
ESC[37mDEBUESC[0m[2023-04-27 07:31:30] [config.go:273] [getDeviceListOfDriverType] Exit discovery.
ESC[37mDEBUESC[0m[2023-04-27 07:31:30] [poolManager.go:327] [startGRPC] afxdp/myPool started serving on /var/lib/kubelet/device-plugins/afxdp-myPool.sock
ESC[36mINFOESC[0m[2023-04-27 07:31:30] [poolManager.go:88] [Init] Pool afxdp/myPool started serving
ESC[37mDEBUESC[0m[2023-04-27 07:31:30] [poolManager.go:123] [ListAndWatch] Pool afxdp/myPool ListAndWatch started
ESC[36mINFOESC[0m[2023-04-27 07:31:30] [poolManager.go:93] [Init] Pool afxdp/myPool registered with Kubelet
ESC[36mINFOESC[0m[2023-04-27 07:34:27] [main.go:115] [main] Received signal "terminated"
ESC[36mINFOESC[0m[2023-04-27 07:34:27] [main.go:117] [main] Terminating myPool
ESC[36mINFOESC[0m[2023-04-27 07:34:27] [poolManager.go:110] [Terminate] afxdp/myPool terminated
ESC[36mINFOESC[0m[2023-04-27 07:35:45] Setting log level: debug
ESC[36mINFOESC[0m[2023-04-27 07:35:45] Switching to debug log format
ESC[36mINFOESC[0m[2023-04-27 07:35:45] [main.go:75] [main] Starting AF_XDP Device Plugin
ESC[36mINFOESC[0m[2023-04-27 07:35:45] [main.go:78] [main] Checking if host meets requirements

afxdp-cni.log:

ESC[37mDEBUESC[0m[2023-04-27 15:42:40] [cni.go:133] [loadConf] loadConf(): Mode is set to: primary
ESC[36mINFOESC[0m[2023-04-27 15:42:40] [cni.go:296] [CmdDel] cmdDel(): getting container network namespace
ESC[36mINFOESC[0m[2023-04-27 15:42:40] [cni.go:306] [CmdDel] cmdDel(): getting default network namespace
ESC[36mINFOESC[0m[2023-04-27 15:42:40] [cni.go:316] [CmdDel] cmdDel(): executing within container network namespace:
ESC[36mINFOESC[0m[2023-04-27 15:42:40] [cni.go:319] [func1] cmdDel(): getting device from name
ESC[31mERROESC[0m[2023-04-27 15:42:40] [cni.go:323] [func1] cmdDel(): failed to find device "enp2s0" in containerNS: Link not found
ESC[37mDEBUESC[0m[2023-04-27 15:42:41] [cni.go:133] [loadConf] loadConf(): Mode is set to: primary
ESC[37mDEBUESC[0m[2023-04-27 15:42:41] [cni.go:156] [CmdAdd] cmdAdd(): loaded config: &{NetConf:{CNIVersion:0.3.0 Name:afxdp-network Type:afxdp Capabilities:map[] IPAM:{Type:whereabouts} DNS:{Nameservers:[] Domain: Search:[] Options:[]} RawPrevResult:map[] PrevResult:<nil>} Device:enp2s0 Mode:primary SkipUnloadBpf:false Queues: LogFile:afxdp-cni.log LogLevel:debug}
ESC[36mINFOESC[0m[2023-04-27 15:42:41] [cni.go:157] [CmdAdd] cmdAdd(): getting container network namespace
ESC[36mINFOESC[0m[2023-04-27 15:42:41] [cni.go:167] [CmdAdd] cmdAdd(): getting device from name
ESC[36mINFOESC[0m[2023-04-27 15:42:41] [cni.go:176] [CmdAdd] cmdAdd(): getting default network namespace
ESC[36mINFOESC[0m[2023-04-27 15:42:41] [cni.go:186] [CmdAdd] cmdAdd(): checking if IPAM is required
ESC[36mINFOESC[0m[2023-04-27 15:42:41] [cni.go:415] [getIPAM] configureIPAM(): running IPAM plugin: whereabouts
ESC[36mINFOESC[0m[2023-04-27 15:42:41] [cni.go:434] [getIPAM] configureIPAM(): converting IPAM result into current result type
ESC[36mINFOESC[0m[2023-04-27 15:42:41] [cni.go:442] [getIPAM] configureIPAM(): checking IPAM plugin returned IP
ESC[37mDEBUESC[0m[2023-04-27 15:42:41] [cni.go:456] [getIPAM] configureIPAM(): setting IPConfig interface
ESC[36mINFOESC[0m[2023-04-27 15:42:41] [cni.go:240] [CmdAdd] cmdAdd(): moving device from default to container network namespace
ESC[36mINFOESC[0m[2023-04-27 15:42:41] [cni.go:248] [CmdAdd] cmdAdd(): executing within container network namespace:
ESC[36mINFOESC[0m[2023-04-27 15:42:41] [cni.go:251] [func1] cmdAdd(): set device to UP state
ESC[36mINFOESC[0m[2023-04-27 15:42:41] [cni.go:464] [setIPAM] configureIPAM(): executing within host netns
ESC[36mINFOESC[0m[2023-04-27 15:42:41] [cni.go:467] [func1] configureIPAM(): setting device IP
ESC[37mDEBUESC[0m[2023-04-27 15:42:43] [cni.go:133] [loadConf] loadConf(): Mode is set to: primary
ESC[36mINFOESC[0m[2023-04-27 15:42:43] [cni.go:296] [CmdDel] cmdDel(): getting container network namespace
ESC[31mERROESC[0m[2023-04-27 15:42:43] [cni.go:300] [CmdDel] cmdDel(): failed to open container netns "": failed to Statfs "": no such file or directory
ESC[37mDEBUESC[0m[2023-04-27 15:42:43] [cni.go:133] [loadConf] loadConf(): Mode is set to: primary
ESC[36mINFOESC[0m[2023-04-27 15:42:43] [cni.go:296] [CmdDel] cmdDel(): getting container network namespace
ESC[31mERROESC[0m[2023-04-27 15:42:43] [cni.go:300] [CmdDel] cmdDel(): failed to open container netns "": failed to Statfs "": no such file or directory
ESC[37mDEBUESC[0m[2023-04-27 15:43:53] [cni.go:133] [loadConf] loadConf(): Mode is set to: primary
ESC[36mINFOESC[0m[2023-04-27 15:43:53] [cni.go:296] [CmdDel] cmdDel(): getting container network namespace
ESC[36mINFOESC[0m[2023-04-27 15:43:53] [cni.go:306] [CmdDel] cmdDel(): getting default network namespace
ESC[36mINFOESC[0m[2023-04-27 15:43:53] [cni.go:316] [CmdDel] cmdDel(): executing within container network namespace:
ESC[36mINFOESC[0m[2023-04-27 15:43:53] [cni.go:319] [func1] cmdDel(): getting device from name
ESC[36mINFOESC[0m[2023-04-27 15:43:53] [cni.go:328] [func1] cmdDel(): moving device from container to default network namespace
ESC[36mINFOESC[0m[2023-04-27 15:43:53] [cni.go:341] [CmdDel] cmdDel(): cleaning IPAM config on device
ESC[36mINFOESC[0m[2023-04-27 15:43:53] [cni.go:349] [CmdDel] cmdDel(): removing BPF program from device
ESC[36mINFOESC[0m[2023-04-27 15:43:53] [bpfWrapper.go:104] [Infof] Clean_bpf: disovering if_index for interface enp2s0
ESC[36mINFOESC[0m[2023-04-27 15:43:53] [bpfWrapper.go:104] [Infof] Clean_bpf: if_index for interface enp2s0 is 3
ESC[36mINFOESC[0m[2023-04-27 15:43:53] [bpfWrapper.go:104] [Infof] Clean_bpf: starting removal of xdp program on interface enp2s0 (3)
ESC[37mDEBUESC[0m[2023-04-27 15:43:54] [cni.go:133] [loadConf] loadConf(): Mode is set to: primary
ESC[36mINFOESC[0m[2023-04-27 15:43:54] [cni.go:296] [CmdDel] cmdDel(): getting container network namespace
ESC[36mINFOESC[0m[2023-04-27 15:43:54] [cni.go:306] [CmdDel] cmdDel(): getting default network namespace
ESC[36mINFOESC[0m[2023-04-27 15:43:54] [cni.go:316] [CmdDel] cmdDel(): executing within container network namespace:

kubectl get pod -A :

default       afxdp-deployment-5655999875-wbx6t         0/1     ContainerCreating   0                3d18h
kube-system   calico-kube-controllers-57b57c56f-8m9pz   1/1     Running             14 (4m53s ago)   4d19h
kube-system   calico-node-sfgqv                         0/1     Running             8 (4m53s ago)    4d19h
kube-system   coredns-59b4f5bbd5-8xn9r                  1/1     Running             8 (4m53s ago)    4d19h
kube-system   helm-install-traefik-crd-ztlgn            0/1     Completed           0                4d19h
kube-system   helm-install-traefik-mtwbz                0/1     Completed           1                4d19h
kube-system   kube-afxdp-device-plugin-gpj8t            1/1     Running             1 (4m53s ago)    3d18h
kube-system   kube-multus-ds-z5lnx                      1/1     Running             1 (4m53s ago)    3d18h
kube-system   local-path-provisioner-76d776f6f9-lscsq   1/1     Running             14 (4m53s ago)   4d19h
kube-system   metrics-server-7b67f64457-vlv66           0/1     Running             14 (4m53s ago)   4d19h
kube-system   svclb-traefik-29d44bad-w7qbv              2/2     Running             16 (4m53s ago)   4d19h
kube-system   traefik-56b8c5fb5c-fvbzd                  1/1     Running             8 (4m53s ago)    4d19h
kube-system   whereabouts-7gwjl                         1/1     Running             1 (4m53s ago)    3d18h

ip -br a:

lo               UNKNOWN        127.0.0.1/8 ::1/128
enp1s0           UP             192.168.122.114/24 fe80::5054:ff:fe2c:31ea/64
enp2s0           UP             192.168.122.139/24 fe80::5054:ff:fe1d:b62/64
tunl0@NONE       UNKNOWN        172.16.124.192/32
calic812c63d2df@if4 UP             fe80::ecee:eeff:feee:eeee/64
calia0f9e9ece0a@if4 UP             fe80::ecee:eeff:feee:eeee/64
caliafa15bd5e4c@if4 UP             fe80::ecee:eeff:feee:eeee/64
calia70de7e4bb0@if4 UP             fe80::ecee:eeff:feee:eeee/64
cali017ff5624a8@if4 UP             fe80::ecee:eeff:feee:eeee/64
caliee83c91241b@if4 UP             fe80::ecee:eeff:feee:eeee/64

ip link show enp2s0:

3: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 52:54:00:1d:0b:62 brd ff:ff:ff:ff:ff:ff
    prog/xdp id 113 name  tag 03b13f331978c78c jited

The interface seems to be not recognized in the pool, but I could not find the reason.

here is my daemonset configuration too:

apiVersion: v1
kind: ConfigMap
metadata:
  name: afxdp-dp-config
  namespace: kube-system
data:
  config.json: |
    {
       "logLevel":"debug",
       "logFile":"afxdp-dp.log",
       "pools":[
          {
             "name":"myPool",
             "UdsTimeout":-1,
             "mode":"primary",
             "devices":[
               {
                   "name":"enp2s0"
               }
             ],
             "drivers":[
                {
                 "name":"virtio_net",
                 "ExcludeDevices":[
                    {
                       "name":"enp1s0"
                    }
                  ]
                }
             ]
          }
       ]
    }

Thank you again for your help.

Aelius

garyloug commented 1 year ago

Hi @aeliusrs,

Thanks for the logs, they're helpful. Still not exactly sure what's going on, but just a few things I can point out:

The device plugin and cni logs are about 8hrs apart. I'm not sure if much happened in the meantime, or if one or both of these logs are before or after the restart. A useful thing to search for in the device plugin logs is this line: Pool afxdp/myPool registered with Kubelet You'll see this line only once (per configured pool) each time the DP starts up. So it's useful for detecting when the plugin (re)started. The DP logs after the node restarted would be the most interesting, to see if it picked up the missing netdev.
Your cni logs are a little different from your original post. Originally you had the error: cmdAdd(): failed to find device: Link not found Now I see: cmdDel(): failed to find device "enp2s0" in containerNS: Link not found Every cni has 2 functions, cmdAdd and cmdDel. Add is called on pod create, Del is called on pod delete. This time it is saying it cannot find the netdev within the pod, while the pod is deleting.

This is behavior above strange, however:

ESC[37mDEBUESC[0m[2023-04-27 15:42:41] [cni.go:156] [CmdAdd] cmdAdd(): loaded config: &{NetConf:{CNIVersion:0.3.0 Name:afxdp-network Type:afxdp Capabilities:map[] IPAM:{Type:whereabouts} DNS:{Nameservers:[] Domain: Search:[] Options:[]} RawPrevResult:map[] PrevResult:<nil>} Device:enp2s0 Mode:primary SkipUnloadBpf:false Queues: LogFile:afxdp-cni.log LogLevel:debug}
ESC[36mINFOESC[0m[2023-04-27 15:42:41] [cni.go:157] [CmdAdd] cmdAdd(): getting container network namespace
ESC[36mINFOESC[0m[2023-04-27 15:42:41] [cni.go:167] [CmdAdd] cmdAdd(): getting device from name
ESC[36mINFOESC[0m[2023-04-27 15:42:41] [cni.go:176] [CmdAdd] cmdAdd(): getting default network namespace
ESC[36mINFOESC[0m[2023-04-27 15:42:41] [cni.go:186] [CmdAdd] cmdAdd(): checking if IPAM is required
ESC[36mINFOESC[0m[2023-04-27 15:42:41] [cni.go:415] [getIPAM] configureIPAM(): running IPAM plugin: whereabouts

These cmdAdd logs immediately after, look good! getting device from name was where the error in your original post occurred. I can see this part is working ok now.

It then goes on to some more cmdDel errors:

ESC[36mINFOESC[0m[2023-04-27 15:42:43] [cni.go:296] [CmdDel] cmdDel(): getting container network namespace
ESC[31mERROESC[0m[2023-04-27 15:42:43] [cni.go:300] [CmdDel] cmdDel(): failed to open container netns "": failed to Statfs "": no such file or directory

...but these errors are expected with a pod stuck on "container creating" state. K8s attempts to delete the stuck pod, calls cni cmdDel as it does so, but the cni reports errors because the container does not exist, there is nothing to cleanup. So this makes sense.

It would be interesting to run a kubectl describe pod afxdp-deployment-5655999875-wbx6t on the hanging pod to see what errors it is reporting now. It doesn't appear to same error as your original post. From the logs you included, it successfully found the netdev on the host.

Your config looks good. The only thing we've not tested ourselves in the whereabouts IPAM plugin. Myself or @patrickog11 will try to test with this one soon.

KR, Gary

aeliusrs commented 1 year ago

Hello Gary,

Sorry, indeed I forget the oc describe and I may have copied the wrong log, I was in a rush.

So the fresh log of today.

the afxdp-dp.log

ESC[36mINFOESC[0m[2023-05-03 01:03:29] Setting log level: debug
ESC[36mINFOESC[0m[2023-05-03 01:03:29] Switching to debug log format
ESC[36mINFOESC[0m[2023-05-03 01:03:29] [main.go:75] [main] Starting AF_XDP Device Plugin
ESC[36mINFOESC[0m[2023-05-03 01:03:29] [main.go:78] [main] Checking if host meets requirements
ESC[37mDEBUESC[0m[2023-05-03 01:03:29] [main.go:171] [checkHost] Checking kernel version
ESC[37mDEBUESC[0m[2023-05-03 01:03:29] [main.go:197] [checkHost] Kernel version: 6.2.12-300.fc38.x86_64 meets minimum requirements
ESC[37mDEBUESC[0m[2023-05-03 01:03:29] [main.go:200] [checkHost] Checking host for Libbpf
ESC[37mDEBUESC[0m[2023-05-03 01:03:29] [host.go:85] [HasLibbpf] Directory /usr/lib64/ does not exist
ESC[37mDEBUESC[0m[2023-05-03 01:03:29] [main.go:207] [checkHost] Libbpf found on host:
ESC[37mDEBUESC[0m[2023-05-03 01:03:29] [main.go:209] [checkHost]        /usr/lib/libbpf.so.0
ESC[37mDEBUESC[0m[2023-05-03 01:03:29] [main.go:209] [checkHost]        /usr/lib/libbpf.so.0.5.0
ESC[36mINFOESC[0m[2023-05-03 01:03:29] [main.go:88] [main] Host meets requirements
ESC[36mINFOESC[0m[2023-05-03 01:03:29] [main.go:91] [main] Getting device pools
ESC[37mDEBUESC[0m[2023-05-03 01:03:29] [config.go:111] [GetPoolConfigs] Unprivileged BPF is allowed on this host
ESC[37mDEBUESC[0m[2023-05-03 01:03:29] [config.go:130] [GetPoolConfigs] lo is not a physical device, removing from list of host devices
ESC[37mDEBUESC[0m[2023-05-03 01:03:29] [config.go:145] [GetPoolConfigs] Host devices:
{
  "enp1s0": {},
  "enp2s0": {}
}
ESC[36mINFOESC[0m[2023-05-03 01:03:29] [config.go:149] [GetPoolConfigs] Processing Pool: myPool
ESC[37mDEBUESC[0m[2023-05-03 01:03:29] [config.go:160] [GetPoolConfigs] UDS timeout is disabled: 0 seconds
ESC[36mINFOESC[0m[2023-05-03 01:03:29] [config.go:264] [getDeviceListOfDriverType] enp2s0 added to pool
ESC[37mDEBUESC[0m[2023-05-03 01:03:29] [config.go:332] [validateDevice] enp1s0 is an excluded device for virtio_net driver
ESC[37mDEBUESC[0m[2023-05-03 01:03:29] [config.go:273] [getDeviceListOfDriverType] Exit discovery.
ESC[37mDEBUESC[0m[2023-05-03 01:03:29] [poolManager.go:327] [startGRPC] afxdp/myPool started serving on /var/lib/kubelet/device-plugins/afxdp-myPool.sock
ESC[36mINFOESC[0m[2023-05-03 01:03:29] [poolManager.go:88] [Init] Pool afxdp/myPool started serving
ESC[37mDEBUESC[0m[2023-05-03 01:03:29] [poolManager.go:123] [ListAndWatch] Pool afxdp/myPool ListAndWatch started
ESC[36mINFOESC[0m[2023-05-03 01:03:29] [poolManager.go:93] [Init] Pool afxdp/myPool registered with Kubelet

the oc descibre of the pod, where the initial error is :

  Normal   AddedInterface          2m2s                  multus             Add eth0 [172.16.124.246/32] from k8s-pod-network
  Normal   AddedInterface          2m2s                  multus             Add net1 [10.99.99.1/24] from kube-system/afxdp-network
  Normal   AddedInterface          2m2s                  multus             Add eth0 [172.16.124.246/32] from multus-cni-network
  Warning  FailedCreatePodSandBox  119s                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "64ff0a4f5cd4612af3f16056299afb9d3dd42847221c27d0d766012a772b6e05": plugin type="multus" name="multus-cni-network" failed (add): [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:afxdp-network]: error adding container to network "afxdp-network": cmdAdd(): failed to find device: Link not found
  Normal   AddedInterface          117s                  multus             Add eth0 [172.16.124.247/32] from k8s-pod-network
  Normal   AddedInterface          117s                  multus             Add net1 [10.99.99.1/24] from kube-system/afxdp-network
  Normal   AddedInterface          117s                  multus             Add eth0 [172.16.124.247/32] from multus-cni-network
  Warning  FailedCreatePodSandBox  115s                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d955579d38f00dfae556e3d183fc1d56a5ec34f66ccc18068b55481f9f63fd57": plugin type="multus" name="multus-cni-network" failed (add): [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:afxdp-network]: error adding container to network "afxdp-network": cmdAdd(): failed to find device: Link not found
  Normal   AddedInterface          114s                  multus             Add eth0 [172.16.124.245/32] from k8s-pod-network
  Normal   AddedInterface          114s                  multus             Add net1 [10.99.99.1/24] from kube-system/afxdp-network
  Normal   AddedInterface          113s                  multus             Add eth0 [172.16.124.245/32] from multus-cni-network
  Warning  FailedCreatePodSandBox  111s                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d915253cae2a6eba591b6fd870ac8fd7468d742025ee3a88a85b052a6733c857": plugin type="multus" name="multus-cni-network" failed (add): [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:afxdp-network]: error adding container to network "afxdp-network": cmdAdd(): failed to find device: Link not found
  Normal   AddedInterface          110s                  multus             Add eth0 [172.16.124.250/32] from k8s-pod-network
  Normal   AddedInterface          110s                  multus             Add net1 [10.99.99.1/24] from kube-system/afxdp-network
  Normal   AddedInterface          110s                  multus             Add eth0 [172.16.124.250/32] from multus-cni-network
  Warning  FailedCreatePodSandBox  107s                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "59a80660977d3cde43df30714bcd9fd4ded4acdb7db5c818474ea67458b01e8e": plugin type="multus" name="multus-cni-network" failed (add): [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:afxdp-network]: error adding container to network "afxdp-network": cmdAdd(): failed to find device: Link not found
  Normal   AddedInterface          106s                  multus             Add eth0 [172.16.124.251/32] from k8s-pod-network
  Normal   AddedInterface          106s                  multus             Add net1 [10.99.99.1/24] from kube-system/afxdp-network
  Normal   AddedInterface          106s                  multus             Add eth0 [172.16.124.251/32] from multus-cni-network
  Warning  FailedCreatePodSandBox  103s                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e801ecc212d7e441b39d5cb8bc6fb7e5556292b54abf3d0eef1518c0ff596d67": plugin type="multus" name="multus-cni-network" failed (add): [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:afxdp-network]: error adding container to network "afxdp-network": cmdAdd(): failed to find device: Link not found
  Normal   AddedInterface          102s                  multus             Add eth0 [172.16.124.253/32] from k8s-pod-network
  Normal   AddedInterface          102s                  multus             Add net1 [10.99.99.1/24] from kube-system/afxdp-network
  Normal   AddedInterface          102s                  multus             Add eth0 [172.16.124.253/32] from multus-cni-network
  Normal   AddedInterface          98s                   multus             Add eth0 [172.16.124.255/32] from k8s-pod-network
  Normal   AddedInterface          97s                   multus             Add net1 [10.99.99.1/24] from kube-system/afxdp-network
  Normal   AddedInterface          97s                   multus             Add eth0 [172.16.124.255/32] from multus-cni-network
  Normal   SandboxChanged          94s (x12 over 2m53s)  kubelet            Pod sandbox changed, it will be killed and re-created.
  Normal   AddedInterface          94s                   multus             Add eth0 [172.16.124.254/32] from k8s-pod-network
  Normal   AddedInterface          94s                   multus             Add net1 [10.99.99.1/24] from kube-system/afxdp-network
  Normal   AddedInterface          94s                   multus             Add eth0 [172.16.124.254/32] from multus-cni-network
  Warning  FailedCreatePodSandBox  91s (x3 over 99s)     kubelet            (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b251ff5e38662af2718fd7d9e347f00c88ec0ee1c831236289fce5554e8e6b60": plugin type="multus" name="multus-cni-network" failed (add): [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:multus-cni-network]: error adding container to network "multus-cni-network": [default/afxdp-deployment-5655999875-cwk2s/dfae3db3-41f3-4e55-8468-ccf778496e1e:afxdp-network]: error adding container to network "afxdp-network": cmdAdd(): failed to find device: Link not found
  Normal   AddedInterface          90s                   multus             Add eth0 [172.16.124.214/32] from k8s-pod-network
  Normal   AddedInterface          90s                   multus             Add net1 [10.99.99.1/24] from kube-system/afxdp-network
  Normal   AddedInterface          90s                   multus             Add eth0 [172.16.124.214/32] from multus-cni-network
  Normal   AddedInterface          85s                   multus             Add eth0 [172.16.124.252/32] from k8s-pod-network
  Normal   AddedInterface          85s                   multus             Add net1 [10.99.99.1/24] from kube-system/afxdp-network
  Normal   AddedInterface          85s                   multus             Add eth0 [172.16.124.252/32] from multus-cni-network
  Normal   AddedInterface          82s                   multus             Add eth0 [172.16.124.213/32] from k8s-pod-network
  Normal   AddedInterface          82s                   multus             Add net1 [10.99.99.1/24] from kube-system/afxdp-network
  Normal   AddedInterface          82s                   multus             Add eth0 [172.16.124.213/32] from multus-cni-network
  Normal   AddedInterface          78s                   multus             Add eth0 [172.16.124.211/32] from k8s-pod-network
  Normal   AddedInterface          78s                   multus             Add net1 [10.99.99.1/24] from kube-system/afxdp-network
  Normal   AddedInterface          78s                   multus             Add eth0 [172.16.124.211/32] from multus-cni-network
  Normal   AddedInterface          74s                   multus             Add eth0 [172.16.124.208/32] from k8s-pod-network
  Normal   AddedInterface          73s                   multus             Add net1 [10.99.99.1/24] from kube-system/afxdp-network
  Normal   AddedInterface          73s                   multus             Add eth0 [172.16.124.208/32] from multus-cni-network
  Normal   AddedInterface          70s                   multus             Add eth0 [172.16.124.209/32] from k8s-pod-network
  Normal   AddedInterface          69s                   multus             Add net1 [10.99.99.1/24] from kube-system/afxdp-network
  Normal   AddedInterface          69s                   multus             Add eth0 [172.16.124.209/32] from multus-cni-network
  Normal   AddedInterface          66s                   multus             Add eth0 [172.16.124.210/32] from k8s-pod-network
  Normal   AddedInterface          65s                   multus             Add net1 [10.99.99.1/24] from kube-system/afxdp-network
  Normal   AddedInterface          65s                   multus             Add eth0 [172.16.124.210/32] from multus-cni-network
  Normal   AddedInterface          62s                   multus             Add eth0 [172.16.124.217/32] from k8s-pod-network
  Normal   AddedInterface          61s                   multus             Add net1 [10.99.99.1/24] from kube-system/afxdp-network
  Normal   AddedInterface          61s                   multus             Add eth0 [172.16.124.217/32] from multus-cni-network
  Normal   AddedInterface          57s                   multus             Add eth0 [172.16.124.218/32] from k8s-pod-network
  Normal   AddedInterface          57s                   multus             Add net1 [10.99.99.1/24] from kube-system/afxdp-network
  Normal   AddedInterface          57s                   multus             Add eth0 [172.16.124.218/32] from multus-cni-network
  Normal   AddedInterface          53s                   multus             Add eth0 [172.16.124.219/32] from k8s-pod-network
  Normal   AddedInterface          53s                   multus             Add net1 [10.99.99.1/24] from kube-system/afxdp-network
  Normal   AddedInterface          53s                   multus             Add eth0 [172.16.124.219/32] from multus-cni-network
  Normal   AddedInterface          49s                   multus             Add eth0 [172.16.124.220/32] from k8s-pod-network
  Normal   AddedInterface          49s                   multus             Add net1 [10.99.99.1/24] from kube-system/afxdp-network
  Normal   AddedInterface          49s                   multus             Add eth0 [172.16.124.220/32] from multus-cni-network
  Normal   AddedInterface          45s                   multus             Add eth0 [172.16.124.193/32] from k8s-pod-network
  Normal   AddedInterface          45s                   multus             Add net1 [10.99.99.1/24] from kube-system/afxdp-network
  Normal   AddedInterface          45s                   multus             Add eth0 [172.16.124.193/32] from multus-cni-network
  Normal   AddedInterface          42s                   multus             Add eth0 [172.16.124.201/32] from k8s-pod-network
  Normal   AddedInterface          41s                   multus             Add net1 [10.99.99.1/24] from kube-system/afxdp-network
  Normal   AddedInterface          41s                   multus             Add eth0 [172.16.124.201/32] from multus-cni-network
  Normal   AddedInterface          37s                   multus             Add eth0 [172.16.124.203/32] from k8s-pod-network
  Normal   AddedInterface          37s                   multus             Add net1 [10.99.99.1/24] from kube-system/afxdp-network
  Normal   AddedInterface          37s                   multus             Add eth0 [172.16.124.203/32] from multus-cni-network
  Normal   AddedInterface          33s                   multus             Add eth0 [172.16.124.202/32] from k8s-pod-network
  Normal   AddedInterface          33s                   multus             Add net1 [10.99.99.1/24] from kube-system/afxdp-network
  Normal   AddedInterface          33s                   multus             Add eth0 [172.16.124.202/32] from multus-cni-network
  Normal   AddedInterface          29s                   multus             Add eth0 [172.16.124.204/32] from k8s-pod-network
  Normal   AddedInterface          28s                   multus             Add net1 [10.99.99.1/24] from kube-system/afxdp-network
  Normal   AddedInterface          28s                   multus             Add eth0 [172.16.124.204/32] from multus-cni-network
  Normal   AddedInterface          24s                   multus             Add eth0 [172.16.124.197/32] from k8s-pod-network
  Normal   AddedInterface          24s                   multus             Add net1 [10.99.99.1/24] from kube-system/afxdp-network
  Normal   AddedInterface          24s                   multus             Add eth0 [172.16.124.197/32] from multus-cni-network
  Normal   AddedInterface          20s                   multus             Add eth0 [172.16.124.225/32] from k8s-pod-network
  Normal   AddedInterface          20s                   multus             Add net1 [10.99.99.1/24] from kube-system/afxdp-network
  Normal   AddedInterface          20s                   multus             Add eth0 [172.16.124.225/32] from multus-cni-network
  Normal   AddedInterface          16s                   multus             Add eth0 [172.16.124.227/32] from k8s-pod-network
  Normal   AddedInterface          16s                   multus             Add net1 [10.99.99.1/24] from kube-system/afxdp-network
  Normal   AddedInterface          16s                   multus             Add eth0 [172.16.124.227/32] from multus-cni-network
  Normal   AddedInterface          12s                   multus             Add eth0 [172.16.124.240/32] from k8s-pod-network
  Normal   AddedInterface          12s                   multus             Add net1 [10.99.99.1/24] from kube-system/afxdp-network
  Normal   AddedInterface          12s                   multus             Add eth0 [172.16.124.240/32] from multus-cni-network
  Normal   AddedInterface          9s                    multus             Add eth0 [172.16.124.226/32] from k8s-pod-network
  Normal   AddedInterface          8s                    multus             Add net1 [10.99.99.1/24] from kube-system/afxdp-network
  Normal   AddedInterface          8s                    multus             Add eth0 [172.16.124.226/32] from multus-cni-network
  Normal   AddedInterface          4s                    multus             Add eth0 [172.16.124.241/32] from k8s-pod-network
  Normal   AddedInterface          4s                    multus             Add net1 [10.99.99.1/24] from kube-system/afxdp-network
  Normal   AddedInterface          4s                    multus             Add eth0 [172.16.124.241/32] from multus-cni-network

you can see that in the end it looping on adding the interface

the afxdp-cni.log show also the error, but with the looping it generate a lot of noise, that why yesterday i was not able to find it :

DEBU[2023-05-03 09:10:18] [cni.go:133] [loadConf] loadConf(): Mode is set to: primary
INFO[2023-05-03 09:10:18] [cni.go:296] [CmdDel] cmdDel(): getting container network namespace
ERRO[2023-05-03 09:10:18] [cni.go:300] [CmdDel] cmdDel(): failed to open container netns "": failed to Statfs "": no such file or directory
DEBU[2023-05-03 09:10:18] [cni.go:133] [loadConf] loadConf(): Mode is set to: primary
INFO[2023-05-03 09:10:18] [cni.go:296] [CmdDel] cmdDel(): getting container network namespace
ERRO[2023-05-03 09:10:18] [cni.go:300] [CmdDel] cmdDel(): failed to open container netns "": failed to Statfs "": no such file or directory
DEBU[2023-05-03 09:10:18] [cni.go:133] [loadConf] loadConf(): Mode is set to: primary
INFO[2023-05-03 09:10:18] [cni.go:296] [CmdDel] cmdDel(): getting container network namespace
ERRO[2023-05-03 09:10:18] [cni.go:300] [CmdDel] cmdDel(): failed to open container netns "": failed to Statfs "": no such file or directory
DEBU[2023-05-03 09:10:18] [cni.go:133] [loadConf] loadConf(): Mode is set to: primary
DEBU[2023-05-03 09:10:18] [cni.go:156] [CmdAdd] cmdAdd(): loaded config: &{NetConf:{CNIVersion:0.3.0 Name:afxdp-network Type:afxdp Capabilities:map[] IPAM:{Type:whereabouts} DNS:{Nameservers:[] Domain: Search:[] Options:[]} RawPrevResult:map[] PrevResult:<nil>} Device:enp2s0 Mode:primary SkipUnloadBpf:false Queues: LogFile:afxdp-cni.log LogLevel:debug}
INFO[2023-05-03 09:10:18] [cni.go:157] [CmdAdd] cmdAdd(): getting container network namespace
INFO[2023-05-03 09:10:18] [cni.go:167] [CmdAdd] cmdAdd(): getting device from name
INFO[2023-05-03 09:10:18] [cni.go:176] [CmdAdd] cmdAdd(): getting default network namespace
INFO[2023-05-03 09:10:18] [cni.go:186] [CmdAdd] cmdAdd(): checking if IPAM is required
INFO[2023-05-03 09:10:18] [cni.go:415] [getIPAM] configureIPAM(): running IPAM plugin: whereabouts
INFO[2023-05-03 09:10:18] [cni.go:434] [getIPAM] configureIPAM(): converting IPAM result into current result type
INFO[2023-05-03 09:10:18] [cni.go:442] [getIPAM] configureIPAM(): checking IPAM plugin returned IP
DEBU[2023-05-03 09:10:18] [cni.go:456] [getIPAM] configureIPAM(): setting IPConfig interface
INFO[2023-05-03 09:10:18] [cni.go:240] [CmdAdd] cmdAdd(): moving device from default to container network namespace
INFO[2023-05-03 09:10:18] [cni.go:248] [CmdAdd] cmdAdd(): executing within container network namespace:
INFO[2023-05-03 09:10:18] [cni.go:251] [func1] cmdAdd(): set device to UP state
INFO[2023-05-03 09:10:18] [cni.go:464] [setIPAM] configureIPAM(): executing within host netns
INFO[2023-05-03 09:10:18] [cni.go:467] [func1] configureIPAM(): setting device IP
DEBU[2023-05-03 09:10:18] [cni.go:133] [loadConf] loadConf(): Mode is set to: primary
DEBU[2023-05-03 09:10:18] [cni.go:156] [CmdAdd] cmdAdd(): loaded config: &{NetConf:{CNIVersion:0.3.0 Name:afxdp-network Type:afxdp Capabilities:map[] IPAM:{Type:whereabouts} DNS:{Nameservers:[] Domain: Search:[] Options:[]} RawPrevResult:map[] PrevResult:<nil>} Device:enp2s0 Mode:primary SkipUnloadBpf:false Queues: LogFile:afxdp-cni.log LogLevel:debug}
INFO[2023-05-03 09:10:18] [cni.go:157] [CmdAdd] cmdAdd(): getting container network namespace
INFO[2023-05-03 09:10:18] [cni.go:167] [CmdAdd] cmdAdd(): getting device from name
ERRO[2023-05-03 09:10:18] [cni.go:171] [CmdAdd] cmdAdd(): failed to find device: Link not found
DEBU[2023-05-03 09:10:18] [cni.go:133] [loadConf] loadConf(): Mode is set to: primary
INFO[2023-05-03 09:10:18] [cni.go:296] [CmdDel] cmdDel(): getting container network namespace
INFO[2023-05-03 09:10:18] [cni.go:306] [CmdDel] cmdDel(): getting default network namespace
INFO[2023-05-03 09:10:18] [cni.go:316] [CmdDel] cmdDel(): executing within container network namespace:
INFO[2023-05-03 09:10:18] [cni.go:319] [func1] cmdDel(): getting device from name
INFO[2023-05-03 09:10:18] [cni.go:328] [func1] cmdDel(): moving device from container to default network namespace
INFO[2023-05-03 09:10:18] [cni.go:341] [CmdDel] cmdDel(): cleaning IPAM config on device
INFO[2023-05-03 09:10:18] [cni.go:349] [CmdDel] cmdDel(): removing BPF program from device
INFO[2023-05-03 09:10:18] [bpfWrapper.go:104] [Infof] Clean_bpf: disovering if_index for interface enp2s0
INFO[2023-05-03 09:10:18] [bpfWrapper.go:104] [Infof] Clean_bpf: if_index for interface enp2s0 is 3
INFO[2023-05-03 09:10:18] [bpfWrapper.go:104] [Infof] Clean_bpf: starting removal of xdp program on interface enp2s0 (3)
DEBU[2023-05-03 09:10:19] [cni.go:133] [loadConf] loadConf(): Mode is set to: primary
INFO[2023-05-03 09:10:19] [cni.go:296] [CmdDel] cmdDel(): getting container network namespace
INFO[2023-05-03 09:10:19] [cni.go:306] [CmdDel] cmdDel(): getting default network namespace
INFO[2023-05-03 09:10:19] [cni.go:316] [CmdDel] cmdDel(): executing within container network namespace:
INFO[2023-05-03 09:10:19] [cni.go:319] [func1] cmdDel(): getting device from name
ERRO[2023-05-03 09:10:19] [cni.go:323] [func1] cmdDel(): failed to find device "enp2s0" in containerNS: Link not found
DEBU[2023-05-03 09:10:19] [cni.go:133] [loadConf] loadConf(): Mode is set to: primary
INFO[2023-05-03 09:10:19] [cni.go:296] [CmdDel] cmdDel(): getting container network namespace
INFO[2023-05-03 09:10:19] [cni.go:306] [CmdDel] cmdDel(): getting default network namespace
INFO[2023-05-03 09:10:19] [cni.go:316] [CmdDel] cmdDel(): executing within container network namespace:
INFO[2023-05-03 09:10:19] [cni.go:319] [func1] cmdDel(): getting device from name
ERRO[2023-05-03 09:10:19] [cni.go:323] [func1] cmdDel(): failed to find device "enp2s0" in containerNS: Link not found
DEBU[2023-05-03 09:10:19] [cni.go:133] [loadConf] loadConf(): Mode is set to: primary
INFO[2023-05-03 09:10:19] [cni.go:296] [CmdDel] cmdDel(): getting container network namespace
INFO[2023-05-03 09:10:19] [cni.go:306] [CmdDel] cmdDel(): getting default network namespace
INFO[2023-05-03 09:10:19] [cni.go:316] [CmdDel] cmdDel(): executing within container network namespace:
INFO[2023-05-03 09:10:19] [cni.go:319] [func1] cmdDel(): getting device from name
ERRO[2023-05-03 09:10:19] [cni.go:323] [func1] cmdDel(): failed to find device "enp2s0" in containerNS: Link not found
DEBU[2023-05-03 09:10:19] [cni.go:133] [loadConf] loadConf(): Mode is set to: primary
INFO[2023-05-03 09:10:19] [cni.go:296] [CmdDel] cmdDel(): getting container network namespace
INFO[2023-05-03 09:10:19] [cni.go:306] [CmdDel] cmdDel(): getting default network namespace
INFO[2023-05-03 09:10:19] [cni.go:316] [CmdDel] cmdDel(): executing within container network namespace:
INFO[2023-05-03 09:10:19] [cni.go:319] [func1] cmdDel(): getting device from name
ERRO[2023-05-03 09:10:19] [cni.go:323] [func1] cmdDel(): failed to find device "enp2s0" in containerNS: Link not found
DEBU[2023-05-03 09:10:19] [cni.go:133] [loadConf] loadConf(): Mode is set to: primary
INFO[2023-05-03 09:10:19] [cni.go:296] [CmdDel] cmdDel(): getting container network namespace
INFO[2023-05-03 09:10:19] [cni.go:306] [CmdDel] cmdDel(): getting default network namespace
INFO[2023-05-03 09:10:19] [cni.go:316] [CmdDel] cmdDel(): executing within container network namespace:
INFO[2023-05-03 09:10:19] [cni.go:319] [func1] cmdDel(): getting device from name
ERRO[2023-05-03 09:10:19] [cni.go:323] [func1] cmdDel(): failed to find device "enp2s0" in containerNS: Link not found
DEBU[2023-05-03 09:10:19] [cni.go:133] [loadConf] loadConf(): Mode is set to: primary
INFO[2023-05-03 09:10:19] [cni.go:296] [CmdDel] cmdDel(): getting container network namespace
INFO[2023-05-03 09:10:19] [cni.go:306] [CmdDel] cmdDel(): getting default network namespace
INFO[2023-05-03 09:10:19] [cni.go:316] [CmdDel] cmdDel(): executing within container network namespace:
INFO[2023-05-03 09:10:19] [cni.go:319] [func1] cmdDel(): getting device from name
ERRO[2023-05-03 09:10:19] [cni.go:323] [func1] cmdDel(): failed to find device "enp2s0" in containerNS: Link not found
DEBU[2023-05-03 09:10:19] [cni.go:133] [loadConf] loadConf(): Mode is set to: primary
INFO[2023-05-03 09:10:19] [cni.go:296] [CmdDel] cmdDel(): getting container network namespace
INFO[2023-05-03 09:10:19] [cni.go:306] [CmdDel] cmdDel(): getting default network namespace
INFO[2023-05-03 09:10:19] [cni.go:316] [CmdDel] cmdDel(): executing within container network namespace:
INFO[2023-05-03 09:10:19] [cni.go:319] [func1] cmdDel(): getting device from name
ERRO[2023-05-03 09:10:19] [cni.go:323] [func1] cmdDel(): failed to find device "enp2s0" in containerNS: Link not found
DEBU[2023-05-03 09:10:19] [cni.go:133] [loadConf] loadConf(): Mode is set to: primary
INFO[2023-05-03 09:10:19] [cni.go:296] [CmdDel] cmdDel(): getting container network namespace
INFO[2023-05-03 09:10:19] [cni.go:306] [CmdDel] cmdDel(): getting default network namespace
INFO[2023-05-03 09:10:19] [cni.go:316] [CmdDel] cmdDel(): executing within container network namespace:
INFO[2023-05-03 09:10:19] [cni.go:319] [func1] cmdDel(): getting device from name
ERRO[2023-05-03 09:10:19] [cni.go:323] [func1] cmdDel(): failed to find device "enp2s0" in containerNS: Link not found
DEBU[2023-05-03 09:10:19] [cni.go:133] [loadConf] loadConf(): Mode is set to: primary
INFO[2023-05-03 09:10:19] [cni.go:296] [CmdDel] cmdDel(): getting container network namespace
INFO[2023-05-03 09:10:19] [cni.go:306] [CmdDel] cmdDel(): getting default network namespace
INFO[2023-05-03 09:10:19] [cni.go:316] [CmdDel] cmdDel(): executing within container network namespace:
INFO[2023-05-03 09:10:19] [cni.go:319] [func1] cmdDel(): getting device from name

is it possible that previous launch of UDS server with timeout at -1 can lead to an inconsistent state of the NIC ? Or maybe some issue of unloading eBPF program or so ?

Thank again for your help !

Aelius

intel / afxdp-plugins-for-kubernetes

CNI doesn't find link after reboot #56