CNI delegation failed due to error:Error delegating ADD to CNI plugin:sriov because:OS exec call faild:netplugin failed with no error message

Is this a BUG REPORT or FEATURE REQUEST?:

bug

What happened:

We have sriov-o1c-host0 cluster network and 4 pods are using that CNI. 1st pod { "clusterNetwork": "sriov-o1c-host0", "ip6": "fdfb:6442:1:673::1", "proutes6": {"::0/0": "fdfb:6442:1:673::7fff"} }, 2nd pod { "clusterNetwork": "sriov-o1c-host0", "ip6": "fdfb:6442:1:673::2", "proutes6": {"::0/0": "fdfb:6442:1:673::7fff"} }, 3rd pod { "clusterNetwork": "sriov-o1c-host0", "ip": "10.0.106.65", "proutes": {"11.0.0.0/20": "10.0.106.72"} } 4th pod {"clusterNetwork": "sriov-o1c-host0", "ip": "10.0.106.72", "ip6": "fdfb:6442:1:673::7fff" },

But when we deploy pods one pod is in container creating state , and when pod is described below error is observed.

Warning FailedCreatePodSandBox 30m kubelet, controller-0 Failed to create pod sandbox: rpc error: code = Unknown desc = faled to setup network for sandbox "cd96d7ad7726d3ff73b4779eb98344eae19c95791cc26583652c5709922af75e": CNI network could not be set up: CNI operaton for network:sriov-o1c-host0 failed with:CNI delegation failed due to error:Error delegating ADD to CNI plugin:sriov because:OS exec call faild:netplugin failed with no error message

What you expected to happen:

All the pods should be up and running

How to reproduce it:

Deploy pods with cluster network as shown above. One pod will not come up. This occurs sometimes

Anything else we need to know?:

Danm cleaner pods are running in the setup

kubectl get pods -A | grep -i danm-cleaner
kube-system               danm-cleaner-5dtgr                                                1/1     Running     0          99m

Environment:

DANM version (use danm -version):

/usr/libexec/cni/danm --version
2021/11/24 05:33:55 DANM binary was built from release: v4.2.1
2021/11/24 05:33:55 DANM binary was built from commit: abd3c48d_dirty

Kubernetes version (use kubectl version):


kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.1", GitCommit:"7879fc12a63337efff607952a323df90cdc7a335", GitTreeState:"archive", BuildDate:"2021-06-10T00:00:03Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.1", GitCommit:"7879fc12a63337efff607952a323df90cdc7a335", GitTreeState:"clean", BuildDate:"2020-04-08T17:30:47Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

- DANM configuration (K8s manifests, kubeconfig files, CNI config file):

cat /etc/cni/net.d/00-danm.conf { "cniVersion": "0.3.1", "name": "danm_meta_cni", "type": "danm", "kubeconfig": "/etc/cni/net.d/danm-kubeconfig", "cniDir": "/etc/cni/net.d", "namingScheme": "legacy" }

cat /etc/cni/net.d/danm-kubeconfig

apiVersion: v1 kind: Config current-context: default clusters:

cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM5akNDQWQ2Z0F3SUJBZ0lJbnA4c0grdHJsemt3RFFZSktvWklodmNOQVFFTEJRQXdJVEVmTUIwR0ExVUUKQXd3V1kyOXRiWE5qYjNCbFlteHlhWEJ6WldOaFpHMXBiakFlRncweU1UQTNNRE14TURVek5UZGFGdzAwTVRBMwpNRE14TURVek5UZGFNQ0V4SHpBZEJnTlZCQU1NRm1OdmJXMXpZMjl3WldKc2NtbHdjMlZqWVdSdGFXNHdnZ0VpCk1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLQW9JQkFRQ3NIU0FWMURSTUFnZ1FtTng1TU5vL0c5V3YKR3hqa2luN0NWOHpYdVV6TFZtRjdjMG5KdjlVaDR1WG1LdzNFR3V4MnQveGx3NlhTN3c3TjdFSzcxNzF5Wm1nVwpXc3RPMWgzeDBDWVhoQ1VxOXVXNUdET2lZVnlxMUljOUZSSUQ0U2N6Z1FNb1hFT0ZhVDhlY1U0MEJNNTJqYmlnCkphb2lDZ0RVYmtNT3dMOWpTdlgzQlhpNjNybVRiZTlRMkM0dmxMUmdQK1lFRnpPaDJEYTU4cm5EdVJqVFUyaWsKOUIvNnVzeXMzZ0VFZkhxZ3R4NGNTQ0RVaFpxNVRjYTZjM2hUYVV6NU9HV3NNN2c1bEFDVDlNQ0tOOEN3R1ZNTwpBVkt2WkhPaDRYTWZlU0x3RjZHdkNBN1V6QzhYYjM3TU04REtVbjVQYnJyVUwvNUdrb1djNzI1R3QrMVJBZ01CCkFBR2pNakF3TUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3SFFZRFZSME9CQllFRkc0ZGg3bTZWcXd1bnZlN1ZKV28KcXJ1dFRmeWFNQTBHQ1NxR1NJYjNEUUVCQ3dVQUE0SUJBUUNXNjlsNlFUMjkwUkl0cXJuRldvZ05SV3dYQTlKbgpWSndCK2JsUmhoLzd2ODlTazZvWFhJZlZob0xGaDlZSEtYcFJaREd3RkVBWERTbFhUWFM1YlB6T3F2dkVmR0dCCm81TGRnZE1PR2VYMVg0OHB2TjNlUmZrRE54MFFEKzRFeVhBaVFobmMva0VJcDV0TTNMS1VEOUp5b2lCek5QL0wKSndZUGxiMkxHWFlQY0ZrRWMwTzJ6ZkZtUEtGL294ZFE0dWtDQU8zbVlFT1BEZEl6ZEI4dEYvdFBqUzQ0aWdlegpSUkRtN1RpR0wwd1VYZHQ1NlB6aTJyVEpwa3dqVHlZWTBscWNWZnBYcWkxc0R0QXBzRTRQTlZZZTVkTFdHYjRnCnpTTDRJSGtQejRMNnZnQUxKTVlWclVzNFBzZTVjZW9Gdm9vbHV6YjQ3SXpHUFljN3FiU09lNGdaCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0= server: https://10.96.0.1:443 name: kubernetes contexts:
context: cluster: kubernetes user: danm name: default users:
name: danm user: token: eyJhbGciOiJSUzI1NiIsImtpZCI6IklseWJIYlBYQlJjOEJ1dHZ0Z3lOd3BLVkp1c3VJUEd3bkVRYm9aUWpFS1UifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYW5tLXRva2VuLXE2ZjZoIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImRhbm0iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIzN2YzYzg1ZS03ZDlkLTQ2NTgtOGU4YS0yOGJmNmMwNjgyYTgiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06ZGFubSJ9.UH1RlL8H3a9as0fTCPQNVU84MwDqI7QbORV8hQyj7EawBB8xi3EZWHUzWaf2K6TaIXZebWc_9x2odZDIrPQucVuQqBoUmTsiI083bPwbFxpita4_N_sjdHC-VwWWyDw7KZpfSWhapLRAMTTRzb_s_VO1-VP9Pv7wGj6kCB50CsnYn-FDQf2d-uPEjjNJlhtlSU9ggwWiuceRylW_kk0n554uAJkexCooW8gfsfv2qQWOiAR9cw7oDSstBcvxDUMGW1iTkpy2b1xHP_LRrXfi6XiJRE91kk4OApB83XaPm04DNcnNnCq0BC8qKjBx1NJmOZyqKHw_qoVbxU1GHQM1jg preferences: {}


- OS (e.g. from /etc/os-release):

cat /etc/os-release NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="7" PRETTY_NAME="CentOS Linux 7 (Core)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:7" HOME_URL="https://www.centos.org/" BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7"


- Kernel (e.g. `uname -a`):

uname -a Linux controller-0 3.10.0-1160.15.2.rt56.1152.el7.tis.4.x86_64 #1 SMP PREEMPT RT Wed Jun 9 20:40:45 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux



- Others:

nokia / danm

CNI delegation failed due to error:Error delegating ADD to CNI plugin:sriov because:OS exec call faild:netplugin failed with no error message #264