search

k8snetworkplumbingwg / sriov-network-device-plugin

SRIOV network device plugin for Kubernetes
Apache License 2.0
387 stars 174 forks source link

While sriov container creation, pod is stuck due to "SRIOV-CNI failed to load netconf: LoadConf(): VF pci addr is required" #490

Closed manaspal2 closed 8 months ago

manaspal2 commented 1 year ago

What happened?

I am planning to use Kubevirt for launching my VM and allocate some VFs to that VM. During that process, i see that pod is stuck due to "SRIOV-CNI failed to load netconf: LoadConf(): VF pci addr is required"

What did you expect to happen?

Ideally, the POD Creation should be successful.

What are the minimal steps needed to reproduce the bug?

  1. Configure the SRIOV and create some VFs in the worker node
  2. On master node, I installed Multus.
  3. After that, I am creating a config map.
  4. Deploy the config map.
  5. Deployed demon-set.
  6. Create a NAD.
  7. After that , once i try to deploy a sriov container. But the pod is stuck @ "SRIOV-CNI failed to load netconf: LoadConf(): VF pci addr is required".

Anything else we need to know?

From worker node:

7: enp152s0f0np0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000 link/ether b8:3f:d2:17:d3:86 brd ff:ff:ff:ff:ff:ff vf 0 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off vf 1 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off vf 2 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off vf 3 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off vf 4 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off vf 5 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off vf 6 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off vf 7 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off 8: enp152s0f1np1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000 link/ether b8:3f:d2:17:d3:87 brd ff:ff:ff:ff:ff:ff vf 0 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off vf 1 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off vf 2 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off vf 3 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off vf 4 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off vf 5 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off vf 6 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off vf 7 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off root@la-lakers [ ~ ]# ethtool -i enp152s0f0np0 driver: mlx5_core version: 5.15.102.1-1.cm2 firmware-version: 22.32.2004 (MT_0000000436) expansion-rom-version: bus-info: 0000:98:00.0 supports-statistics: yes supports-test: yes supports-eeprom-access: no supports-register-dump: no supports-priv-flags: yes root@la-lakers [ ~ ]# ethtool -i enp152s0f1np1 driver: mlx5_core version: 5.15.102.1-1.cm2 firmware-version: 22.32.2004 (MT_0000000436) expansion-rom-version: bus-info: 0000:98:00.1 supports-statistics: yes supports-test: yes supports-eeprom-access: no supports-register-dump: no supports-priv-flags: yes root@la-lakers [ ~ ]# lspci -D | grep Mellanox 0000:98:00.0 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6 Dx] 0000:98:00.1 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6 Dx] 0000:98:00.2 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function 0000:98:00.3 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function 0000:98:00.4 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function 0000:98:00.5 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function 0000:98:00.6 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function 0000:98:00.7 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function 0000:98:01.0 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function 0000:98:01.1 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function 0000:98:01.2 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function 0000:98:01.3 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function 0000:98:01.4 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function 0000:98:01.5 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function 0000:98:01.6 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function 0000:98:01.7 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function 0000:98:02.0 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function 0000:98:02.1 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function

`root [ /home/admin ]# cat configMap.yaml apiVersion: v1 kind: ConfigMap metadata: name: sriovdp-config namespace: kube-system data: config.json: | { "resourceList": [{ "resourceName": "mlnx_sriov_rdma", "resourcePrefix": "mellanox.com", "selectors": { "vendors": ["15b3"], "devices": ["101e"], "drivers": ["mlx5_core"], "isRdma": false } }, { "resourceName": "netxtreme_sriov_rdma", "resourcePrefix": "broadcom.com", "selectors": { "vendors": ["14e4"], "devices": ["16dc"], "drivers": ["bnxt_en"], "isRdma": false } } ] } root [ /home/admin ]# kubectl get pods -A | grep multus kube-system kube-multus-ds-97gnb 1/1 Running 0 53m kube-system kube-multus-ds-dhltr 1/1 Running 0 53m

root [ /home/admin ]# kubectl get pods -A | grep sriov kube-system kube-sriov-cni-ds-amd64-xf8dr 1/1 Running 1 (25h ago) 43h kube-system kube-sriov-device-plugin-amd64-dvjpj 1/1 Running 1 (25h ago) 43h

root [ /home/admin ]# cat sriov_nad_fxp.yml apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: sriovnet-fxp annotations: k8s.v1.cni.cncf.io/resourceName: mellanox.com/mlnx_sriov_rdma spec: config: '{ "type": "sriov", "cniVersion": "0.3.1", "name": "sriovnet-fxp", "vlan": 1000, "ipam": { "type": "host-local", "subnet": "100.100.100.0/24", "routes": [{ "dst": "0.0.0.0/0" }], "gateway": "100.100.100.1" } }'

root [ /home/admin ]# kubectl get network-attachment-definitions NAME AGE sriovnet-fxp 58m sriovnet-left 58m sriovnet-right 58m

root [ /home/admin ]# cat test_pod.yml apiVersion: v1 kind: Pod metadata: name: testpod1 annotations: k8s.v1.cni.cncf.io/networks: sriovnet-fxp spec: containers:

root [ /home/admin ]# kubectl get pod NAME READY STATUS RESTARTS AGE testpod1 0/1 ContainerCreating 0 56m root [ /home/admin ]# kubectl describe pod testpod1 Name: testpod1 Namespace: default Priority: 0 Service Account: default Node: la-lakers.englab.juniper.net/40.40.40.2 Start Time: Wed, 07 Jun 2023 22:55:32 +0000 Labels: Annotations: k8s.v1.cni.cncf.io/networks: sriovnet-fxp Status: Pending IP: IPs: Containers: appcntr1: Container ID: Image: centos/tools Image ID: Port: Host Port: Command: /bin/bash -c

Args:
  while true; do sleep 300000; done;
State:          Waiting
  Reason:       ContainerCreating
Ready:          False
Restart Count:  0
Limits:
  mellanox.com/mlnx_sriov_rdma:  1
Requests:
  mellanox.com/mlnx_sriov_rdma:  1
Environment:                     <none>
Mounts:
  /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-stlcf (ro)

Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: kube-api-access-stlcf: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: BestEffort Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message

Normal Scheduled 56m default-scheduler Successfully assigned default/testpod1 to la-lakers.englab.juniper.net Normal AddedInterface 55m multus Add eth0 [10.244.1.15/24] from cbr0 Warning FailedCreatePodSandBox 55m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "10715fb67539cbb18ba689652648923804fd48275f2dc4266ffe76695fd6aa51": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: '&{ContainerID:10715fb67539cbb18ba689652648923804fd48275f2dc4266ffe76695fd6aa51 Netns:/var/run/netns/cni-a3cbc1d7-e29f-782c-3a6f-7159ee52e352 IfName:eth0 Args:IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NAME=testpod1;K8S_POD_INFRA_CONTAINER_ID=10715fb67539cbb18ba689652648923804fd48275f2dc4266ffe76695fd6aa51;K8S_POD_UID=e390078d-2881-4b40-962d-ade653cd2d46 Path: StdinData:[123 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 123 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 116 114 117 101 125 44 34 99 108 117 115 116 101 114 78 101 116 119 111 114 107 34 58 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 47 49 48 45 102 108 97 110 110 101 108 46 99 111 110 102 108 105 115 116 34 44 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 108 111 103 76 101 118 101 108 34 58 34 118 101 114 98 111 115 101 34 44 34 110 97 109 101 34 58 34 109 117 108 116 117 115 45 99 110 105 45 110 101 116 119 111 114 107 34 44 34 115 111 99 107 101 116 68 105 114 34 58 34 47 104 111 115 116 47 114 117 110 47 109 117 108 116 117 115 47 34 44 34 116 121 112 101 34 58 34 109 117 108 116 117 115 45 115 104 105 109 34 44 10 32 32 32 32 34 99 104 114 111 111 116 68 105 114 34 58 32 34 47 104 111 115 116 114 111 111 116 34 44 10 32 32 32 32 34 99 111 110 102 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 10 32 32 32 32 34 108 111 103 76 101 118 101 108 34 58 32 34 118 101 114 98 111 115 101 34 44 10 32 32 32 32 34 115 111 99 107 101 116 68 105 114 34 58 32 34 47 104 111 115 116 47 114 117 110 47 109 117 108 116 117 115 47 34 44 10 32 32 32 32 34 99 110 105 86 101 114 115 105 111 110 34 58 32 34 48 46 51 46 49 34 44 10 32 32 32 32 34 99 110 105 67 111 110 102 105 103 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 10 32 32 32 32 34 109 117 108 116 117 115 67 111 110 102 105 103 70 105 108 101 34 58 32 34 97 117 116 111 34 44 10 32 32 32 32 34 109 117 108 116 117 115 65 117 116 111 99 111 110 102 105 103 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 10 125 10]} {ContainerID:10715fb67539cbb18ba689652648923804fd48275f2dc4266ffe76695fd6aa51 Netns:/var/run/netns/cni-a3cbc1d7-e29f-782c-3a6f-7159ee52e352 IfName:eth0 Args:IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NAME=testpod1;K8S_POD_INFRA_CONTAINER_ID=10715fb67539cbb18ba689652648923804fd48275f2dc4266ffe76695fd6aa51;K8S_POD_UID=e390078d-2881-4b40-962d-ade653cd2d46 Path: StdinData:[123 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 123 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 116 114 117 101 125 44 34 99 108 117 115 116 101 114 78 101 116 119 111 114 107 34 58 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 47 49 48 45 102 108 97 110 110 101 108 46 99 111 110 102 108 105 115 116 34 44 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 108 111 103 76 101 118 101 108 34 58 34 118 101 114 98 111 115 101 34 44 34 110 97 109 101 34 58 34 109 117 108 116 117 115 45 99 110 105 45 110 101 116 119 111 114 107 34 44 34 115 111 99 107 101 116 68 105 114 34 58 34 47 104 111 115 116 47 114 117 110 47 109 117 108 116 117 115 47 34 44 34 116 121 112 101 34 58 34 109 117 108 116 117 115 45 115 104 105 109 34 44 10 32 32 32 32 34 99 104 114 111 111 116 68 105 114 34 58 32 34 47 104 111 115 116 114 111 111 116 34 44 10 32 32 32 32 34 99 111 110 102 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 10 32 32 32 32 34 108 111 103 76 101 118 101 108 34 58 32 34 118 101 114 98 111 115 101 34 44 10 32 32 32 32 34 115 111 99 107 101 116 68 105 114 34 58 32 34 47 104 111 115 116 47 114 117 110 47 109 117 108 116 117 115 47 34 44 10 32 32 32 32 34 99 110 105 86 101 114 115 105 111 110 34 58 32 34 48 46 51 46 49 34 44 10 32 32 32 32 34 99 110 105 67 111 110 102 105 103 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 10 32 32 32 32 34 109 117 108 116 117 115 67 111 110 102 105 103 70 105 108 101 34 58 32 34 97 117 116 111 34 44 10 32 32 32 32 34 109 117 108 116 117 115 65 117 116 111 99 111 110 102 105 103 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 10 125 10]} ERRORED: error configuring pod [default/testpod1] networking: [default/testpod1/e390078d-2881-4b40-962d-ade653cd2d46:sriovnet-fxp]: error adding container to network "sriovnet-fxp": SRIOV-CNI failed to load netconf: LoadConf(): VF pci addr is required

root@pf-cloud$ kubectl describe pods/kube-multus-ds-97gnb -n kube-system Name: kube-multus-ds-97gnb Namespace: kube-system Priority: 0 Service Account: multus Node: pf-cloud.spglab.juniper.net/10.157.69.43 Start Time: Wed, 07 Jun 2023 22:49:15 +0000 Labels: app=multus controller-revision-hash=5c8fc5b975 name=multus pod-template-generation=1 tier=node Annotations: Status: Running IP: 10.157.69.43 IPs: IP: 10.157.69.43 Controlled By: DaemonSet/kube-multus-ds Init Containers: install-multus-binary: Container ID: containerd://8504f72c4cef8f78161f5c67c37ffd45a468cd9de1069c0ef690b49ebb4a7ec8 Image: ghcr.io/k8snetworkplumbingwg/multus-cni:snapshot-thick Image ID: ghcr.io/k8snetworkplumbingwg/multus-cni@sha256:a292271ebeb9e1650e4b24248c4a29133e26ad33f532faeb8ef258638e57fe04 Port: Host Port: Command: cp /usr/src/multus-cni/bin/multus-shim /host/opt/cni/bin/multus-shim State: Terminated Reason: Completed Exit Code: 0 Started: Wed, 07 Jun 2023 22:49:16 +0000 Finished: Wed, 07 Jun 2023 22:49:16 +0000 Ready: True Restart Count: 0 Requests: cpu: 10m memory: 15Mi Environment: Mounts: /host/opt/cni/bin from cnibin (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fnl99 (ro) Containers: kube-multus: Container ID: containerd://733317908ae6f0520b0b37a17412d1e205d8ff043e7539495e103d0905f9dc8a Image: ghcr.io/k8snetworkplumbingwg/multus-cni:snapshot-thick Image ID: ghcr.io/k8snetworkplumbingwg/multus-cni@sha256:a292271ebeb9e1650e4b24248c4a29133e26ad33f532faeb8ef258638e57fe04 Port: Host Port: Command: /usr/src/multus-cni/bin/multus-daemon State: Running Started: Wed, 07 Jun 2023 22:49:18 +0000 Ready: True Restart Count: 0 Limits: cpu: 100m memory: 50Mi Requests: cpu: 100m memory: 50Mi Environment: Mounts: /etc/cni/net.d/multus.d from multus-daemon-config (ro) /host/etc/cni/net.d from cni (rw) /host/run from host-run (rw) /hostroot from hostroot (rw) /run/k8s.cni.cncf.io from host-run-k8s-cni-cncf-io (rw) /run/netns from host-run-netns (rw) /var/lib/cni/multus from host-var-lib-cni-multus (rw) /var/lib/kubelet from host-var-lib-kubelet (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fnl99 (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: cni: Type: HostPath (bare host directory volume) Path: /etc/cni/net.d HostPathType: cnibin: Type: HostPath (bare host directory volume) Path: /opt/cni/bin HostPathType: hostroot: Type: HostPath (bare host directory volume) Path: / HostPathType: multus-daemon-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: multus-daemon-config Optional: false host-run: Type: HostPath (bare host directory volume) Path: /run HostPathType: host-var-lib-cni-multus: Type: HostPath (bare host directory volume) Path: /var/lib/cni/multus HostPathType: host-var-lib-kubelet: Type: HostPath (bare host directory volume) Path: /var/lib/kubelet HostPathType: host-run-k8s-cni-cncf-io: Type: HostPath (bare host directory volume) Path: /run/k8s.cni.cncf.io HostPathType: host-run-netns: Type: HostPath (bare host directory volume) Path: /run/netns/ HostPathType: kube-api-access-fnl99: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: :NoSchedule op=Exists :NoExecute op=Exists node.kubernetes.io/disk-pressure:NoSchedule op=Exists node.kubernetes.io/memory-pressure:NoSchedule op=Exists node.kubernetes.io/network-unavailable:NoSchedule op=Exists node.kubernetes.io/not-ready:NoExecute op=Exists node.kubernetes.io/pid-pressure:NoSchedule op=Exists node.kubernetes.io/unreachable:NoExecute op=Exists node.kubernetes.io/unschedulable:NoSchedule op=Exists Events: root@pf-cloud$

root@pf-cloud$ kubectl describe pods/kube-multus-ds-dhltr -n kube-system Name: kube-multus-ds-dhltr Namespace: kube-system Priority: 0 Service Account: multus Node: la-lakers.englab.juniper.net/40.40.40.2 Start Time: Wed, 07 Jun 2023 22:50:29 +0000 Labels: app=multus controller-revision-hash=5c8fc5b975 name=multus pod-template-generation=1 tier=node Annotations: Status: Running IP: 40.40.40.2 IPs: IP: 40.40.40.2 Controlled By: DaemonSet/kube-multus-ds Init Containers: install-multus-binary: Container ID: containerd://a41eb81b2e89a24329132a6b6d37c74011a01af252f746e754ee42e8d6fa962f Image: ghcr.io/k8snetworkplumbingwg/multus-cni:snapshot-thick Image ID: ghcr.io/k8snetworkplumbingwg/multus-cni@sha256:a292271ebeb9e1650e4b24248c4a29133e26ad33f532faeb8ef258638e57fe04 Port: Host Port: Command: cp /usr/src/multus-cni/bin/multus-shim /host/opt/cni/bin/multus-shim State: Terminated Reason: Completed Exit Code: 0 Started: Wed, 07 Jun 2023 22:50:29 +0000 Finished: Wed, 07 Jun 2023 22:50:29 +0000 Ready: True Restart Count: 0 Requests: cpu: 10m memory: 15Mi Environment: Mounts: /host/opt/cni/bin from cnibin (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gd9xs (ro) Containers: kube-multus: Container ID: containerd://91e691e09222840e43aff4cee90928bccf5318015ad2f45208b8571e9f06b717 Image: ghcr.io/k8snetworkplumbingwg/multus-cni:snapshot-thick Image ID: ghcr.io/k8snetworkplumbingwg/multus-cni@sha256:a292271ebeb9e1650e4b24248c4a29133e26ad33f532faeb8ef258638e57fe04 Port: Host Port: Command: /usr/src/multus-cni/bin/multus-daemon State: Running Started: Wed, 07 Jun 2023 22:50:30 +0000 Ready: True Restart Count: 0 Limits: cpu: 100m memory: 50Mi Requests: cpu: 100m memory: 50Mi Environment: Mounts: /etc/cni/net.d/multus.d from multus-daemon-config (ro) /host/etc/cni/net.d from cni (rw) /host/run from host-run (rw) /hostroot from hostroot (rw) /run/k8s.cni.cncf.io from host-run-k8s-cni-cncf-io (rw) /run/netns from host-run-netns (rw) /var/lib/cni/multus from host-var-lib-cni-multus (rw) /var/lib/kubelet from host-var-lib-kubelet (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gd9xs (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: cni: Type: HostPath (bare host directory volume) Path: /etc/cni/net.d HostPathType: cnibin: Type: HostPath (bare host directory volume) Path: /opt/cni/bin HostPathType: hostroot: Type: HostPath (bare host directory volume) Path: / HostPathType: multus-daemon-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: multus-daemon-config Optional: false host-run: Type: HostPath (bare host directory volume) Path: /run HostPathType: host-var-lib-cni-multus: Type: HostPath (bare host directory volume) Path: /var/lib/cni/multus HostPathType: host-var-lib-kubelet: Type: HostPath (bare host directory volume) Path: /var/lib/kubelet HostPathType: host-run-k8s-cni-cncf-io: Type: HostPath (bare host directory volume) Path: /run/k8s.cni.cncf.io HostPathType: host-run-netns: Type: HostPath (bare host directory volume) Path: /run/netns/ HostPathType: kube-api-access-gd9xs: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: :NoSchedule op=Exists :NoExecute op=Exists node.kubernetes.io/disk-pressure:NoSchedule op=Exists node.kubernetes.io/memory-pressure:NoSchedule op=Exists node.kubernetes.io/network-unavailable:NoSchedule op=Exists node.kubernetes.io/not-ready:NoExecute op=Exists node.kubernetes.io/pid-pressure:NoSchedule op=Exists node.kubernetes.io/unreachable:NoExecute op=Exists node.kubernetes.io/unschedulable:NoSchedule op=Exists Events: root@pf-cloud$

`

Component Versions

Please fill in the below table with the version numbers of components used.

Component Version
SR-IOV Network Device Plugin vfio-pci
SR-IOV CNI Plugin
Multus
Kubernetes 1.26
OS CBL Mariner

Config Files

Config file locations may be config dependent.

Device pool config file location (Try '/etc/pcidp/config.json')
Multus config (Try '/etc/cni/multus/net.d')
CNI config (Try '/etc/cni/net.d/')
Kubernetes deployment type ( Bare Metal, Kubeadm etc.)
Kubeconfig file
SR-IOV Network Custom Resource Definition

Logs

SR-IOV Network Device Plugin Logs (use kubectl logs $PODNAME)
Multus logs (If enabled. Try '/var/log/multus.log' )
Kubelet logs (journalctl -u kubelet)
SchSeba commented 1 year ago

for kubevirt if I remember you need to vf to be on vfio-pci and in the device-pluing I see you search for the regular mlx driver.

please check the kubevirt documentation. also you can try to use the sriov-operator to make the deployment and configuration of the virtual functions much easier

SchSeba commented 1 year ago

any update? or we can close this issue?

SchSeba commented 8 months ago

closing this issue feel free to reopen if needed