Is this plug in support intel nic supporting rdma?

thsmfe001 commented 5 months ago

My server environment will use connectx-7 vpi and intel 810-XXV nic card. So i need to register intel nic into rdma resource pool. Is it possible to use intel nic? If so, could you provide config.map to filter them to register on rdma resource pool?

adrianchiris commented 5 months ago

if 810-XXV supports RDMA then it should be possible. i have not tested on intel hardware

just specify intel vendor ID and device ID as selectors in the resource pool and try it out :)

thsmfe001 commented 5 months ago

Please refer to my configmap.yaml file and plug-in logs.

kind: ConfigMap metadata: name: rdma-devices namespace: kube-system data: config.json: | { "periodicUpdateInterval": 300, "configList": [ { "resourceName": "intel_rdma_devices", "rdmaHcaMax": 1000, "selectors": { "vendors": ["8086"], "deviceIDs": ["0d00", "b500", "a000"], "ifNames": ["eno16595", "eno17105", "enp13s0f0", "enp160s0f1"], "drivers": ["irdma"], "linkTypes": ["ethernet"] } } ] }

kubectl logs rdma-shared-dp-ds-5gpr9 -n kube-system 2024/06/25 04:11:20 Starting K8s RDMA Shared Device Plugin version= master 2024/06/25 04:11:20 resource manager reading configs 2024/06/25 04:11:20 Reading /k8s-rdma-shared-dev-plugin/config.json Using Kubelet Plugin Registry Mode 2024/06/25 04:11:20 loaded config: [{ResourceName:intel_rdma_devices ResourcePrefix: RdmaHcaMax:1000 Devices:[] Selectors:{Vendors:[8086] DeviceIDs:[0d00 b500 a000] Drivers:[irdma] IfNames:[eno16595 eno17105 enp13s0f0 enp160s0f1] LinkTypes:[ethernet]}}] 2024/06/25 04:11:20 periodic update interval: +300 2024/06/25 04:11:20 Discovering host devices 2024/06/25 04:11:20 discovering host network devices 2024/06/25 04:11:20 DiscoverHostDevices(): device found: 0000:01:00.0 02 Broadcom Inc. and... NetXtreme BCM5720 Gigabit Ethernet PCIe 2024/06/25 04:11:20 DiscoverHostDevices(): device found: 0000:01:00.1 02 Broadcom Inc. and... NetXtreme BCM5720 Gigabit Ethernet PCIe 2024/06/25 04:11:20 DiscoverHostDevices(): device found: 0000:0d:00.0 02 Intel Corporation Ethernet Controller E810-XXV for SFP 2024/06/25 04:11:20 DiscoverHostDevices(): device found: 0000:0d:00.1 02 Intel Corporation Ethernet Controller E810-XXV for SFP 2024/06/25 04:11:20 DiscoverHostDevices(): device found: 0000:0d:01.0 02 Intel Corporation Ethernet Adaptive Virtual Function 2024/06/25 04:11:20 DiscoverHostDevices(): device found: 0000:22:00.0 02 Intel Corporation I350 Gigabit Network Connection 2024/06/25 04:11:20 DiscoverHostDevices(): device found: 0000:22:00.1 02 Intel Corporation I350 Gigabit Network Connection 2024/06/25 04:11:20 DiscoverHostDevices(): device found: 0000:22:00.2 02 Intel Corporation I350 Gigabit Network Connection 2024/06/25 04:11:20 DiscoverHostDevices(): device found: 0000:22:00.3 02 Intel Corporation I350 Gigabit Network Connection 2024/06/25 04:11:20 DiscoverHostDevices(): device found: 0000:b5:00.0 02 Intel Corporation Ethernet Controller E810-XXV for SFP 2024/06/25 04:11:20 DiscoverHostDevices(): device found: 0000:b5:00.1 02 Intel Corporation Ethernet Controller E810-XXV for SFP 2024/06/25 04:11:20 DiscoverHostDevices(): device found: 0000:b5:11.0 02 Intel Corporation Ethernet Adaptive Virtual Function 2024/06/25 04:11:20 Initializing resource servers 2024/06/25 04:11:20 Resource: &{ResourceName:intel_rdma_devices ResourcePrefix:rdma RdmaHcaMax:1000 Devices:[] Selectors:{Vendors:[8086] DeviceIDs:[0d00 b500 a000] Drivers:[irdma] IfNames:[eno16595 eno17105 enp13s0f0 enp160s0f1] LinkTypes:[ethernet]}} 2024/06/25 04:11:20 error creating new device: "missing RDMA device spec for device 0000:01:00.0, RDMA device \"issm\" not found" 2024/06/25 04:11:20 error creating new device: "missing RDMA device spec for device 0000:01:00.1, RDMA device \"issm\" not found" 2024/06/25 04:11:20 error creating new device: "missing RDMA device spec for device 0000:0d:00.0, RDMA device \"issm\" not found" 2024/06/25 04:11:20 error creating new device: "missing RDMA device spec for device 0000:0d:00.1, RDMA device \"issm\" not found" 2024/06/25 04:11:20 error creating new device: "missing RDMA device spec for device 0000:0d:01.0, RDMA device \"issm\" not found" 2024/06/25 04:11:20 error creating new device: "missing RDMA device spec for device 0000:22:00.0, RDMA device \"issm\" not found" 2024/06/25 04:11:20 error creating new device: "missing RDMA device spec for device 0000:22:00.1, RDMA device \"issm\" not found" 2024/06/25 04:11:20 error creating new device: "missing RDMA device spec for device 0000:22:00.2, RDMA device \"issm\" not found" 2024/06/25 04:11:20 error creating new device: "missing RDMA device spec for device 0000:22:00.3, RDMA device \"issm\" not found" 2024/06/25 04:11:20 error creating new device: "missing RDMA device spec for device 0000:b5:00.0, RDMA device \"issm\" not found" 2024/06/25 04:11:20 error creating new device: "missing RDMA device spec for device 0000:b5:00.1, RDMA device \"issm\" not found" 2024/06/25 04:11:20 error creating new device: "missing RDMA device spec for device 0000:b5:11.0, RDMA device \"issm\" not found" 2024/06/25 04:11:20 Warning: no devices in device pool, creating empty resource server for intel_rdma_devices 2024/06/25 04:11:20 Warning: no Rdma Devices were found for resource intel_rdma_devices 2024/06/25 04:11:20 Starting all servers... 2024/06/25 04:11:20 starting rdma/intel_rdma_devices device plugin endpoint at: intel_rdma_devices.sock 2024/06/25 04:11:20 rdma/intel_rdma_devices device plugin endpoint started serving 2024/06/25 04:11:20 All servers started. 2024/06/25 04:11:20 Listening for term signals 2024/06/25 04:11:20 Starting OS watcher. 2024/06/25 04:11:21 intel_rdma_devices.sock gets registered successfully at Kubelet 2024/06/25 04:11:21 ListAndWatch called by kubelet for: rdma/intel_rdma_devices 2024/06/25 04:11:21 Updating "rdma/intel_rdma_devices" devices 2024/06/25 04:11:21 exposing "0" devices 2024/06/25 04:16:20 discovering host network devices 2024/06/25 04:16:20 DiscoverHostDevices(): device found: 0000:01:00.0 02 Broadcom Inc. and... NetXtreme BCM5720 Gigabit Ethernet PCIe 2024/06/25 04:16:20 DiscoverHostDevices(): device found: 0000:01:00.1 02 Broadcom Inc. and... NetXtreme BCM5720 Gigabit Ethernet PCIe 2024/06/25 04:16:20 DiscoverHostDevices(): device found: 0000:0d:00.0 02 Intel Corporation Ethernet Controller E810-XXV for SFP 2024/06/25 04:16:20 DiscoverHostDevices(): device found: 0000:0d:00.1 02 Intel Corporation Ethernet Controller E810-XXV for SFP 2024/06/25 04:16:20 DiscoverHostDevices(): device found: 0000:0d:01.0 02 Intel Corporation Ethernet Adaptive Virtual Function 2024/06/25 04:16:20 DiscoverHostDevices(): device found: 0000:22:00.0 02 Intel Corporation I350 Gigabit Network Connection 2024/06/25 04:16:20 DiscoverHostDevices(): device found: 0000:22:00.1 02 Intel Corporation I350 Gigabit Network Connection 2024/06/25 04:16:20 DiscoverHostDevices(): device found: 0000:22:00.2 02 Intel Corporation I350 Gigabit Network Connection 2024/06/25 04:16:20 DiscoverHostDevices(): device found: 0000:22:00.3 02 Intel Corporation I350 Gigabit Network Connection 2024/06/25 04:16:20 DiscoverHostDevices(): device found: 0000:b5:00.0 02 Intel Corporation Ethernet Controller E810-XXV for SFP 2024/06/25 04:16:20 DiscoverHostDevices(): device found: 0000:b5:00.1 02 Intel Corporation Ethernet Controller E810-XXV for SFP 2024/06/25 04:16:20 DiscoverHostDevices(): device found: 0000:b5:11.0 02 Intel Corporation Ethernet Adaptive Virtual Function 2024/06/25 04:16:20 error creating new device: "missing RDMA device spec for device 0000:01:00.0, RDMA device \"issm\" not found" 2024/06/25 04:16:20 error creating new device: "missing RDMA device spec for device 0000:01:00.1, RDMA device \"issm\" not found" 2024/06/25 04:16:20 error creating new device: "missing RDMA device spec for device 0000:0d:00.0, RDMA device \"issm\" not found" 2024/06/25 04:16:20 error creating new device: "missing RDMA device spec for device 0000:0d:00.1, RDMA device \"issm\" not found" 2024/06/25 04:16:20 error creating new device: "missing RDMA device spec for device 0000:0d:01.0, RDMA device \"issm\" not found" 2024/06/25 04:16:20 error creating new device: "missing RDMA device spec for device 0000:22:00.0, RDMA device \"issm\" not found" 2024/06/25 04:16:20 error creating new device: "missing RDMA device spec for device 0000:22:00.1, RDMA device \"issm\" not found" 2024/06/25 04:16:20 error creating new device: "missing RDMA device spec for device 0000:22:00.2, RDMA device \"issm\" not found" 2024/06/25 04:16:20 error creating new device: "missing RDMA device spec for device 0000:22:00.3, RDMA device \"issm\" not found" 2024/06/25 04:16:20 error creating new device: "missing RDMA device spec for device 0000:b5:00.0, RDMA device \"issm\" not found" 2024/06/25 04:16:20 error creating new device: "missing RDMA device spec for device 0000:b5:00.1, RDMA device \"issm\" not found" 2024/06/25 04:16:20 error creating new device: "missing RDMA device spec for device 0000:b5:11.0, RDMA device \"issm\" not found" 2024/06/25 04:16:20 no changes to devices for "rdma/intel_rdma_devices" 2024/06/25 04:16:20 exposing "0" devices 2024/06/25 04:21:20 discovering host network devices 2024/06/25 04:21:20 DiscoverHostDevices(): device found: 0000:01:00.0 02 Broadcom Inc. and... NetXtreme BCM5720 Gigabit Ethernet PCIe 2024/06/25 04:21:20 DiscoverHostDevices(): device found: 0000:01:00.1 02 Broadcom Inc. and... NetXtreme BCM5720 Gigabit Ethernet PCIe 2024/06/25 04:21:20 DiscoverHostDevices(): device found: 0000:0d:00.0 02 Intel Corporation Ethernet Controller E810-XXV for SFP 2024/06/25 04:21:20 DiscoverHostDevices(): device found: 0000:0d:00.1 02 Intel Corporation Ethernet Controller E810-XXV for SFP 2024/06/25 04:21:20 DiscoverHostDevices(): device found: 0000:0d:01.0 02 Intel Corporation Ethernet Adaptive Virtual Function 2024/06/25 04:21:20 DiscoverHostDevices(): device found: 0000:22:00.0 02 Intel Corporation I350 Gigabit Network Connection 2024/06/25 04:21:20 DiscoverHostDevices(): device found: 0000:22:00.1 02 Intel Corporation I350 Gigabit Network Connection 2024/06/25 04:21:20 DiscoverHostDevices(): device found: 0000:22:00.2 02 Intel Corporation I350 Gigabit Network Connection 2024/06/25 04:21:20 DiscoverHostDevices(): device found: 0000:22:00.3 02 Intel Corporation I350 Gigabit Network Connection 2024/06/25 04:21:20 DiscoverHostDevices(): device found: 0000:b5:00.0 02 Intel Corporation Ethernet Controller E810-XXV for SFP 2024/06/25 04:21:20 DiscoverHostDevices(): device found: 0000:b5:00.1 02 Intel Corporation Ethernet Controller E810-XXV for SFP 2024/06/25 04:21:20 DiscoverHostDevices(): device found: 0000:b5:11.0 02 Intel Corporation Ethernet Adaptive Virtual Function 2024/06/25 04:21:20 error creating new device: "missing RDMA device spec for device 0000:01:00.0, RDMA device \"issm\" not found" 2024/06/25 04:21:20 error creating new device: "missing RDMA device spec for device 0000:01:00.1, RDMA device \"issm\" not found" 2024/06/25 04:21:20 error creating new device: "missing RDMA device spec for device 0000:0d:00.0, RDMA device \"issm\" not found" 2024/06/25 04:21:20 error creating new device: "missing RDMA device spec for device 0000:0d:00.1, RDMA device \"issm\" not found" 2024/06/25 04:21:20 error creating new device: "missing RDMA device spec for device 0000:0d:01.0, RDMA device \"issm\" not found" 2024/06/25 04:21:20 error creating new device: "missing RDMA device spec for device 0000:22:00.0, RDMA device \"issm\" not found" 2024/06/25 04:21:20 error creating new device: "missing RDMA device spec for device 0000:22:00.1, RDMA device \"issm\" not found" 2024/06/25 04:21:20 error creating new device: "missing RDMA device spec for device 0000:22:00.2, RDMA device \"issm\" not found" 2024/06/25 04:21:20 error creating new device: "missing RDMA device spec for device 0000:22:00.3, RDMA device \"issm\" not found" 2024/06/25 04:21:20 error creating new device: "missing RDMA device spec for device 0000:b5:00.0, RDMA device \"issm\" not found" 2024/06/25 04:21:20 error creating new device: "missing RDMA device spec for device 0000:b5:00.1, RDMA device \"issm\" not found" 2024/06/25 04:21:20 error creating new device: "missing RDMA device spec for device 0000:b5:11.0, RDMA device \"issm\" not found" 2024/06/25 04:21:20 no changes to devices for "rdma/intel_rdma_devices" 2024/06/25 04:21:20 exposing "0" devices 2024/06/25 04:26:20 discovering host network devices 2024/06/25 04:26:20 DiscoverHostDevices(): device found: 0000:01:00.0 02 Broadcom Inc. and... NetXtreme BCM5720 Gigabit Ethernet PCIe 2024/06/25 04:26:20 DiscoverHostDevices(): device found: 0000:01:00.1 02 Broadcom Inc. and... NetXtreme BCM5720 Gigabit Ethernet PCIe 2024/06/25 04:26:20 DiscoverHostDevices(): device found: 0000:0d:00.0 02 Intel Corporation Ethernet Controller E810-XXV for SFP 2024/06/25 04:26:20 DiscoverHostDevices(): device found: 0000:0d:00.1 02 Intel Corporation Ethernet Controller E810-XXV for SFP 2024/06/25 04:26:20 DiscoverHostDevices(): device found: 0000:0d:01.0 02 Intel Corporation Ethernet Adaptive Virtual Function 2024/06/25 04:26:20 DiscoverHostDevices(): device found: 0000:22:00.0 02 Intel Corporation I350 Gigabit Network Connection 2024/06/25 04:26:20 DiscoverHostDevices(): device found: 0000:22:00.1 02 Intel Corporation I350 Gigabit Network Connection 2024/06/25 04:26:20 DiscoverHostDevices(): device found: 0000:22:00.2 02 Intel Corporation I350 Gigabit Network Connection 2024/06/25 04:26:20 DiscoverHostDevices(): device found: 0000:22:00.3 02 Intel Corporation I350 Gigabit Network Connection 2024/06/25 04:26:20 DiscoverHostDevices(): device found: 0000:b5:00.0 02 Intel Corporation Ethernet Controller E810-XXV for SFP 2024/06/25 04:26:20 DiscoverHostDevices(): device found: 0000:b5:00.1 02 Intel Corporation Ethernet Controller E810-XXV for SFP 2024/06/25 04:26:20 DiscoverHostDevices(): device found: 0000:b5:11.0 02 Intel Corporation Ethernet Adaptive Virtual Function 2024/06/25 04:26:20 error creating new device: "missing RDMA device spec for device 0000:01:00.0, RDMA device \"issm\" not found" 2024/06/25 04:26:20 error creating new device: "missing RDMA device spec for device 0000:01:00.1, RDMA device \"issm\" not found" 2024/06/25 04:26:20 error creating new device: "missing RDMA device spec for device 0000:0d:00.0, RDMA device \"issm\" not found" 2024/06/25 04:26:20 error creating new device: "missing RDMA device spec for device 0000:0d:00.1, RDMA device \"issm\" not found" 2024/06/25 04:26:20 error creating new device: "missing RDMA device spec for device 0000:0d:01.0, RDMA device \"issm\" not found" 2024/06/25 04:26:20 error creating new device: "missing RDMA device spec for device 0000:22:00.0, RDMA device \"issm\" not found" 2024/06/25 04:26:20 error creating new device: "missing RDMA device spec for device 0000:22:00.1, RDMA device \"issm\" not found" 2024/06/25 04:26:20 error creating new device: "missing RDMA device spec for device 0000:22:00.2, RDMA device \"issm\" not found" 2024/06/25 04:26:20 error creating new device: "missing RDMA device spec for device 0000:22:00.3, RDMA device \"issm\" not found" 2024/06/25 04:26:20 error creating new device: "missing RDMA device spec for device 0000:b5:00.0, RDMA device \"issm\" not found" 2024/06/25 04:26:20 error creating new device: "missing RDMA device spec for device 0000:b5:00.1, RDMA device \"issm\" not found" 2024/06/25 04:26:20 error creating new device: "missing RDMA device spec for device 0000:b5:11.0, RDMA device \"issm\" not found" 2024/06/25 04:26:20 no changes to devices for "rdma/intel_rdma_devices" 2024/06/25 04:26:20 exposing "0" devices

adrianchiris commented 1 month ago

are you using the latest k8s-rdma-shared-device-plugin ? we dont require issm rdma device anymore

[1] https://github.com/Mellanox/k8s-rdma-shared-dev-plugin/blob/master/pkg/resources/rdma_device_spec.go#L19

Mellanox / k8s-rdma-shared-dev-plugin

Is this plug in support intel nic supporting rdma? #113