smartxworks / virtink

Lightweight Virtualization Add-on for Kubernetes
Apache License 2.0
503 stars 42 forks source link

feat: Add supports for vDPA interface #98

Closed makoto126 closed 10 months ago

makoto126 commented 10 months ago

Tested with SR-IOV Network Device Plugin and modified kube-ovn(ovs-dpdk offload to our DPU). Live migration previously tested in Kubevirt(with qemu).

VM spec

apiVersion: virt.virtink.smartx.com/v1alpha1
kind: VirtualMachine
spec:
  instance:
    interfaces:
      - name: pod
      - name: vdpa1
        vdpa: { numQueues: 9, iommu: False }
      - name: vdpa2
        vdpa: { numQueues: 9, iommu: False }
  networks:
    - name: pod
      pod: {}
    - name: vdpa1
      multus:
        networkName: offload-ovn1
    - name: vdpa2
      multus:
        networkName: offload-ovn2

SR-IOV device plugin config

apiVersion: v1
kind: ConfigMap
metadata:
  name: sriovdp-config
  namespace: kube-system
data:
  config.json: |
    {
        "resourceList": [
            {
                "resourceName": "jaguar",
                "resourcePrefix": "jaguarmicro.com",
                "selectors": {
                    "vendors": ["1f53"],
                    "devices": ["1000"],
                    "drivers": ["jaguar"],
                    "vdpaType": "vhost"
                }
            }
        ]
    }

NetworkAttachmentDefinition Spec

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: offload-ovn1
  namespace: default
  annotations:
    k8s.v1.cni.cncf.io/resourceName: jaguarmicro.com/jaguar
spec:
  config: >-
    {
        "cniVersion": "0.3.0",
        "type": "kube-ovn",
        "server_socket": "self.dpu.com:12345",
        "provider": "offload-ovn1.default.ovn"
    }
---
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: offload-ovn2
  namespace: default
  annotations:
    k8s.v1.cni.cncf.io/resourceName: jaguarmicro.com/jaguar
spec:
  config: >-
    {
        "cniVersion": "0.3.0",
        "type": "kube-ovn",
        "server_socket": "self.dpu.com:12345",
        "provider": "offload-ovn2.default.ovn"
    }

Pod network status

apiVersion: v1
kind: Pod
metadata:
  annotations:
    k8s.v1.cni.cncf.io/network-status: |-
      [{
          "name": "cbr0",
          "interface": "eth0",
          "ips": [
              "10.244.2.45"
          ],
          "mac": "5a:61:29:39:0d:29",
          "default": true,
          "dns": {},
          "gateway": [
              "10.244.2.1"
          ]
      },{
          "name": "default/offload-ovn1",
          "interface": "net1",
          "ips": [
              "10.1.0.20"
          ],
          "mac": "00:00:00:03:6C:84",
          "dns": {},
          "device-info": {
              "type": "vdpa",
              "version": "1.1.0",
              "vdpa": {
                  "parent-device": "test1",
                  "driver": "vhost",
                  "path": "/dev/vhost-vdpa-2",
                  "pci-address": "0000:68:00.5"
              }
          }
      },{
          "name": "default/offload-ovn2",
          "interface": "net2",
          "ips": [
              "10.2.0.2"
          ],
          "mac": "00:00:00:50:A4:81",
          "dns": {},
          "device-info": {
              "type": "vdpa",
              "version": "1.1.0",
              "vdpa": {
                  "parent-device": "test2",
                  "driver": "vhost",
                  "path": "/dev/vhost-vdpa-3",
                  "pci-address": "0000:68:00.6"
              }
          }
      }]

VM after dhclient

ubuntu@ubuntu-container-rootfs:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 5a:61:29:39:0d:29 brd ff:ff:ff:ff:ff:ff
    altname enp0s4
    inet 10.244.2.45/24 metric 100 brd 10.244.2.255 scope global ens4
       valid_lft forever preferred_lft forever
3: ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:00:00:03:6c:84 brd ff:ff:ff:ff:ff:ff
    altname enp0s6
    inet 10.1.0.20/24 brd 10.1.0.255 scope global dynamic ens6
       valid_lft 2305sec preferred_lft 2305sec
4: ens7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:00:00:50:a4:81 brd ff:ff:ff:ff:ff:ff
    altname enp0s7
    inet 10.2.0.2/24 brd 10.2.0.255 scope global dynamic ens7
       valid_lft 2305sec preferred_lft 2305sec
CLAassistant commented 10 months ago

CLA assistant check
All committers have signed the CLA.

codecov[bot] commented 10 months ago

Codecov Report

Attention: 3 lines in your changes are missing coverage. Please review.

Comparison is base (30780fe) 34.11% compared to head (0f3d995) 34.08%.

:exclamation: Current head 0f3d995 differs from pull request most recent head a49cb4e. Consider uploading reports for the commit a49cb4e to get more accurate results

Files Patch % Lines
pkg/controller/vm_webhook.go 50.00% 2 Missing and 1 partial :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #98 +/- ## ========================================== - Coverage 34.11% 34.08% -0.04% ========================================== Files 6 6 Lines 1952 1957 +5 ========================================== + Hits 666 667 +1 - Misses 1182 1185 +3 - Partials 104 105 +1 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.