Feasibility and Support for NVMe-over-Fabrics (NVMeoF) in KubeVirt VMs

I need to utilize NVMe-over-Fabrics (NVMeoF) devices within my KubeVirt VMs. Currently, my PVCs are backed by NVMeoF devices. However, I am encountering an issue where these NVMe devices are being presented as SCSI devices (e.g., /dev/sda, /dev/sdb) inside the VMs.

I expect it to be NVMe devices in side the kubevirt VMs

csi driver that support NVMeOF and storage is required, need to use the storage class for root disk and additional disks to reproduce this issue

Environment:

KubeVirt version (use virtctl version): 1.1
Kubernetes version (use kubectl version): 1.28
VM or VMI specifications: `$ k get vmi aisaacs-agent-1 -o yaml apiVersion: kubevirt.io/v1 kind: VirtualMachineInstance metadata: annotations: kubevirt.io/latest-observed-api-version: v1 kubevirt.io/storage-observed-api-version: v1 kubevirt.io/vm-generation: "1" prometheus.io/port: "9100" prometheus.io/scrape: "true" creationTimestamp: "2024-07-11T05:13:03Z" finalizers:
- kubevirt.io/virtualMachineControllerFinalize
- foregroundDeleteVirtualMachine generation: 15 labels: devops.ddn.com/expose: virt devops.ddn.com/kubevirt: aisaacs-agent-1 devops.ddn.com/name: aisaacs-agent-1 devops.ddn.com/network: kubert devops.ddn.com/owner: 543029207-aisaacs kubevirt.io/nodeName: ampere-kv-03 name: aisaacs-agent-1 namespace: virts ownerReferences:
- apiVersion: kubevirt.io/v1 blockOwnerDeletion: true controller: true kind: VirtualMachine name: aisaacs-agent-1 uid: 438bb15d-08f9-4aa9-8e34-3e7df0712fe5 resourceVersion: "814977" uid: 80af877d-ee8f-4a85-9a3c-d42a2b4c83ad spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms:
  - matchExpressions:
    - key: devops.ddn.com/dedicated_cpu operator: In values:
    - "true" podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution:
  - podAffinityTerm: labelSelector: matchExpressions:
    - key: devops.ddn.com/owner operator: In values:
      - 543029207-aisaacs topologyKey: kubernetes.io/hostname weight: 10
  - podAffinityTerm: labelSelector: matchExpressions:
    - key: devops.ddn.com/kubevirt operator: Exists topologyKey: kubernetes.io/hostname weight: 10 architecture: arm64 dnsPolicy: ClusterFirst domain: chassis: asset: 543029207-aisaacs manufacturer: DDN Devops serial: "1720674303" sku: red-ready-u22 cpu: cores: 1 model: host-passthrough sockets: 4 threads: 1 devices: blockMultiQueue: true disks:
  - bootOrder: 1 disk: bus: scsi name: root-disk serial: root-543029207-7 tag: root-disk
  - disk: bus: virtio readonly: true name: cloudinitdisk serial: cidata-543029207-8 tag: cloud-init disk
  - disk: bus: scsi name: dv-aisaacs-agent-1-1-1720674303 serial: dv-aisaacs-agent-1-1-1720674303-543029207-9
  - disk: bus: scsi name: dv-aisaacs-agent-1-2-1720674303 serial: dv-aisaacs-agent-1-2-1720674303-543029207-10
  - disk: bus: scsi name: dv-aisaacs-agent-1-3-1720674303 serial: dv-aisaacs-agent-1-3-1720674303-543029207-11
  - disk: bus: scsi name: dv-aisaacs-agent-1-4-1720674303 serial: dv-aisaacs-agent-1-4-1720674303-543029207-12 hostDevices:
  - deviceName: devices.kubevirt.io/mellanox-sriov name: device interfaces:
  - bridge: {} macAddress: ea:1f:f7:e9:c9:4e name: default rng: {} tpm: {} features: acpi: enabled: true kvm: {} firmware: bootloader: efi: secureBoot: false serial: "1720674303" uuid: 2b8cac74-52ea-4eb9-a452-4ed8035f5779 machine: type: virt memory: guest: 16Gi resources: limits: cpu: "4" memory: 17Gi requests: cpu: "4" memory: 17Gi evictionStrategy: None hostname: aisaacs-agent-1 networks:
- name: default pod: {} priorityClassName: system-cluster-critical subdomain: vms volumes:
- dataVolume: name: dv-aisaacs-agent-1-root-1720674303 name: root-disk
- cloudInitNoCloud: networkData: | ethernets: id0: dhcp4: true match: driver: virtio-pci macaddress: ea:1f:f7:e9:c9:4e mtu: 1500 set-name: dontuseme0 id1: addresses:
  - 172.25.116.69/23 dhcp4: false match: driver: mlx5_core mtu: 9000 set-name: eth1 version: 2 secretRef: name: vm-cloudinit-aisaacs-agent-1 name: cloudinitdisk
- dataVolume: name: dv-aisaacs-agent-1-1-1720674303 name: dv-aisaacs-agent-1-1-1720674303
- dataVolume: name: dv-aisaacs-agent-1-2-1720674303 name: dv-aisaacs-agent-1-2-1720674303
- dataVolume: name: dv-aisaacs-agent-1-3-1720674303 name: dv-aisaacs-agent-1-3-1720674303
- dataVolume: name: dv-aisaacs-agent-1-4-1720674303 name: dv-aisaacs-agent-1-4-1720674303 status: activePods: fa34dc84-952d-4dd9-867f-0897ae5b03d0: ampere-kv-03 conditions:
- lastProbeTime: null lastTransitionTime: "2024-07-11T05:13:05Z" status: "True" type: Ready
- lastProbeTime: null lastTransitionTime: null message: 'cannot migrate VMI: PVC dv-aisaacs-agent-1-root-1720674303 is not shared, live migration requires that all PVCs must be shared (using ReadWriteMany access mode)' reason: DisksNotLiveMigratable status: "False" type: LiveMigratable
- lastProbeTime: "2024-07-11T05:15:27Z" lastTransitionTime: null status: "True" type: AgentConnected currentCPUTopology: cores: 1 sockets: 1 threads: 1 guestOSInfo: id: ubuntu kernelRelease: 6.2.0-39-generic kernelVersion: '#40~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Nov 16 12:53:29 UTC 2' name: Ubuntu prettyName: Ubuntu 22.04.4 LTS version: "22.04" versionId: "22.04" interfaces:
- infoSource: domain, guest-agent interfaceName: dontuseme0 ipAddress: 250.58.95.218 ipAddresses:
- 250.58.95.218
- fe80::e81f:f7ff:fee9:c94e mac: ea:1f:f7:e9:c9:4e name: default queueCount: 1
- infoSource: guest-agent interfaceName: eth1 ipAddress: 172.25.116.69 ipAddresses:
- 172.25.116.69
- fe80::4c0f:b8ff:fe6f:c35 mac: 4e:0f:b8:6f:0c:35
- infoSource: guest-agent interfaceName: virbr0 ipAddress: 192.168.122.1 ipAddresses:
- 192.168.122.1 mac: 52:54:00:7c:bd:4e
- infoSource: guest-agent interfaceName: docker0 ipAddress: 172.17.0.1 ipAddresses:
- 172.17.0.1 mac: 02:42:a8:d5:e4:a9 launcherContainerImageVersion: quay.io/kubevirt/virt-launcher:v1.1.0 machine: type: virt-rhel9.2.0 memory: guestAtBoot: 16Gi guestCurrent: 16Gi guestRequested: 16Gi migrationMethod: BlockMigration migrationTransport: Unix nodeName: ampere-kv-03 phase: Running phaseTransitionTimestamps:
- phase: Pending phaseTransitionTimestamp: "2024-07-11T05:13:03Z"
- phase: Scheduling phaseTransitionTimestamp: "2024-07-11T05:13:03Z"
- phase: Scheduled phaseTransitionTimestamp: "2024-07-11T05:13:05Z"
- phase: Running phaseTransitionTimestamp: "2024-07-11T05:13:09Z" qosClass: Guaranteed runtimeUser: 107 selinuxContext: none virtualMachineRevisionName: revision-start-vm-438bb15d-08f9-4aa9-8e34-3e7df0712fe5-1 volumeStatus:
- name: cloudinitdisk size: 1048576 target: vda
- name: dv-aisaacs-agent-1-1-1720674303 persistentVolumeClaimInfo: accessModes:
  - ReadWriteOnce capacity: storage: 128Gi filesystemOverhead: "0.055" requests: storage: 128Gi volumeMode: Filesystem target: sdb
- name: dv-aisaacs-agent-1-2-1720674303 persistentVolumeClaimInfo: accessModes:
  - ReadWriteOnce capacity: storage: 128Gi filesystemOverhead: "0.055" requests: storage: 128Gi volumeMode: Filesystem target: sdc
- name: dv-aisaacs-agent-1-3-1720674303 persistentVolumeClaimInfo: accessModes:
  - ReadWriteOnce capacity: storage: 128Gi filesystemOverhead: "0.055" requests: storage: 128Gi volumeMode: Filesystem target: sdd
- name: dv-aisaacs-agent-1-4-1720674303 persistentVolumeClaimInfo: accessModes:
  - ReadWriteOnce capacity: storage: 128Gi filesystemOverhead: "0.055" requests: storage: 128Gi volumeMode: Filesystem target: sde
- name: root-disk persistentVolumeClaimInfo: accessModes:
  - ReadWriteOnce capacity: storage: 100Gi filesystemOverhead: "0.055" requests: storage: 100Gi volumeMode: Filesystem target: sda `
Cloud provider or hardware configuration: N/A
OS (e.g. from /etc/os-release): Ubuntu 24.04
Kernel (e.g. uname -a): 6.8

kubevirt / user-guide

Feasibility and Support for NVMe-over-Fabrics (NVMeoF) in KubeVirt VMs #822