ani-sinha
commented
11 months ago You can add NVMe controllers using instructions in https://learn.microsoft.com/en-gb/azure/virtual-machines/enable-nvme-faqs#overview .
Open ani-sinha opened 11 months ago
ani-sinha
commented
11 months ago You can add NVMe controllers using instructions in https://learn.microsoft.com/en-gb/azure/virtual-machines/enable-nvme-faqs#overview .
ani-sinha
commented
11 months ago Downstream RHEL ticket https://issues.redhat.com/browse/RHEL-7288 .
ani-sinha
commented
11 months ago This systemd patch is relevant https://github.com/systemd/systemd/pull/3136/commits/8a44b47bbbc07d5591d10148270818552e959901 .
Also this is the entire usev config from 60-persistent-storage.rules from systemd:
# NVMe
KERNEL=="nvme*[0-9]n*[0-9]", ATTR{wwid}=="?*", SYMLINK+="disk/by-id/nvme-$attr{wwid}"
KERNEL=="nvme*[0-9]n*[0-9]p*[0-9]", ENV{DEVTYPE}=="partition", ATTRS{wwid}=="?*", SYMLINK+="disk/by-id/nvme-$attr{wwid}-part%n"
KERNEL=="nvme*[0-9]n*[0-9]", ENV{DEVTYPE}=="disk", ATTRS{serial}=="?*", ENV{ID_SERIAL_SHORT}="$attr{serial}"
KERNEL=="nvme*[0-9]n*[0-9]", ENV{DEVTYPE}=="disk", ATTRS{wwid}=="?*", ENV{ID_WWN}="$attr{wwid}"
KERNEL=="nvme*[0-9]n*[0-9]", ENV{DEVTYPE}=="disk", ATTRS{model}=="?*", ENV{ID_MODEL}="$attr{model}"
KERNEL=="nvme*[0-9]n*[0-9]", ENV{DEVTYPE}=="disk", ENV{ID_MODEL}=="?*", ENV{ID_SERIAL_SHORT}=="?*", \
ENV{ID_SERIAL}="$env{ID_MODEL}_$env{ID_SERIAL_SHORT}", SYMLINK+="disk/by-id/nvme-$env{ID_SERIAL}"
KERNEL=="nvme*[0-9]n*[0-9]p*[0-9]", ENV{DEVTYPE}=="partition", ATTRS{serial}=="?*", ENV{ID_SERIAL_SHORT}="$attr{serial}"
KERNEL=="nvme*[0-9]n*[0-9]p*[0-9]", ENV{DEVTYPE}=="partition", ATTRS{model}=="?*", ENV{ID_MODEL}="$attr{model}"
KERNEL=="nvme*[0-9]n*[0-9]p*[0-9]", ENV{DEVTYPE}=="partition", ENV{ID_MODEL}=="?*", ENV{ID_SERIAL_SHORT}=="?*", \
ENV{ID_SERIAL}="$env{ID_MODEL}_$env{ID_SERIAL_SHORT}", SYMLINK+="disk/by-id/nvme-$env{ID_SERIAL}-part%n"Not sure if the above will help as we also seem to need kernel support.
cjp256
commented
11 months ago This is something I've been working on external to the cloud-init rules. I'm curious as you mention azure_root, what do you use it for?
ani-sinha
commented
3 months ago This is something I've been working on external to the cloud-init rules. I'm curious as you mention azure_root, what do you use it for?
It is used by /usr/lib/udev/rules.d/66-azure-ephemeral.rules
$ cat /usr/lib/udev/rules.d/66-azure-ephemeral.rules
# Azure specific rules
ACTION!="add|change", GOTO="cloud_init_end"
SUBSYSTEM!="block", GOTO="cloud_init_end"
ATTRS{ID_VENDOR}!="Msft", GOTO="cloud_init_end"
ATTRS{ID_MODEL}!="Virtual_Disk", GOTO="cloud_init_end"
# Root has a GUID of 0000 as the second value on Gen1 instances
# The resource/resource has GUID of 0001 as the second value
ATTRS{device_id}=="?00000000-0000-*", ENV{fabric_name}="azure_root", GOTO="ci_azure_names"
ATTRS{device_id}=="?00000000-0001-*", ENV{fabric_name}="azure_resource", GOTO="ci_azure_names"
# Azure well known SCSI controllers on Gen2 instances
ATTRS{device_id}=="{f8b3781a-1e82-4818-a1c3-63d806ec15bb}", ENV{fabric_scsi_controller}="scsi0", GOTO="azure_datadisk"
# Do not create symlinks for scsi[1-3] or unmatched device_ids
ATTRS{device_id}=="{f8b3781b-1e82-4818-a1c3-63d806ec15bb}", ENV{fabric_scsi_controller}="scsi1", GOTO="cloud_init_end"
ATTRS{device_id}=="{f8b3781c-1e82-4818-a1c3-63d806ec15bb}", ENV{fabric_scsi_controller}="scsi2", GOTO="cloud_init_end"
ATTRS{device_id}=="{f8b3781d-1e82-4818-a1c3-63d806ec15bb}", ENV{fabric_scsi_controller}="scsi3", GOTO="cloud_init_end"
GOTO="cloud_init_end"
# Map scsi#/lun# fabric_name to azure_root|resource on Gen2 instances
LABEL="azure_datadisk"
ENV{DEVTYPE}=="partition", PROGRAM="/bin/sh -c 'readlink /sys/class/block/%k/../device|cut -d: -f4'", ENV{fabric_name}="$env{fabric_scsi_controller}/lun$result"
ENV{DEVTYPE}=="disk", PROGRAM="/bin/sh -c 'readlink /sys/class/block/%k/device|cut -d: -f4'", ENV{fabric_name}="$env{fabric_scsi_controller}/lun$result"
ENV{fabric_name}=="scsi0/lun0", ENV{fabric_name}="azure_root", GOTO="ci_azure_names"
ENV{fabric_name}=="scsi0/lun1", ENV{fabric_name}="azure_resource", GOTO="ci_azure_names"
GOTO="cloud_init_end"
# Create the symlinks
LABEL="ci_azure_names"
ENV{DEVTYPE}=="disk", SYMLINK+="disk/cloud/$env{fabric_name}"
ENV{DEVTYPE}=="partition", SYMLINK+="disk/cloud/$env{fabric_name}-part%n"
LABEL="cloud_init_end"
@cjp256 ☝️
Enhancement
For Azure VM sizes that use NVMe as OS disk, the
66-azure-ephemeral.rulescannot detect root partitions. No 'azure_root' links under /dev/disk/cloud:This is similar to the ticket https://bugs.launchpad.net/cloud-init/+bug/1797480 where we enabled support for SCSI disks for Gen2 VMs. It was resolved with https://git.launchpad.net/cloud-init/commit/?id=3b332c93 in cloud-init.
I spawned two azure VMs with NVMe and ran
udevadm info -q all /sys/class/block/nvme0n1*. Here are the resilts:VM1:
VM2:
I do not see any similarities in
device_idsacross VMs. Outputs are slightly different across VMs since VM2 is RHEL 9 and VM1 is RHEL 8.On similar note, does WALA needs to be updated too?
cc: @blackboxsw cc: @raharper