ani-sinha
commented
1 year ago You can add NVMe controllers using instructions in https://learn.microsoft.com/en-gb/azure/virtual-machines/enable-nvme-faqs#overview .
Open ani-sinha opened 1 year ago
ani-sinha
commented
1 year ago You can add NVMe controllers using instructions in https://learn.microsoft.com/en-gb/azure/virtual-machines/enable-nvme-faqs#overview .
ani-sinha
commented
1 year ago Downstream RHEL ticket https://issues.redhat.com/browse/RHEL-7288 .
ani-sinha
commented
1 year ago This systemd patch is relevant https://github.com/systemd/systemd/pull/3136/commits/8a44b47bbbc07d5591d10148270818552e959901 .
Also this is the entire usev config from 60-persistent-storage.rules from systemd:
# NVMe
KERNEL=="nvme*[0-9]n*[0-9]", ATTR{wwid}=="?*", SYMLINK+="disk/by-id/nvme-$attr{wwid}"
KERNEL=="nvme*[0-9]n*[0-9]p*[0-9]", ENV{DEVTYPE}=="partition", ATTRS{wwid}=="?*", SYMLINK+="disk/by-id/nvme-$attr{wwid}-part%n"
KERNEL=="nvme*[0-9]n*[0-9]", ENV{DEVTYPE}=="disk", ATTRS{serial}=="?*", ENV{ID_SERIAL_SHORT}="$attr{serial}"
KERNEL=="nvme*[0-9]n*[0-9]", ENV{DEVTYPE}=="disk", ATTRS{wwid}=="?*", ENV{ID_WWN}="$attr{wwid}"
KERNEL=="nvme*[0-9]n*[0-9]", ENV{DEVTYPE}=="disk", ATTRS{model}=="?*", ENV{ID_MODEL}="$attr{model}"
KERNEL=="nvme*[0-9]n*[0-9]", ENV{DEVTYPE}=="disk", ENV{ID_MODEL}=="?*", ENV{ID_SERIAL_SHORT}=="?*", \
ENV{ID_SERIAL}="$env{ID_MODEL}_$env{ID_SERIAL_SHORT}", SYMLINK+="disk/by-id/nvme-$env{ID_SERIAL}"
KERNEL=="nvme*[0-9]n*[0-9]p*[0-9]", ENV{DEVTYPE}=="partition", ATTRS{serial}=="?*", ENV{ID_SERIAL_SHORT}="$attr{serial}"
KERNEL=="nvme*[0-9]n*[0-9]p*[0-9]", ENV{DEVTYPE}=="partition", ATTRS{model}=="?*", ENV{ID_MODEL}="$attr{model}"
KERNEL=="nvme*[0-9]n*[0-9]p*[0-9]", ENV{DEVTYPE}=="partition", ENV{ID_MODEL}=="?*", ENV{ID_SERIAL_SHORT}=="?*", \
ENV{ID_SERIAL}="$env{ID_MODEL}_$env{ID_SERIAL_SHORT}", SYMLINK+="disk/by-id/nvme-$env{ID_SERIAL}-part%n"Not sure if the above will help as we also seem to need kernel support.
cjp256
commented
1 year ago This is something I've been working on external to the cloud-init rules. I'm curious as you mention azure_root, what do you use it for?
ani-sinha
commented
5 months ago This is something I've been working on external to the cloud-init rules. I'm curious as you mention azure_root, what do you use it for?
It is used by /usr/lib/udev/rules.d/66-azure-ephemeral.rules
$ cat /usr/lib/udev/rules.d/66-azure-ephemeral.rules
# Azure specific rules
ACTION!="add|change", GOTO="cloud_init_end"
SUBSYSTEM!="block", GOTO="cloud_init_end"
ATTRS{ID_VENDOR}!="Msft", GOTO="cloud_init_end"
ATTRS{ID_MODEL}!="Virtual_Disk", GOTO="cloud_init_end"
# Root has a GUID of 0000 as the second value on Gen1 instances
# The resource/resource has GUID of 0001 as the second value
ATTRS{device_id}=="?00000000-0000-*", ENV{fabric_name}="azure_root", GOTO="ci_azure_names"
ATTRS{device_id}=="?00000000-0001-*", ENV{fabric_name}="azure_resource", GOTO="ci_azure_names"
# Azure well known SCSI controllers on Gen2 instances
ATTRS{device_id}=="{f8b3781a-1e82-4818-a1c3-63d806ec15bb}", ENV{fabric_scsi_controller}="scsi0", GOTO="azure_datadisk"
# Do not create symlinks for scsi[1-3] or unmatched device_ids
ATTRS{device_id}=="{f8b3781b-1e82-4818-a1c3-63d806ec15bb}", ENV{fabric_scsi_controller}="scsi1", GOTO="cloud_init_end"
ATTRS{device_id}=="{f8b3781c-1e82-4818-a1c3-63d806ec15bb}", ENV{fabric_scsi_controller}="scsi2", GOTO="cloud_init_end"
ATTRS{device_id}=="{f8b3781d-1e82-4818-a1c3-63d806ec15bb}", ENV{fabric_scsi_controller}="scsi3", GOTO="cloud_init_end"
GOTO="cloud_init_end"
# Map scsi#/lun# fabric_name to azure_root|resource on Gen2 instances
LABEL="azure_datadisk"
ENV{DEVTYPE}=="partition", PROGRAM="/bin/sh -c 'readlink /sys/class/block/%k/../device|cut -d: -f4'", ENV{fabric_name}="$env{fabric_scsi_controller}/lun$result"
ENV{DEVTYPE}=="disk", PROGRAM="/bin/sh -c 'readlink /sys/class/block/%k/device|cut -d: -f4'", ENV{fabric_name}="$env{fabric_scsi_controller}/lun$result"
ENV{fabric_name}=="scsi0/lun0", ENV{fabric_name}="azure_root", GOTO="ci_azure_names"
ENV{fabric_name}=="scsi0/lun1", ENV{fabric_name}="azure_resource", GOTO="ci_azure_names"
GOTO="cloud_init_end"
# Create the symlinks
LABEL="ci_azure_names"
ENV{DEVTYPE}=="disk", SYMLINK+="disk/cloud/$env{fabric_name}"
ENV{DEVTYPE}=="partition", SYMLINK+="disk/cloud/$env{fabric_name}-part%n"
LABEL="cloud_init_end"
@cjp256 ☝️
Enhancement
For Azure VM sizes that use NVMe as OS disk, the
66-azure-ephemeral.rulescannot detect root partitions. No 'azure_root' links under /dev/disk/cloud:This is similar to the ticket https://bugs.launchpad.net/cloud-init/+bug/1797480 where we enabled support for SCSI disks for Gen2 VMs. It was resolved with https://git.launchpad.net/cloud-init/commit/?id=3b332c93 in cloud-init.
I spawned two azure VMs with NVMe and ran
udevadm info -q all /sys/class/block/nvme0n1*. Here are the resilts:VM1:
VM2:
I do not see any similarities in
device_idsacross VMs. Outputs are slightly different across VMs since VM2 is RHEL 9 and VM1 is RHEL 8.On similar note, does WALA needs to be updated too?
cc: @blackboxsw cc: @raharper