How to properly map Images in Hyper-V Hosts

[matschges1cht]

Regarding to the official Ceph Docs "The disk number and path will be picked by Windows". Unfortunately this leads to VM Guests with randomly attached RBDs. How can i archive a predictable mounting of these RBDs so they persist a reboot?

Thank you

[petrutlucian94]

Hi,

This is a Hyper-V/Windows limitation and it also affects iSCSI and Fibre Channel disks. The issue is that Hyper-V identifies disks by number instead of the scsi id, while the disk number can change across host reboots.

We hit this issue in Openstack and our solution was to:

• set the vm AutomaticStartAction to Nothing (e.g. Set-VM $vmName -AutomaticStartAction "Nothing") so that when the host reboots, the vms don't get powered back on by default. This is quite important, preventing the vms from booting with the wrong disks, which has security implications
• have the Openstack Nova service check and update the attached disks before powering the vms back on: https://github.com/openstack/compute-hyperv/blob/1971a6719e89eac9addd4048312c028a2bfbcd03/compute_hyperv/nova/volumeops.py#L221-L238

That being considered, you'd have the following options:

• use an NTFS/ReFS partition to store VHDX images instead of directly attaching the RBD image - might be a bit inconvenient and there might be some performance penalty
• use the same workflow as we did in Openstack - set the vm automatic start action and have a script that updates the vm attachments at boot time. This might be a bit more complicated. Let's say that after a reboot, your hyper-v vm is using a different disk. You have to be able to tell which is the disk that was supposed to be attached to that vm. One way of doing it is to label disk attachments using Msvm_StorageAllocationSettingData.ElementName like we do here: https://github.com/openstack/os-win/blob/5.5.0/os_win/utils/compute/vmutils.py#L581-L585
• use Openstack

It's great that you've mentioned this. We should at least document this Hyper-V limitation. Although the issue isn't caused by WNBD, I guess we can leave it open until we update the Ceph docs.

Btw, I think we might consider adding a Windows service and CLI command to safely attach RBD disks to hyper-v vms (e.g. using Openstack's logic w/o actually requiring Openstack). We'll have to discuss with the Ceph community to see if this service/cli can be part of Ceph. I can't provide any guarantees about when/if this will get implemented.

Thanks, Lucian

[matschges1cht]

Hi Lucian, Thanks for your quick reply. Since this is only a small customer site, Openstack isn't really an option. Could you clarify your solution regarding Msvm_StorageAllocationSettingData.ElementName a bit further? Unfortunately, I'm not that much into Python.

The only solution i came up with is a Powershell script to clear all disks from the VMs, look the real device number up via rbd device ls, attach them accordingly. and then boot the VMs out of that Powershell script

Thanks

[petrutlucian94]

VM disk attachments can be identified and modified using Msvm_StorageAllocationSettingData [1] WMI [2] objects. Labeling the attachments using the ElementName field can simplify disk attachment lookups and it can also be done using Powershell.

The only solution i came up with is a Powershell script to clear all disks from the VMs, look the real device number up via rbd device ls, attach them accordingly. and then boot the VMs out of that Powershell script

Sounds good. If you're able to identify the disk attachments without labels, you can skip the step that I was mentioning above. You should be careful with vms that have multiple attachments, you'll probably have to preserve the disk order.

[1] https://docs.microsoft.com/en-us/windows/win32/hyperv_v2/msvm-storageallocationsettingdata [2] https://docs.microsoft.com/en-us/windows/win32/wmisdk/wmi-start-page

[petrutlucian94]

I've submitted a PR against the Ceph docs: https://github.com/ceph/ceph/pull/43079