Feature request: Attach the disk of specified size to Batch VM

[tonybendis]

The Batch VM is currently selected based on resource requirements of the Cromwell task. The code selects the cheapest VM that has CPU count, memory and disk size that are (all three) equal or larger than requested. For tasks requiring large disk, this increases the cost of the VM because larger disk requirement will selected the VM that also has CPU/memory much higher that wanted.

Proposed solution: In light of the fact that TES now supports out of the box local NVMe drives which are found on more than just series L SKUs but the code does NOT use the NVMe drive presence in the selection code described above, the following is the new proposal:

1. Add total drive size of NVMe drives in any given SKU to the current VM whitelist.
2. Add Standard LRS SSD storage to the PriceApi client, and capture this in the GenerateBatchVmSkus tool
3. Change the logic to select the VM by using the greater of ResourceDiskSizeInGiB and NVMe drives total size for all SKUs where that value is >= the requested disk size, and add the additional cost of the requested disk size to the cost of the SKU for all other SKUs.
4. When generating the pool specification, if the requested disk size is greater than the larger of the ResourceDiskSizeInGiB and NVMe drives total size, add a Standard LRS disk of the requested size to the pool spec and format and mount it to the same mount point as the current NVMe start-task.

Estimated effort: 3 days

Note: Premium SSD is not available for all SKUs, Premium v2 is not available through Batch (same with Ultra). To simplify the implementation and maintain costs, Standard SSD was chosen.

Consider dividing the requested capacity by the number of additional drives that can be attached in order to pool them to improve disk I/O.

Previous proposed solution:

Change the logic to select the VM based on CPU count and memory only, then attach the disk of requested size (up to a limit, settable in the configuration?). Additionally, to further improve disk I/O, attach multiple smaller disks and pool them together into single logical volume.

See https://learn.microsoft.com/en-us/azure/virtual-machines/premium-storage-performance

Example code to run on the batch VM to prepare the volume (assumes that 4 disks are attached):

1. Create physical volumes: pvcreate /dev/sdc /dev/sdd /dev/sde /dev/sdf OR pvcreate /dev/sd[c-f]

2. Create volume group (group of physical volumes): vgcreate vg1 /dev/sdc /dev/sdd /dev/sde /dev/sdf OR vgcreate vg1 /dev/sd[c-f]

3. Create logical volume, striped over the volumes in the volume group: lvcreate -i 4 -I 512k -n lv1 -L $(vgs vg1 -o vg_size --noheadings) vg1 (4 is the number of stripes, needs to be equal to number of disks, 512k is the stripe size) OR lvcreate --extents 100%FREE --stripes 4 -I 512k --name lv1 vg1

4. Create the file system on the logical volume: mkfs -t xfs /dev/vg1/lv1

5. Mount the volume: mkdir -p /mnt1 mount /dev/vg1/lv1 /mnt1

[patmagee]

@tonybendis have you created a branch for this as of yet that I can poke around in?

[ngambani]

@BMurri this looks like a long pending issue open since 2021, are we actively working on this or can this be closed?

[BMurri]

@ngambani This is another one that would be great for someone in the community to pitch in on. It can help control costs with some types of workflows that don't need more cores but still need more storage space. microsoft/ga4gh-tes#454 in a crude way would do the same thing, but without as much cost savings nor the more precise control.