[ci.jenkins.io] Switch system disk to SSD

[dduportal]

As per the Azure Advisor Monitor, we should switch the system disk of ci.jenkins.io to an SSD.

The VM has 2 disks today:

• The system disk is a 50 Gb HDD (hardware encrypted, 500 IOPS, 60 Mbit/s) :

• This resource could be managed through an import (+ setup) of the VM ci.jenkins.io (system disk is tied to the virtual machine resource) in jenkins-infra/azure
• Checking if the change can be made without restarting (but I doubt it)

• The JENKINS_HOME of ci.jenkins.io is stored on a data disk (Premium SSD, hardware encrypted, 500Gb, P20 - 2300 IOPS, 150 MBps)
  • This resource could be imported and managed by Terraform, and eventually snapshoted daily
  • Please note that this sizing could be decreased in the future if we are succesful in implementing #3496

=> The Azure metrics (ref. https://learn.microsoft.com/en-us/azure/virtual-machines/disks-metrics) shown above does not show any contention on the system disk in the past 7 days, which does not prompt us to switch to an SSD. => However, the datadog VM metrics shows that the latency is clearly higher (/dev/sdc is the system disk, /dev/sdb is the Premium SSD datadisk) which is a good reason to follow the advisor message:

=> Please note that the system disk hosts the system, but also the temp dirs and the logs (along with all the container overlays).

[dduportal]

A word on the pricing:

• The HDD costs us, currently, around $4 per month: $3.01 + ~1$ per month of transactions

  • It's a custom size of 50Gb and could be set to 64Gb for the same price as per the HDD table at https://azure.microsoft.com/en-us/pricing/details/managed-disks/ (Category S6)

• Switching to Standard SSD should increase slighlty: baseline of $4.80 (64 Gb, in the E6 category) + eventual transactions (limited to ~$12 monthly), same IOPS and bandwidth, and $0.47 per mount

  • It would also allow bursting (credit-based) above this IOPS limit

• Switching to Premium SSD should increase a bit more: baseline of $10 (64 Gb, in the P6 category), lower IOPS (250) and bandwidth (50 Mbits/s) but no transaction variable part, and still $0.47 per mount

Current cost for ci.jenkins.io storage for the past 6 month, to have an order of magnitude:

![Uploading Capture d’écran 2023-04-19 à 19.50.22.png…]()

[dduportal]

Let's go for a Standard SSD \o/

[dduportal]

• Snapshot taken (named 20230419-helpdesk-3526)
• ci.jenkins.io is put in shutdown mode

[dduportal]

• Stopped VM (deallocation)
• System disk changed to Standard SSD' with a size of64Gb` (category E6)
• Started VM
• Upgraded system and rebooted a second time

back to work ✅

=> let's check the metrics in ~24h

[dduportal]

This change did not had any visible effect except the Azure Advisor does not alert us on the VM anymore :'(

• Disk latency still the same (on the screenshot, the vertical bar is when the change was applied)

• The new disk size is seen at the block device, but it's not taken in account by the cloudinit resize process during the boot phase:

root@ci:~# lsblk /dev/sdc
NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sdc       8:32   0   64G  0 disk 
├─sdc1    8:33   0 49.9G  0 part /
├─sdc14   8:46   0    4M  0 part 
└─sdc15   8:47   0  106M  0 part /boot/efi
root@ci:~# fdisk -l /dev/sdc
GPT PMBR size mismatch (104857599 != 134217727) will be corrected by w(rite).
Disk /dev/sdc: 64 GiB, 68719476736 bytes, 134217728 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: F9A32187-D01B-465B-AEC5-3D0FE6FADD47

Device      Start       End   Sectors  Size Type
/dev/sdc1  227328 104857566 104630239 49.9G Linux filesystem
/dev/sdc14   2048     10239      8192    4M BIOS boot
/dev/sdc15  10240    227327    217088  106M EFI System

Partition table entries are not in disk order.

=> the partition table seems in weird shape, which could explain a lot. => Please note the VM generation is v1 which could also have an impact (because not an UEFI boot as per https://learn.microsoft.com/en-us/azure/virtual-machines/generation-2)

[dduportal]

Closing in favor of https://github.com/jenkins-infra/helpdesk/issues/3535