[ci.jenkins.io] Use a new VM instance type

[dduportal]

What is the problem?

The current VM for ci.jenkins.io starts to show issues:

• Latency for packed builds such as https://github.com/jenkinsci/bom/pull/1969#issuecomment-1512932582
• Latency on the system disk: https://github.com/jenkins-infra/helpdesk/issues/3526
• Network latency and errors: https://github.com/jenkins-infra/helpdesk/issues/3481#issuecomment-1497021548

Also, this VM was sized a few years ago with a slighlty different context: JDK8 for running the controller (e.g. less CPU usage but more memory usage), no UEFI bootloader (v1 generation), Ubuntu 18.04.

Finally, managing this VM is manually managed for the infrasrtucture layer (initially created with Terraform, but then changed to manual management).

What should we do

There are numerous tasks for this VM:

• Fixing its system disk for latencies
• Checking the correct sizing (with the JDK11 assumption today, and then JDK17 with even less memory used)
• Migrating this VM to Ubuntu 22.04 as per #2982
• Importing this VM in the jenkins-infra/azure Terraform management (https://github.com/jenkins-infra/helpdesk/issues/3527 sugests to import it in Terraform again. and #2981
• Migrate to a version2 generation

How could we do it

Proposal: to avoid any maintenance overhead and migration risk, the infra team thought of the following plan:

• Create a brand new VM for ci.jenkins.io:
  • With Terraform from the beginning, using modern syntax as per @smerle33 's work in #1101
  • Use an instance with better sizing (e.g. more CPUs or at least more powerfull, less RAM, etc.) with generation V2 and a brand new SSD
  • Based on Ubuntu 22.04
  • Already in the new https://github.com/jenkins-infra/helpdesk/issues/3257 public network with no overlap nor performance issue and closer to the Azure ACP and IPv6 support
  • Managed by puppet with the name ci.jenkins.io (reminder: ci.jenkins.io used to be the named under Puppet for the AWS VM, the current name of the Azure VM is azure.ci.jenkins.io)

=> this would avoid disrupting the current ci.jenkins.io service until the effective migration

Validation steps would be:

• Checking an empty controller is set up under Ubuntu 22.04 by Puppet management

• Checking that a manual bom build (https://github.com/jenkinsci/bom/pull/1969#issuecomment-1512932582) is not worse than the current (ideally, it should have a bit less overhead for sh steps)

• Then, the next step would be to add a data disk, smaller than the current one (thanks to #3492 and #3496 we are able to use less disk space) to decrease costs

  • Testing with an initial snapshot from today's datadisk
  • ⚠️ find a way to block outgoing requests to github.com to avoid updating build status for nothing, as it could be disruprtive for developers

• Finally migrating the service:

  • announce to developers
  • stop ci.jenkins.io service
  • rsync the content of jenkins_home (incremental copy)
  • migrate the public IP

[dduportal]

Should use a backup policy (merged #3527):

The goal is to ensure we have a daily backup of the JENKINS_HOME of ci.jenkins.io

Azure provides a Backup System, than can be used specifically for managed disks such as this one: https://learn.microsoft.com/en-us/azure/backup/backup-managed-disks.

We don't (and should not) need a VM-level backup as we use Puppet to manage the system: disaster recovery for ci.jenkins.io is to install a blank new VM and mount the resotre of the datadisk for Jenkins.

As per https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/data_protection_backup_instance_disk, we can define this using Terraform which implies importing ci.jenkins.io VM once for all.

A word about encryption:

• The backup vault is, like the VM disks, encrypted at rest with an Azure PMK key (hardware level).
• We can keep this behavior (encryption at rest with PMK) for the backup, as ci.jenkins.io deos not have any senstivie data (eventually credentials for GH org, but that is all).
• Note: This encryption could be provided a custom key private for sensitvie backups such as trusted.ci's

[dduportal]

• Creating the new VM in Terraform: https://github.com/jenkins-infra/azure/pull/348
• Setting up security groups to allow access: https://github.com/jenkins-infra/azure/pull/349

[dduportal]

Delaying as it's blocked by the network peering accesses part of #3351 and by the work on the new trusted machines in #3486

[dduportal]

Current status: we have to set up the Azure VM agents to inbound mode (thanks to https://github.com/jenkinsci/azure-vm-agents-plugin/pull/406) to ensure that migrating either the controller first or the ephemeral agents first won't break the connections (otherwise we would need to use public IPs for SSH agent temporarly)

[dduportal]

Update: ci.jenkins.io is now using inbound agent running from the new virtual network.

• Created required resources for the ephemeral agents in the network public-vnet (without overlap):
  • https://github.com/jenkins-infra/azure/pull/413
  • https://github.com/jenkins-infra/azure/pull/416
• Switched the Azure VM agents of ci.jenkins.io to the new network, with the inbound mode (required to reach the current ci.jenkins.io controller)
  • https://github.com/jenkins-infra/jenkins-infra/pull/2925
  • https://github.com/jenkins-infra/jenkins-infra/pull/2926
  • https://github.com/jenkins-infra/jenkins-infra/pull/2927
  • https://github.com/jenkins-infra/jenkins-infra/pull/2928

Watching the builds (ping @lemeurherve not urgent but I'll try to check the CI integration in datadog to see if any pattern arise here - #3573)

[dduportal]

Next step: bootstraping a fully operational VM for the new ci.jenkins.io

• [x] Prepare bootstrap - https://github.com/jenkins-infra/azure/pull/421 (with fixups of course)
  • https://github.com/jenkins-infra/azure/pull/422
  • https://github.com/jenkins-infra/azure/commit/d2c2c162a7ae0c3594e9718a71a70b8294cd0952
  • https://github.com/jenkins-infra/azure/commit/2f16ff85fb80db8902d0effdc6ded65e9690fd19
  • https://github.com/jenkins-infra/azure/commit/fb06dcb0e63ddcc6d8f569abad7db41e12bb181d
• [x] Announce
  • [x] to developers (no BOM builds during the migration) on mailing list - https://groups.google.com/g/jenkinsci-dev/c/WS8r1ZVdlec
  • [x] in status.jenkins.io - https://github.com/jenkins-infra/status/pull/340
• [x] Stop ci.jenkins.io VM (to allow data to be frozen)
• [x] Initial Puppet agent run
  • https://github.com/jenkins-infra/jenkins-infra/pull/2950
  • fixup: disable letsencrypt - https://github.com/jenkins-infra/jenkins-infra/commit/a09eaa5e606a0dc27d88c9de09432e447effe283
  • fixup - Docker-CE to jammy - https://github.com/jenkins-infra/jenkins-infra/commit/cc9456293f0fa5a5c87a9614ee5fd6f744dbf5ce
• [-] Initial migration of the data with a snapshot, manually mounted in the bootsrapped VM
  • [x] Snapshot of prod-ci 's data disk created
  • [x] Disk created from the snapshot, in the new resource group
  • [x] Mounting disk on the new VM
  • [-] Rsync in the machine of both disks (in progress ⌚️)

⚠️ The old JENKINS_HOME seems to have inode issues (wether the old disk or the snapshots). Despite copying the full disk yesterday, the rsync sees all files as changed: the copy is being done but will take multiple hours (~ 12 hours). ci.jenkins.io will remain down until 5 July.

• [x] Restart of the service
• [x] Migrate DNS
• [x] Puppet run with letsencrypt

[dduportal]

Update (4th of July):

• Migration finished late (european time)
• Removed the cache*/, **/*.tmp, plugins/, war/ and log*/* elements from the new JENKINS_HOME
• Service started with success ("Jenkins is up and running") and valid locally (curl -v http://localhost:8080)
• ci.jenkins.io CNAME record created manually to allow letsencrypt
  • Initial failure as Apache expecte the file to be there. Copied the /etc/letsencrypt from former VM
  • letsencrypt enabled with success after this
• Agent failures due to missing NSGs: added manually
• Missing plugins which were not part of the puppet "as code" setup
  • startup works (e.g. all JCasc required plugins are defined as code)
  • It's only missing features or pipeline runtime errors (No such DSL method 'xxxx' found among steps <...>)

[dduportal]

Update (5th of July):

• Fixups in the Puppet code:

  • https://github.com/jenkins-infra/jenkins-infra/commit/b4093a22defa25a47c582853d15c6d44ab5b32f1 (hotfix to enable letsencrypt)
  • https://github.com/jenkins-infra/jenkins-infra/pull/2954 => Ubuntu 22.04 only for Docker-CE packages
  • https://github.com/jenkins-infra/jenkins-infra/pull/2955 => ensure no more EC2 plugins/setup
  • https://github.com/jenkins-infra/jenkins-infra/pull/2958 => missing plugin
  • https://github.com/jenkins-infra/jenkins-infra/pull/2959 => missing plugin
  • https://github.com/jenkins-infra/jenkins-infra/pull/2960 => enable AWS Cloud (to allow BOM builds)
  • https://github.com/jenkins-infra/jenkins-infra/pull/2962 => missing plugin

• Fixups in the Azure Terraform:

  • https://github.com/jenkins-infra/azure/pull/422 => NSGs fixup
  • Yet another NSG fixup: https://github.com/jenkins-infra/azure/commit/d2c2c162a7ae0c3594e9718a71a70b8294cd0952
  • Yet another NSG fixup: https://github.com/jenkins-infra/azure/commit/2f16ff85fb80db8902d0effdc6ded65e9690fd19
  • Yet another NSG fixup: https://github.com/jenkins-infra/azure/commit/fb06dcb0e63ddcc6d8f569abad7db41e12bb181d
  • https://github.com/jenkins-infra/azure/pull/423 => NSG and DNS records

[dduportal]

Update (5th of July)

• Problems yet to be solved caused by the migration:

  • https://github.com/jenkins-infra/helpdesk/issues/3648
  • https://github.com/jenkins-infra/helpdesk/issues/3650

• Cleanup/Post migration tasks:

  • [x] Update runbooks - https://github.com/jenkins-infra/runbooks/commit/69772fbdf0fef4e360c9fefea79255c4d17c600a
  • [x] Close operation
  • [x] status - https://github.com/jenkins-infra/status/pull/341
  • [x] email thread

[dduportal]

Todo list to close this issue:

• [x] Move the ACI workload to the new resource group and network - https://github.com/jenkins-infra/jenkins-infra/pull/2974

• [x] Cleanup leftovers of the former VM in code (azure.ci.jenkins.io)

  • https://github.com/jenkins-infra/jenkins-infra/pull/2973

• [x] Remove the DNS records pointing to 10.0.2.4 (or add to terraform for "as code" management and update them)

  • [x] Removed ci.private.jenkins.io
  

• [x] Create a snapshot of the 2 disks of the old VM in the new resource group `` (to be archived later as per <>):

  • [x] Rootfs if need to retrieve older system files such as letsencrypt certificates
  • [x] Datadisk (of the old JENKINS_HOME)
  

• [x] Remove the PRODJENKINSCI resource group and its resources (old VM, 2 old disks, old network interface, old public_ip)

  

• [x] Remove the former resource group of the agents (should only contains the storage account of the former azure VM agents once ACI workload is migrated) - https://github.com/jenkins-infra/azure/pull/434

[dduportal]

Closing the issue as the work is finished