riarenas commented 1 year ago

Motivation

Currently, we generate all of our Azure VM images by making calls to the Image Factory service directly. We maintain the infrastructure to do this in the helix-machines repository.

Image factory is currently used for three scenarios:

Generating the non-standard base images like Mariner, VS, and windows: Everything in this folder: https://dnceng.visualstudio.com/internal/_git/dotnet-helix-machines?path=/base-images. We generate these through the Image Factory CLI and the image generation pipeline. These images never use any of our custom artifacts, and use Image factory's base artifact source in order to install all the windows updates and the vs components.
Generating the helix images: These are additional calls to image factory on top of both the public gallery, and the base images we generate above to install all the Helix artifacts and make these machines Helix-ready. This happens in the helix-machines main pipeline, and once they are generated, they are sent to an Azure compute gallery where the different Helix scale sets pull the image versions from.
Generating the 1ES Hosted pool images: This goes through the same path as 2. But once the image is ready, we add it to a 1ES image so that the hosted pools can use them. This is a bit of a waste, because the managed image is only used as a container for the image we already generated manually by calling Image Factory. An example of such image is the build.ubuntu.1804.amd64 1ES image. These 1ES images we generate for hosted pools are of the public gallery type.

Since the time when we adopted Image Factory and 1ES hosted pools, some developments have made it so that we are using unsupported or deprecated models.

Mainly, The Image Factory team is working on a V3 implementation of their service which requires changes in our infrastructure to adopt, and the 1ES team has deprecated our current model for hosting the 1ES hosted pools in favor for a model where the pools are hosted in their managed subscriptions.

1ES offers the service to generate managed images, which:

Make all calls to image factory completely transparent to the user, meaning we don't need to maintain infrastructure for communicating with Image Factory
Support copying the underlying image to VM Compute galleries so that we can use the images for our existing scenarios.
Provide automatic updates to the image when there is an OS update or the artifacts added to the images changes
Can be used as-is with other 1ES infrastructure offerings such as Cloudtest and 1ES hosted pools.

As part of our journey to integrate our infrastructure with 1ES' we should take advantage of this offering and look into reducing our own custom infrastructure.

Business objectives

[ ] Align with 1ES offerings by moving VM image generation for Build and test machines into 1ES managed images, in an effort to align our infrastructure with Microsoft.
[ ] Reduce the number of direct dependencies our services rely on by removing our direct dependency on Image Factory APIs.
[ ] A process is in place for dealing with individual image generation failures.
[ ] Separate base OS image updates from artifact updates. Base OS image updates are handled by 1ES, while artifact updates are still handled by our team.

One Pager

use 1es managed images for image generation in helix machines dnceng 536

Phases

Individual POCs

Test out the individual pieces of 1ES infrastructure and how they fit into the helix-machines image generation.

Verify that we are able to generate a 1ES managed image based on one of the base image definitions in helix-machines, using ARM template and bicep templates. We intend to use custom artifacts that match our helix-machines specifications.
Verify that we can replicate a 1ES image to a Compute Gallery that can be later used by a scale set to create Helix VMs. This will cover the additional steps required for helix-image workflow.

End to end Prototype of Image Generator

Build upon the individual learnings to create an Image generator that uses 1ES managed images end to end instead of calling into image factory, from a test/build image YAML definition to usage inside a hosted pool / helix queue.

Image Migration

Build upon the Image generator to migrate all existing images generated via Image Factory to 1ES managed images

Both systems will run during this phase
Staggered migration of images, starting with the least used
Create alert flow for image generation failures
Create validation pipelines

Decommission Image Factory CLI

Once all traffic is going through 1ES managed images, decommission the Image Factory CLI and the monitoring set up around it.

Timeline

Phase	ETA
Individual PoCs	Completed
End to end Prototype of Image Generator	2/23
Image Migration	3/29
Decommission ImageFactoryCLI	4/12

riarenas commented 11 months ago

This has been merged with epic #376. I added this issue to that milestone for posterity.

dotnet-eng-status[bot] commented 11 months ago

Sorry! Could not close or remove the 'Epic' label from this issue because there are still open issues associated with it. Close or remove open issues in the related milestone and try again.