Improve helix-machines Infrastructure

[garath]

Business objectives

• [ ] Telemetry and monitoring is in place to understand the image generation status and current versions deployed for any given Helix queue / hosted pool image.
• [ ] The image generation process allows for atomic deployments of individual images as they are ready for update. Any image that is ready for deployment will be deployed, and does not depend on the status of other images.
• [ ] Align with 1ES offerings by moving VM image generation for Build and test machines into 1ES managed images, in an effort to align our infrastructure with Microsoft.
• [ ] Artifacts are always kept up to date both in on-prem and VM scenarios.
• [ ] A process is in place for dealing with individual image generation failures.
• [ ] Separate base OS image updates from artifact updates. Base OS image updates are handled by 1ES, while artifact updates are still handled by our team.
• [ ] CG/S360 compliance and security fixes to base images happen in lockstep with 1ES.
• [ ] Stop using deprecated hosting models by moving infrastructure to 1ES managed subscriptions.
• [ ] Improve the reliability of PR and official builds, especially through use of the 1ES APIs to leverage broader learnings about image generation and deployment

One Pager

use 1es managed images for image generation in helix machines dnceng 536

[dougbu]

@riarenas b/4 the break, we chatted about expanding the name of this epic to cover more than "image generation". Some of the fixes suggested in the associated milestone are more about "Improve the reliability of PR and official build" (plus usability of the repo) in general than "Image generation infrastructure" in particular. I somewhat prefer the previous "Helix Machines Improvements" name. Thoughts❔

Separately, should we close the #229 and #536 epics and, for #536, fold the still-open issues into this epic❔

[riarenas]

I don't have a strong preference for the name of the epic, so I'm good with reverting the name if we think that covers the work better. I'll deal with the other epics!

[dougbu]

I don't have a strong preference for the name of the epic, so I'm good with reverting the name if we think that covers the work better.

didn't actually revert the name b/c "infrastructure" seems core to this work.

I'll deal with the other epics!

thanks. I also closed #229 and the milestone for #536

[dougbu]

I'm adding more issues to this epic. While I'm trying to avoid issues that may (pending Triage) fit better under Ops or perhaps a general security push, the general feeling I have is most will be ease of use (i.e., improve our dev work in the repo) or speed and reliability (i.e., improve the pipelines) issues.

[dougbu]

/cc @markwilkie ^^