[Feature] Support custom images for nodes in AKS managed clusters

nclaeys commented 1 year ago

Is your feature request related to a problem? Please describe. It would be very useful for us if we could customize the images used by the AKS nodes, as already exists on EKS or GKE. We have 2 main usecases for this:

packaging pre pulled images for our daemonsets/common used images and thus speeding up the startup of a node significantly
building a custom image with sysbox installed, again to drastically speeding up the startup of these nodes (on eks: 8min -> 3min). Now we need to depend on the sysbox daemonset to configure the node but this is really slow when dynamically scaling nodes.

Describe the solution you'd like Allow nodepools to specify a custom image instead of only supporting the aks images. Open source the building of the aks images such that people can extend them.

Describe alternatives you've considered We looked at the CAPI project but do not want to create non-managed clusters. Other alternatives are hard to come by as the AKS nodes are quite closed source and it is very difficult to figure out how to build an Azure image as well as there is no way to configure custom images in nodepools.

esolace88 commented 7 months ago

I agree with this feature add, and if possible, allow incorporating the Azure STIG Templates for Linux - Preview https://learn.microsoft.com/en-us/azure/azure-government/documentation-government-stig-linux-vm. Thanks!

Dhruvin4530 commented 1 month ago

But how to use this in AKS?

esolace88 commented 1 month ago

Hmm good question, maybe via az Cli preview extension like adding an additional option. For customer image maybe accept a VHD for along with OS.SKU flag. For STIG OS allow the STIG template to be used on existing OS.SKU options.

[--os-sku {AzureLinux, CBLMariner, Mariner, Ubuntu}]

AZ Preview https://learn.microsoft.com/en-us/cli/azure/aks?view=azure-cli-latest#az-aks-create(aks-preview)

STIG VM Creation https://learn.microsoft.com/en-us/azure/azure-government/documentation-government-stig-linux-vm

Dhruvin4530 commented 1 month ago

I checked the docs and terraform official AKS module but it is not mentioned anything about custom image 😢

microsoft-github-policy-service[bot] commented 4 weeks ago

Issue needing attention of @Azure/aks-leads

microsoft-github-policy-service[bot] commented 2 weeks ago

Issue needing attention of @Azure/aks-leads

Azure / AKS

[Feature] Support custom images for nodes in AKS managed clusters #3751