[Feature] Enable aks-custom-headers in ARM templates

headconnect commented 12 months ago

Is your feature request related to a problem? Please describe. When wanting to create AKS clusters with GPU support, registering an experimental feature is required - and passing the aks-custom-headers with UseGPUDedicatedVHD=true to node-pool creation.

Describe the solution you'd like I would like the ability to provide aks-custom-headers as a key-value pair in ARM templates so as to be able to leverage experimental features.

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered. This has already been mentioned in https://github.com/Azure/AKS/issues/1638, but the resolution is unsatisfactory - specifically, I would say the resolution is not valid due to new context over time. My exact issue is in fact mentioned in a comment by @slaiyer two days before the issue was automatically closed (I, too, use pulumi to create and manage clusters as part of everything-infrastructure-as-code - and we cannot allow for manual az cli-based operations as that would cause configuration drift. ).

Honestly, alternatives at this point are genuinely dropping AKS and building & managing kubernetes with custom VMs (or simply hosting our kubernetes clusters on another cloud provider). There is currently not sufficient problems with using AKS to do this, but prolonged lack of essential features like these genuinely tip the balance, especially as the world suddenly needs GPUs on everything.

The Daemonset approach is not ideal, as it is somewhat flaky, and highly intrusive (security isn't too happy about it).

Additional context It's important to understand that using experimental features as part of IaC lifecycle is just as important as having features be experimental on the AKS side. Making this available to ARM does not mean that it will be thrown into production - it means that it is possible to verify infrastructure feasibility alongside your experiments, just as you intend by making it available to the public at all. Most organizations who don't care about feature maturity likely maintain their infrastructure manually anyway, so preventing this feature from reaching ARM just excludes those who would use it properly rather than those who use it improperly today.

KarlisAG commented 10 months ago

Yes, I would also be very interested in this feature! Currently we are forced to use az-cli just for this nodepool while nothing else in our IaC uses that. And we don't want to use the DS approach as that resulted in the workload not working as expected