search

MicrosoftDocs / azure-docs

Open source documentation of Microsoft Azure
https://docs.microsoft.com/azure
Creative Commons Attribution 4.0 International
10.21k stars 21.36k forks source link

AKS with virtual nodes limitations #113892

Closed vivuu1989 closed 2 months ago

vivuu1989 commented 1 year ago

We are using AKS v1.26 and current AKS clusters are configured with AZURE CNI network policy and using the vmms based nodepools. we have Istio service mesh enabled with this cluster and using CSI driver plugin to mount secrets to the pods from keyvault. Also we have a number of daemonset pods running which needed for secretstore mounting and metric server based plugns etc..

Now we have a plan to use the Virtual nodes instead of the current nodepools and when we went through the MS docs, we are not sure whether the virtualnodes are production ready as its having a lot of limitations as listed below.

Using service principal to pull ACR images. Workaround is to use Kubernetes secrets
Virtual Network Limitations including VNet peering, Kubernetes network policies, and outbound traffic to the internet with network security groups.
Init containers
Host aliases
Arguments for exec in ACI
DaemonSets won't deploy pods to the virtual nodes
Virtual nodes support scheduling Linux pods. You can manually install the open source Virtual Kubelet ACI provider to schedule Windows Server containers to ACI.
Virtual nodes require AKS clusters with Azure CNI networking.
Using api server authorized ip ranges for AKS.
Volume mounting Azure Files share support General-purpose V2 and General-purpose V1. However, virtual nodes currently don't support Persistent Volumes and Persistent Volume Claims. Follow the instructions for mounting a volume with Azure Files share as an inline volume.
Using IPv6 isn't supported.
Virtual nodes don't support the Container hooks feature.

We have following query while planning for the virtual node implementation but the docs are not giving much information.

Document link: https://learn.microsoft.com/en-us/azure/aks/virtual-nodes#known-limitations

Document Details

⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

RamanathanChinnappan-MSFT commented 1 year ago

@vivuu1989 It would be great if you could add a link to the documentation you are following for these steps? This would help us redirect the issue to the appropriate team. Thanks!!

vivuu1989 commented 1 year ago

@vivuu1989 It would be great if you could add a link to the documentation you are following for these steps? This would help us redirect the issue to the appropriate team. Thanks!! https://learn.microsoft.com/en-us/azure/aks/virtual-nodes#known-limitations Documents link added.

RamanathanChinnappan-MSFT commented 1 year ago

@vivuu1989 Thanks for your feedback! We will investigate and update as appropriate.

SaibabaBalapur-MSFT commented 1 year ago

@vivuu1989 Thanks for bringing this to our attention. I'm going to assign this to the document author so they can take a look at it accordingly.

@MGoedtel Can you please check and add your comments on this doc update request as applicable.

vivuu1989 commented 1 year ago

Any update on the above?

MGoedtel commented 1 year ago

Assigning to #reassign:nickomang

vivuu1989 commented 1 year ago

Its pending for long time. Any update on this?

rayoef commented 2 months ago

Thank you for your dedication to our documentation. Unfortunately, at this time we have been unable to review your issue in a timely manner, and we sincerely apologize for the delayed response. The requested updates have not been made since the creation of this issue, so we've created an internal work item to incorporate your suggestions. We are closing this issue for now, but feel free to comment here as necessary.

please-close