Deploy Cluster without Kube-Proxy

[chasewilson]

Customer would like to deploy Cluster without Kube-Proxy.

Example scenario:

Customers would like to deploy a cluster with BYO CNI + Cilium and not have the cluster contain Kueb-Proxy.

Stems from #1846

[thewisenerd]

currently, just specifying

kubeProxyConfig:
  enabled: false

throws a validation error

{"code": "KubeProxyConfigDisabledBYOCNIOnly", "message": "Preflight validation check for resource(s) for container service {myAks} in resource group {myResourceGroup} failed. Message: Disabling kube-proxy is only allowed for bring your own (BYO) CNI clusters. For more information, please check https://aka.ms/byo-cni. Details: "}

is this check going to go away?

[bartwitkowski]

@chasewilson Below command create AKS with BYO CNI and without kube-proxy:

kube-proxy-config.json
{
  "enabled": false,
  "mode": "IPVS",
  "ipvsConfig": {
    "scheduler": "LeastConnection",
    "TCPTimeoutSeconds": 900,
    "TCPFINTimeoutSeconds": 120,
    "UDPTimeoutSeconds": 300
  }
}

az aks create -n $clusterName -g $resourceGroup --location $location `
    --kube-proxy-config .\kube-proxy-config.json `
    --nodepool-name "system" `
    --vm-set-type VirtualMachineScaleSets `
    --node-count 2 `
    --node-vm-size Standard_B2as_v2 `
    --max-pods 50 `
    --load-balancer-sku standard `
    --enable-private-cluster `
    --dns-name-prefix $clusterName `
    --kubernetes-version 1.29.7 `
    --vnet-subnet-id $podSubnetId `
    --service-cidr "10.255.0.0/16" `
    --dns-service-ip "10.255.0.10" `
    --network-plugin none

@thewisenerd And I believe that is correct. If you use BYO CNI then you can disable installing kube-proxy. With Azure CNI powered by Cilium kube-proxy is not being installed by default.

[kc8421]

@chasewilson Below command create AKS with BYO CNI and without kube-proxy:

kube-proxy-config.json
{
  "enabled": false,
  "mode": "IPVS",
  "ipvsConfig": {
    "scheduler": "LeastConnection",
    "TCPTimeoutSeconds": 900,
    "TCPFINTimeoutSeconds": 120,
    "UDPTimeoutSeconds": 300
  }
}

az aks create -n $clusterName -g $resourceGroup --location $location `
  --kube-proxy-config .\kube-proxy-config.json `
  --nodepool-name "system" `
  --vm-set-type VirtualMachineScaleSets `
  --node-count 2 `
  --node-vm-size Standard_B2as_v2 `
  --max-pods 50 `
  --load-balancer-sku standard `
  --enable-private-cluster `
  --dns-name-prefix $clusterName `
  --kubernetes-version 1.29.7 `
  --vnet-subnet-id $podSubnetId `
  --service-cidr "10.255.0.0/16" `
  --dns-service-ip "10.255.0.10" `
  --network-plugin none

@thewisenerd And I believe that is correct. If you use BYO CNI then you can disable installing kube-proxy. With Azure CNI powered by Cilium kube-proxy is not being installed by default.

That is for sure the known (but not well documented) workaround, but there are few things making it troublesome. Firstly it does depend on #1846 which is not in GA and its latest update was "GA ETA: Retiring" due to Cilium, which at least in my mind does create worst case scenario that it might be removed in future API changes. Secondly Terraform for example doesn't get from API proper information to be able to track its state. Third, deploying Cilium with AKS with built-in feature, does deploy very old version of it. I and my customers would prefer controlling Cilium version independently with Helm as we are doing already.