Support for kube-proxy-config

EppO commented 1 year ago

Is there an existing issue for this?

[X] I have searched the existing issues

Community Note

Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

In order to setup kube-proxy in IPVS mode (instead of iptables) or completely disable kube-proxy (BYOCNI use case) at provisioning time, you need to specify a custom kube-proxy-config when creating the AKS cluster.

New or Affected Resource(s)/Data Source(s)

azurerm_kubernetes_cluster

Potential Terraform Configuration

resource "azurerm_kubernetes_cluster" "example" {
  name                = "example-aks1"
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name
  dns_prefix          = "exampleaks1"

  default_node_pool {
    name       = "default"
    node_count = 1
    vm_size    = "Standard_D2_v2"
  }

  kube_proxy_config {
    enabled = true
    mode = "IPVS"
    ipvs_config {
      scheduler = "LeastConnection"
      tcp_timeout_seconds =  900
      tcp_fin_timeout_seconds = 120
      udp_timeout_seconds = 300
    }
  }

  identity {
    type = "SystemAssigned"
  }

  tags = {
    Environment = "Production"
  }
}

References

https://learn.microsoft.com/en-us/azure/aks/configure-kube-proxy

stephybun commented 1 year ago

Closed by #19567

brunomiguel-teixeira commented 1 year ago

Is there any news here?

It seems it was merged and then reverted https://github.com/hashicorp/terraform-provider-azurerm/pull/20063 It didnt give much context

stephybun commented 1 year ago

The context is in the original PR https://github.com/hashicorp/terraform-provider-azurerm/pull/19567

sycured commented 8 months ago

Do you have any plan/ETA for this support? I'm looking to create AKS clusters with BYOCNI and without kube-proxy because Cilium can fully replace it

tynsh commented 7 months ago

It seems like modifying the configuration is supported using the API now.

janlauber-swissgrid commented 4 months ago

What is the plan to support this feature in the provider?

acuteaura commented 4 months ago

Unfortunately very often it boils down to "whenever someone with commit access finds time to look at 2 year old issues at random" or "whenever a hashicorp enterprise customer asks for it". And usually a PR doesn't change the priorities too much.

We just wrapped our Terraform (tofu now) in nushell and ensure the kube-proxy-config from there.

flavio-fernandes commented 4 months ago

It seems like modifying the configuration is supported using the API now.

Can you please clarify or give an example? I attempted it and it did not seem to work at all. I even tried using the az command:

$ az aks update -g $RG -n $CL --kube-proxy-config disable.json
unrecognized arguments: --kube-proxy-config disable.json

Examples from AI knowledge base:
az aks update --resource-group MyResourceGroup --name MyManagedCluster --load-balancer-managed-outbound-ip-count 2
Update a kubernetes cluster with standard SKU load balancer to use two AKS created IPs for the load balancer outbound connection usage.

az aks update --resource-group MyResourceGroup --name MyManagedCluster --api-server-authorized-ip-ranges 0.0.0.0/32
Restrict apiserver traffic in a kubernetes cluster to agentpool nodes.

az version
Show the versions of Azure CLI modules and extensions in JSON format by default or format configured by --output (autogenerated)

https://docs.microsoft.com/en-US/cli/azure/aks#az_aks_update
Read more about the command in reference docs

jusdor commented 4 months ago

Hi, I think your command looks fine in essence, but there are some prerequisites which need to be met:

If using the Azure CLI, you need the aks-preview extension. See Install the aks-preview Azure CLI extension.
If using ARM or the REST API, the AKS API version must be 2022-08-02-preview or later.
You need to register the KubeProxyConfigurationPreview feature flag. See Register the KubeProxyConfigurationPreview feature flag.

See Azure docs

Pionerd commented 4 months ago

Not the prettiest code I ever wrote, but it gets the job done (sorry I mixed and matched the below from different tf files, please adjust for your own situation)

# Enable Preview functionalities for AKS
resource "null_resource" "enable_kube_proxy_config" {
  provisioner "local-exec" {
    command = "az feature register --namespace \"Microsoft.ContainerService\" --name \"KubeProxyConfigurationPreview\" --subscription ${local.merged["azurerm_subscription_id"]}"
    when    = create
  }
}
resource "null_resource" "enable_encryption_at_host" {
  provisioner "local-exec" {
    command = "az feature register --name EncryptionAtHost  --namespace Microsoft.Compute --subscription ${local.merged["azurerm_subscription_id"]}"
    when    = create
  }
}
resource "null_resource" "check_kube_proxy" {
  provisioner "local-exec" {
    command = "while  [ \"$(az feature show --namespace \"Microsoft.ContainerService\" --name \"KubeProxyConfigurationPreview\" -o json --query 'properties.state' --subscription ${local.merged["azurerm_subscription_id"]})\" != '\"Registered\"' ]; do sleep 10 ; done"
    when    = create
  }
  depends_on = [null_resource.enable_kube_proxy_config]
}
resource "null_resource" "check_encryption_at_host" {
  provisioner "local-exec" {
    command = "while  [ \"$(az feature show --name EncryptionAtHost --namespace Microsoft.Compute -o json --query 'properties.state' --subscription ${local.merged["azurerm_subscription_id"]})\" != '\"Registered\"' ]; do sleep 10 ; done"
    when    = create
  }
  depends_on = [null_resource.enable_encryption_at_host]
}

resource "null_resource" "renew_containerservice" {
  provisioner "local-exec" {
    command = "az provider register --namespace Microsoft.ContainerService --subscription ${local.merged["azurerm_subscription_id"]}"
    when    = create
  }
  depends_on = [null_resource.check_kube_proxy]
}

resource "null_resource" "renew_compute" {
  provisioner "local-exec" {
    command = "az provider register --namespace Microsoft.Compute --subscription ${local.merged["azurerm_subscription_id"]}"
    when    = create
  }
  depends_on = [null_resource.check_encryption_at_host]
}

######## other file inside module

# After cluster creation, it can take a few seconds for the Control Plane to become available.
resource "time_sleep" "wait_60_seconds" {
  count = var.network_plugin == "none" ? 1 : 0
  #TODOAKS check in private cluster
  depends_on = [
    module.aks,
    azurerm_kubernetes_cluster_node_pool.failover,
    azurerm_kubernetes_cluster_node_pool.spot
  ]

  create_duration = "60s"
}

data "azurerm_subscription" "current" {
}

resource "null_resource" "disable_kube_proxy" {
  count = var.network_plugin == "none" ? 1 : 0
  provisioner "local-exec" {
    command = "az aks update -g ${var.resource_group_name} -n ${var.cluster_name} --kube-proxy-config ${path.module}/kube-proxy.json --subscription ${data.azurerm_subscription.current.subscription_id}"
    when    = create
  }

  depends_on = [time_sleep.wait_60_seconds[0]]
}

######### json

{
  "enabled": false
}

flavio-fernandes commented 4 months ago

Not the prettiest code I ever wrote, but it gets the job done (sorry I mixed and matched the below from different tf files, please adjust for your own situation)

# Enable Preview functionalities for AKS
resource "null_resource" "enable_kube_proxy_config" {
  provisioner "local-exec" {
    command = "az feature register --namespace \"Microsoft.ContainerService\" --name \"KubeProxyConfigurationPreview\" --subscription ${local.merged["azurerm_subscription_id"]}"
    when    = create
  }
}
resource "null_resource" "enable_encryption_at_host" {
  provisioner "local-exec" {
    command = "az feature register --name EncryptionAtHost  --namespace Microsoft.Compute --subscription ${local.merged["azurerm_subscription_id"]}"
    when    = create
  }
}
resource "null_resource" "check_kube_proxy" {
  provisioner "local-exec" {
    command = "while  [ \"$(az feature show --namespace \"Microsoft.ContainerService\" --name \"KubeProxyConfigurationPreview\" -o json --query 'properties.state' --subscription ${local.merged["azurerm_subscription_id"]})\" != '\"Registered\"' ]; do sleep 10 ; done"
    when    = create
  }
  depends_on = [null_resource.enable_kube_proxy_config]
}
resource "null_resource" "check_encryption_at_host" {
  provisioner "local-exec" {
    command = "while  [ \"$(az feature show --name EncryptionAtHost --namespace Microsoft.Compute -o json --query 'properties.state' --subscription ${local.merged["azurerm_subscription_id"]})\" != '\"Registered\"' ]; do sleep 10 ; done"
    when    = create
  }
  depends_on = [null_resource.enable_encryption_at_host]
}

resource "null_resource" "renew_containerservice" {
  provisioner "local-exec" {
    command = "az provider register --namespace Microsoft.ContainerService --subscription ${local.merged["azurerm_subscription_id"]}"
    when    = create
  }
  depends_on = [null_resource.check_kube_proxy]
}

resource "null_resource" "renew_compute" {
  provisioner "local-exec" {
    command = "az provider register --namespace Microsoft.Compute --subscription ${local.merged["azurerm_subscription_id"]}"
    when    = create
  }
  depends_on = [null_resource.check_encryption_at_host]
}

######## other file inside module

# After cluster creation, it can take a few seconds for the Control Plane to become available.
resource "time_sleep" "wait_60_seconds" {
  count = var.network_plugin == "none" ? 1 : 0
  #TODOAKS check in private cluster
  depends_on = [
    module.aks,
    azurerm_kubernetes_cluster_node_pool.failover,
    azurerm_kubernetes_cluster_node_pool.spot
  ]

  create_duration = "60s"
}

data "azurerm_subscription" "current" {
}

resource "null_resource" "disable_kube_proxy" {
  count = var.network_plugin == "none" ? 1 : 0
  provisioner "local-exec" {
    command = "az aks update -g ${var.resource_group_name} -n ${var.cluster_name} --kube-proxy-config ${path.module}/kube-proxy.json --subscription ${data.azurerm_subscription.current.subscription_id}"
    when    = create
  }

  depends_on = [time_sleep.wait_60_seconds[0]]
}

######### json

{
  "enabled": false
}

First of all, thank you very much for writing this here, @Pionerd !

However, one caveat in your example I still cannot get past is that az aks update does not accept the --kube-proxy-config flag.

The doc on that may be wrong wrong because if I were to try it, I can clearly see it does not work. That flag is only available on creation, right?

$ export RG=$(terraform output -raw resource_group_name) ; \
  export CL=$(terraform output -raw kubernetes_cluster_name)

$ cat <<EOT >> disable.json
{
  "enabled": false
}
EOT

$ az version
{
  "azure-cli": "2.58.0",
  "azure-cli-core": "2.58.0",
  "azure-cli-telemetry": "1.1.0",
  "extensions": {}
}
$ az aks update -g $RG -n $CL --kube-proxy-config disable.json --subscription ${TF_VAR_subscription_id}
unrecognized arguments: --kube-proxy-config disable.json

Examples from AI knowledge base:
az aks update --resource-group MyResourceGroup --name MyManagedCluster --load-balancer-managed-outbound-ip-count 2
Update a kubernetes cluster with standard SKU load balancer to use two AKS created IPs for the load balancer outbound connection usage.

az aks update --resource-group MyResourceGroup --name MyManagedCluster --api-server-authorized-ip-ranges 0.0.0.0/32
Restrict apiserver traffic in a kubernetes cluster to agentpool nodes.

az version
Show the versions of Azure CLI modules and extensions in JSON format by default or format configured by --output (autogenerated)

https://docs.microsoft.com/en-US/cli/azure/aks#az_aks_update
Read more about the command in reference docs

Or, am I missing something?

https://learn.microsoft.com/en-US/cli/azure/aks?view=azure-cli-latest#az-aks-update

Pionerd commented 4 months ago

I do not see any reference of you installing (or having enabled) the preview functionality in your cli installation

flavio-fernandes commented 4 months ago

I do not see any reference of you installing (or having enabled) the preview functionality in your cli installation

Does that look right? Maybe I'm not being patient enough?

$ az feature register --namespace Microsoft.ContainerService --name KubeProxyConfigurationPreview --subscription ${TF_VAR_subscription_id}
Once the feature 'KubeProxyConfigurationPreview' is registered, invoking 'az provider register -n Microsoft.ContainerService' is required to get the change propagated
{
  "id": "/subscriptions/XXX/providers/Microsoft.Features/providers/Microsoft.ContainerService/features/KubeProxyConfigurationPreview",
  "name": "Microsoft.ContainerService/KubeProxyConfigurationPreview",
  "properties": {
    "state": "Registering"
  },
  "type": "Microsoft.Features/providers/features"
}

$ az feature register --name EncryptionAtHost  --namespace Microsoft.Compute --subscription ${TF_VAR_subscription_id}
Once the feature 'EncryptionAtHost' is registered, invoking 'az provider register -n Microsoft.Compute' is required to get the change propagated
{
  "id": "/subscriptions/XXX/providers/Microsoft.Features/providers/Microsoft.Compute/features/EncryptionAtHost",
  "name": "Microsoft.Compute/EncryptionAtHost",
  "properties": {
    "state": "Registering"
  },
  "type": "Microsoft.Features/providers/features"
}
$ az provider register --namespace Microsoft.ContainerService --subscription ${TF_VAR_subscription_id}
$ az provider register --namespace Microsoft.Compute --subscription ${TF_VAR_subscription_id}

$ export RG=$(terraform output -raw resource_group_name) ; \
>   export CL=$(terraform output -raw kubernetes_cluster_name)
$ cat <<EOT >> disable.json
> {
>   "enabled": false
> }
> EOT
$ k get pod -A | grep -i prox
kube-system      kube-proxy-6bg6v                      1/1     Running   0               8m26s
kube-system      kube-proxy-fv98d                      1/1     Running   0               8m26s
kube-system      kube-proxy-qw7vn                      1/1     Running   0               8m27s

$ az aks update -g $RG -n $CL --kube-proxy-config disable.json --subscription ${TF_VAR_subscription_id}
unrecognized arguments: --kube-proxy-config disable.json

Examples from AI knowledge base:
az aks update --resource-group MyResourceGroup --name MyManagedCluster --load-balancer-managed-outbound-ip-count 2
Update a kubernetes cluster with standard SKU load balancer to use two AKS created IPs for the load balancer outbound connection usage.

az aks update --resource-group MyResourceGroup --name MyManagedCluster --api-server-authorized-ip-ranges 0.0.0.0/32
Restrict apiserver traffic in a kubernetes cluster to agentpool nodes.

az version
Show the versions of Azure CLI modules and extensions in JSON format by default or format configured by --output (autogenerated)

https://docs.microsoft.com/en-US/cli/azure/aks#az_aks_update
Read more about the command in reference docs
$

Pionerd commented 4 months ago

Please check the check_kube_proxy resource.

The above code is being used on a daily basis to create new clusters, so it should work. If it still dysfunctions, only thing I can think of is an old az cli version.

flavio-fernandes commented 4 months ago

Please check the check_kube_proxy resource.

The above code is being used on a daily basis to create new clusters, so it should work. If it still dysfunctions, only thing I can think of is an old az cli version.

You are correct, @Pionerd . I was missing a step to enable the aks-preview extension. I was able to make it work, but remain wishing that (1) Terraform supported it natively in the azurerm_kubernetes_cluster resource and (2) that was not a preview.

Ref: https://github.com/MicrosoftDocs/azure-docs/issues/120608

$ kubectl get pod -A | grep prox
kube-system   kube-proxy-2q2vp                      1/1     Running   0          6m50s
kube-system   kube-proxy-8fvkg                      1/1     Running   0          6m55s
kube-system   kube-proxy-bvfgx                      1/1     Running   0          6m44s

$ az extension update --name aks-preview --allow-preview true
The extension aks-preview is not installed. Please install the extension via `az extension add -n aks-preview`.

$ az extension add --name aks-preview --allow-preview true
The installed extension 'aks-preview' is in preview.

$ az feature register --namespace "Microsoft.ContainerService" --name "KubeProxyConfigurationPreview"
Once the feature 'KubeProxyConfigurationPreview' is registered, invoking 'az provider register -n Microsoft.ContainerService' is required to get the change propagated
{
  "id": "/subscriptions/XXX/providers/Microsoft.Features/providers/Microsoft.ContainerService/features/KubeProxyConfigurationPreview",
  "name": "Microsoft.ContainerService/KubeProxyConfigurationPreview",
  "properties": {
    "state": "Registered"
  },
  "type": "Microsoft.Features/providers/features"
}

$ az provider register --namespace Microsoft.ContainerService

Working!

$ export RG=$(terraform output -raw resource_group_name)
$ export CL=$(terraform output -raw kubernetes_cluster_name)

$ cat <<EOT > disable.json
{
  "enabled": false
}
EOT

$ az aks update -g $RG -n $CL --kube-proxy-config disable.json --subscription ${TF_VAR_subscription_id}

The behavior of this command has been altered by the following extension: aks-preview
...
  "workloadAutoScalerProfile": {
    "keda": null,
    "verticalPodAutoscaler": null
  }
}
$ echo $?
0
$ kubectl get pod -A | grep prox | wc -l
0

MaxAnderson95 commented 2 months ago

Here is a workaround using the azapi provider instead:

resource "azapi_update_resource" "kube_proxy_disabled" {
  resource_id = azurerm_kubernetes_cluster.aks.id
  type        = "Microsoft.ContainerService/managedClusters@2024-02-02-preview"
  body = jsonencode({
    properties = {
      networkProfile = {
        kubeProxyConfig = {
          enabled = false
        }
      }
    }
  })
}

hashicorp / terraform-provider-azurerm