Support for configuring application gateway tier in azurerm_kubernetes_cluster resource

josedev-union commented 8 months ago

Is there an existing issue for this?

[X] I have searched the existing issues

Community Note

Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment and review the contribution guide to help.

Description

TLDR; Affected resource is azurerm_kubernetes_cluster. Purpose is to control WAF(Web Application Firewall) enablement of the AKS cluster's application gateway.

Dive deeper When I deploy an AKS cluster using azurerm_kubernetes_cluster resource, I can create an application gateway by configuring ingress_application_gateway field but it creates Standard_v2 gateway in default and no way to use other tiers, for example, WAF_v2. My expectation is that ingress_application_gateway field has a few more fields like tier and firewall_policy_id so users can create WAF enabled application gateway out of the box during the AKS cluster creation.

New or Affected Resource(s)/Data Source(s)

azurerm_kubernetes_cluster

Potential Terraform Configuration

resource "azurerm_kubernetes_cluster" "example" {
  name                = "example-aks1"
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name

  ingress_application_gateway {
    subnet_cidr = "10.52.1.0/24"
    tier = "WAF_v2"
    firewall_policy_id = azurerm_web_application_firewall_policy.example.id
  }

  tags = {
    Environment = "Production"
  }
}

References

No response

harshavmb commented 8 months ago

Hi @josedev-union ,

Have you tried passing gateway_id? This way I believe you can create app-gw of you choice tier. I don't see the support for this either with az-cli.

az aks enable-addons -a ingress_application_gateway -h
Application Gateway Arguments
    --appgw-id                       : Resource Id of an existing Application Gateway to use with
                                       AGIC. Use with ingress-azure addon.
    --appgw-name                     : Name of the application gateway to create/use in the node
                                       resource group. Use with ingress-azure addon.
    --appgw-subnet-cidr              : Subnet CIDR to use for a new subnet created to deploy the
                                       Application Gateway. Use with ingress-azure addon.
    --appgw-subnet-id                : Resource Id of an existing Subnet used to deploy the
                                       Application Gateway. Use with ingress-azure addon.
    --appgw-watch-namespace          : Specify the namespace, which AGIC should watch. This could be
                                       a single string value, or a comma-separated list of
                                       namespaces.

josedev-union commented 8 months ago

Hi @josedev-union ,

Have you tried passing gateway_id? This way I believe you can create app-gw of you choice tier. I don't see the support for this either with az-cli.

az aks enable-addons -a ingress_application_gateway -h
Application Gateway Arguments
    --appgw-id                       : Resource Id of an existing Application Gateway to use with
                                       AGIC. Use with ingress-azure addon.
    --appgw-name                     : Name of the application gateway to create/use in the node
                                       resource group. Use with ingress-azure addon.
    --appgw-subnet-cidr              : Subnet CIDR to use for a new subnet created to deploy the
                                       Application Gateway. Use with ingress-azure addon.
    --appgw-subnet-id                : Resource Id of an existing Subnet used to deploy the
                                       Application Gateway. Use with ingress-azure addon.
    --appgw-watch-namespace          : Specify the namespace, which AGIC should watch. This could be
                                       a single string value, or a comma-separated list of
                                       namespaces.

@harshavmb thanks for yoru reply. It means we need to create AGW first and let AKS to consume the pre-existing AGW. I think this can be a solution for new clusters, but for our case, the concern is the AKS cluster and AGW were created already by using this one azurerm_kubernetes_cluster resource. I need to check the impacts first when I set gateway_id so the AGW created by azurerm_kubernetes_cluster will be deleted and that gateway will be used. The ideal minimum impact is just the public ip change of AGW but the worst case can lead the whole cluster recreation. But this minimum impact is also too much because several production systems are using that public IP.

aristosvo commented 8 months ago

Hi @josedev-union! Have you raised this question to the AKS team or Azure Support whether this scenario is supported? Please take it to the AKS team or Azure Support first, as we cannot for certain confirm whether this scenario is supported or not.

Based on their response and suggestion how to do it without Terraform we can better decide if it possible to implement this in azurerm.

josedev-union commented 8 months ago

Another problem with the workaround suggested by @harshavmb is we cannot know the auto-generated resource group name of aks cluster in advance. It means we have to change the tf codebase after the cluster and agw creation to set the agw id.

josedev-union commented 8 months ago

az aks enable-addons -a ingress_application_gateway -h
Application Gateway Arguments
    --appgw-id                       : Resource Id of an existing Application Gateway to use with
                                       AGIC. Use with ingress-azure addon.
    --appgw-name                     : Name of the application gateway to create/use in the node
                                       resource group. Use with ingress-azure addon.
    --appgw-subnet-cidr              : Subnet CIDR to use for a new subnet created to deploy the
                                       Application Gateway. Use with ingress-azure addon.
    --appgw-subnet-id                : Resource Id of an existing Subnet used to deploy the
                                       Application Gateway. Use with ingress-azure addon.
    --appgw-watch-namespace          : Specify the namespace, which AGIC should watch. This could be
                                       a single string value, or a comma-separated list of
                                       namespaces.

@aristosvo thanks for your suggestion and here it is https://github.com/Azure/AKS/issues/4050

rcskosir commented 7 months ago

Thanks for taking the time to open this issue. It looks like the behavior you requested is not supported by the underlying Azure API so I am going to label this issue as such and close it for now. Thank you for creating https://github.com/Azure/AKS/issues/4050. When it gets added, we can reopen this request or you can create a new one.

github-actions[bot] commented 4 months ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

hashicorp / terraform-provider-azurerm