v4 removed private vnet integration for API server in azurerm_kubernetes_cluster

LiamLeane commented 1 week ago

Is there an existing issue for this?

[X] I have searched the existing issues

Community Note

Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave comments along the lines of "+1", "me too" or "any updates", they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment and review the contribution guide to help.

Terraform Version

Tofu 1.8.2

AzureRM Provider Version

4.4.0

Affected Resource(s)/Data Source(s)

azurerm_kubernetes_cluster

Terraform Configuration Files

resource "azurerm_kubernetes_cluster" "k8s" {
  api_server_access_profile {
    subnet_id                = var.api_server_subnet_id # Does not exist
    vnet_integration_enabled = true # Does not exist
  }
}

Debug Output/Panic Output

│ Error: Unsupported argument
│ 
│   on ..\..\k8.tf line 118, in resource "azurerm_kubernetes_cluster" "k8s":
│  118:     subnet_id                = var.api_server_subnet_id
│ 
│ An argument named "subnet_id" is not expected here.
╵
╷
│ Error: Unsupported argument
│ 
│   on ..\..\k8.tf line 119, in resource "azurerm_kubernetes_cluster" "k8s":
│  119:     vnet_integration_enabled = true
│ 
│ An argument named "vnet_integration_enabled" is not expected here.

Expected Behaviour

Can vnet integrate API server

Actual Behaviour

Can't integrate API server.

I understand why this occurred with the migration to the stable API. However, this specific case should have been exempted as API vnet integration is required for key_management_service.key_vault_network_access = "Private" which is required for every security & compliance standard that exists.

Unless egress is using a user defined resource the IP that will originate the KMS calls will not be known until after the cluster is created which requires an unsafe azurerm_key_vault configuration with a default allow rule. Having public internet access enabled to AKV at all is inherently unsafe but this compounds that problem.

As it currently stands this resource is not usable in professional cloud settings (currently entirely unusable in gov cloud, it wont meet 800-53) and azapi or ARM/bicep has to be used in place of this.

Steps to Reproduce

No response

Important Factoids

No response

References

No response

stephybun commented 4 days ago

Hey @LiamLeane

Unfortunately we can't pick and choose which bits of the AKS resource use a preview version and which a stable without breaking the entire resource, so exempting this particular preview feature was not a viable option for us. We're dismayed by the loss of functionality that we can provide in the resource caused by the migration to the stable API, in particular for features that relate to security and compliance. This was a joint decision made at the request of the AKS Service Team, I suggest getting in touch with them over on Azure/AKS to see if there is a timeline for when this will go GA so we can add it back into the provider.

iverberk commented 4 days ago

This is a showstopper for us as well, exactly for the reasons outlined by @LiamLeane. It's very disappointing that we now have to move away from Terraform and I imagine a lot of others as well, that require this feature for compliance and security reasons.

tspearconquest commented 3 days ago

I want to point out that Microsoft is also going to be disabling the APIs that the v3 AzureRM provider uses in the near future so we won't be able to keep creating clusters using that provider once that happens.

https://learn.microsoft.com/en-us/azure/aks/concepts-preview-api-life-cycle#upcoming-deprecations

@stephybun can you confirm what preview API version was used by the last v3.x provider release, so that we can at least know which date from the above linked table will cause this to happen and plan accordingly? Hopefully we can also reference this issue, the timeline and the API version I am asking about when talking to MS support about this because it's going to really hit people hard when they are forced onto v4.

tspearconquest commented 2 days ago

I did some more digging over on Azure/AKS. I found something positive.

https://github.com/Azure/AKS/issues/2729

GA ETA: Q1 2025*

*ETAs are only estimates and can be subject to change

Both AKS Public and Private Cluster use a Tunnel, also a Private Cluster is currently based on Private Link. API Server VNET Integration will be enabled for both Public and Private clusters, and will allow for upgrade capabilities from Public to Private and Private to Public if required, without a redeploy.

stephybun commented 2 days ago

Thanks for finding and linking that info @tspearconquest! That is certainly promising news.

The preview version used by the last v3.x release was 2023-09-02-preview which doesn't appear to have a deprecation date yet. Judging by the table it's safe to say it will be post March 10.

I'll pass this on to our contacts at MSFT to see if we can avoid a situation where the preview version is deprecated, but the feature hasn't been released and added to the provider yet.

hashicorp / terraform-provider-azurerm