Support for Custom CA Certificates

TimJongerius commented 1 year ago

Is there an existing issue for this?

[X] I have searched the existing issues

Description

Add an option to upload additional ca certificates during cluster creation like it is already possible using the Cli (https://learn.microsoft.com/en-us/azure/aks/custom-certificate-authority)

New or Affected Resource(s)/Data Source(s)

azurerm_kubernetes_cluster

Potential Terraform Configuration

No response

References

No response

zioproto commented 1 year ago

Hello, this is a preview feature, currently tracked with https://github.com/Azure/AKS/issues/2259

As soon as the feature is GA and the Terraform Provider supports the feature, we can start the implementation in the module.

TimJongerius commented 1 year ago

@zioproto Okay thanks for the update. I can see that it's possible to activate the custom ca daemonset for additional nodepools. Is there a reason why I can't specify it for the default_node_pool?

TimJongerius commented 1 year ago

@zioproto it seems to be available now https://registry.terraform.io/providers/hashicorp/azurerm/3.64.0/docs/resources/kubernetes_cluster#custom_ca_trust_certificates_base64

When would it be possible to update the module?

lonegunmanb commented 1 year ago

Hi @TimJongerius, according to this post, the feature hasn't an ETA for GA yet, are you sure that this feature is GA already?

TimJongerius commented 1 year ago

Hi @lonegunmanb,

according to this link https://learn.microsoft.com/en-us/azure/aks/custom-certificate-authority it seems to be still in preview.

Also for the azure cli it's only available after enabling aks-preview.

However the terraform provider started to support it by adding the custom_ca_trust_certificates_base64 property from 3.63 upwards. https://github.com/hashicorp/terraform-provider-azurerm/blob/v3.63.0/CHANGELOG.md

Before, to work around this limitation without the need to deployment a very complex daemonset I used a terraform provisioner to upload the certificate with the cli + aks preview after the aks deployment. Because the custom_ca_trust_certificates_base64 property wasn't known to the terraform provider it didn't change that property when I redeployed the module, hence the nodepools didn't get drained.

With 3.63 this behavior changed since the provider is know removing this property and I have no way to supply it with the aks module. The only way to avoid this is to fixate the provider on a version < 3.63.0.

Why do we have to wait for GA if the azurerm provider has already started to support it?

lonegunmanb commented 1 year ago

Thanks for asking @TimJongerius, a preview feature might be changed or even removed totally at any time, so when the provider introduces a preview feature it also introduces the corresponding risk, it happened before and it would happen again. This Aks module is one of our "verified" modules. We'd like to keep these verified modules as stable as possible, so we decide that we should release the major version upgrade which contains breaking changes every six months.

I fully understand the reason you want this feature in this module, and thanks for using our modules. We don't have a best practice on balance between stability and capability, do you have any suggestions?

HouseDamage commented 1 year ago

Any idea when this feature will go Globally Available?

Have been tracking this for a long time but unable to find out when it's planned for GA release.

Thanks!

zioproto commented 1 year ago

Any idea when this feature will go Globally Available?

Have been tracking this for a long time but unable to find out when it's planned for GA release.

Thanks!

The correct place to ask this question is https://github.com/Azure/AKS/issues/2259

HouseDamage commented 1 year ago

@zioproto - I know, but the commenting is closed! :(