feat: (IAC-654) GKE add support for K8s 1.24

[jarpat]

Background:

Starting from Kubernetes version 1.24.0 service account token is not automatically generated, thus it has to be created separately. The following resources were updated by the provider hashicorp/kubernetes to handle this change in v2.13.0: d/kubernetes_service_account, r/kubernetes_default_service_account, r/kubernetes_service_account. For Kubernetes clusters running v1.24+ default_secret_name will be empty. A warning message will be printed once any of the above resources are in use.

Changes

• To support 1.24+ Kubernetes version in GKE, a secret is manually created for the service account
• The default K8s value is updated from 1.22.9-gke.1500 to 1.23.8-gke.1900
• The default kubectl value is updated from 1.22.10 to 1.23.8
• The terraform-google-modules/kubernetes-engine/google//modules/private-cluster had to be updated to 15.0.2, since its version constraint allows support for hashicorp/kubernetes version ~> 2.0. The older 14.3.0 did not allow it.
  • Note: I specifically chose version 15.0.2 not something and newer since it it introduces a very little change to the module, on our end we don't have to do too many code updates.
  • I went through the module's changelog to ensure no other changes were needed in the code base https://github.com/terraform-google-modules/terraform-google-kubernetes-engine/blob/v15.0.2/CHANGELOG.md
  • In the future we will be doing a general update where we update the modules/providers to the newest working version.
• Updated hashicorp/kubernetes to 2.13.0
  • Removed load_config_file from the provider parameters, that feature was deprecated in hashicorp/kubernetes v2 and we were setting that value to false anyways so that behavior was not being used.

Tests

More details and artifacst in interal tickets

Peformed the following

1. Created a v1.24.2-gke.1900 cluster with the updated code and successfully deployed Viya
2. Created a v1.23.8-gke.1900 cluster with the updated code and successfully deployed Viya
3. Created a v1.22.11-gke.400 cluster with the updated code and successfully deployed Viya
4. Created a v1.24.2-gke.1900 cluster with create_static_kubeconfig=false and verifed that the code workflow to create a provider based kube config was not modifed. I was able to use the provider based kube config to view cluster resources.

[jarpat]

@thpang This is still a WIP, running into some timing issues that I am working on resolving

• Running into some timing issues with the service account and secret creation. Currently testing adding a couple of depends_on so that the order is sa>sa secret>kubeconfig sa

  • This seems to be working so far, I have not run into the Error: Provider produced inconsistent result after apply anymore

• Timing issues retrieving the token & ca.crt during initial secret creation

  • Although removed in in the AKS PR, this change will require the data "kubernetes_secret" "sa_secret" call after the secret is initially created. The behavior I have been consistently seeing in the debug log is that after resource "kubernetes_secret" "sa_secret" creates the secret, the Kubernetes API's response (from GET /api/v1/namespaces/kube-system/secrets/jarpatgcpb-sa-secret) does NOT have a data field (which contains the required ca.crt + token values).
  • Adding back the data "kubernetes_secret" "sa_secret" call right after the secret is created results in a kubernetes API response that actually has a populated data field.
  • With the data values back, the kubeconfig template can now be successfully rendered and produce a working kubeconfig.conf
  • So far testing in my dev testing, this update has resolved this issue, pushing it here soon.