BigtableInstance does not allow to switch back from autoscaling node allocation to manual node allocation

diogospt commented 11 months ago

Checklist

[X] I did not find a related open issue.
[X] I did not find a solution in the troubleshooting guide: (https://cloud.google.com/config-connector/docs/troubleshooting)
[X] If this issue is time-sensitive, I have submitted a corresponding issue with GCP support.

Bug Description

BigtableInstance ConfigConnector resource does not allow a end-user to switch back from using autoscalingConfig to numNodes. Switching from a BigtableInstance from numNodes: 1 to autoscalingConfig { cpuTarget: 10, minNodes: 1, maxNodes: 5 } works as expected. But once we try to roll back the change, disabling autoscaling, the API returns an error.

Error: API response status: Failure
Tue 26 Sep 2023 12:46:22 [INFO    ] puppet: Gcp_project           Exec[run_terraform_for_project_resources]:     1: resource "kubernetes_manifest" "bigtable_instance_bigtable" {
Tue 26 Sep 2023 12:46:22 [INFO    ] puppet: Gcp_project           Exec[run_terraform_for_project_resources]:
Tue 26 Sep 2023 12:46:22 [INFO    ] puppet: Gcp_project           Exec[run_terraform_for_project_resources]:  BigtableInstance.bigtable.cnrm.cloud.google.com "bigtable" is invalid:
Tue 26 Sep 2023 12:46:22 [INFO    ] puppet: Gcp_project           Exec[run_terraform_for_project_resources]:  [spec.cluster[0].autoscalingConfig.cpuTarget: Required value,
Tue 26 Sep 2023 12:46:22 [INFO    ] puppet: Gcp_project           Exec[run_terraform_for_project_resources]:  spec.cluster[0].autoscalingConfig.maxNodes: Required value,
Tue 26 Sep 2023 12:46:22 [INFO    ] puppet: Gcp_project           Exec[run_terraform_for_project_resources]:  spec.cluster[0].autoscalingConfig.minNodes: Required value,
Tue 26 Sep 2023 12:46:22 [INFO    ] puppet: Gcp_project           Exec[run_terraform_for_project_resources]:  spec.cluster[1].autoscalingConfig.cpuTarget: Required value,
Tue 26 Sep 2023 12:46:22 [INFO    ] puppet: Gcp_project           Exec[run_terraform_for_project_resources]:  spec.cluster[1].autoscalingConfig.maxNodes: Required value,
Tue 26 Sep 2023 12:46:22 [INFO    ] puppet: Gcp_project           Exec[run_terraform_for_project_resources]:  spec.cluster[1].autoscalingConfig.minNodes: Required value]
Tue 26 Sep 2023 12:46:22 [INFO    ] puppet: Gcp_project           Exec[run_terraform_for_project_resources]:  Error: Kubernetes API Error: Invalid BigtableInstance.bigtable.cnrm.cloud.google.com [bigtable]
Tue 26 Sep 2023 12:46:22 [INFO    ] puppet: Gcp_project           Exec[run_terraform_for_project_resources]:
Tue 26 Sep 2023 12:46:22 [INFO    ] puppet: Gcp_project           Exec[run_terraform_for_project_resources]:    with kubernetes_manifest.bigtable_instance_bigtable,
Tue 26 Sep 2023 12:46:22 [INFO    ] puppet: Gcp_project           Exec[run_terraform_for_project_resources]:    on bigtable_bigtable.tf line 1, in resource "kubernetes_manifest" "bigtable_instance_bigtable":
Tue 26 Sep 2023 12:46:22 [INFO    ] puppet: Gcp_project           Exec[run_terraform_for_project_resources]:     1: resource "kubernetes_manifest" "bigtable_instance_bigtable" {
Tue 26 Sep 2023 12:46:22 [INFO    ] puppet: Gcp_project           Exec[run_terraform_for_project_resources]:
Tue 26 Sep 2023 12:46:22 [INFO    ] puppet: Gcp_project           Exec[run_terraform_for_project_resources]:
Tue 26 Sep 2023 12:46:22 [INFO    ] puppet: Gcp_project           Exec[run_terraform_for_project_resources]:
Tue 26 Sep 2023 12:46:22 [INFO    ] puppet: Gcp_project           Exec[run_terraform_for_project_resources]:  Error: spec.cluster[0].autoscalingConfig.cpuTarget
Tue 26 Sep 2023 12:46:22 [INFO    ] puppet: Gcp_project           Exec[run_terraform_for_project_resources]:
Tue 26 Sep 2023 12:46:22 [INFO    ] puppet: Gcp_project           Exec[run_terraform_for_project_resources]:    with kubernetes_manifest.bigtable_instance_bigtable,
Tue 26 Sep 2023 12:46:22 [INFO    ] puppet: Gcp_project           Exec[run_terraform_for_project_resources]:    on bigtable_bigtable.tf line 1, in resource "kubernetes_manifest" "bigtable_instance_bigtable":
Tue 26 Sep 2023 12:46:22 [INFO    ] puppet: Gcp_project           Exec[run_terraform_for_project_resources]:     1: resource "kubernetes_manifest" "bigtable_instance_bigtable" {
Tue 26 Sep 2023 12:46:22 [INFO    ] puppet: Gcp_project           Exec[run_terraform_for_project_resources]:
Tue 26 Sep 2023 12:46:22 [INFO    ] puppet: Gcp_project           Exec[run_terraform_for_project_resources]:  Required value
Tue 26 Sep 2023 12:46:22 [INFO    ] puppet: Gcp_project           Exec[run_terraform_for_project_resources]:
Tue 26 Sep 2023 12:46:22 [INFO    ] puppet: Gcp_project           Exec[run_terraform_for_project_resources]:
Tue 26 Sep 2023 12:46:22 [INFO    ] puppet: Gcp_project           Exec[run_terraform_for_project_resources]:  Error: spec.cluster[0].autoscalingConfig.maxNodes
Tue 26 Sep 2023 12:46:22 [INFO    ] puppet: Gcp_project           Exec[run_terraform_for_project_resources]:
Tue 26 Sep 2023 12:46:22 [INFO    ] puppet: Gcp_project           Exec[run_terraform_for_project_resources]:    with kubernetes_manifest.bigtable_instance_bigtable,
Tue 26 Sep 2023 12:46:22 [INFO    ] puppet: Gcp_project           Exec[run_terraform_for_project_resources]:    on bigtable_bigtable.tf line 1, in resource "kubernetes_manifest" "bigtable_instance_bigtable":
Tue 26 Sep 2023 12:46:22 [INFO    ] puppet: Gcp_project           Exec[run_terraform_for_project_resources]:     1: resource "kubernetes_manifest" "bigtable_instance_bigtable" {
Tue 26 Sep 2023 12:46:22 [INFO    ] puppet: Gcp_project           Exec[run_terraform_for_project_resources]:
Tue 26 Sep 2023 12:46:22 [INFO    ] puppet: Gcp_project           Exec[run_terraform_for_project_resources]:  Required value

Doing the same change using Terraform google_bigtable_instance resource performs the change accordingly.

Additional Diagnostic Information

No info.

Kubernetes Cluster Version

v1.25.10-gke.2700

Config Connector Version

1.108.0

Config Connector Mode

cluster mode

Log Output

No response

Steps to reproduce the issue

Create a BigtableInstance resource with ConfigConnector, specifying the numNodes to 1. Once the resource is created, configure the autoscalingConfig. As an example:

  spec:
    cluster:
    - autoscalingConfig:
        cpuTarget: 10
        maxNodes: 5
        minNodes: 1
        storageTarget: 2560
      clusterId: bigtable-cluster
      numNodes: 1
      storageType: SSD
      zone: europe-west4-b

Then, once the second change is applied, try to disable the autoscaling configuration by removing it from the YAML.

YAML snippets

No response

diviner524 commented 11 months ago

This could be related to a feature gap in Config Connector, which is how the current controller interpret "removal".

https://cloud.google.com/config-connector/docs/troubleshooting#unable_to_remove_fields_from_resource_spec

Since autoscalingConfig is an object field, we may need to introduce a proper feature or syntax to indicate the field needs to be set to NULL/empty.

diogospt commented 9 months ago

A public tracker has been created for this bug. https://issuetracker.google.com/issues/315988174

GoogleCloudPlatform / k8s-config-connector