CloudBuildTriggers in UpdateFailed state after v1.96.0 upgrade

[npitts0811]

Checklist

• [X] I did not find a related open issue.
• [X] I did not find a solution in the troubleshooting guide: (https://cloud.google.com/config-connector/docs/troubleshooting)
• [X] If this issue is time-sensitive, I have submitted a corresponding issue with GCP support.

Bug Description

We upgraded to v1.96.0 today; after the upgrade, our CloudBuildTrigger resources that were created prior to v1.96.0 are now in an UpdateFailed state. The error states Update call failed: cannot make changes to immutable field(s): [location]; we are not setting/managing the location attribute yet as support is introduced with v1.96.0. Presumably, this issue will occur with all CloudBuildTriggers that are updated from previous versions to v1.96.0.

Additional Diagnostic Information

na

Kubernetes Cluster Version

v1.22.12-gke.2300

Config Connector Version

1.96.0

Config Connector Mode

namespaced mode (default)

Log Output

Describe output for a CloudBuildTrigger resource post upgrade (truncated for readability/privacy); it was not failing prior to KCC upgrade. Note, location is not being defined.

Name:         my-example-trigger
Namespace:    example-namespace
Labels:       app.kubernetes.io/managed-by=Helm
  ...
API Version:  cloudbuild.cnrm.cloud.google.com/v1beta1
Kind:         CloudBuildTrigger
Metadata:
  ...
Spec:
  Build:
    ...
  Description:  trigger for the [repo] [branch] branch
  Github:
    ...
  Include Build Logs:  INCLUDE_BUILD_LOGS_WITH_STATUS
Status:
  Conditions:
    Last Transition Time:  2022-11-04T16:43:09Z
    Message:               Update call failed: cannot make changes to immutable field(s): [location]; please refer to our troubleshooting doc: https://cloud.google.com/config-connector/docs/troubleshooting
    Reason:                UpdateFailed
    Status:                False
    Type:                  Ready
  Create Time:             2022-02-03T17:16:29.991002515Z
  Observed Generation:     7
  Trigger Id:              [trigger ID]
Events:
  Type     Reason        Age                   From                          Message
  ----     ------        ----                  ----                          -------
  Warning  UpdateFailed  8m22s (x19 over 34m)  cloudbuildtrigger-controller  Update call failed: cannot make changes to immutable field(s): [location]; please refer to our troubleshooting doc: https://cloud.google.com/config-connector/docs/troubleshooting
  Warning  UpdateFailed  7s (x9 over 6m16s)    cloudbuildtrigger-controller  Update call failed: cannot make changes to immutable field(s): [location]; please refer to our troubleshooting doc: https://cloud.google.com/config-connector/docs/troubleshooting

### Steps to reproduce the issue

Starting with a KCC version prior to v1.96.0 (we were on v1.95.0):
1. Create a CloudBuildTrigger resource that does not define `location` (not supported until v1.96.0)
2. Upgrade to v1.96.0
3. Post upgrade, the CloudBuildTrigger resource is in a 'UpdateFailed' state as shown in the 'Log Output' section of this page.

### YAML snippets

_No response_

[diviner524]

@npitts0811 Could you please try adding the location as "global" explicitly in the spec, and see if the error goes away?

spec:
  ...
  location: "global"
  ...

[npitts0811]

@diviner524 we've since worked around the issue by recreating our triggers, but prior to that, the testing we did included adding location: global to an existing trigger as a possible fix. Unfortunately, it did not resolve the issue for us- we were still met with the same immutable field message. The only way we could get around the issue was to recreate the CloudBuiltTrigger resource.

[diviner524]

Providing a bit more context on what is happening under the hood:

Terraform code backfills the location "global" during resource import (instead of resource read):

https://github.com/hashicorp/terraform-provider-google-beta/blob/main/google-beta/resource_cloudbuild_trigger.go#L1597

And this location value of "global" is reset to zero value in Config Connector controller before the controller performs a resource read.

https://github.com/GoogleCloudPlatform/k8s-config-connector/blob/master/pkg/krmtotf/fetchlivestate.go#L149

We are looking into a proper change to solve this case.

[hsuabina]

Is there any update on this issue? I'm still facing this problem (using KCC version 1.102.0) and to complicate things further, every time I need to update a CloudBuildTrigger (which requires recreating the resource) they get stuck on DeleteFailed and I have to manually remove the finalizers, which is obviously not ideal!