GoogleCloudPlatform / k8s-config-connector

GCP Config Connector, a Kubernetes add-on for managing GCP resources
https://cloud.google.com/config-connector/docs/overview
Apache License 2.0
882 stars 215 forks source link

SQLDatabase Resource stuck in an UpdateFailed state with an empty error reason #2549

Open jasonneurohr-stake opened 3 weeks ago

jasonneurohr-stake commented 3 weeks ago

Checklist

Bug Description

We have multiple SQLDatabase resources which are successfully created, however, KCC is reporting them in an UpdateFailed state with a less than useful status message

status:
  conditions:
  - lastTransitionTime: "2024-08-21T06:17:43Z"
    message: "Update call failed: error applying desired state: summary: \n"
    reason: UpdateFailed
    status: "False"
    type: Ready
  observedGeneration: 1

Additional Diagnostic Information

.

Kubernetes Cluster Version

v1.30.3

Config Connector Version

1.121.0

Config Connector Mode

namespaced mode (default)

Log Output

{ "insertId": "x32kybhy914148po", "jsonPayload": { "controller": "sqldatabase-controller", "error": "Update call failed: error applying desired state: summary: \n", "msg": "Reconciler error", "reconcileID": "300a52ee-2e9a-4602-8c5c-1b68b60d8028", "controllerGroup": "sql.cnrm.cloud.google.com", "namespace": "config-control", "controllerKind": "SQLDatabase", "SQLDatabase": { "name": "my-sqldatabase", "namespace": "config-control" }, "name": "my-sqldatabase", "timestamp": "2024-08-21T04:26:20.076Z" }, "resource": { "type": "k8s_container", "labels": { "project_id": "prj-c-hs-infra-iac", "container_name": "manager", "namespace_name": "cnrm-system", "cluster_name": "krmapihost-gke-c-hs-us-e1-cc", "location": "us-east1", "pod_name": "cnrm-controller-manager-c7s6gncgkgt830b2q53g-0" } }, "timestamp": "2024-08-21T04:26:20.079645408Z", "severity": "ERROR", "labels": { "k8s-pod/controller-revision-hash": "cnrm-controller-manager-c7s6gncgkgt830b2q53g-8668b66c58", "k8s-pod/apps_kubernetes_io/pod-index": "0", "k8s-pod/cnrm_cloud_google_com/system": "true", "k8s-pod/cnrm_cloud_google_com/scoped-namespace": "config-control", "k8s-pod/statefulset_kubernetes_io/pod-name": "cnrm-controller-manager-c7s6gncgkgt830b2q53g-0", "compute.googleapis.com/resource_name": "gke-krmapihost-gke-c-krmapihost-gke-c-032b62d7-6dci", "k8s-pod/cnrm_cloud_google_com/component": "cnrm-controller-manager" }, "logName": "projects/my-project/logs/stdout", "receiveTimestamp": "2024-08-21T04:26:23.551469335Z" }

Steps to reproduce the issue

create the resource as describe, wait for it to sync, and observe the issue.

Have tried without the state-into-spec (originally defaulted to merge). Neither mode makes a difference.

YAML snippets

apiVersion: sql.cnrm.cloud.google.com/v1beta1
kind: SQLDatabase
metadata:
  annotations:
    cnrm.cloud.google.com/deletion-policy: abandon
    cnrm.cloud.google.com/project-id: my-project
    cnrm.cloud.google.com/state-into-spec: absent
  labels:
    cost-centre: abcd
    environment: prod
    project-id: my-project
  name: my-sqldatabase
  namespace: config-control
spec:
  deletionPolicy: ABANDON
  instanceRef:
    name: my-sqlinstance
  resourceID: my-sqldatabase
ewoutp commented 1 week ago

Have the same problem