ContainerCluster: Update call failed: cannot make changes to immutable field(s) (Autopilot)

[flunderpero]

Checklist

• [x] I did not find a related open issue.
• [x] I did not find a solution in the troubleshooting guide: (https://cloud.google.com/config-connector/docs/troubleshooting)
• [ ] If this issue is time-sensitive, I have submitted a corresponding issue with GCP support.

Bug Description

I create an Autopilot Cluster like this:

---
apiVersion: container.cnrm.cloud.google.com/v1beta1
kind: ContainerCluster
metadata:
  name: beta
spec:
  enableAutopilot: true
  location: europe-west1
  releaseChannel:
    channel: REGULAR
  minMasterVersion: 1.21.5-gke.1302

Everything is fine, but after a while, the resource moves to status UpdateFailed with the message: Update call failed: cannot make changes to immutable field(s): [nodeConfig.0.Taint.0.Value nodeConfig.0.Taint.# nodeConfig.0.Taint.0.Effect nodeConfig.0.Taint.0.Key].

As you can see, I did not specify any taints. I use Spot-VMs and as a result, the nodes receive a taint during runtime. But that's nothing I could specify here because it relies purely on the actual pods/deployments running in the cluster.

I upgraded ConfigConnector to the latest version, but the problem still exists.

Additional Diagnostic Information

Kubernetes Cluster Version

Client Version: v1.20.8-dispatcher Server Version: v1.20.10-gke.1600

Config Connector Version

1.67.0

Config Connector Mode

cluster

Steps to Reproduce

See above.

[jcanseco]

Hi @flunderpero, apologies for the late reply. I believe this might be due to a known issue where users cannot let list fields (and only list fields) be externally managed due to technical limitations with K8s that are a bit too complex to get into here.

Today, Config Connector takes ownership of list fields even if the user did not specify them in their original YAML, and pins the list to the last-read value. This means that if the list were to change under-the-hood, Config Connector will try to change it back, which results in an error in this case.

We have a feature that should allow you to bypass this issue, but we haven't rolled it out to all resources yet, so I'll note down this issue as another problem that can be fixed by the feature.

The feature allows users to annotate resources with state-into-spec: absent which tells Config Connector not to reflect non-user-specified state into the spec (i.e. keep the desired and observed states separate). This should fix your problem since it'll allow you tell Config Connector not to reflect list fields you did not specify yourself into the spec, allowing you to keep them externally managed.

Please let us know if that works for you. We'll update this thread when we've rolled out the feature to ContainerCluster.

[flunderpero]

Hi @jcanseco, that would certainly work for us and in many cases, this would be exactly the behavior I'd expect.

[jcanseco]

Sounds great, thanks for confirming. I'll raise your need up internally and will let you know when we have rolled out the feature for ContainerCluster.

[jcanseco]

@flunderpero FYI: it seems there is an issue with the latest release (KCC v1.68.0) that causes your config above to fail to create. Please avoid upgrading to KCC v1.68.0. We'll see if we can put out a fix soon.

We should hopefully also be able to put out support for state-into-spec: absent for ContainerCluster soon which is currently in the works.

[xiaobaitusi]

@flunderpero, with the release of 1.69.0, we have rolled out support for "state-into-spec: absent" to ContainerCluster. Feel free to give it try, and let me know if that satisfies your use case.