Cannot delete ingress.ui's annotations once created

What version of operator are you using? commit 561cf47d783c368fd8795acb82a5a39099a35984 (HEAD -> master)

What operating system and processor architecture are you using (kubectl version)? Ubuntu. 20.04

kubectl version Output

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-03T13:46:05Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"darwin/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.9", GitCommit:"6df4433e288edc9c40c2e344eb336f63fad45cd2", GitTreeState:"clean", BuildDate:"2022-05-19T19:53:08Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.24) and server (1.22) exceeds the supported minor version skew of +/-1

What did you do?

We are trying to delete the annotations of the ingress.ui by setting the ingress.ui.annotations to null I first deployed the operator using the example yaml file and crd yaml file. Then I created the cluster by applying the following custom resource file:

apiVersion: crdb.cockroachlabs.com/v1alpha1
kind: CrdbCluster
metadata:
  name: test-cluster
spec:
  additionalLabels:
    crdb: is-cool
  dataStore:
    pvc:
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        volumeMode: Filesystem
  image:
    name: cockroachdb/cockroach:v21.2.10
  ingress:
    ui:
      annotations:
        mykey: myvalue
      host: localhost
  nodes: 3
  resources:
    limits:
      cpu: 2
      memory: 2Gi
    requests:
      cpu: 100m
      memory: 1Gi
  tlsEnabled: true

(All files were applied using kubectl apply -f <filename> -n cockroach-operator-system)

Then I deleted the ingress.ui.annotation field (basically, deleting "annotations"), as follows:

apiVersion: crdb.cockroachlabs.com/v1alpha1
kind: CrdbCluster
metadata:
  name: test-cluster
spec:
  additionalLabels:
    crdb: is-cool
  dataStore:
    pvc:
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        volumeMode: Filesystem
  image:
    name: cockroachdb/cockroach:v21.2.10
  ingress:
    ui:
      annotations: null
      host: localhost
  nodes: 3
  resources:
    limits:
      cpu: 2
      memory: 2Gi
    requests:
      cpu: 100m
      memory: 1Gi
  tlsEnabled: true

What did you expect to see and what did you see instead?

I expect to see annotations being deleted, but they are still present.

Root cause The way annotations are updated is merging new annotations into old ones, as this function suggests. However, when a field is deleted (i.e. being changed to null), the key and value pair will not show up in the new annotations, resulting in the old annotations leaving unchanged.

cockroachdb / cockroach-operator

Cannot delete ingress.ui's annotations once created #927