[percona-server-mongodb-operator] Unable to delete the annotations on pods by deleting them in spec.replsets.annotations field

[tylergu]

What happened? We found it impossible to remove the additional annotations by unsetting the key-value pairs under the field spec.replsets.annotations. We first added a label key: value under the spec.replsets.annotations field, and PSMDB operator correctly adds the annotation to the pods. However, later we wanted to remove the key: value label from the pods by deleting it from the CR. However, by doing so, PSMDB operator does not remove the annotations from the pods.

Did you expect to see something different? We expected that the CR represents the desired state of the application, based on the declarative model of Kubernetes. So we expected that when remove the label from the spec.replsets.annotations field, PSMDB operator would automatically remove this annotation from the pods.

How to reproduce it (as minimally and precisely as possible):

1. Deploy PSMDB operator

2. Deploy a minimal MongoDB cluster using minimal.yaml: kubectl apply -f example.yaml

   example.yaml

   ```yaml apiVersion: psmdb.percona.com/v1-12-0 kind: PerconaServerMongoDB metadata: name: minimal-cluster spec: crVersion: 1.12.0 image: percona/percona-server-mongodb:5.0.7-6 allowUnsafeConfigurations: true upgradeOptions: apply: 5.0-never schedule: "0 2 * * *" secrets: users: minimal-cluster replsets: - name: rs0 size: 1 volumeSpec: persistentVolumeClaim: resources: requests: storage: 3Gi sharding: enabled: true configsvrReplSet: size: 1 volumeSpec: persistentVolumeClaim: resources: requests: storage: 3Gi mongos: size: 1 ```

3. Add additional annotations key1: value1 to replica set by applying the cr1.yaml

   cr1.yaml

   ```yaml apiVersion: psmdb.percona.com/v1-12-0 kind: PerconaServerMongoDB metadata: name: minimal-cluster spec: crVersion: 1.12.0 image: percona/percona-server-mongodb:5.0.7-6 allowUnsafeConfigurations: true upgradeOptions: apply: 5.0-never schedule: "0 2 * * *" secrets: users: minimal-cluster replsets: - name: rs0 annotations: key1: value1 size: 1 volumeSpec: persistentVolumeClaim: resources: requests: storage: 3Gi sharding: enabled: true configsvrReplSet: size: 1 volumeSpec: persistentVolumeClaim: resources: requests: storage: 3Gi mongos: size: 1 ```
4. Observe that the seed service now has the additional annotation key1: value1
5. Delete the additional annotation by applying the example CR again
6. Observe that the seed service still has the label key1: value1

Environment

• Kubernetes version information:

  Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.1", GitCommit:"86ec240af8cbd1b60bcc4c03c20da9b98005b92e", GitTreeState:"clean", BuildDate:"2021-12-16T11:41:01Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.9", GitCommit:"6df4433e288edc9c40c2e344eb336f63fad45cd2", GitTreeState:"clean", BuildDate:"2022-05-10T18:26:54Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind: Kind cluster

Anything else we need to know?: The root cause is inside the https://github.com/percona/percona-server-mongodb-operator/blob/88f47b92483e54f4eeb070c46408b5fb42830482/pkg/controller/perconaservermongodb/psmdb_controller.go#L1402 where the annotations in the existing statefulSet are merged into the desired statefulSet. The reason of the merge is because mongodb operator does not want to mess up the annotations that are added by other sources. A possible fix is to keep track of the annotations that are added by the user using the spec.replsets.annotations field, so that when users delete some annotations from the field, mongodb operator can delete these from the statefulSet.

[tylergu]

Issued: https://jira.percona.com/browse/K8SPSMDB-696

[tylergu]

Fixed