[percona-server-mongodb-operator] Updating the `spec.sharding.mongos.expose.serviceAnnotations` get silently rejected

Description

We are trying to update the service annotations of the mongos service by updating the spec.sharding.mongos.expose.serviceAnnotations field. However, we found that updating this field has no effect, the operator never updating the annotations of the service no matter what changes are done to the field spec.sharding.mongos.expose.serviceAnnotations.

We later dug into source code and discovered that the annotations are intentionally left out for service objects: https://github.com/percona/percona-server-mongodb-operator/blob/f7c117cf58830a42166c655c0b89249cb730b04a/pkg/controller/perconaservermongodb/psmdb_controller.go#L1693

We think this behavior is very confusing, since there is no message passed to the users. It would be much better if there is an error message passed to user when they try to change the serviceAnnotations. We suggest to make use of the new validation feature in Kubernetes: https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#validation-rules to directly reject users input at kubectl level. This feature is alpha is 1.23, and will be beta in 1.25. You can use the CEL expression to express immutability.

Expectation

We expected that our changes to the spec.sharding.mongos.expose.serviceAnnotations field will be reflected to the mongos service.

Reproduce

Deploy the percona-server-mongodb-operator

Deploy a mongodb cluster instance through some minimum CR, with some initial spec.sharding.mongos.expose.serviceAnnotations set:

apiVersion: psmdb.percona.com/v1-12-0
kind: PerconaServerMongoDB
metadata:
name: minimal-cluster
spec:
crVersion: 1.12.0
image: percona/percona-server-mongodb:5.0.7-6
allowUnsafeConfigurations: true
upgradeOptions:
apply: 5.0-never
schedule: "0 2 * * *"
secrets:
users: minimal-cluster
replsets:

- name: rs0
size: 1
volumeSpec:
  persistentVolumeClaim:
    resources:
      requests:
        storage: 3Gi

sharding:
enabled: true

configsvrReplSet:
  size: 1
  volumeSpec:
    persistentVolumeClaim:
      resources:
        requests:
          storage: 3Gi

mongos:
  size: 1
  expose:
    exposeType: ClusterIP
    serviceAnnotations:
      key1: key2
mongod:
security:
  encryptionKeySecret: "my-cluster-name-mongodb-encryption-key"
storage:
  engine: inMemory
  inMemory:
    engineConfig:
      inMemorySizeRatio: 0.9
  wiredTiger:
    engineConfig:
      cacheSizeRatio: 0.5
      directoryForIndexes: false
      journalCompressor: snappy
    collectionConfig:
      blockCompressor: snappy
    indexConfig:
      prefixCompression: true

Change the spec.sharding.mongos.expose.serviceAnnotations to some new annotation kv pair, and apply the CR

apiVersion: psmdb.percona.com/v1-12-0
kind: PerconaServerMongoDB
metadata:
name: minimal-cluster
spec:
crVersion: 1.12.0
image: percona/percona-server-mongodb:5.0.7-6
allowUnsafeConfigurations: true
upgradeOptions:
apply: 5.0-never
schedule: "0 2 * * *"
secrets:
users: minimal-cluster
replsets:

- name: rs0
size: 1
volumeSpec:
  persistentVolumeClaim:
    resources:
      requests:
        storage: 3Gi

sharding:
enabled: true

configsvrReplSet:
  size: 1
  volumeSpec:
    persistentVolumeClaim:
      resources:
        requests:
          storage: 3Gi

mongos:
  size: 1
  expose:
    exposeType: ClusterIP
    serviceAnnotations:
      key1: key3
mongod:
security:
  encryptionKeySecret: "my-cluster-name-mongodb-encryption-key"
storage:
  engine: inMemory
  inMemory:
    engineConfig:
      inMemorySizeRatio: 0.9
  wiredTiger:
    engineConfig:
      cacheSizeRatio: 0.5
      directoryForIndexes: false
      journalCompressor: snappy
    collectionConfig:
      blockCompressor: snappy
    indexConfig:
      prefixCompression: true

Describe the service and check that the annotation is not updated

Root cause

We checked the source code and found that the annotations of service object are intentionally left out.

Environment

Kubernetes version information:

Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.1", GitCommit:"86ec240af8cbd1b60bcc4c03c20da9b98005b92e", GitTreeState:"clean", BuildDate:"2021-12-16T11:41:01Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.9", GitCommit:"6df4433e288edc9c40c2e344eb336f63fad45cd2", GitTreeState:"clean", BuildDate:"2022-05-10T18:26:54Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

xlab-uiuc / acto