search

mongodb / mongodb-kubernetes-operator

MongoDB Community Kubernetes Operator
Other
1.21k stars 493 forks source link

Operator falsely errors and does not let to upgrade MongoDB cluster replicas #1613

Closed KarooolisZi closed 1 month ago

KarooolisZi commented 1 month ago

Hello,

I was trying to adjust replicas number in the CR MongoDB yaml manifest. Only change to current CR was changing 'replicas' from 2 to 3.

That is strange because according to operator I should be able to do this. My statefulset is not scaling. I checked last applied configurations, last success apply and no any differences were there. I compared these configuration to my VCS configuration. No changes were detected.

Using 6.0.4 MongoDB community version. Using 0.7.8 operator version.

I tried removing annotations for existing CRD which states as 'failed' after apply and reapply again - no result. Nothing was changed so previous setup is still online and working. However, it prompts errors on operator without any reasons even after applying same configuration with 2 'replicas' again.

I have another environment with same specific operator and MongoDB versions. I was able to add replica and even arbiter to specs there. That was also the only change made to CR of MongoDB.

The error I get: ERROR controllers/mongodb_status_options.go:104 Error deploying MongoDB ReplicaSet: error creating/updating StatefulSet: error creating/updating StatefulSet: StatefulSet.apps “mongodb” is invalid: spec: Forbidden: updates to statefulset spec for fields other than ‘replicas’, ‘ordinals’, ‘template’, ‘updateStrategy’, ‘persistentVolumeClaimRetentionPolicy’ and ‘minReadySeconds’ are forbidden

What did you do to encounter the bug? Steps to reproduce the behavior:

  1. Changed my CR MongoDB cluster yaml manifest database.yaml spec.members: 2 to spec.members: 3
  2. Use kubectl apply -f database.yaml
  3. Checked MongoDB statefulset pods (no replica was added).
  4. Checked MongoDB community operator logs. Observed the error.

What did you expect? I expected operator to add an additional member to existing MongoDB cluster in statefulset. Making members number 3 instead of existing 2.

What happened instead? Statefulset of MongoDB still had 2 members. MongoDB operator threw error (pasted in description).

Screenshots If applicable, add screenshots to help explain your problem.

Operator Information

Kubernetes Cluster Information

If possible, please include:

❯ k get mdbc NAME PHASE VERSION mongo Running 4.4.0

 - yaml definitions of your MongoDB Deployment(s):
 - `kubectl get mdbc -oyaml`

apiVersion: v1 items:

nammn commented 1 month ago

hey @KarooolisZi thanks for opening an Issue!

looking at your applied sts and applied cr one can see that you've increased the storage of the pvcClaim.

mdbc

        volumeClaimTemplates:
        - metadata:
            name: data-volume
          spec:
            accessModes:
            - ReadWriteOnce
            resources:
              requests:
                storage: 70G <---- look at me!
            storageClassName: ebs-sc
        - metadata:
            name: logs-volume
          spec:
            accessModes:
            - ReadWriteOnce
            resources:
              requests:
                storage: 10G

vs

sts

    volumeClaimTemplates:
    - apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        creationTimestamp: null
        name: data-volume
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 50G  <---- look at me!
        storageClassName: ebs-sc
        volumeMode: Filesystem
      status:
        phase: Pending
    - apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        creationTimestamp: null
        name: logs-volume
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 10G
        storageClassName: ebs-sc
        volumeMode: Filesystem

I suggest that you update your claim to be equal to the one in the sts and you should be fine

KarooolisZi commented 1 month ago

@nammn Thank you for such a fast response. I might missed this one. I can see my PVs are now 70GB yet sts config has 50GB. I remember changing storage because of the urgent need. Is there any chance to make sts use these PVs already created by mdbc. Or the only resolution is to either get back to old size in mdbc or recreate sts?

nammn commented 1 month ago

@KarooolisZi you can try the following to have the operator use the new pvc sizes, please note that this is a limitation of sts. You cannot resize the storage used by an sts. Read more here: https://github.com/kubernetes/enhancements/pull/4651/commits/763b35fd9272edff361162227377c4670f79d8ce

steps:

KarooolisZi commented 1 month ago

@nammn Yes, I am aware of limitation just thinking about the workaround. Would these steps guarantee sts works as previously and attaches required pvc?

nammn commented 1 month ago

yes, since the pvcs are having the same name and therefore the sts will re-attach them, all assuming you didn't change the name.

Closing this one, since the issue itself was a miss-configuration.

KarooolisZi commented 1 month ago

Yes, great misconfiguration. I have definitely spent too much time on it.. Thank you for your patience and swift response

KarooolisZi commented 1 month ago

@nammn After doing so, my sts shows as 0/0 with replicas 2 and does not recreate pods. I have one pod but it is not in sts somehow.

It looks like adding an arbiter crashes operator. Trying to do 2 members and 1 arbiter. 2 members work perfectly.

After getting it to work, I get 0/2 as mongo agent is not ready after adding arbiter. Strange behaviour

nammn commented 3 weeks ago

@KarooolisZi please open a new issue and provide the required information as given in the issue template

KarooolisZi commented 3 weeks ago

@nammn Did it: https://github.com/mongodb/mongodb-kubernetes-operator/issues/1615