arbiter not getting added to replicaset and and arbiter pod restart at first

achar007 commented 1 week ago

Name and Version

bitnami/mongodb

What architecture are you using?

amd64

What steps will reproduce the bug?

helm -n mongo-ha1 install my-mongo-ha1 bitnami/mongodb -f values-mon-ha1.yml

$ kubectl get all -n mongo-ha1
NAME                                 READY   STATUS    RESTARTS      AGE
pod/my-mongo-ha1-mongodb-0           1/1     Running   0             73m
pod/my-mongo-ha1-mongodb-1           1/1     Running   0             72m
pod/my-mongo-ha1-mongodb-arbiter-0   1/1     Running   1 (72m ago)   73m

NAME                                            TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)           AGE
service/my-mongo-ha1-mongodb-0-external         LoadBalancer   10.43.186.138   10.11.180.111   27017:30282/TCP   73m
service/my-mongo-ha1-mongodb-1-external         LoadBalancer   10.43.129.117   10.11.180.112   27017:32074/TCP   73m
service/my-mongo-ha1-mongodb-arbiter-headless   ClusterIP      None            <none>          27017/TCP         73m
service/my-mongo-ha1-mongodb-headless           ClusterIP      None            <none>          27017/TCP         73m

NAME                                            READY   AGE
statefulset.apps/my-mongo-ha1-mongodb           2/2     73m
statefulset.apps/my-mongo-ha1-mongodb-arbiter   1/1     73m

Are you using any custom parameters or values?

Here are my values.yml

architecture: replicaset
image:
  debug: true
auth:
  rootPassword: "mypassword"
externalAccess:
  enabled: true
  service:
    loadBalancerIPs:
      - 10.11.180.111
      - 10.11.180.112
persistence:
  storageClass: "nfs-lab"
arbiter:
  enabled: true
  initContainers:
    - name: wait-for-primary
      image: busybox
      command:
        - sh
        - -c
        - >
          until nslookup my-mongo-ha1-mongodb-0.my-mongo-ha1-mongodb-headless.mongo-ha1.svc.cluster.local &&
          nslookup my-mongo-ha1-mongodb-1.my-mongo-ha1-mongodb-headless.mongo-ha1.svc.cluster.local;
          do echo "Waiting for primary and secondary..."; sleep 5; done

What is the expected behavior?

$ mongosh -u root -p "mypassword"  --host=10.11.180.11 --port=27017 --eval 'rs.status()' | grep name
      name: '10.11.180.111:27017',
      name: '10.11.180.112:27017',

This is missing the arbiter node.

What do you see instead?

It is missing out arbiter node in rs.status() output moreover arbiter is trying to connect to fqdn of node name which is incorrect...

$ kubectl logs -f my-mongo-mongodb-arbiter-0 -n mongo-ha1
mongodb 10:49:03.38 INFO  ==>
mongodb 10:49:03.38 INFO  ==> Welcome to the Bitnami mongodb container
mongodb 10:49:03.38 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
mongodb 10:49:03.38 INFO  ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
mongodb 10:49:03.39 INFO  ==> Upgrade to Tanzu Application Catalog for production environments to access custom-configured and pre-packaged software components. Gain enhanced features, including Software Bill of Materials (SBOM), CVE scan result reports, and VEX documents. To learn more, visit https://bitnami.com/enterprise
mongodb 10:49:03.39 INFO  ==>
mongodb 10:49:03.39 INFO  ==> ** Starting MongoDB setup **
mongodb 10:49:03.40 INFO  ==> Validating settings in MONGODB_* env vars...
mongodb 10:49:03.45 INFO  ==> Initializing MongoDB...
mongodb 10:49:03.53 INFO  ==> Writing keyfile for replica set authentication...
mongodb 10:49:03.54 INFO  ==> Deploying MongoDB from scratch...
MongoNetworkError: connect ECONNREFUSED 10.42.203.45:27017
mongodb 10:49:05.00 INFO  ==> Creating users...
mongodb 10:49:05.00 INFO  ==> Users created
mongodb 10:49:05.02 INFO  ==> Enabling authentication...
mongodb 10:49:05.06 INFO  ==> Configuring MongoDB replica set...
mongodb 10:49:05.07 INFO  ==> Stopping MongoDB...
mongodb 10:49:08.41 INFO  ==> Trying to connect to MongoDB server my-mongo-mongodb-0.my-mongo-mongodb-arbiter-headless.narayana.svc.cluster.local...
cannot resolve host "my-mongo-mongodb-0.my-mongo-mongodb-arbiter-headless.narayana.svc.cluster.local": lookup my-mongo-mongodb-0.my-mongo-mongodb-arbiter-headless.narayana.svc.cluster.local: no such host
cannot resolve host "my-mongo-mongodb-0.my-mongo-mongodb-arbiter-headless.narayana.svc.cluster.local": lookup my-mongo-mongodb-0.my-mongo-mongodb-arbiter-headless.narayana.svc.cluster.local: no such host
cannot resolve host "my-mongo-mongodb-0.my-mongo-mongodb-arbiter-headless.narayana.svc.cluster.local": lookup my-mongo-mongodb-0.my-mongo-mongodb-arbiter-headless.narayana.svc.cluster.local: no such host
cannot resolve host "my-mongo-mongodb-0.my-mongo-mongodb-arbiter-headless.narayana.svc.cluster.local": lookup my-mongo-mongodb-0.my-mongo-mongodb-arbiter-headless.narayana.svc.cluster.local: no such host
cannot resolve host "my-mongo-mongodb-0.my-mongo-mongodb-arbiter-headless.narayana.svc.cluster.local": lookup my-mongo-mongodb-0.my-mongo-mongodb-arbiter-headless.narayana.svc.cluster.local: no such host
cannot resolve host "my-mongo-mongodb-0.my-mongo-mongodb-arbiter-headless.narayana.svc.cluster.local": lookup my-mongo-mongodb-0.my-mongo-mongodb-arbiter-headless.narayana.svc.cluster.local: no such host
cannot resolve host "my-mongo-mongodb-0.my-mongo-mongodb-arbiter-headless.narayana.svc.cluster.local": lookup my-mongo-mongodb-0.my-mongo-mongodb-arbiter-headless.narayana.svc.cluster.local: no such host
mongodb 10:49:43.51 ERROR ==> Unable to connect to host my-mongo-mongodb-0.my-mongo-mongodb-arbiter-headless.narayana.svc.cluster.local
mongodb 10:49:43.51 INFO  ==> Stopping MongoDB...

env variable MONGODB_INITIAL_PRIMARY_HOST on arbiter node and secondary should be different as their headless service name is different

$ env | grep MONGODB_INITIAL_PRIMARY_HOST
MONGODB_INITIAL_PRIMARY_HOST=my-mongo-ha1-mongodb-0.my-mongo-ha1-mongodb-arbiter-headless.mongo-ha1.svc.cluster.local

$ env | grep MONGODB_INITIAL_PRIMARY_HOST
MONGODB_INITIAL_PRIMARY_HOST=my-mongo-ha1-mongodb-0.my-mongo-ha1-mongodb-headless.mongo-ha1.svc.cluster.local

The actual value of MONGODB_INITIAL_PRIMARY_HOST on arbiter is supposed to be my-mongo-ha1-mongodb-0.my-mongo-ha1-mongodb-headless.mongo-ha1.svc.cluster.local

somewhere it's getting modified to use the local service name on the arbiter which is incorrect and hence container restarted and not able to add to replicaset.

carrodher commented 4 days ago

Thank you for bringing this issue to our attention. We appreciate your involvement! If you're interested in contributing a solution, we welcome you to create a pull request. The Bitnami team is excited to review your submission and offer feedback. You can find the contributing guidelines here.

Your contribution will greatly benefit the community. Feel free to reach out if you have any questions or need assistance.

achar007 commented 4 days ago

following change would help fixing this issue, I appreciate if maintainers can review this change in _helpers.tpl

{{/*
Get initial primary host to configure MongoDB cluster.
*/}}
{{- define "mongodb.initialPrimaryHost" -}}
{{- if contains "arbiter" (include "mongodb.fullname" .) -}}
  {{/* Use the service name of the first pod in the MongoDB StatefulSet when this is the arbiter */}}
  {{ ternary 
       ( printf "%s-0.%s-headless.$(MY_POD_NAMESPACE).svc.%s" 
         (include "mongodb.fullname" . | replace "arbiter" "") 
         (include "mongodb.fullname" . | replace "arbiter" "") 
         .Values.clusterDomain ) 
       ( first .Values.externalAccess.service.publicNames ) 
       ( empty .Values.externalAccess.service.publicNames ) 
  }}
{{- else -}}
  {{/* Use the standard service name for MongoDB secondaries or primary */}}
  {{ ternary 
       ( printf "%s-0.$(K8S_SERVICE_NAME).$(MY_POD_NAMESPACE).svc.%s" 
         (include "mongodb.fullname" .) 
         .Values.clusterDomain ) 
       ( first .Values.externalAccess.service.publicNames ) 
       ( empty .Values.externalAccess.service.publicNames ) 
  }}
{{- end -}}
{{- end -}}

bitnami / charts