Mongo Kubernetes Replica Set Sidecar

This project is as a PoC to setup a mongo replica set using Kubernetes. It should handle resizing of any type and be resilient to the various conditions both mongo and kubernetes can find themselves in.

How to use it

The docker image is hosted on docker hub and can be found here: https://hub.docker.com/r/cvallance/mongo-k8s-sidecar/

An example kubernetes replication controller can be found in the examples directory on github here: https://github.com/cvallance/mongo-k8s-sidecar

There you will also find some helper scripts to test out creating the replica set and resizing it.

Settings

In its default configuration the sidecar uses the pods' IPs for the MongodDB replica names. Here is a trimmed example:

[ { _id: 1,
   name: '10.48.0.70:27017',
   stateStr: 'PRIMARY',
   ...},
 { _id: 2,
   name: '10.48.0.72:27017',
   stateStr: 'SECONDARY',
   ...},
 { _id: 3,
   name: '10.48.0.73:27017',
   stateStr: 'SECONDARY',
   ...} ]

If you want to use the StatefulSets' stable network ID, you have to make sure that you have the KUBERNETES_MONGO_SERVICE_NAME environmental variable set. Then the MongoDB replica set node names could look like this:

[ { _id: 1,
   name: 'mongo-prod-0.mongodb.db-namespace.svc.cluster.local:27017',
   stateStr: 'PRIMARY',
   ...},
 { _id: 2,
   name: 'mongo-prod-1.mongodb.db-namespace.svc.cluster.local:27017',
   stateStr: 'SECONDARY',
   ...},
 { _id: 3,
   name: 'mongo-prod-2.mongodb.db-namespace.svc.cluster.local:27017',
   stateStr: 'SECONDARY',
   ...} ]

StatefulSet name: mongo-prod. Headless service name: mongodb. Namespace: db-namespace.

Read more about the stable network IDs here.

An example for a stable network pod ID looks like this: $(statefulset name)-$(ordinal).$(service name).$(namespace).svc.cluster.local. The statefulset name + the ordinal form the pod name, the service name is passed via KUBERNETES_MONGO_SERVICE_NAME, the namespace is extracted from the pod metadata and the rest is static.

A thing to consider when running a cluster with the mongo-k8s-sidecar is that it will prefer the stateful set stable network ID to the pod IP. It is however compatible with replica sets, configured with the pod IP as identifier - the sidecar should not add an additional entry for it, nor alter the existing entries. The mongo-k8s-sidecar should only use the stable network ID for new entries in the cluster.

Finally if you have a preconfigured replica set you have to make sure that:

• the names of the mongo nodes are their IPs
• the names of the mongo nodes are their stable network IDs (for more info see the link above)

Example of compatible mongo replica names:

10.48.0.72:27017 # Uses the default pod IP name
mongo-prod-0.mongodb.db-namespace.svc.cluster.local:27017 # Uses the stable network ID

Example of not compatible mongo replica names:

mongodb-service-0 # Uses some custom k8s service name. Risks being a duplicate entry for the same mongo.

If you run the sidecar alongside such a cluster, it may lead to a broken replica set, so make sure to test it well before going to production with it (which applies for all software).

MongoDB Command

The following is an example of how you would update the mongo command enabling ssl and using a certificate obtained from a secret and mounted at /data/ssl/mongodb.pem

Command

        - name: my-mongo
          image: mongo
          command:
            - mongod
            - "--replSet"
            - heroku
            - "--bind_ip"
            - 0.0.0.0
            - "--smallfiles"
            - "--noprealloc"
            - "--sslMode"
            - "requireSSL"
            - "--sslPEMKeyFile"
            - "/data/ssl/mongodb.pem"
            - "--sslAllowConnectionsWithoutCertificates"
            - "--sslAllowInvalidCertificates"
            - "--sslAllowInvalidHostnames"

Volume & Volume Mount

          volumeMounts:
            - name: mongo-persistent-storage
              mountPath: /data/db
            - name: mongo-ssl
              mountPath: /data/ssl
        - name: mongo-sidecar
          image: cvallance/mongo-k8s-sidecar:latest
          env:
            - name: MONGO_SIDECAR_POD_LABELS
              value: "role=mongo,environment=prod"
            - name: MONGO_SSL_ENABLED
              value: 'true'
      volumes:
        - name: mongo-ssl
          secret:
            secretName: mongo

Creating Secret for SSL

Use the Makefile:

export MONGO_SECRET_NAME=mongo-ssl
export KUBECTL_NAMESPACE=default
cd examples && make generate-certificate

or

Generate them on your own and push the secrets kube create secret generic mongo --from-file=./keys where keys is a directory containing your SSL pem file named mongodb.pem

Debugging

TODO: Instructions for cloning, mounting and watching

Still to do

• Add tests!
• Add to circleCi
• Alter k8s call so that we don't have to filter in memory