search

mongodb / mongodb-kubernetes-operator

MongoDB Community Kubernetes Operator
Other
1.23k stars 501 forks source link

Operator doesn't start on IPv4 only k8s cluster if IPv6 enabled #715

Closed mkruliv closed 3 years ago

mkruliv commented 3 years ago

We need to create MongoDB clusters for Kubernetes that will work on IPv4 and IPv6 platforms, but when we set

  additionalMongodConfig:
    net.ipv6: true
    net.bindIpAll: true

the cluster doesn't start on IPv4 platform, for IPv6 only platform it works well Is it possible to have one config for both platforms?

What did you expect? We expect that the Kubernetes operator with that parameters should work on both ipv4 only and ipv6 only Kubernetes cluster

What happened instead? MongoDB Kubernetes operator with IP6 enabled doesn't start on IPv4 k8s cluster

Operator Information

Kubernetes Cluster Information

rodrigovalin commented 3 years ago

Hello @mkruliv

Thanks for reporting! We have not done any tests on dual-stack ipv4/ipv6 Kubernetes, but I would like to definitely try.

mkruliv commented 3 years ago

Hi @rodrigovalin! Thanks for quick response

  1. Currently we are using Kubernetes on VMware vSphere Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-25T14:50:46Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}
  2. CR YAML: 
    ---
apiVersion: mongodbcommunity.mongodb.com/v1
kind: MongoDBCommunity
metadata:
name: tst-mongo-service
namespace: default
spec:
members: 3
type: ReplicaSet
version: "4.4.8"
security:
authentication:
  modes: ["SCRAM-SHA-1"]
users:
- name: useradmin
  db: admin
  passwordSecretRef: 
    name: tst-useradmin-password
  roles:
    - name: userAdmin
      db: admin
  scramCredentialsSecretName:  tst-mongo-service
additionalMongodConfig:
net.ipv6: true
net.bindIpAll: true
storage.wiredTiger.engineConfig.cacheSizeGB: 1.2
mkruliv commented 3 years ago

If we don't set ipv6 parameter cluster starts successfully

rodrigovalin commented 3 years ago

Hello @mkruliv

I managed to reproduce your configuration using kind with the following configuration:

# kind_config.yaml
---
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  ipFamily: dual

And starting a cluster with:

kind create cluster --config kind_config.yaml

I've tested your configuration and I can confirm that the Operator does not work on dual-stack Kubernetes with ipv6 parameter. I've been investigating this a bit and the agent won't get to running state as you mention.

The problem seems to be the type of Service the operator creates:

$ kubectl get services
NAME                    TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)     AGE
tst-mongo-service-svc   ClusterIP   None         <none>        27017/TCP   1s

I managed to make the MongoDB resource to get to a running state by making the following changes on the Service (kubectl edit service/tst-mongo-service-svc). You have to add an entry to ipFamilies and change ipFamilyPolicy to PreferDualStack:

apiVersion: v1
kind: Service
metadata:
  name: tst-mongo-service-svc
  namespace: mongodb
spec:
  clusterIP: None
  clusterIPs:
  - None
  ipFamilies:
  - IPv6   #  <-- add this one here
  - IPv4
  ipFamilyPolicy: PreferDualStack   # <-- and change this one here
  ports:
  - name: mongodb
    port: 27017
    protocol: TCP
    targetPort: 27017
  publishNotReadyAddresses: true
  selector:
    app: tst-mongo-service-svc
  sessionAffinity: None
  type: ClusterIP

After a few minutes the MongoDB resource will reach "Running" state.

I will discuss with the team about supporting this particular IPv6 scenario, or maybe only document the process (manual configuration) for running the Operator in dual stack Kubernetes clusters.

rodrigovalin commented 3 years ago

Hey @mkruliv

I have edited this comment to only include relevant information

Unfortunatelly, the Operator, if running with net.ipv6: true, will always bind to :: no matter what you have in bindIp.

So the following configuration:

net.ipv6: true
net.bindIpAll: true

and

net.ipv6: true
net.bindIp: "::,0.0.0.0"

Will result in mongod binding to :: only. This is the behaviour of our automation agent, and nothing we can do about for now.

This means that if ipv6 is enabled, mongod will only bind to ipv6 addresses.

mkruliv commented 3 years ago

I can't find the MongoDB automation agent source code. Is it free or proprietary?

rodrigovalin commented 3 years ago

The agent's code is not open source unfortunately.

I understand you might have a very good reason to enable ipv6 on a SingleStack Kube cluster, but for now, and to make your resource to work, it has to be disabled.

mkruliv commented 3 years ago

@rodrigovalin thanks for the explanation

mkruliv commented 3 years ago

I'm going to close the ticket as I'm able to fix that issue by editing the mongo Kubernetes operator Go code (related to container run command) and building a new operator docker image and it works as expected (I need a few tests to confirm it). Not sure that I need to create PR with my changes here because it looks like a workaround. If someone needs the same please ping me. Thanks.

rodrigovalin commented 3 years ago

I think I'm interested in seeing your solution; if you don't mind, create that PR and we can discuss about that being a solution for this particular use case and how we can include it in the future.

mkruliv commented 3 years ago

@rodrigovalin I've pushed these changes to my fork, so you can check it there