Possibility to choose nodeport used port

[MikeDevresse]

Hi, I want to deploy a database instance and expose it, my idea was to use a nodeport but I also wanted to choose the port, bur from what I saw there is no way to do that. I am using the helm chart to deploy my instance but from what I saw it seems to be an issue coming from the operator. I want to use a nodeport to make things easier and I want to choose the port to keep it consistant between my clusters

[spron-in]

Hello @MikeDevresse ,

well, we thought about it. There are a couple of problems with static NodePort:

1. Multiple Replica Set Pods on the same k8s node. It basically means that instead of simple expose section:

   expose:
   exposeType: NodePort
   port: 11111

   we will need to specify unique port for each replica set node. Something like this:

   expose:
   exposeType: NodePort
   ports:
   rs0-0: 11111
   rs0-1: 22222
   rs0-2: 33333

2. Another problem is horizontal scaling - adding more nodes into the replica set. If you don't specify NodePort, it means that new node will have a random port. Is it desired? Or we can pre-define ports, which is suboptimal as these ports might be used already at the moment of scaling.

As you see there are bunch of problems and logical loopholes. Do you have any thoughts here?

[MikeDevresse]

Why do you need to set a port for each instance ? you only need one port for each RS right ? I think it can be done like this:

replsets:
  - name: rs0
    size: 3
    expose:
      exposeType: NodePort
      nodePort: 30004
  - name: rs1
    size: 3
    expose:
      exposeType: NodePort
      nodePort: 30003
sharding:
  enabled: true
  balancer:
    enabled: true
  configrs:
    size: 3
    expose:
      exposeType: NodePort
      nodePort: 30002
  mongos:
    size: 2
    expose:
      exposeType: NodePort
      nodePort: 30001

I don't even think it is required for configdb and replicaSets but why not. My use case is to set it on the mongos, and with a static port that I choose which are consistant between environments, I can set approprietly my VPN rules. Right now it is impossible to predict and the port will always be different between environments.

[spron-in]

  mongos:
    size: 2
    expose:
      exposeType: NodePort
      nodePort: 30001

The problem will occur if these mongos instances land on the same k8s node. Most probably you are going to avoid that through affinity rules, but it does not mean that the problem is not there.

We can think of implementing it for a narrow use case. Let me discuss it with the team.

Meanwhile, could you pls confirm that you rely on affinity?

[MikeDevresse]

I'm not sure I understand, the nodeport is unique no matter how many mongos we set, it's the kube-proxy that redirects so it doesn't matter if we have anti-affinity or not.

[spron-in]

Sorry, it is my bad. It will not be an issue for mongos. But it will be a problem for replica sets - where we have a service per pod. So mongos is an option to have it.

Do you think you can send a PR or work with us on it?

[MikeDevresse]

Can try but not sure at all

[ath88]

This would also be interesting for me. When I run Percona Server for MongoDB on a local Kind cluster for development, I would like to expose just a single rs instance as a NodePort, to be able to reach it from outside of the cluster. With an unpredictable port number this becomes cumbersome.

[spron-in]

Hey. So we are planning to have a realease this quarter for Operator for MongoDB. We will review the PR that @MikeDevresse shared and see if it can be included.

There is a caveat though. Currently you can expose mongos in two different ways:

1. Through a single service
2. Through a service per pod (where each mongos pod gets its own service).

The solution that you request is about having a nodeport for option (1). Which will work. But, we introduced service per pod for mongos for a reason. When the cluster is accessed through mongos and multiple threads execute a transaction using the same driver instance, a transaction may end up being executed on different mongos. This results in the following error:

$$$ Command insert failed: cannot continue txnId 530 for session 8694022c-4d3f-4926-8e9d-9e6581831511 - uzGxMP6dwuKBbzh5yTEf0uqrg97E2XVlv3D6zMI0/QE= with txnId 536.

See more here: https://perconadev.atlassian.net/browse/K8SPSMDB-599

So technically, it is recommended to use service per pod for mongos, but there are cases where single service will work fine for you.

[MikeDevresse]

why not add an option to decide whether we want a single service or service per pod ?