Highly available master components (Kubernetes on Kubernetes)

[HristoA]

What would you like to be added: Option to put Master Nodes(cluster control-planes) inside K8s cluster that they control

Why is this needed: For purpose of HA we need minimum 3 Masters and 3 Workers nodes. We can say that we had 2 clusters: 1 from Master nodes and 1 from Worker nodes

• Negative of existing approach:
  • needs to manage 2 cluster
  • with 6 VPS/BM we achieve only FT=1
    • bad utilization of server power in case of BM for master nodes (in case of needs to setup private K8s from BM)

• Positives of suggested approach:
  • manage only one cluster of VPS/BM
  • with the same amount of VPS/BM= 6 we achieve double FT=2
  • more good utilization of server power (master nodes are inside K8s)
  • auto healing of master component thanks to K8s specific
  • easy scaling of master nodes(pods)

Use cases:

• Proposal is very useful when business needs private BM architecture.

Example cluster.yml:

• Add new key type: default| ruler. This will allow using existing approach and suggested one. If this key value is ruler no need for key role

hosts:

• address: 10.0.0.1 user: root type: ruler ssh_key_path: ~/.ssh/my_key

[jnummelin]

Option to put Master Nodes(cluster control-planes) inside K8s cluster that they control

I'm not really sure if I understand this correctly. :/ Do you mean there would be one cluster where we could run another set of control planes for other clusters? I.e. deploy all the control plane components into existing K8S cluster and expose the API out somehow?

Proposal is very useful when business needs private BM architecture.

What do you mean with the term BM?

If you do mean running possibly many control planes in some existing cluster, I'm not really sure how it fits into Pharos tooling and distribution "vision". Maybe just write a Helm chart to deploy all the needed things (etcd + api-server + controller + scheduler + ...) and expose the API via Ingress

[HristoA]

First thanks @jnummelin for replay! May be i miss something in architecture concept of K8s but let me try to describe my idea again:

BM - Bare Metal

Control planes is deployed to the same cluster that they control and in general they will control himself also. We benefit with:

• easy scaling for control planes(master nodes) just like other deployments
• one unit less for worry ( master nodes servers will not be needed) I hope that i am more clear now Thanks

[jakolehm]

One option might be microvm pods (like firecracker). Then we could use "normal" pharos up command for managing those control-planes if they just open a ssh port via NodePort (or similar).