Extend usage of ControlPlaneEndpoint for HA

fabriziopandini commented 6 years ago

The goal of this issue is to discuss possible changes to the current implementation of ControlPlaneEndpoint that will facilitate the implementation of HA in kubeadm

/cc @timothysc @stevesloka @mattkelly @mattkelly @mbert @jamieh

Background

Issue https://github.com/kubernetes/kubeadm/issues/411 and related PR https://github.com/kubernetes/kubernetes/pull/59288 introduced a new configuration entry named API.ControlPlaneEndpoint. This entry allows to specify a DNS name that should be used for composing the API server address that kubeadm embeds in several components - scheduler.conf, controller-manager.conf, admin.conf + kubelet.conf, kube-proxy configMap, kubadm-public configMap (used for discovery/TLS bootstrap).

The kube-apiserver manifest instead continues to use the API.AdvertiseAddress entry.

This issue focuses on the possibility to use the API.ControlPlaneEndpoint to explicitly declare the load balancer address in an HA scenario.

User Stories

User story 1. Single node cluster, upgradable to HA

A user creates an initial master node with kubeadm init passing the following advanced configuration options in the master configuration file :

the address of an external load balancer in the API.ControlPlaneEndpoint entry
the address of an external etcd cluster in the API.etcd entry

The resulting single master cluster could be upgraded to HA at any time by joining additional master nodes using kubeadm join —master.

New master nodes will join the cluster and advertise their-self with their own advertise address, that is the IP of the default network interface if not otherwise specified using the kubeadm join --master --advertise-address flag.

User story 2. Single node cluster, not upgradable to HA

If a user creates an initial master node without setting the API.controlPlaneEndpoint address or without declaring an etcd cluster via the API.etcd entry , the cluster can't be upgraded to HA and kubeadm join —master will return an error.

How this will work?

When joining a new master node we will generate a dedicates API server certificate configured for serving request addressed to the ControlPlaneEndpoint and to the advertise-address of the specific node only.

The existence of many different API server certificates is considered an acceptable trade-off for supporting the desired kubeadm join --master dynamic workflow.

Proposed changes

make the ControlPlaneEndpoint accept both IP and DNS
improve HA KEP including the possibility to provide the --advertise-address for the joining master node and adapting the flow to the condition described above ( API.controlPlaneEndpoint and API.etcd entry must be set).

Open points to be addressed Self hosting with certificates stored in secrets doesn't allow to have different API server certificates for each master nodes.

As a tactical solution, in case of self hosting with certificates stored in secrets , we will allow kubeadm join --master only in case of the joining master node has advertise-address equal to ControlPlaneEndpoint address, and thus the existing certificate can be used without changes.

As a long term solution, we will implement a logic that creates a unique certificate with all the advertise addresses (existing masters + joining master), and replaces the existing API server certificate with the new one.

erik-stephens commented 6 years ago

Apologies if this isn't the best forum for this comment. I don't see support for specifying the port. Trying to include the port in the value gives a validation error. Looks like 6443 is hard-coded somewhere. In general, when I see the word "endpoint", I think "scheme://host:port/path". I don't know if helpful - feel free to ignore.

rbtr commented 6 years ago

@erik-stephens that's a separate config entry: API.BindPort. the default is 6443

erik-stephens commented 6 years ago

Surprising that something called "bind ..." does more than impact what port a service will listen on. Based on my experience with both 1.10 & 1.11, I'm seeing that impact both the port the kube-apiserver will listen on as well the port clients will use to connect to the control plane endpoint.

Wed Jul 18 22:27:03 root@xenial1-minion:/etc/kubernetes
netstat -plunt | grep 443
tcp6       0      0 :::443                  :::*                    LISTEN      10671/kube-apiserver

Wed Jul 18 22:27:08 root@xenial1-minion:/etc/kubernetes
grep server admin.conf
    server: https://k8s-test:443

Wed Jul 18 22:27:12 root@xenial1-minion:/etc/kubernetes
head config.yaml
api:
    advertiseAddress: 10.167.0.3
    # advertiseAddress: 10.167.0.2
    bindPort: 443
    controlPlaneEndpoint: k8s-test
...

Trying to summarize my expectations as someone on-prem with not so much kubernetes experience:

Any "bind" related fields should govern the interface & port that the kube-apiserver will listen on.
Any "advertise" or "control plane" related fields should refer to the external load balancer.
Any "endpoint" field should encapsulate the scheme://host:port/path which should cover most all cases for reverse-proxying a service.

timothysc commented 6 years ago

I'm closing this one as the other PRs are up regarding clusterstatus and controlplane endpoint has moved.

kubernetes / kubeadm

Extend usage of ControlPlaneEndpoint for HA #726