RFE: Control plane support for various ways to provide a stable API endpoint

[dlipovetsky]

User Story

As an operator, I would like to choose my own way to provide a stable API endpoint, as required by the default control plane management implementation.

Detailed Description

The default control plane management implementation requires a stable API endpoint. This endpoint

• MUST route API requests to a live kube-apiserver replica, if one is available.
• MUST be defined before the first control plane replica is created, so that it is added to the SubjectAltNames of the kube-apiserver server certificate.
• MUST stay the same for the entire life of the cluster.
• MUST be a DNS record, or an IP.

For reference: Today, the stable API endpoint is provided either by the operator (e.g. using an existing LB, or a VIP backed by keepalived), or by the infra provider (e.g. an ELB created by CAPA). There is a proposal to make an explicit provider for this endpoint: #1250.

/cc @detiber @randomvariable

[vincepri]

/kind proposal

[ncdc]

@dlipovetsky the proposed KubeadmControlPlane has .spec.kubeadmConfigSpec.clusterConfiguration.controlPlaneEndpoint. I believe a user should be able to fill that in with any value. How does that figure in to this RFE - does it address it?

[dlipovetsky]

@dlipovetsky the proposed KubeadmControlPlane has .spec.kubeadmConfigSpec.clusterConfiguration.controlPlaneEndpoint. I believe a user should be able to fill that in with any value. How does that figure in to this RFE - does it address it?

Yes.

--

We may want to call out the above requirements in the proposal (#1613 . In any case, @detiber asked me to open an issue to anchor the discussion, rather than leaving comments on the PR.

[ncdc]

I wonder if we need to at least include something in the CAEP to have kubeadmControlPlane.spec.kubeadmConfigSpec.clusterConfiguration.controlPlaneEndpoint be populated from cluster.status.apiEndpoints if the controlPlaneEndpoint is empty? Or, to ask a related question, how do we envision something like CAPA working with KubeadmControlPlane where CAPA creates the ELB + the cluster's api endpoint when the AWSCluster is reconciled - how does ...controlPlaneEndpoint get filled in? Or is that manual?

[detiber]

Or is that manual?

That would be a major regression from previous releases for providers that support Load Balancers...

[ncdc]

Right, so until (if) we have a proper LoadBalancer story, should we update the CAEP to add some text to describe how we maintain the existing behavior w/o requiring manual intervention to set the endpoint?

[moshloop]

be populated from cluster.status.apiEndpoints if the controlPlaneEndpoint is empty

Why not have the infra provider create the LB and update the controlPlaneEndpoint if it is empty and then remove cluster.status.apiEndpoints or make it a read-only mirror of controlPlaneEndpoint and then you would have consistent behaviour for both user and infra provided LB's

[detiber]

Why not have the infra provider create the LB and update the controlPlaneEndpoint if it is empty and then remove cluster.status.apiEndpoints or make it a read-only mirror of controlPlaneEndpoint and then you would have consistent behaviour for both user and infra provided LB's

I think the concern here is that it would require tight coupling with the kubeadmControlPlane rather than being a more generic mechanism that would work across different controlplane and bootstrapping providers.

[ncdc]

Exactly. I'd like there to be an eventual interaction between a LoadBalancer (LB) resource and the KubeadmControlPlane (KCP), and maybe the Cluster too, where the KCP knows to pull the LB's endpoint if it exists (could be KCP pulls from LB, or KCP gets Cluster, see it has an LBRef, and follows that).

[moshloop]

So then let's make controlPlaneEndpoint a first-level field of cluster itself?

KCP could then wait until it is populated by a LB

[ncdc]

then you would have consistent behaviour for both user and infra provided LB's

Also, assuming we proceed with your LB proposal @moshloop, I think a natural progression is that there are no more "infra provided LBs" in the sense that the "cluster + machine infra provider" is separate from "LB provider"

[ncdc]

xref #1197 - I assume that ties into this

[dlipovetsky]

So then let's make controlPlaneEndpoint a first-level field of cluster itself?

KCP could then wait until it is populated by a LB

This seems reasonable.

If an LB provider did exist, I suppose it would, after creating the load balancer, update the status of a CR that the Cluster object would reference. Until we figure out the schema of that CR and use a reference, an inline field works.

[vincepri]

/priority critical-urgent /lifecycle active