Investigate support for externally managed etcd clusters

[nprokopic]

Motivation

Cluster API is getting support for externally managed etcd clusters:

• https://github.com/kubernetes-sigs/cluster-api/issues/7399
• https://github.com/kubernetes-sigs/cluster-api/pull/7525

Todos

- [ ] List all the benefits of externally managed etcd clusters and if those would apply to our existing clusters
- [ ] List all the downsides of externally managed etcd clusters and if those would apply to our existing clusters

Outcome

• Understand if we want to support externally managed etcd clusters

[njuettner]

I haven't looked at the current idea how upstream wants to implement it but here are my thoughts

Some benefits which I can see:

• In large clusters we could store things like K8s events into a different etcd cluster (optionally), which would solve some incidents in the past
• Increase the etcd cluster without increasing control plane instances
• Updating etcd without touching anything else

Some downsides which I can see:

• Increases the complexity of clusters (who is creating the Loadbalancer for etcd, who is creating DNS records, how do we update etcd, how do we store backups, ...)

[puja108]

From a product perspective I think that's a decision that Turtles should be taking based on data and use cases around our etcd / api server reliability, scalability, and performance, most probably you'd want to start also measuring these things before moving to external etcd(s)

Generally, from what I've seen in the community, sooner or later you won't get around that at scale. Question is if we need it already, as like Nick mentioned it will increase complexity and is definitely some significant work to be done.

[stone-z]

Shield has some use cases which go one step further to involve external storage, which is not necessarily etcd.

Short version

We'd like to be able to send certain cluster resources to an alternative "etcd" to avoid impacting cluster operations and give us more storage and usage flexibility.

Long version

For security tools to be useful, they have to generate data. That data must be stored somewhere, so most tools produce reports which are currently stored as cluster resources. The reports themselves do not require many of the guarantees provided by etcd, but storing them as cluster resources has tremendous benefits for the platform as a whole.

The problem we have is similar to the problem with Events. Like Events, report data can be prone to high volumes of noisy operations (e.g. heavy scanning activity produces lots of reports + events) which can degrade the performance of etcd for more important operations.

Additionally, there are clusters which are large and complex enough to produce enough data to completely fill etcd (>8GB) if we stored all the data we wanted.

If possible, we would configure the API server to use a secondary etcd instance to store those reports. This secondary etcd instance would actually be kine or a similar service impersonating etcd, but which would give us more flexibility (e.g. no 8GB limit).

Some other discussion in this slack thread.

[bavarianbidi]

For the specific etcd topic it might be also an option to have e.g. one central etcd-event-cluster (probably in the MC itself) and all the WC-clusters are sending the events (stored under /events) to that central etcd-event-cluster:

--etcd-servers="https://etcd-0:2379,https://etcd-1:2379,https://etcd-2:2379" \
--etcd-servers-overrides="/events#https://etcd-3:2379,https://etcd-4:2379,https://etcd-5:2379"

source

[puja108]

AFAIK you can use the technique Mario mentioned also for other paths, so as long as you can isolate the paths that are used by your resources that might be a very clean way to "shard" our etcd usage. I've heard companies like alibaba using this technique at scale

[stone-z]

Yes we're familiar with the possibility and it has been tried at Giant Swarm, but it currently has tradeoffs https://github.com/giantswarm/giantswarm/issues/26106. Official support during cluster creation might help