Health of etcd project

[dims]

The etcd project is a critical dependency of Kubernetes. The project has been struggling with both turnover of critical folks and lack of new folks with sufficient knowledge/experience to offset the loss of members.

original email from maintainers of etcd: https://groups.google.com/a/kubernetes.io/g/steering/c/e-O-tVSCJOk/m/N9IkiWLEAgAJ

More recent health check: https://groups.google.com/a/kubernetes.io/g/steering/c/e-O-tVSCJOk/m/GHUB6jl5BAAJ

Proposal for mentorship: https://github.com/etcd-io/etcd/issues/14243

last governance change: https://github.com/etcd-io/etcd/pull/14053

Deep dive from one of the new maintainers: Benjamin Wang from VMware, had presented to Runtime TAG in June 2022. https://docs.google.com/presentation/d/e/2PACX-1vRGyr3gSUJVm23Zv6bQNmZRaCeU-2nQD1U1vBEbZJZc2vhEbN4w8Rxf6e-01L1B8w/pub?start=false&loop=false&delayms=3000&slide=id.p1

[dims]

Process to cleanup maintainer list: https://github.com/etcd-io/etcd/pull/14238

[ahrtr]

Update versioning support :

• https://github.com/etcd-io/website/pull/601
• https://github.com/etcd-io/etcd/pull/14306

[serathius]

Update on maintainer cleanup: We introduced procedure for removing maintainers, analysed activity and proposed to remove 6 inactive maintainers https://github.com/etcd-io/etcd/pull/14373, leaving etcd with 6 maintainers.

Seems like enough but I expect that within next 6 months another 2 maintainers will leave the project.

[dims]

Chart from devstats

Things are looking up!

[mrbobbytables]

I think that graph hides a lot of details - those contributions are up, but most are from 2 people (looking at PRs + PR reviews)

https://etcd.devstats.cncf.io/d/9/developer-activity-counts-by-repository-group-table?orgId=1&var-period_name=Last%20year&var-metric=reviews&var-repogroup_name=All&var-country_name=All

It becomes even more evident when you look at the etcd repo itself (where contributors are REALLY needed) and start looking at commits / PRs etc

https://etcd.devstats.cncf.io/d/9/developer-activity-counts-by-repository-group-table?orgId=1&var-period_name=Last%20year&var-metric=commits&var-repogroup_name=etcd&var-country_name=All

[caniszczyk]

Bob is definitely right here, I think you need to look at health in multiple ways (not just contributions but # of active maintainers)

On Tue, May 2, 2023 at 9:23 AM Bob Killen @.***> wrote:

I think that graph hides a lot of details - those contributions are up, but most are from 2 people (looking at PRs + PR reviews)

https://etcd.devstats.cncf.io/d/9/developer-activity-counts-by-repository-group-table?orgId=1&var-period_name=Last%20year&var-metric=reviews&var-repogroup_name=All&var-country_name=All

It becomes even more evident when you look at the etcd repo itself (where contributors are REALLY needed) and start looking at commits / PRs etc

https://etcd.devstats.cncf.io/d/9/developer-activity-counts-by-repository-group-table?orgId=1&var-period_name=Last%20year&var-metric=commits&var-repogroup_name=etcd&var-country_name=All

— Reply to this email directly, view it on GitHub https://github.com/cncf/toc/issues/898#issuecomment-1531573670, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAPSIPOTYS54VGLSATWPYDXEEKGZANCNFSM56S5DS6Q . You are receiving this because you are subscribed to this thread.Message ID: @.***>

-- Cheers,

Chris Aniszczyk https://aniszczyk.org

[logicalhan]

I agree with Bob. Two people account for 70% of the merged PRs, which definitely doesn't seem healthy.

[jmhbnz]

Sharing a couple of quick thoughts as a new contributor that has been making an effort to get involved this year. I think the adoption of a more structured and intentional membership ladder has been a step in the right direction.

There has definitely been an uptick in contributors and lately, and some of the basic maintenance for the project is now shared amongst contributors rather than just being handled by one maintainer which is positive.

It was also heartening to see a decent turnout at the etcd contribfest at kubecon eu although most of the contributions merged there and shortly afterwards were very simple or docs maintenance. The interesting thing will be seeing if any of those brand new contributors stick around. One has three pr's merged now and is working on their fourth.

To finish, the challenge with etcd in my mind is the complexity of the codebase combined with the subject matter expertise. Some positive steps have been taken recently for the project, though it just takes quite a while for folks to get to the point of being able to confidently operate as a maintainer I think. If we can keep up the momentum from recently then I'm optimistic the project health will improve.

[ahrtr]

I think it (etcd's health status) is definitely a little better now.

• We recently on-boarded 7 new members, and most of them keep active in the past a couple of months. They help to triage & fix issues, answer questions, fix flaky test cases, maintain document, etc. They definitely share the burden of maintainers.
• Given the complexity of the etcd codebase, it may need some time for some new members & contributors to dig into the etcd core (let alone the bbolt and raft). I believe it gets better and better because basically all the new members and some contributors have big passion on etcd, and some members & contributors already have very deep understanding on etcd's codebase.

[jberkus]

We've also started a stepped-up contributor/reviewer recruitment program, kicked off at the recent Kubecon EU. Our only real way out of this bind is organized contributor development, so that's what we'll be doing. Look for a full program proposal soon.

Kubecon was really good as a kick-off; we got a bunch of new contributors interested/involved. Now we need to keep the momentum going.

[dims]

xref: https://github.com/etcd-io/etcd/issues/15875

[shreemaan-abhishek]

How about lfx/gsoc mentorship projects under etcd?

[amye]

Closing, this is no longer active with TOC. (Ping me if you have other comments!)

[BenTheElder]

[EDIT: Speaking for myself only as a member of The Kubernetes Steering Committee]

Linking back: https://github.com/kubernetes/community/pull/7372

The Kubernetes Steering Committee has reviewed & unanimously approved a proposal to bring etcd into Kubernetes as a top-level Special Interest Group, etcd will continue to be run by the etcd maintainers but will receive support from the relevant Kubernetes teams (SIS K8s Infra for CI/Hosting resources etc, Security Response Committee for triaging vulnerability reports and running the bug bounty program, Code of Conduct Committee, the GitHub Management subproject, ... etc).

We hope that deduplicating this sort of meta-organization support work and raising more visibility to Kubernetes vendors will help the etcd project, however for clarity this does not mean etcd is "controlled by kubernetes" or "only supporting kubernetes" and the technical direction will continue to be set by existing etcd leadership, etcd will be an independent SIG on the same tier as any other technical or non-technical ownership group in the broader Kubernetes organization.

As part of this etcd has bootstrapped leaders for the SIGs and some more folks have stepped back up to support this effort.

With respect to technical direction between etcd and SIG API-Machinery: there is also a new sub-project under SIG Etcd to define the currently implicit technical contract between the Kubernetes API Server and etcd.

There are still some ongoing procedural bits to sort out, but https://github.com/kubernetes/community/pull/7372 has merged and things are currently moving forward.