Community Meeting May 18, 2021

[imjasonh]

Tuesday May 18, at noon Eastern, 9am Pacific, 4pm UTC; find your time

Meet link: https://meet.google.com/squ-dtxk-xdi

This issue will collect prospective agenda topics. Add topics you'd like to discuss below.

• progress update; changes this week
• how do people want to run kcp? options include:
  • embedded as a library
  • separate kcp / cluster-controller-with-syncer ("push")
  • separate kcp / cluster-controller with in-cluster ("pull") syncers
  • separate kcp / cluster-controller with external/remote syncers ("none"-mode)

Add more topics below!

[imjasonh]

A couple more potential topics:

Syncer will need some notion of filters/transforms to apply to resources, beyond just only syncing objects with the cluster label. How should we express those?

• filter out any resources in syncer-system (https://github.com/kcp-dev/kcp/issues/83)
• transform namespace names before applying to the leaf cluster
• @smarterclayton 's comment in Slack thread:

some kind of pod spec defaulting spec -> fill out security defaults -> spec would be a specific transformation ... the reason to have separate syncers is because you can more tightly control the pipeline. admission doesn’t have to be part of the source cluster - it might be part of the syncer. this has other advantages (strong control over what is admitted to the target cluster). admission on create loses context that a sync pipeline has that the control plane may not

---

There are already a few "syncer"-like things out there, in CAPN, OCM, etc. -- how much should we focus on kcp including its own syncer impl, vs making sure existing syncers can be made to work with kcp to retain their own syncer behavior/semantics. cc @mdelder

[imjasonh]

Jason DeTiberus's KubeCon talk, "From Tweet to Badidea: Creating an Embeddable Kubernetes Style API Server", is up on YouTube: https://www.youtube.com/watch?v=fxqV24h_ocs 👀

[smarterclayton]

Clayton's biased meeting notes:

• Will sync with Jason this week on minimal api server and see about queueing up a discussion with some early users
• Lots of people (CAPN, vcluster, karmada, ocm/acm) have done library for syncing - targeted for their use cases
  • in theory a "library" or "framework" should match the key use cases and be easily pivotable
  • transparent multi-cluster (have an object at the top level and an object on the cluster) is our unique focus and we'll assess components based on that
  • this should be a new investigation thread (discussion about adding a new line of investigation around "syncer as a reusable library / framework for multiple use cases")
• Unique contributions that kcp prototype can bring to the syncer narrative
  • CRD normalization is an attempt to solve "how do i evolve an API across clusters" - sometimes that is adding optional fields (how kube handles v1). Sometimes that is detecting version breaks, or conversion. If we can solve this problem more effectively, that changes what syncers can rely on (namely that direction a -> direction b is always safe)
• Discussion on transparent multi cluster
  • In theory transparent multi-cluster is intended to give us the tools to make "regional failure" and "partition" normal to test, emulate, or simulate in production since movement relies on those tools surviving those mechanics. By making these behaviors normal, we can give operational teams a new toolset for simulating and surviving failure at each of these domains, which requires us to think very closely about how we design our APIs.
  • Phase 2 of demo is crd normalization - annotations on objects (where they are placed), a placer / scheduler operating on whole logical cluster / whole namespace domain, and conditions as status. After that we can explore some other variations that may drive other use cases.
  • What is our "health check" for the failure modes we expect the demo to explore (and can we align that with hard operational challenges today that incentivize implementations to appear)?
• Along with minimal apiserver, we want to identify points of inadequate extension that are tolerable in kube-apiserver, but needed for kcp
  • eg. custom crd subresources, webhook inadequacy, complexity of adding aggregated apis
  • minimal apiserver might enable better inline code mutation
  • a number of explorations of better in-process extension (lua, javascript, wasm, CEL) have been mooted before - could we leverage those for some of the use cases and avoid the need to inline code
• Net new investigation suggestions this week
  • "running a control plane as a high scale service" - can kcp / logical clusters / improvements to apiserver enable better use of kube-apiserver at scale (logical clusters kind of demand it)

[imjasonh]

Apologies for sending these notes out so late.

Recording: https://youtu.be/wRjdY5VAbhQ

@imjasonh: --push_mode added

• other modes are --pull_mode (run syncer in-cluster) and "none-mode" (run syncer manually, for debugging)

AI(@imjasonh): need a general field study of syncers in the wild

• common patterns
• where existing syncers can be used with kcp toward its goals
• kcp syncer tries to be very generic; syncs/negotiates CRDs and any of those objects
• hive syncs YAML definitions in the syncset, assumes types are valid in the target cluster

@smarterclayton: kcp syncer improves state-of-the-art by allowing fluid and even inconsistent CRD definitions

• if they're different-but-compatible, sync least common denominator
• if they're not, bubble up status to an operator to diagnose/rollback
• CRD type compatibility is a scheduling constraint, if a leaf cluster gets an incompatible CRD version, objects of those types should be evacuated to another cluster (and notify operator)

possible demo:

• two clusters, same CRD definition
• one upgrades to get a new field -- can't set that field in kcp
• other upgrades to get that field -- now can set that field in kcp
• one cluster rolls back(!!!) -- kcp moves those objects to a cluster that accepts that field
• prioritize some availability over spread conditions (SLA risk reduced, but app isn't down)

CRD LCD is structural type comparison, not semver

• people mess up semver all the time
• behavioral changes aren't signalled/signallable whether it's structural or semver

kcp can help API evolution tests too, detecting/preventing breaking API changes

transparent multicluster, when it works, needs a chaos monkey to verify your app is resilient

• simulate a whole-region outage (this region's capacity is now zero)
• simulate a dependency outage (because apps declare their deps to kcp)

K8s exploring other extension points

• lua, wasm, cel

@mdelder OCM uses Hive to provision a cluster

• management is decoupled from provisioning

Expressing scheduling constraints should probably look like taints/tolerations/nodeSelectors/spread/antiAffinity

• not necessarily because that's the platonic best API, but because it's familiar

How do you manage health of an app spread across lots of clusters?

• you'd have healthcheck against pods, but not healthcheck of a multicluster loadbalancer
• transparent multicluster observability (e.g., aggregating metrics, logs) is a prereq, needs more thought

[imjasonh]

Next: https://github.com/kcp-dev/kcp/issues/91