Kubernetes API and Controller Development Docs

[johnbelamaric]

As part of Nephio, we are designing CRD-based APIs for network functions and network topology. We are also developing open source controllers and development kits for vendors to build their own controllers. Given that CRDs are only a few years old, very few people have extensive experience developing high quality APIs and controllers. Thus, a series of talks, blogs, and other educational initiatives is necessary to ensure API and controller quality. Ideas:

• Designing Extensible Kubernetes APIs
  • Design patterns and lessons learned from Gateway, PV/PVC/SC, etc.
  • Principles of K8s API design
  • Conventions of K8s API design
  • Discoverability of API support in a given implementation
  • Maintaining layers of independence
  • Consider user privilege required in designing API resources and RBAC scope
  • Best practices around Status and Conditions
  • Ensuring CaD / automation friendly APIs
• Building Kubernetes Controllers
  • Develop tools / best practices (e.g., resource state interaction table)

There may well be some prior art on this (certainly there are the published K8s API conventions, for example). So first thing should be to try to identify that. Next, we can work with the K8s community on developing some of this material, as it is applicable outside of Nephio as well.

[adetalhouet]

The following presentation captures additional consideration when it comes to API design and identifies some actions we need to address as a community to define development best practices.

https://docs.google.com/presentation/d/1DpMxnGYCmO1gPdLYGQICrl5ocJalBLMdZ9JOcCp-laI/edit?usp=sharing

[johnbelamaric]

Some more notes / best practice thoughts around K8s API design:

• Follow the Kubernetes API Conventions
• Abstractions with multiple implementations
  • Very difficult to get the right abstraction, often results in lowest common denominator
  • Needs flexibility and extensibility
  • Composition may be a better alternative in many cases
  • Compare Ingress vs Gateway, for example
• Partition interfaces to separate concerns (different resources can have different RBAC)
  • Class / Claim Pattern is an example of this
  • PersistentVolume, PersistentVolumeClaim, StorageClass (link)
  • Gateway, GatewayClass (link)
  • Segregate vendor or other implementation details from consumption
  • Resource management and allocation via claims
• Multiple specialized abstractions built on top of lower-level primitives
  • Deployment, StatefulSet
  • In Nephio we might want something like FiveGCoreTopology, FiveGRANTopology built on top of NFTopology which is built on top of NFInstanceSet which in turn is on top of PackageVariant
• Other tools around reducing configuration complexity
  • Composable mutations
  • Decouple data producers and consumers (Endpoints)
  • Automation (HorizontalPodAutoscaler)
  • Convention over configuration
  • Defaults, when a single value makes sense for most use cases

[johnbelamaric]

The Configuration-as-Data methodology has the goal to make the hydration process:

• Scalable - we can create specialized variants of packages across thousands of clusters
• Collaborative - multiple actors can contribute iteratively to each package configuration (facilitated interaction between actors)
• Manageable - we can make changes to upstream packages and easily propagate those changes across all variants

To that end, there are some best practices that are helpful when designing CRDs and kpt packages:

• Avoid package-specific interfaces and encapsulation
  • Most operations relate to resource types, not really packages
  • Rely on the native APIs of K8s objects rather than wrapping them in a new API
  • The same resource type is used across many packages, so code operationg on resource types is more re-usable
  • Allow actors visibility into and access to mutate the underlying resources (increases collaborative configuration)
  • Corollaries:
  • Avoid setters in kpt when possible
  • Avoid package-specific CRDs
  • Operators that just effectively render helm charts are not helpful
• Decouple the API needed for automations from package content
  • Re-usable kpt functions serve as the bridge between the automation interfaces and standard Kubernetes resources
  • Example: ClusterScaleProfile in https://github.com/nephio-project/nephio-controller-poc
• Shift configuration to the management cluster, operating on config in storage rather than operators interacting directly with the K8s API server

[johnbelamaric]

/area docs

[johnbelamaric]

Doc from K8s community (David Eads mostly): https://github.com/kubernetes/community/blob/master/contributors/devel/sig-api-machinery/controllers.md