Following aspects need to be kept in mind while implementing Kubernetes custom resources:
Exercise granular control by defining composable custom resources
e.g. Kind Deployment can compose kind ReplicaSet which in turn composes kind Pod
Similarly kind Job composes kind Pod
Right mix of Convention vs. Configuration principles
It is quite common to expose every option via resource specifications and make it configurable
However explosion of tunables make the specifications hard to follow, release & maintain
Operators will try as much as possible to limit exposing configurations unless it is really needed
e.g. exposing every Pod specifications via a custom resource that is supposed to manage this Pod makes the custom resource hard to manage
Convention on the other hand refers to standard settings (read default logic) that can be applied without depending on configurations.
Have a clear separation of responsibility to mutate custom resource’s fields
e.g. Operator will avoid setting the spec fields of the custom resource it is managing
This will let the higher order tool or human to edit the fields without bothering about lost updates
Operator will notify the state of reconciliation via status fields of the custom resource
Operator will add or update the metadata fields (e.g. labels, finalizers, annotations, etc.) if required while reconciling the desired state against the actual state
Operators need to work smoothly at scale. There will be Kubernetes environments that can span across 100 or more nodes and hence will lead to Kubernetes managing millions of custom resource instances. This might impact the responsiveness of Kubernetes API server.
The detailed implementation aspects needed to handle operations at scale is yet to be finalized
However operators will embrace clean architectural principles to easily adapt to scaled requirements
This is expected to happen in a phased manner without impacting existing implementations
Clean architectural constructs should also help implementing operator specifications for non Kubernetes environments as well
Treat RBAC options such as Namespace, ServiceAccount, ClusterRole, ClusterRoleBinding as orthogonal concerns of life cycle operator
In other words these need to be created & managed by users before deploying life cycle operators
Operations that demand finer control or those that needs more handling may be designed as separate custom resources
For example draining node via a drain policy aka custom resource
Note that draining a node is a useful feature of auto scale
Similarly moving Pod from one node to another may be orchestrated by a specific policy
Avoid managing cluster scoped resources from namespace scoped resources
In cases where this is inevitable, namespace resources should not own cluster resources
Namespace scoped resource’s controller should manage resource(s) belonging to the resource’s namespace
Design Principles
Following aspects need to be kept in mind while implementing Kubernetes custom resources: