search

AmitKumarDas / Decisions

Apache License 2.0
10 stars 3 forks source link

K8s: Amazing Controllers ~ K8s to CSI communication #223

Open AmitKumarDas opened 4 years ago

AmitKumarDas commented 4 years ago

Motivation: At the end of this subject, one will be able to appreciate how specifications driven design helps one to solve cross cutting problems.

This article will focus on communication aspects of Kubernetes with that of CSI drivers.

This also brings home the point that CRDs are the new APIs.

AmitKumarDas commented 4 years ago

refer - https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/container-storage-interface.md

Master to CSI Driver Communication

Because CSI volume driver code is considered untrusted, it might not be allowed to run on the master. Therefore, the Kube controller manager (responsible for create, delete, attach, and detach) can not communicate via a Unix Domain Socket with the “CSI volume driver” container. Instead, the Kube controller manager will communicate with the external “CSI volume driver” through the Kubernetes API.

More specifically, some external component must watch the Kubernetes API on behalf of the external CSI volume driver and trigger the appropriate operations against it. This eliminates the problems of discovery and securing a channel between the kube-controller-manager and the CSI volume driver.

To enable easy deployment of an external containerized CSI volume driver on Kubernetes, without making the driver Kubernetes aware, Kubernetes will provide a sidecar “Kubernetes to CSI” proxy container that will watch the Kubernetes API and trigger the appropriate operations against the “CSI volume driver” container.

The external component watching the Kubernetes API on behalf of the external CSI volume driver must handle provisioning, deleting, attaching, and detaching.

AmitKumarDas commented 4 years ago

One needs to understand that a Kubernetes API has something extra than that of typical APIs. This additional behaviour of K8s API gives it a lot of flexibility to accommodate various architectural patterns. Now, what is this wow factor in K8s APIs? I have tried to list them as following items:

A K8s API request is same as the desired specifications. This desired state is stored into Kubernetes store i.e. etcd after which reconcile hooks get triggered. These hooks are the ones which are supposed to serve the original API request. One can immediately imagine this to be similar to async handling of requests. However, this is not just async handling but also provides an abstraction between the request and its handle. Let us try to understand this abstraction further in next section.

In above communication, if Kubernetes controller manager had tried to invoke the gRPC based CSI APIs directly without creating CRDs (or native resources), it would have been difficult to personalise with custom domain logic. With resources as an abstraction, one can write custom reconcile logic i.e. controllers to react to these resources in addition to default K8s logic that handles this request. A custom reconcile logic can even modify this resource with defaults and let K8s to implement the low level logic of arriving at desired state. This is done without bothering about order of implementation.

In addition, as detailed in earlier notes, K8s APIs via its resource first design, solves the discovery and security issues typically required for communicating between two applications. One just need to adopt (read watch) to right abstractions (read native / custom resources) to achieve this. Gone are the days, when one had to implement security to be considered as secure. Instead piggyback on security based abstractions and align to the security needs.

AmitKumarDas commented 4 years ago

Summary By considering a API request as a desired specification that is in turn durable (stored in etcd), Kubernetes provides a nice abstraction that can be used to customise Kubernetes. If these customisation is fed back to the system by updating the same API i.e. specification, it leads to a continuous feedback system that runs constantly to change the current state of the system to its desired state.