feat: jsonnet as a kubernetes controller

[AmitKumarDas]

Problem Statement: Most of my requirements deal with extracting & updating json elements. I don't want to use a full blown programming language to deal with these. Jsonnet solves my purpose very well, is easy to learn, build & test. Over time my effort to build Kubernetes controllers decreases.

Assumption: I would like to build something on top of Metac that already abstracts away Kubernetes controllers.

Goal: This solution should provide further levels of abstraction to build workflows that are native to Kubernetes.

[AmitKumarDas]

Design Draft 0

Design of a Recipe

This is a sample recipe to check if openebs components are running

# A custom resource understood by metac
# Internal to controller binary
kind: GenericController
metadata:
  name: are-openebs-components-running
spec:
  watch:
    resource: recipes
    selector:
      name: are-openebs-components-running
  attachments:
  - resource: deployments
  - resource: pods

# user facing
kind: Recipe
metadata:
  name: are-openebs-components-running
spec:
  file:
    name: are-openebs-components-running.jsonnet
status:

// internal to controller binary
// this is the 'are-openebs-components-running.jsonnet' file
// jsonnet code

This is a sample recipe to install openebs components

# A custom resource understood by metac
# internal to controller binary
kind: GenericController
metadata:
  name: install-openebs-components
spec:
  watch:
    resource: recipes
    selector:
      name: install-openebs-components
  attachments:
  - resource: deployments
  - resource: pods
  - resource: configmaps
  - resource: cstorpoolclusters

# user facing
kind: Recipe
metadata:
  name: install-openebs-components
spec:
  file:
    name: install-openebs-components.jsonnet
status:

// internal to controller binary
// this is the 'install-openebs-components.jsonnet' file
// jsonnet code

Design of a Suite

# A custom resource understood by metac
# internal to controller binary
kind: GenericController
metadata:
  name: e2e-for-install-openebs
spec:
  watch:
    resource: suites
    selector:
      name: e2e-for-install-openebs
  attachments:
  - resource: recipes

# user facing
kind: Suite
metadata:
  name: e2e-for-install-openebs
spec:
  steps:
  - are-openebs-components-running
  - install-openebs-components

NOTES:

• A suite implementation will correspond to its own docker image
  • i.e. suite:dockerimage::1:1
• This image will have jsonnet files & GenericController yamls as configs
• This image can be deployed as a Kubernetes STS
• User/Tool will apply the Suite & Recipe custom resources against the K8s cluster

References:

• https://github.com/AmitKumarDas/metac/
• https://github.com/AmitKumarDas/metac/blob/master/examples/gctl/install-uninstall-crd/hooks/finalize-crd.jsonnet
• https://github.com/AmitKumarDas/metac/blob/master/examples/gctl/install-uninstall-crd/hooks/sync-crd.jsonnet

[AmitKumarDas]

As per latest thoughts, I am not pursuing jsonnet to design kubernetes controller. It seems feasible that a declarative controller can be implemented just with yaml(s).

[AmitKumarDas]

Back with requirements for jsonnet!

There are cases when given a set of kubernetes resources one can implement a jsonnet based logic to achieve a higher order requirement. The next question that comes to our mind is whether jsonnet based logic suffices? Yes. It will be sufficient if the logic is limited to extracting values from available kubernetes resources (read json) and applying them against kubernetes resources (again json). In other words, jsonnet based logic works if this logic needs to deal only with json resource(s).

NOTE: Apply implies either create or update of Kubernetes resources

Use Case - CStor Volume Migration

As a cstor admin, I would like to move one of my cstor volume replicas from Pool1 to Pool2. Note that this cstor volume has three replicas spread over three pools where each pool resides on a particular Kubernetes node. I have custom resources defined for cstor volume replica & cstor pool. In addition, kubernetes node is already a native resource.

[AmitKumarDas]

Design Draft 1

NOTE: kind: Recipe is already taken in this project NOTE: kind: Chef will be the name given to this new custom resource

# We shall use Chef as the name of this new resource. Chef implies
# someone who cooks food from the ingredients. Here ingredients imply
# one or more kubernetes resources (read json documents). Chef will make
# use of jsonnet and metac against these json documents to eventually arrive
# at the desired kubernetes state.
apiVersion: diy.metacontroller.io/v1alpha1
kind: Chef
metadata:
  name: cstor-volume-migration
  namespace: openebs
  labels:
    diy.metacontroller.io/storage-engine: cstor
    diy.metacontroller.io/feature: volume-migration
spec:
  configmapRef: # this configmap will hold the jsonnet logic
    name:
    namespace:
  values:
    key1: val2
    key2: val2

NOTE: We shall be using metac to take care of reconciliation needs NOTE: We shall make use of GenericController from metac

kind: GenericController
metadata:
  name: migrate-cstor-volume
  labels:
    diy.metacontroller.io/storage-engine: cstor
    diy.metacontroller.io/feature: volume-migration
spec:
  watch:
    kind: kitchen.metacontroller.io/v1alpha1
    resource: chefs
  attachments:
  - kind: v1
    resource: nodes
  - kind: cstor.openebs.io/v1
    resource: cstorvolumeconfigs
  - kind: cstor.openebs.io/v1
    resource: cstorpoolinstances

NOTE: A binary named diy will be provided by d-operators NOTE: DIY expands to Do It Yourself NOTE: DIY code will have the reconciliation logic for Chef custom resource NOTE: DIY code will understand executing jsonnet document NOTE: Jsonnet logic will be provided with a known set of inputs NOTE: Jsonnet logic should respond with a known set of outputs NOTE: Input & output used by all Jsonnet logic will conform to GenericController's hook API NOTE: One or more GenericController configs will be provided by individuals / teams NOTE: These GenericController config(s) will be used with kitchen as the base image to build a new image NOTE: Teams will deploy this newly built image as a StatefulSet or Deployment NOTE: Each GenericController config instance will get executed as a Kubernetes controller from above running image

NOTE: In future, we shall have UberLoop custom resource that removes the need for separate binary as well as separate config files.

NOTE: We might just want to implement UberLoop only instead of this temporary approach