Add CustomResourceDefinitions (CRDs) to acorn

[cjellick]

This has come up a few times in feedback from users. We are investing different approaches to solving this.

Our challenge here is that because every CRD has its own unique impact and lifecycle, Acorn cannot reliably and predictably manage CustomResources generically.

We are kicking around different ideas, including bespoke support for different CRD-based systems (like various service meshes and ingresses) or a plugin framework.

If you have a specific CRD usecase, please share it here!

[denist-huma]

Propose a new Acornfile syntax for Custom Resources

Use case

We are in R&D team describe resources for our workloads in Kubernetes with CRDs. The resources currently are the AWS S3 bucket, Redis and MongoDBUser (db + user pair) in-cluster provided by KubeDB. There are several CRs we use with our operators at huma.com.

apiVersion: db.operator.huma.com/v1alpha1
kind: MongoDBUser
metadata:
  name: ppserver-2513-db
spec:
  clusterRef:
    name: pr-preview-mongodb-cluster-2
    namespace: kubedb
  dataLifecycle: delete
  databaseName: ppserver-2513-db
  provider: kubedb
  roleName: readWrite
---
apiVersion: redis.operator.huma.com/v1alpha1
kind: RedisClaim
metadata:
  name: ppserver-rc
spec:
  haEnabled: false
  redisVersion: 6.2.5
  storage: 1Gi
---
apiVersion: storage.huma.com/v1alpha1
kind: S3Bucket
metadata:
  name: ppserver-2513-bucket
spec:
  region: eu-west-2

Custom Resources syntax proposal

Used a Authoring Acornfiles guide

cr: {
    "db": {
        name: "ppserver-2513-db",
        apiVersion: "db.operator.huma.com/v1alpha1",
        kind: "MongoDBUser",
        spec: {
            "clusterRef": {
                "name": "pr-preview-mongodb-cluster-2",
                "namespace": "kubedb",
            },                
            "dataLifecycle": "delete",
            "databaseName": "ppserver-2513-db",
            "provider": "kubedb",
            "roleName": "readWrite",                
        },
    },
}

1. Custom Resources contained in an object to allow to reference their fields later like "cr.db.name".
2. There could be 2 Custom Resources with same name but different apiVersion/Kind, so such objects need a name field.
3. I see name, apiVersion, kind, spec present in all my Custom Resources. Only but spec can be empty. For example, an empty spec in my S3Bucket would create an S3 bucket in the default "us-east-1" region.

cr: {
    "bucket": {
        name: "ppserver-2513-bucket",
        apiVersion: "storage.huma.com/v1alpha1",
        kind: "S3Bucket",
        spec: {},
    },
}

[denist-huma]

Watching for Custom Resources Status

Acorn can watch events with Custom Resources waiting for them to become "Ready". Upon CR reaching the "Ready" phase, the dependent container(s) can be deployed. That gives an opportunity to declare in a container dependsOn: [ "cr.bucket", "cr.db", "cr.redis"].

Custom Resources in general have some fields inside their Status that represent a state when it is good to go. The difference is in the field name and value, because there is up to CRD designer.

• apiVersion "kubedb.com/v1alpha2", kind "Redis", has ".status.phase: Ready"
• my "RedisClaim" has ".status.status: Ready"
• my "MongoDBUser" has ".status.status: Available"

RBAC

Acorn need viewer ClusterRole for each Custom Resources, example:

# permissions for end users to view mongodbusers.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: mongodbuser-viewer-role
rules:
- apiGroups:
  - db.operator.huma.com
  resources:
  - mongodbusers
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - db.operator.huma.com
  resources:
  - mongodbusers/status
  verbs:
  - get

[Westixy]

Proposal to use "plugins" to abstract CRD as well

What if we use something like providers in terraform that allow you to to extend the config.

Since CRDs are inherently custom, we cannot know beforehand what are the important part without being explicit but I do not want to map in the new dsl what is in yaml.

It is more a request for comments than a proposal but I think it can be valuable here.

Example

Here is an simple example where I use a plugin for KafkaTopics CRD:

@import { KafkaTopic } "https://github.com/acorn/crds/KafkaTools#v1.0.0"

cr: KafkaTopic: {
  name: "some"
  cluster: "my-cluster"
}

And here the definition of the plugin:

folder structure:

.
├── manifest.cue
└── kafka-topic
   ├── schema.cue
   └── crd.yaml # as go template

manifest.cue:

version: 1.0.0
name: KafkaTools

exports: KafkaTopic: {
  schema: "./kafka-topic/schema.cue"
  templates: [ "./kafka-topic/crd.yaml" ]
}

schema.cue:

inputs: {
  name: {
    description: "Name of the kafka topic"
    type: "string"
  }
  cluster: {
    description: "Name of the cluster"
    optional: true
    type: "string"
  }
}

check: ready: {
  path: ".status.phase"
  condition: "equal"
  value: "Ready"
}

crd.yaml (go template):

apiVersion: kafka.strimzi.io/v1
kind: KafkaTopic
metadata:
  name: {{.inputs.name.value}}
  {{if .inputs.cluster.defined }}
  labels:
    strimzi.io/cluster: {{.inputs.cluster.value}}
  {{end}}
spec:
  ...

Well I hope you get the point.

BR, Esteban

[cjellick]

Plugins are interesting. I recently had the thought of a plugin system that would allow one to add additional content to an acorn image and then interact with it in cli extensions provided by the cli. Like an "SBOM" plugin that put the content in the image as an additional indexed item that could the be retrieved via acorn sbom <image> (but the sbom command not being baked into the cli).

I got a 🤔 from @ibuildthecloud. Not sure if it was a good thinking face or bad one though.

[ksondere]

Just dropping in another use case. I need to extend acorn AML so that I can release rancher-logging flows/outputs with my app.

Need to be able to do similar things with creating auth/authz in istio

[cjellick]

We are supporting this "indirectly" through service acorns. You could (at least in theory) create a service that creates a CRD. I'm not sure how far I want to go with claiming support for this, but it is at least theoretically possible.

@sangee2004 I dont really want you putting much effort into testing this.

@cloudnautique will we ship any docs/examples that get close to this?

[cloudnautique]

An example of using services and k8s permissions to create a custom resource can be found here: https://github.com/acorn-io/examples/tree/main/services/crd

We won't be documenting this use case because it requires Kubernetes permissions that are only exposed to self-hosted/self-managed clusters with the runtime.

It is best to leverage this capability in the service Acorn construct because it isolates the app author from the implementation details of writing out the Custom Resource and it allows the Acorn app to be more portable by making it easier to swap out if the deployment target doesn't have that CRD present.