CloudStation k8s Operator

[SeanCheatham]

Implement a Kubernetes Operator which manages CloudStation projects within a cluster. The operator should not duplicate the efforts and functionality of the CloudState operator; it should instead complement it.

The CloudStation Operator is responsible for:

• Managing a CloudState operator
• Managing a stateful store (i.e. a Cassandra or Postgresql deployment)
• Managing a CloudState entity that manages User Project state
• Managing a CI/CD-like infrastructure which manages Compile/Test/Deploy infrastructure

[sleipnir]

Let me try to understand this here. Are we going to have some type of CRD that the user will generate? I often see operators used to monitor CRD deployments, how will this work with CloudStation? Another question is how after the user defines his logic via the frontend we will generate the artifacts for the user's specific language? There must be a stage of building these files for the desired programming language and subsequent generation of the docker image and publication in the user's registry, right? Do you already have any idea how to do this? Maybe, and only maybe, we should use some tool similar to the CloudState Community CLI that can generate the real project in the end. What language and/or framework do you want to create the Operator with? With Go, Quarkus, Scala?

[SeanCheatham]

All good questions that are still TBD! But where I'm currently leaning:

• A CRD "CloudStationProject". This CRD contains the image tags needed for each of the functions/entities defined in the project (Event Sourced Entities, Replicated Entities, and Actions). It's essentially a like a "deployment set" for the project.
• An operator that is installed in the cluster and manages the CloudStationProject CRDs. Anytime there's a change to a CRD, the operator invokes the CloudState operator (probably using the CloudState Community CLI)
• From a UX perspective, I'm thinking there will be two buttons: Deploy to Test and Promote to Prod.
• For each CloudStation Project, there will be two namespaces in the cluster (one for Test and one for Prod). The CloudStation Operator will deploy the CRD to these namespaces (depending on which button the user clicked)
• When a user presses the Deploy to Test button, the backend generates a project file structure (perhaps using CloudState CLI as you mentioned), generates .proto definitions from the user's Model definitions, and puts the user's code where it needs to be. Included in this is a generated Dockerfile which then gets built, tagged, and published to a container repository.
  • NOTE: To follow recommended CloudState practices, each entity (Event Sourced, Replicated, or Action) is its own independent project directory, and only code relating to that entity is included. So for each of the entities defined in the project will have their own Docker image
• When all of the project's images are successfully built, the CloudStation operator deploys them to the test environment (using the CloudState CLI)
• When the user presses "Promote to Prod", the same image tags are then deployed to the production namespace.

My hope, as far as "backend" goes, is to actually implement as much as I can using CloudState itself. So for example, I got started on a "Project" Event Sourced Entity that keeps track of Project state (models, event sourced entities, replicated entities, and actions). For the step where CloudStation needs to build and deploy to Test, I was hoping to use CloudState Actions to handle that process. There would be one Action per supported language (so a CloudState Action meant specifically for generating/building/compiling for Java, another for JavaScript, another for Dart, etc.)

As far as language choice for the k8s Operator - this is new territory for me personally. I'm familiar with what Operators and CRDs are and have used them, but I have 0 experience actually building them. In that light, I would personally choose Scala to implement the operator since it's my most familiar language. But I see they are frequently implemented in Go so I may end up learning a new language.

[sleipnir]

I specifically liked the idea of using CloudState itself to generate some steps in the process. Regarding the Operator I can make a prototype in Kotlin / Quarkus Native, that would be quick, I'm not a big fan of Scala for native compilation and I think Quarkus has all the tools for building an operator built in so I can try to go that way if you want help.

[SeanCheatham]

I hacked together some rough backend implementation code. I think we can make it so the CloudState CLI does most of the heavy lifting like you suggested. I think we can simplify down several of the steps I listed above into create -> build -> deploy using the CLI. The rough (and incomplete) implementation I have so far is an Action written in Dart. The Action's main purpose is to transform the internal model into actual code, but its other purpose is to invoke the CloudState CLI create, build, and deploy commands.

I'm actually wondering - if we lean on the CloudState CLI heavily enough, we may not even need a CRD. If the "Deploy CloudStation Project" Action interacts with the CloudState CLI which interacts directly with CloudState, then a CRD may not be necessary.

I think perhaps instead of an operator, this could all be simplified down to a Helm chart that does the following:

• Deploy an internal Docker registry (private, can only be accessed from within the cluster)
• Install CloudState
• Deploy the "CloudStation Project" Event Sourced Entity
• Deploy the "Deploy CloudStation Project" Action

This is all just brainstorming though. Any thoughts on that?

[sleipnir]

I think we can have some quick success that way but I will need to invest some time in the CLI again because it is downtime. Another thing I can do is expose the CLI's functionality in the form of API's so we would deploy something like cloudstate --server in a container and install it in the cluster via HELM as suggested and it would react to your action. A question how did you use Action with Dart?