Using Micronaut as an Operator framework

[jbrisbin]

Although Quarkus and CDI is a great option for building operators on Kubernetes, I would really like to see an Operator framework for Micronaut that would enable a developer to implement an Operator to control their application(s) using the same framework they build their service layer with.

To be clear: this is not proposing a framework to allow applications to integrate with Kubernetes like with service discovery but to integrate an Operator with the Kubernetes environment like managing custom resources, being notified of changes via watches, creating new resources with a cluster based on templated specs, etc... All those things an Operator typically does by just creating a client (e.g. Fabric8) and manipulating the cluster manually. We have built an Operator on Quarkus and I think there are some meaningful abstractions that could be extracted into a reusable Micronaut component that would make writing Operators using Micronaut really, really easy.

[graemerocher]

Hi Jon! Good to hear from you! Sounds like you have some great ideas in this area. Would you be willing to contribute? 😉

[jbrisbin]

Would you be willing to contribute?

Hi Graeme! I was hoping you'd ask. ;) Not sure on the timeline as it would be more exploratory at first, just to validate some of the assumptions. But if that works out, I'd be happy to contribute.

[graemerocher]

Sounds good 👍

[jbrisbin]

Started looking at this last night. We decided to just expose the Fabric8 models directly in our own work since it seems unnecessary to duplicate all those models and copy them back-and-forth. It might be possible to exchange the client with another implementation, but AFAIK the not-yet-stable Kubernetes client doesn't support OpenShift, so it's really only Fabric8 to choose from for maximum compatibility.

It might be possible to construct the API in a way that genericizes the models so you're not tied into a particular client but IMO that's unneeded complexity. For a first iteration I think it makes sense to just rely on Fabric8 and the models for all the types without creating a copy with a different package name. This can be optimized for portability later if that's really required for some specific reason.

[jbrisbin]

Another option is to "hide" Fabric8 and communicate to user code via annotated POJOs that map e.g. @CustomResource and @ObjectReference to their counterparts in the Fabric8 models. This is more flexible at the cost of greater complexity. That might increase testability as well.

[graemerocher]

Not sure I fully understand all the problems but I assume you are referring to the Fabric8 kubernetes client. Note that there is also a official Java kubernetes client that includes models https://github.com/kubernetes-client/java/tree/master/kubernetes/src/main/java/io/kubernetes/client/openapi/models

[jbrisbin]

TL;DR in order to use custom resources with Fabric8 you need to extend and expose Fabric8-only models in your code: https://github.com/instana/instana-agent-operator/blob/master/src/main/java/com/instana/operator/customresource/InstanaAgent.java

In order to get away from a strong tie to the Fabric8 client and models, we'd either need to genericize the API or use annotations and POJOs to allow for extracting information from the Kubernetes resources (things like the OwnerReference, ConfigMap, or other internal Kubernetes resources). Now that I think about it, annotations are probably cleaner and make for a better approach for exposing handling code for events on custom resources (Operators are managed by creating, modifying, etc... these custom resources which trigger your operator code).

The official client AFAIK doesn't support OpenShift so we'll need to expose those client models (including DeploymentConfig) to the user anyway so I don't see how we can really get around providing some mapping facility from the model the client uses to represent the Kubernetes resource to the model the user chooses to represent the same thing inside their operator code.

[alvarosanchez]

Hi there, sorry that I join late to this conversation, been on leave for a while.

Micronaut has its own Kubernetes client, as one of the design principles of Micronaut is to have (almost) zero dependencies. The existing client only covers a small surface of the Kubernetes API (the one needed to implement service discovery and distributed configuration). Note that it does support watching config map changes, for example.

That being said, if the idea is to make the whole Kubernetes API available, then I agree that switching to an existing client might be a more reasonable decision. I actually use the Fabric8 client for the integration tests. With regard to the official Kubernetes client, it's auto generated from an OpenAPI specification.

However, with the discussion about the client aside, what concrete features do you think Micronaut should offer? So far, you gave me the idea to make the watch events available as Micronaut events using the EventPublisher, so that users can consume them in a framework-standard fashion.

But for other things like "managing custom resources, creating new resources with a cluster based on templated specs", why can't users simply grab the Fabric8 client themselves and use it directly? What should Micronaut offer at the framework level?

[jbrisbin]

@alvarosanchez I have been thinking about this a little more and I see a pretty huge opportunity to make writing Operators easier (which is what Quarkus will do for certain). There are specifics for each of the following categories that bear their own discussion, but here are the top-level things I see:

1) Leader election. It's critical to implement leader election in a reusable way and the first thing you can do is perform leader election for your operator so you can have failover to manage your services/applications and also ensure only 1 operator is ever "in charge" at a time. This is relatively simple but is all boilerplate and can easily be implemented by the framework and configured via runtime or annotation. A good leader election API could be extended to your application components as well and implementations provided by the user.

2) Events watching. It's important to abstract out the details of starting a watch with a specific resourceVersion (or the last known) so you don't re-start a watch that starts from the beginning every time the operator starts or takes over leadership. You also need resiliency in your watches so that if the API server kills your watch you can re-set it. It would be good to provide a filter-like API for events so you could fine-tune the events you're interesting in handling in a particular controller class. Exposing the event itself to interested parties can be done in a very Micronaut-y way as well, though there's the issue of mapping the event data.

3) Object mapping. You should be able to put some annotations on a POJO model class and have the framework map from, say, a Fabric8 ObjectMetadata object to your POJOs so you don't have to rely on a specific client and don't need Fabric8 in the classpath of your model code to make it more portable and, more importantly, more testable.

4) Resource handling. It would be awesome to have a micronaut-data-inspired way to handle k8s resources. You want to be able to list, delete, create, etc... these resources and it would be awesome if you could deal with them without having to rely on Fabric8 models and API calls directly, but instead use something like a Repository approach for custom resources (or any k8s resource, for that matter...e.g. Deployments/DeploymentConfigs, etc...).

5) Testing. You can't easily mock a user's interaction with their own applications (e.g. calling APIs and the like) but you should be able to use a helper to set up your operator for testing without having a live k8s API server to talk to. The IO should be abstracted and mocked sufficiently for the user to test their operator in unit tests and not require integration tests with external KIND clusters, etc...

[jbrisbin]

A good place to see what all is possible/needed in an Operator SDK is to look at the golang official SDK sample controller which includes almost all boilerplate and very little actual logic:

http://github.com/kubernetes/sample-controller/blob/master/controller.go

[alvarosanchez]

@jbrisbin sorry for a delay in my response. Due to holidays I haven't been able to discuss this internally until today.

We agree that your ideas are definitely useful, and would like to discuss how can we help you in contributing them. I will reach out to you by email to schedule a call.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[miguelaferreira]

Hi everyone. I'm very interested in this issue and how will micronaut move forward in this space.

After using the official k8s java library to build a k8s operator, I'm moving towards a more light-weight approach based on watching for changes on resources using the micronaut k8s client. For my use case I need to watch secrets, be able to create and update secrets, configmaps and pods. I've been extending the micronaut k8s client in this project to be able to fully replace the official k8s library in my controller, even thought I will be missing features like leader election for now. I'm counting on opening a PR to this project to contribute back the changes I'm making to the client and underlying model.

[graemerocher]

Contributions welcome!

[miguelaferreira]

Great to hear that!

My main challenge at the moment is that I haven't managed to configure the k8s client using a kubeconfig file. Which means I can only test in the cluster. I've looked at a few implementations that parse the kubeconfig file and set up the client, but I was wondering if you guys have a way to do this that would not involve writing lots of code.

[alvarosanchez]

No, we haven't looked at parsing the kubeconfig file. However, for testing you can simply use kubectl proxy and point your client to localhost:8001.

That being said, I doubt that our current built-from-scratch client is a viable solution going forward, since it lacks many features like kubeconfig parsing, but also SSL certs configuration, etc etc. Not to mention that the (ridiculously large) Kubernetes API coverage is minimal.

What it would make sense is to have a generic API (it could be the current KubernetesOperations, or something along that) which is then backed by the fabric8 Kubernetes client. However, we are far from that, since it would require spinning off a kubernetes-core project and then having a kubernetes-client-fabric8 or something like that.

[miguelaferreira]

Thanks for the tip @alvarosanchez.

I completely agree with your point. I only considered extending the micronaut client because my use-case uses a very small subset of the k8s API. After reading this thread I have a better understanding of the limitations in the approach, and I will think a bit more about how would it make sense to me to move forward.

[vroyer]

We just released a Kubernetes operator for Elassandra based on micronaut on https://github.com/strapdata/elassandra-operator. It uses the java kubernetes client to watch k8s ressources, and it can run in the cluster, or outside, it detects the micronaut env to properly k8s configure the k8s client, as shown in the configuration examples.

The kubernetes-client-fabric8 is also used because it provides JSON serializer/deserializer required to implement the k8s admission webhook (implemented by a micronaut controller, see ValidationController ).

More generally, the micronaut reactive architecture + REST client/serveur + management endpoints gives a nice framework to implement a k8s operator.

PS: Elassandra is also a good backend for microservices, offering both the Elasticsearch and Cassandra API for the same dataset, as shown here