Create a generic configuration model

[iocanel]

Currently, jkube, dekorate and quarkus use different configuration models, that are exposed to users through different mechanisms. We should propose a single model that should be applicable through all projects, as this helps us align user experience, features etc.

[Sgitario]

To add some context, the configuration model for each tool is:

• jkube uses a Maven plugin to provide the configuration
• dekorate uses annotation binding and/or configuration properties
• quarkus uses configuration properties

As far I see, the least requirements for the new configuration model should be:

1. Have a programmatically API to be used by a Maven plugin (jkube), a session instance (Dekorate), and Quarkus (build step).
2. properties prefix agnostic, so when using it in Dekorate, the prefix for the properties should be dekorate.kubernetes...., and for Quarkus quarkus.kubernetes.xxx.
3. Ability to provide custom properties at build time (for annotations in Dekorate, or other build steps in Quarkus).
4. The generic configuration model should be excluded at runtime (so this dependency and its transitive dependencies should not be incorporated to the final jar - the user application).

Please, correct me if anything I wrote before is wrong or inaccurate or if further clarification is needed.

[iocanel]

At this point I am more interested in defining a model capable of capturing all the information that is required. I am not so much concerned about how we will map it to various use cases (e.g. to xml, annotation configuration etc) or how we will consume it. I was considering the mapping / adapting part as something that we cloud possibly deal with in a following iteration.

Of course the iterative approach I had in my mind is not necesserily the right approach, so I am really open about it.

[manusa]

IMHO we should start by defining the API part (Interface(s)) and the set of configuration options that this model should represent.

I'd keep the implementation of this API as a separate module. Downstream projects can then optionally reuse this model, but must offer the possibility to interact with the common API.

[manusa]

jkube uses a Maven plugin to provide the configuration

JKube uses System/Maven/Gradle properties, XML (Maven) configuration, DSL (Gradle) configuration, and depending on the case additional options that can be provided through YAML files, application.properties, and so on.

[Sgitario]

IMHO we should start by defining the API part (Interface(s)) and the set of configuration options that this model should represent.

I'd keep the implementation of this API as a separate module. Downstream projects can then optionally reuse this model, but must offer the possibility to interact with the common API.

I also think this is the right path. Perhaps, we should create a very simple use case, for example: "populate deployment with replicas" and see how it fits with jkube, dekorate, quarkus.

[iocanel]

After pondering a while on this, I am starting to lean towards of having no config-model at all. Instead we should allow users to specify partial configuration that our tools will apply to the manifests. In other words I am suggesting generating the best possible manifests for the application (based on the things we can detect) and for customization purposes to use partial configuration.

Why ?

• Users shouldn't have to learn a limited kubernetes dialect.
• Us shouldn't be spending time maintaining the config-model.
• There is a growing tendency on using external tools like kustomize and being close to that model gives familiarity points.

@manusa @Sgitario Thoughts?

[Sgitario]

I'm missing most of the picture here. But are you suggesting that our model is a YAML file similarly as what kustomize does? Maybe an end-to-end example would help.

[iocanel]

Not necessarily a yaml. It can be yml, properties, xml or anything. I am just saying that the structure could be the one of the actual resources. So instead of us coming down with something that maps to the actual model, to use the actual model directly.

Examples incoming

[iocanel]

Setting a custom port

To set a custom container port for our container in dekorate we use soemthing like:

dekorate.kubernetes.ports[0].name=http
dekorate.kubernetes.ports[0].containerPort=8080

If we used the actual model it could be instead:

spec:
  template:
    spec:
      containers:
          ports:
            - containerPort: 8080
              name: http 

Or expressed in properties:

spec.template.spec.containers[0].ports[0].name=http
spec.template.spec.containers[0].ports[0].containerPort=8080

Or in a quarkus style properties dialect:

spec.template.spec.containers.my-container.ports.http.containerPort=8080

We may or may not be able to further simplify it:

containers.my-container.ports.http.containerPort=8080

The biggest pros of this approach:

• WYSIWYG: What you SET is what you get.
• No need to learn a new dialect
• Less maintainance from our part
• Can be a smooth migration path. Support this style in all our tools in parallel with what we have. And slowly deprecated the existing stuff if we feel like.

The cons I see:

• Increased verbosity
• Not ideal in property style.

[Sgitario]

I see the benefit of this approach, mostly the "You get what you set" and not having to learn new dialects statements.

However, except for the properties dialects, I see this is pretty much what Kustomize already does (and Kustomize can be used directly from the kubectl command line which is a big pro vs kooperate).

Apart from this, I don't see we can smoothly migrate both kubernetes-client and dekorate in an easy way to support the old pattern and the new one. I'm not saying it's impossible, but I see it harder than easier (either we invent something to map the new pattern to the Dekorate config style, so all the dekorate classes keep working, or we duplicate a lot of things). About kubernetes-client, I have no idea about it would be hard or easy though.

[iocanel]

I see the benefit of this approach, mostly the "You get what you set" and not having to learn new dialects statements.

The ugly truth is that users often need to troubleshoot something that is a black box to them and there are way less resources around to thelp them in their struggle. You get what you set makes things more transparent. On top of that kubernetes is becoming a comodity so dealing with manifests is not that hard. That doesn't mean our tooling has little value. IMHO, it means that our tooling should primarily focus on providing the best possible manifest for the app (based on the extracted information) and use a more standard way of customization.

However, except for the properties dialects, I see this is pretty much what Kustomize already does (and Kustomize can be used directly from the kubectl command line which is a big pro vs kooperate).

That's true and from my pov that's not necesserily a bad thing. Given that's part of kubectl makes it pretty much standard. So, us supporting a standard or part of a standard. I recall @maxandersen suggesting that we might want to do that anyway (supporting kustomize).

Apart from this, I don't see we can smoothly migrate both kubernetes-client and dekorate in an easy way to support the old pattern and the new one. I'm not saying it's impossible, but I see it harder than easier (either we invent something to map the new pattern to the Dekorate config style, so all the dekorate classes keep working, or we duplicate a lot of things). About kubernetes-client, I have no idea about it would be hard or easy though.

From kubernetes-client 6 onwards the Visitors will have full access to the path they operate. This might be something we could possibly use in our advantage and help us implement something like a KustomizingDecorator or Kustomizer if you prefer that would be responsilbe of applying aribtrary maps where it makes sense. Note, these are just ideas coming to my head. Thre may be gotchas I can't see right now.

[cmoulliard]

The idea you are proposing is interesting and should also maybe impose us to review what is the role of dekorate as that could also guide the way that you will design this config model.

I assume that, no matter what we will develop, the main objective of dekorate is still to Generate YAML/JSON manifests k8s compliant. Nevertheless, it could be interesting to investigate if that could make sense or not to continue to develop dekorate as it is or in a more smarter way.

Why ? As the difficulty when we work on a k8s is to find the appropriate GVK (e.g ingress api: extensions/v1beta1 or networking.k8s.io/v1 or networking.k8s.io/v1beta) to be used like its associated API spec (= list of fields, type, ...), we should perhaps put in place a mechanism when dekorate generates the resources to:

• Query the cluster to get a list of the GVKs
• According to the modules defined within the pom.xml, filter the list of the GVKs to only keep the needed
• Generate the manifests when no additional configurations are defined by the user

By configuration, I mean some fields/parameters passed by the user to customize the generated manifests using annotations/properties.

So, when we got some customizations(s) from the user, and instead of expecting that they will pass the exact path to set a field (e.g https://kubernetes.io/docs/reference/kubernetes-api/service-resources/ingress-v1/#IngressSpec -> rules.host)

kind: Ingress
metadata:
  name: ingress-1
spec:
  rules:
  - host: "my-hot.example.127.0.0.1.nip.io"
    http:
      paths:
      - pathType: Prefix
        path: "/nginx"
        backend:
          service:
            name: nginx-svc
            port:
              number: 80

why dont we use an intermediate model (e.g. devfile, Kubernetes application CRD, Halkyon CRD...) able to match a user's request to the final k8s GVK.

Example:

Application
  name: my-app
  host: my-hot.example.127.0.0.1.nip.io

--> that dekorate will use as custom model to generate the final resource

kind: Ingress
metadata:
  name: ingress-1
spec:
  rules:
  - host: "myapp.com"
    http:
      paths:
      - pathType: Prefix
        path: "/nginx"
        backend:
          service:
            name: nginx-svc
            port:
              number: 80

That will avoid to have to know the exact syntax of the API, path to set the field/value, to improve the DevExp by proposing a model supported by our different frameworks, tools, ...

WDYT: @iocanel @Sgitario @manusa @aureamunoz

[metacosm]

Maybe a stupid question but if the kustomise approach is selected, why not use kustomise then instead of re-inventing more or less the same thing?

[cmoulliard]

Maybe a stupid question but if the kustomise approach is selected, why not use kustomise then instead of re-inventing more or less the same thing?

This is also an option to keep in mind. We use Dekorate to generate the manifests according top the spec and if some customizations are needed, then we use another external tool such as kustomize even if another tool will be needed

[iocanel]

Maybe a stupid question but if the kustomise approach is selected, why not use kustomise then instead of re-inventing more or less the same thing?

• Kustomize does not generate manifests
• Kustomize requires the presenence of kubectl
• Kustomize itself is still more complicated than we would like.

Also, from my point of view, this is not re-inventing. Fragments (which is pretty similar to what kustomize does) is soemthing that is around in FMP for years (before kustomize) and partially supported by dekorate and jkube.

Edit: @metacosm the question is not stupid at all. It's perfectly valid and one that I've asked myself. What I wrote above were enough to convience me. Eager to hear your thoughts.

[iocanel]

• Query the cluster to get a list of the GVKs

Given, that all three tools of interest are hooking up in the actual build process, they should to be idemponent. Always produce the same output for the same project. Even without querying the cluster we are not 100% there (see [1]). FMP v1 did query the cluster in an effort to be smart. The result was unpredictable, really hard for the users to understand and for maintainers to troubleshoot.

• According to the modules defined within the pom.xml, filter the list of the GVKs to only keep the needed.

If you are refering to multimodule support, this is something that is interesting.

• Generate the manifests when no additional configurations are defined by the user

Yeah, we still do a great job at generating a great zero config manifest and we should keep it that way.

[manusa]

First some thoughts on your comments:

After pondering a while on this, I am starting to lean towards of having no config-model at all. Instead we should allow users to specify partial configuration that our tools will apply to the manifests.

I'm not sure if you're aware of this, but in your comment (s) you're partially describing one of the features of JKube Fragments.

Though I like the flexibility of this approach (it helps us cope with feature that we might have not yet implemented), the general feedback I get from users is that they prefer to use XML or something familiar to them rather than K8s YAMLs.

Following this approach, users will need to know about k8s specifics too. If they want to change the number of replicas for a controller (Deployment, ReplicaSet, Job, and so on), they'll need to configure resource-specific values for all of these resources instead of managing a single "replicas" configuration. Which expands on what Jose shared. However, in case of JKube, this is already covered. We partially cover most of Kustomize's features (fragments for multiple environments, and so on).

GVK

This is something we have considered in JKube, especially when selecting the kind of resources to generate. This is what we could call the online mode, since it requires an active cluster to work.

This is great for the inner-loop, but it's something that shouldn't be required for the outer-loop. In general, our users run the pipelines in an isolated environment with no clusters available. They generate and push the container image, and they generate and push their Helm charts. Any GVK configuration for outer-loop should be configurable.

Maybe a stupid question but if the kustomize approach is selected, why not use kustomise then instead of re-inventing more or less the same thing?

(there are no stupid questions)

One of JKube's goals is that this is the only tool you need. No need to install Docker, kubectl, kustomize, or any other CLI. You should only need a JVM.

Also, JKube already covers part of what Kustomize does.

why dont we use an intermediate model (e.g. devfile, Kubernetes application CRD, Halkyon CRD...) able to match a user's request to the final k8s GVK.

I think that config-model should define the API for this intermediate model which should be more focused on the application's requirements rather than k8s resource internals.

---

Now some opinions on what I think this model should provide:

For me the model should represent a collection of features that all of the implementing tools (Quarkus, Dekorate, JKube) should allow and are compatible with.

For example, in Dekorate you have an AddLabelDecorator that I assume adds labels to all of the HasMetadata instances.

https://github.com/dekorateio/dekorate/blob/84b7b50812e523acfd10608660dfad1836eaab24/core/src/main/java/io/dekorate/kubernetes/decorator/AddLabelDecorator.java#L57

In JKube we have a similar DefaultMetadataEnricher that does more or less the same thing.

However, each of these projects reads this configuration from different places.

Having a common API to read this configuration:

// Labels for every HasMetadata (`dekorate.kubernetes.labels`)
kooperateConfig.getCommon().getLabels();
// Labels for controller kinds (Deployment, ReplicaSet, StatefulSet, Jobs...) 
kooperateConfig.getController().getLabels();
// Number of replicas controller kinds (Deployment, ReplicaSet, StatefulSet, Jobs...) ( `dekorate.kubernetes.replicas`)
kooperateConfig.getController().getReplicas();

would be a very good first step to align the projects, at least feature-wise. The configuration model could serve as the spec that all implementing projects should follow (regardless of how the config API is implemented).

[iocanel]

I'm not sure if you're aware of this, but in your comment (s) you're partially describing one of the features of JKube Fragments.

I am well aware of that. See my response to Chris where I mention that too.

[iocanel]

Well, if we have feedback from users that they still prefer to use non-yaml as a configuratio delivery mechanism then this is something we do need to take into consideration.

However, my suggestion is not limited to yaml. The same information can be still be represented in xml, properties or anything that can be used to represent arbitrary maps (as already mentioned in the original post).

[manusa]

I am well aware of that. See my response to Chris where I mention that too.

Yes, I saw that one later :sweat_smile:, sorry for the noise

Well, if we have feedback from users that they still prefer to use non-yaml as a configuration delivery mechanism then this is something we do need to take into consideration.

I keep getting this feedback from SAs and community users in general. Our use of fragments is quite extended, and we also tend to recommend solutions based on fragments to workaround our missing features. Maybe this is why we get more feedback in this regard :shrug:.

However, my suggestion is not limited to yaml. The same information can be still be represented in xml, properties or anything that can be used to represent arbitrary maps (as already mentioned in the original post).

Yes, I understood that. My point is that regardless of the format in which you specify those values (YAMLs, property files that map to YAMLs, and so on), they are still k8s-resource-dependent. Meaning that users need to know the k8s configuration internals for those resources they want to config. Which is kind of contrary to what in part most of our projects do: automatically generate manifests for your application and allow its deployment to k8s in a transparent way. "Need more replicas? need to add some configuration option? just provide this (agnostic) config option and everything will be taken care of"

[cmoulliard]

If you are refering to multimodule support, this is something that is interesting.

I was referring to the maven GAVs of dekorate declared in a POM.xml file

[cmoulliard]

The result was unpredictable, really hard for the users to understand and for maintainers to troubleshoot.

The current situation is not even comfortable as a user as no idea when it is connected to a cluster what dekorate will generate as the code generated depends on the dekorate version used.

[cmoulliard]

would be a very good first step to align the projects, at least feature-wise.

I like this approach as a first step before maybe to embrace more complex stuffs such as to define a common model or let's an intermediate model able to be used by the different tools and representing what an application, service to be build/deployed on k8s is and in order to avoid that a user has to know what the k8s path to access a field of a GVK is.

Such a Kubernetes Application model could also be used as the format supported by Devfiles too. We should contact Tim DeBoer and invite him here to discuss with us.

[deboer-tim]

I've tried to read through all the comments, and hope I understand them all. Two things that stand out to me:

1. +1 to a common model between the tools. Most users aren't going to switch between them (so we don't gain anything by familiarity) and we're unlikely to execute faster by sharing code, but I think it'll be a benefit in consistency, learning from each other, and understanding what patterns work well in one tool and applying them to the others.

2. As a tool creator the Kubernetes yaml fragments is a really slick option - there's no disconnect from what's actually going to run, and no missing features. However, I think Marc is correct that the average user of these tools is specifically trying to avoid Kubernetes yaml, and even an xml or properties-based form of it isn't what they want. They want something simpler, in Java/developer/application terms.

If we can make it work, I think this actually leans most toward the current JKube/fragment approach: keep the model simple and in user/application terms, but allow Kubernetes fragments as well. Normal users can stick to the model, but anyone who needs something we don't support (yet) or has a Kube snippet from elsewhere can use the yaml directly. This also frees us up a bit, we can focus on supporting the golden paths and reject requests for more obscure or 1-off Kubernetes features because there's always the yaml escape route.