Open bavarianbidi opened 1 month ago
cc @gabriel-samfira - as you already have similar thoughts about managing garm itself via the operator: would be nice if you could drop some of your thoughts about this in here :pray:
I've actually started playing around with this here: https://github.com/gabriel-samfira/garm-operator/tree/add-garm-server-controller
However I've been sidetracked a bit. Hopefully I can resume soon.
The basic idea is as follows:
This is just a rough idea so far. What do you think?
i'm not sure if i got the entire idea :see_no_evil:
The first few bullet-points in your list are awesome - especially as you probably know best how it should be :sweat_smile:
are you even go one step further to manage the garm-server
Deployment object (incl. Service
, ... ) or is it more like "generating and creating/updating the configmap in k8s"?
If so, the "state" of the garm-server
Deployment is managed by two dedicated controllers/components:
garm-server
belongs to another tool (e.g. flux/argocd or a semi-automatic github action deploy workflow)The controller would manage a StatefulSet which is the garm-server itself. Not sure if it's worth having separate controllers for config map management. I think we could fit that into the reconciliation loop of the garm server controller.
the reconciliation loop would basically use the CRD settings to compose the config map, save the config map, update the stateful set. If we add CRs for credentials and providers, we just add those two types in the loop as well. List al creds/providers, get CRD settings, compose config map, update stateful set.
The controller can watch for changes to credentials, providers and the garm server CRD (which it owns) and react accordingly.
For credentials and providers we need separate controllers/webhooks.
I think the best way to go about it would be to create a WiP branch and discuss it once it's ready. Doesn't seem to difficult to write a controller using kubebuilder. An initial draft would be without tests and not necessarily correct, but would give everyone a view of the rough idea.
hey gabriel, first of all - thanks for the detailed explanation and all the work you already put into garm
:heart:
we can also jump into a dedicated call and take some time to talk about garm
, garm-operator
and other related stuff to get a better feeling what make sense from different point of views.
just some thoughts from our side which came into our mind:
Your desired approach feels very similar to the prometheus-operator
implementation.
The entire configuration of the prometheus-stack (different parts of application configuration + parts of the pod.spec) is done in multiple CRs. This makes offering Prometheus-as-a-Service
super handy.
Taking this similarity and reflecting to your approach means, it will be possible to offer Garm-as-a-Service
very easy (which is awesome)
Pros:
Garm-as-a-Service
Cons:
networkpolicies
beside the garm-application - so we still need an additional way to deploy resources which belong to the managed garm-instance
(adding this to the operator, will also increase complexity and probably ending up re-implementing existing solutions - like helm-chart support, ...)garm-instance
(please note, the current operator doesn't support multiple garm-instances yet)Our preferred approach is still the following:
garm
side via the API (e.g. github configuration
, ...) will became a CRD
where the operator takes care ofgarm-server
is deployed/configured/managed outside the operator (if wanted, we can try to contribute our existing helm
chart back to cloudbase/garm
)You're right. That makes a lot of sense. A proper helm chart is definitely preferable to reinventing the wheel. In that case, we should be fine with just implementing the resources that were moved to the DB (gh endpoints and credentials - potentially providers in the future), and potentially offer a helm chart that folks can use.
It's so easy to over complicate things. Thanks for the reality check!
Edit: It makes sense to have any helm chart live in the operator project. GARM should just do what it does. At most it should worry about packaging (deb, rpm, snap, flatpack, plain binary releases, etc). But automation should be separate.
What is the feature you would like to have?
With the recent changes in
garm
(and also future ideas in the backlog) it will be possible to "configure" agarm-server
instance in a declarative way. The current version ofgarm-operator
already takes care of the circumstance if a garm-instance is initialized or not and takes care of the initialization if needed.As we already check on every api-request for an initialed garm-server (func Ensured Auth) we should at least decouple this implementation and use some existing primitives of kubernetes to make the code a bit more easier to read.
I would like to see a dedicated controller with is responsible for the generic garm-server management and controllers for scoped resources (Separation of concerns).
Current known resources
Anything else you would like to add?
As the configuration of the github backend has already moved out of a config file and moved into the api (with https://github.com/cloudbase/garm/pull/243) and we have an issue for this (https://github.com/mercedes-benz/garm-operator/issues/127) we should agree on the separation level first.
How could the new CRs look like? Maybe we can "design" the CRs first before start implementing :pray: