How to manage multiple realms with distinct list of clients on kubernetes

[ReginaldoSantos]

Problem Statement

I'm working on a multi-tenant application, built by several microservices which, in turn, are clients in keycloak. There's a contract that rules which microservices a tenant is allowed to use, so not all the tenants have all clients available. The clients have very similiar configuration, but differ in name, uris (redirect-uri, logout-uri, ...) and authorization policy.

The applications and all required tools (e.g. keycloak) are deployed in kubernetes via GitOps with ArgoCD (a k8s app which is in sync with git repo applying manifests whenever they change) and we would like to manage keycloak configuration in a similar way.

This means that changes made on JSON/property files on our keycloak-config git repository should trigger keycloak-config-cli running on kubernetes and update keycloak configuration on the fly.

As far as we undertood, keycloak-config-cli could be deployed as a Job in kubernetes. This job could be bound to a configMap generated from the JSON/property file and then, changes on the file would trigger the job execution, because this is the way ArgoCD already works.

This solution would be great for a single realm without many variations, but that is not the case.

We need that realm updates occur for each realm whenever some general realm property change. In the same way, changes in a client templates or properties, should be applied to all realms having that client and adding new realms should be as easy as adding a new realm name and properties file.

From a list of clients we would like to apply/create them to a list of realms and, if possible, reusing the very same templates. However, variable substitution occurs only once per execution in keycloak-config-cli and it's not possible to reuse the same templates with different values in the same execution. I mean, we can't submit a list of Realms with their respective properties to keycloak-config-cli and use a single realm template to generate N different realms from the same original template. This sends us towards very redundant, big files and thus, hard to maintain.

Finally, we are looking for advice in how to deploy and handle this multi-tenancy configuration with keycloak-config-cli on kubernetes. I believe this kind of scenario is very common and someone might have a good hint on how we could deal with that.

[ChristianCiach]

We use "one realm per namespace", so we don't encounter this issue, since we are just deploying our job for each namespace separately.

This doesn't sound like an option to you, so let's approach this differently: How about just not using variable substitution? Since a few releases keycloak-config-cli supports multi-yaml-documents. I never actually used that, but according to the source code it should be supported.

This means that you could prepare a multi-yaml-document (just multiple regular yaml documents separated by --- in a single file) and feed it to keycloak-config-cli. You could use an init-container to prepare this document on the fly.

[ReginaldoSantos]

Hi @ChristianCiach, first of all tks for your reply.

I will explain it better.

Context

We have some conventions for backend clients (and authorization), frontend clients, groups, and so on. Each "entity" has already have its own json file. So the creation of a new realm is a matter of build the puzzle "which Jsons shall we put together for this realm".

And this information comes from a admin tool we've built: so a admin user creates a new contract with new customer and there we have the informations like realm name, bundle of clients, default groups, etc... therefore, our first idea was to use the keycloak admin REST API to handle that (this is the current solution).

Even though it works at first, maintaining and assuring that everything is as it should be in keycloak is not easy. Users might change configuration via Keycloak UI, new configurations might be necessary, for instance, to all backend clients and so we need to change their json files and reapply them though all realms. And there are parameters that are configurable by a user in our admin tool and others that are part of our own convention that are hardcoded in our json files. So for the first, we need to use placeholders in our files.

Under these circumstances, as we already have a good experience with GitOps, we are trying to handle keycloak config in a similar way.

Status and Doubts

In one hand, I have a GitOps repo with Json templates for backend clients, frontend clients, different types of authorizations, groups, etc, and some Job manifests for keycloak-config-cli. The Jobs use each a generated configMap pointing to a specific json file and currently there are some hardcoded env vars in each Job for var substitution in the templates. Any change in the json files will regenerate the configMap which will trigger the Job again. I can also change the env vars in the manisfest to trigger a new execution with different parameters. This "works" for one realm at a time, one client at time, etc. However, if I want to create several clients in the same realm, only with different names, its already difficult, because I would need to replicate the client json and use a different var for its name in each json or do it in separated executions. Really cumbersome, but separated executions looks like the way to go.

So, what I'm trying to achieve now is a way to execute this Job on demand, but sending values for the env vars in the job creation itself, so each execution could use it's own env vars (this is more a kubernetes Job related problem). But this way, I could manually trigger changes to all my realms from changes in the GitOps repo without changing the files itself. More than that, it will be possible to make our admin tool invoke/create Jobs in kubernetes with the same purpose consuming the very same json files (single source of truth).

As you can see, with var subst I can maintain a single Json for each kind of keycloak element that I care of and this is way easier to maintain than having a big file representing the whole keycloak realm configuration for each realm.

Moreover, I don't have realm names, client names and parameters and other variables from keycloak configuration in the json files in the GitOps repo. Neither in kubernetes. It's all tenant agnostic and this information is centralized in the admin tool during tenant creation and maintenance.

This is why I don't think using a initContainer and build a huge yaml before keycloak-config-cli execution fits well in my case.

Tks again for your reply and I hope you our somebody else could give me some more insights :-)

[ReginaldoSantos]

Just an additional note: I've just realized what I do need is a way to dynamically build full realm json/yaml files based on templates and variables and the keycloak-config-cli is more in the area of ensuring configurations (with some limited variable substitution capabilities). So I will need to build some final realm json/yaml on the fly by my own before submit it to keycloak-config-cli. I guess FreeMarker might be handy. That's the plan now.

[raffis]

For kubernetes I recommend you have a look at https://github.com/DoodleScheduling/keycloak-controller

[ReginaldoSantos]

Thank you @raffis, interesting indeed.

However, I ended up building a FreeMarker "processor" to assemble the final realm.json.

The whole new realm flow goes like this:

1. On admin console user sets realm name and choose bundles of available clients;
2. The admin console triggers a "NewRealmPipeline" via REST;
3. NewRealmPipeline uses FreeMarker processor;
4. FreeMarker processor has some opinionated json templates for realm and clients;
5. FreeMarker processor uses its templates with the input data coming from admin console to build final realm.json;
6. NewRealmPipeline add realm.json to gitops repository;
7. NewRealmPipeline adds an entrance for the new realm.json to a configMap manifest on gitops repository;
8. The configMap is linked to a Job for keycloak-config-cli;
9. ArgoCD watches gitops repository and when NewRealmPipeline pushes chances to the configMap manifest the Job is triggered;
10. keycloak-config-cli runs and creates the new realm;

The only possible updates via user admin console are add clients, enable/disable clients, enable/disable realms. For updates, the pipeline flow is pretty much the same.

Anything else could be done by editing realm files in the gitops repository (we avoid doing changes using keycloak ui). By the way, this is great, specially when you need to spread some changes through several realms at once. But whenever the change is a new default we need to reflect the changes to the opinionated templates.

Not ideal and a little error-prone for sure, but reliable as the templates are stable and all realm's json files are on git.