adorsys / keycloak-config-cli

Import YAML/JSON-formatted configuration files into Keycloak - Configuration as Code for Keycloak.
Apache License 2.0
705 stars 133 forks source link

Normalized Export #799

Open mlhartme opened 1 year ago

mlhartme commented 1 year ago

Problem Statement

(I'm not sure if this feature request is best placed here or in keycloak itself ... I'll first try here ...)

We use keycloak-config-cli to properly define the configuration of our Keycloak server in my-realm.json. That's great!

But we use manual configuration as well, usually because we need try and error to find the proper configuration. To get manual changes into my-realm.json (and thus get a well-defined state again), we diff keycloaks export (kcadm.sh create realms/my-realm/partial-export) before and after the manual change.

Problem with that is: the diff gets big, mostly because ordering in some of keycloaks arrays doesn't matter, and keycloak returns entries in different order. And because we regularly delete and create the realm from my-realm.json to play save - which causes Keycloak to choose new id s.

Proposed Solution

Is it possible to provide normalized realm exports that:

We currently use jq to manually do some normalization (see below). That helps us a lot, but it feels like a poor solution, especially replacing all id s with XXX ...

Best regards, Michael

!/bin/sh

if [ "$#" -ne 1 ] ; then echo "Normalize arrays and ids" echo "usage: normalize.sh " exit 1 fi set -e

json=$(cat "$1") json=$(echo "${json}" | jq '.clientScopes = (.clientScopes | sort_by(.name))') json=$(echo "${json}" | jq '(.. | select(.protocolMappers?) | .protocolMappers) |= sort_by(.name)') json=$(echo "${json}" | jq '(.components | .["org.keycloak.keys.KeyProvider"]) |= sort_by(.name)') json=$(echo "${json}" | jq '(.components | .["org.keycloak.storage.UserStorageProvider"] | .[] | .subComponents | .["org.keycloak.storage.ldap.mappers.LDAPStorageMapper"]) |= sort_by(.name)') json=$(echo "${json}" | jq '(.components | .["org.keycloak.services.clientregistration.policy.ClientRegistrationPolicy"]) |= sort_by(.name, .subType)') json=$(echo "${json}" | jq '(.. | select(.["allowed-protocol-mapper-types"]?) | .["allowed-protocol-mapper-types"] ) |= sort') json=$(echo "${json}" | jq '(.. | select(.id?) | .id) |= "XXX" ') echo "${json}" | jq --sort-keys

Environment

sonOfRa commented 1 year ago

I've started a draft implementation of this feature over at https://github.com/adorsys/keycloak-config-cli/pull/818.

The general idea is to use Javers to diff the objects, comparing them to a "baseline" realm export for any given Keycloak version, and then using that diff to produce a normalized/minimized yaml file that can later be imported using keycloak-config-cli.

Any inputs on particular choices or ideas on how to do it better are much appreciated!

katoquro commented 1 year ago

Hello! @sonOfRa, how is it going with normalization? I have a similar requirement: show diff before deployment of the new configuration. I've started but I've found this issue and think that it would be better to wait for your implementation and then tune it for my case.

sonOfRa commented 1 year ago

Hi!

I'm making progress, but it's relatively slow going. There's a few edge cases that pop up every now and then when trying the current state out with different realm configurations. Most of the important features of the keycloak config are already present in the normalized config now, but one big block that is still missing is the "components": [] block, which is a quite complex type that isn't very constrained, and as such a bit difficult to "normalize" without accidentally breaking things.

katoquro commented 1 year ago

I see. components are like Map<String, Object> 😄. Currently, I think that we can just make a deep ordering for the keys of that maps and it may cover most cases. At least, it will be reproducible.

SebastianKunz commented 10 months ago

I am also interested in this feature. @sonOfRa can you elaborate what the state of progress is? Are there any blockers for you?

dominiktopp commented 9 months ago

We had a similar problem. Our solution was to replace ids (search by regex, replace with XXX) and use a JSON diff tool like https://www.jsondiff.com/. That reduces the problem to the order of elements in arrays but this is not very hard to check by hand. After a few roundtrips we had enough experience and examples in our realm.json to do most of the configuration directly in the realm.json.

But for the migration from "configuration by admin ui" to keycloak-config-cli a tool like https://github.com/adorsys/keycloak-config-cli/pull/818 would have been great 😄