Exporting realms & users from Dev Services Keycloak container

[sberyozkin]

Discussed in https://github.com/quarkusio/quarkus/discussions/30071

^{Originally posted by **franck102** December 24, 2022} Hi, First of all congratulations for the outstanding documentation around OIDC support in Quarkus! I am using OIDC Dev Services and one issue I have is that my entire Keycloak configuration (public realm, users with specific roles...) is lost every time I restart quarkus:dev. I have been able to partially export the configuration from the admin console, however exporting everything including users requires running kc.sh export from the container. I have tried to do that using the -Djboss.socket.binding.port-offset mentioned elsewhere to avoid the port conflict with the running Keycloak, but this doesn't seem to work in the quay.io container, I am still getting a `io.quarkus.runtime.QuarkusBindException` and the export fails. I can start the container manually with the docker label expected by Dev Services.. but I eventually run into the same issue when the container is eventually stopped: does anyone have a good suggestion to successfully export the whole configuration from that Keycloak container? Thanks!

[quarkus-bot[bot]]

/cc @geoand(devservices), @pedroigor(keycloak), @stuartwdouglas(devservices)

[geoand]

Would it not make sense to have something in the DevUI that could do this?

[sberyozkin]

Hi @geoand, to have the users and roles included in a keycloak realm during the export, a sequence like the one described in https://github.com/quarkusio/quarkus/discussions/30071#discussioncomment-4491437 is required. So as proposed in the follow up comment, that sequence perhaps can be supported with TestContainers API during the container shutdown

[geoand]

I really think that the Keycloak container (at least in a development mode) should make it much easier to get those files. Doing that container dance will be incredibly brittle.

[sberyozkin]

@geoand See, the only way with Keycloak to have users and roles included in the exported realm is to do just that, run the command in the container, and then copy the realm file. It is just the way it is. If it were possible to let users press, in DevUI, Export realm, and then internally DevServices for Keycloak fetches the realm from Keycloak and saves locally, then I'd gladly do it. But it won't work like that. We do run various entry point commands for containers on their startup, in Kafka, etc, so running something like /opt/keycloak/bin/kc.sh export --users realm_file --dir /tmp/export on the container shutdown and then copying it to the user configured path on the host should work ?

[geoand]

It is just the way it is

Sure, I understand that's the way things are currently, but what I am saying is that IMO, we can work with the Keycloak team to make it easier and then consume whatever API we come up with from our side.

[sberyozkin]

@geoand FYI, DevUI already offers an indirect option to export the realm, there is an AdminURL link there and the users can go directly to Keycloak Admin Console and press Export Realm - but some sensitive information can only be exported via the command line.

but what I am saying is that IMO, we can work with the Keycloak team to make it easier and then consume whatever API we come up with from our side.

It is a design choice of Keycloak which I do not expect to be changed anytime soon, they have an API to export realm, it is just that they don't want to include certain information when an export is requested via this API.

To be honest I'm not sure what the problem is with running a command in the container, this is a normal option we use when we need to.

[geoand]

they have an API to export realm, it is just that they don't want to include certain information when an export is requested via this API

Would it not be an option to start Keycloak with a flag that allows for this information to be exported?

To be honest I'm not sure what the problem is with running a command in the container, this is a normal option we use when we need to.

It's not a problem, it's just that it will be super brittle. If anything changes on the Keycloak side, our "integration" will fail.

[sberyozkin]

Would it not be an option to start Keycloak with a flag that allows for this information to be exported?

Makes sense to check with the team, sure, I don't think it is available now, but indeed, good enhancement idea; not sure it will be enough to include such info in the realm returned via Keycloak Admin API but is worth a try.

It's not a problem, it's just that it will be super brittle. If anything changes on the Keycloak side, our "integration" will fail.

Not really, this is what Keycloak supports, we won't be doing some internal API calls.

I think your idea to request supporting the users/roles export at the start up is nice, and hopefully Keycloak team will consider it (I'll get in touch with CC to you soon), but in meantime, running a command in the container should not be dismissed IMHO

[geoand]

but in meantime, running a command in the container should not be dismissed IMHO

I agree, we should do this in meantime. My point is more to make sure that we try to make our lives easier in the future

[sberyozkin]

Makes sense, I'm not really fancying doing updating DevServices for KC to run commands in the docker container :-)

FYI, https://www.keycloak.org/server/importExport#_exporting_a_realm_to_a_file.

I'll do a few checks a bit later to double check that exporting a realm via Admn API/Keycloak console right now produces a partial export only, was the case as far as I recall when I was migrating KC realms in quickstarts when updating to KC 19...

[geoand]

Thanks!

[michalvavrik]

This can't be done unless KC team make changes. Export is only supported when KC server is stopped according to docs. If you don't listen and try it anyway, only master realm is exported and errors are logged due to conflict as "export" command tried to start new server instance. You can work around this by setting '--http-port' for original instance used by Quarkus, but we use in-memory database and every new instance creates it's own database instance, therefore no changes made by Quarkus can be exported. If we were using external database, there would be multiple options (e.g. starting new instance solely for export as suggested by Vaclav Muzikar, etc.).

[sberyozkin]

Right, if we can convince the Keycloak team to make the full export possible via Admin API then it would be best

[michalvavrik]

I think it would be much easier if Keycloak instead of starting new Keycloak instance allowed you to set port and use the existing (local) one. I'll open an issue there. This way, it shouldn't be big job.

[HerrDerb]

Right, if we can convince the Keycloak team to make the full export possible via Admin API then it would be best

maybe only enable the export over the api on start-dev