Enhancement : Send commands to extension in the dev console

[survivant]

Description

I'm using the extension oidc. I'm running to application with mvn quarkus:dev

I would like to export my realms from keycloak. I can't use the UI because the users and secrets won't be exported from the admin UI. So I need to launch a command to docker to export the realm followed by a command to retrieve the file.

The commands look like that

docker exec -it 172a22de783b /opt/jboss/keycloak/bin/standalone.sh "-Djboss.socket.binding.port-offset=100" "-Dkeycloak.migration.action=export" "-Dkeycloak.migration.provider=singleFile" "-Dkeycloak.migration.file=/tmp/export-realm.json"

docker cp 172a22de783b:/tmp/export-realm.json export-realm.json

I would like to be able to send a command to the extension from quarkus console.

When I press "h" to get help, I can see few commands

The following commands are currently available:

== Continuous Testing

[r] - Resume testing
[o] - Toggle test output (disabled)

== HTTP

[w] - Open the application in a browser
[d] - Open the Dev UI in a browser

== System

[s] - Force restart
[i] - Toggle instrumentation based reload (disabled)
[l] - Toggle live reload (enabled)
[j] - Toggle log levels (INFO)
[h] - Shows this help
[q] - Quits the application

--
Tests paused
Press [r] to resume testing, [o] Toggle test output, [h] for more options>

We could have something like

[x] Send command to extension

x 

Please enter which extension you want to send the command too [l] : to list all the extensions

oidc

Please enter the command  or [h] to list the commands supported

h

[e] export realms
.....

in this example, I'm talking about exporting the realms, but the functionnality have to be implemented later

Implementation ideas

No response

[maxandersen]

interesting idea - we've discussed in past to have api endpoints in devui we could access from a terminal ...this is a bit more where you can navigate list of commands of extensions.

@stuartwdouglas wdyt?

[sberyozkin]

@survivant DevServicies for Keycloak start their own container - so not sure about starting another standalone container from the Dev Console - but other than that, indeed, being able to request Dev Services to do some actions from the console is interesting as Max also said.

In meantime, you can export the realms using a Keycloak Admin option in the Dev UI.

@stuartwdouglas I wonder if #20628 is somehow related (may be not...)

[survivant]

@sberyozkin I'm not talking about launching a new container, but rather sending commands to container launched by DevServices for Keycloak

for Keycloak, we can really used the export in the UI because the export won't include users and secrets. To get a full export we need to connect into the container and use the export option . I have a example that is working.

main script launch on the host

CONTAINERID=$(docker ps -f "ancestor=quay.io/keycloak/keycloak:15.0.2" -q)

# copy the script into the docker container
docker cp export-realm.sh $CONTAINERID:/tmp/export-realm.sh

# execute the export
docker exec -it $CONTAINERID bash /tmp/export-realm.sh

# retrieve the export file
docker cp $CONTAINERID:/tmp/realms-export-single-file.json export.json

the script that will do the export

# docker-exec-cmd.sh taken from https://stackoverflow.com/questions/60766292/how-to-get-keycloak-to-export-realm-users-and-then-exit

set -o errexit
set -o errtrace
set -o nounset
set -o pipefail

# If something goes wrong, this script does not run forever, but times out
TIMEOUT_SECONDS=300
# Logfile for the keycloak export instance
LOGFILE=/tmp/standalone.sh.log
# destination export file
JSON_EXPORT_FILE=/tmp/realms-export-single-file.json

# Remove files from old backups inside the container
# You could also move the files or change the name with timestamp prefix
rm -f ${LOGFILE} ${JSON_EXPORT_FILE}

# Start a new keycloak instance with exporting options enabled.
# Use the port offset argument to prevent port conflicts
# with the "real" keycloak instance.
timeout ${TIMEOUT_SECONDS}s \
    /opt/jboss/keycloak/bin/standalone.sh \
        -Dkeycloak.migration.action=export \
        -Dkeycloak.migration.provider=singleFile \
    -Dkeycloak.migration.strategy=OVERWRITE_EXISTING \
    -Dkeycloak.migration.realmName=quarkus \
        -Dkeycloak.migration.file=${JSON_EXPORT_FILE} \
        -Djboss.socket.binding.port-offset=99 \
    > ${LOGFILE} &

# Grab the keycloak export instance process id
PID="${!}"

# Wait for the export to finish
# It will wait till it sees the string, which indicates
# a successful finished backup.
# If it will take too long (>TIMEOUT_SECONDS), it will be stopped.
timeout ${TIMEOUT_SECONDS}s \
    grep -m 1 "Export finished successfully" <(tail -f ${LOGFILE})

# Stop the keycloak export instance
kill ${PID}

the script is not complete yet, but it works.

[sberyozkin]

@survivant Sure, you typed the command which starts a separate container so I assumed this is what you wanted to do - but I see it was just an example. OK, so as I said I agree it would be interesting to request something from Dev Services via the console and hopefully Stuart will figure out how it can be done in a generic way; perhaps each DevServices can provide a resource listing the supported commands and the corresponding handler paths (export realm - /export, etc).

As far as the realm export is concerned - you can do it by clicking a Keycloak Admin - it is in the top right Dev UI corner, you can see how it looks at https://quarkus.io/guides/security-openid-connect-dev-services#keycloak-initialization and just export it directly from Keycloak Admin Console. We can't really ship various scripts in DevServices for Keycloak - but we can try to support some main actions via a loose Keycloak Admin API requests as we do for creating the realm right now for example (using quarkus-keycloak-admin directly is problematic due to its dependency on JAX-RS) - we can review the list of commands to support once the feature is supported at the Dev Services level in general

[quarkus-bot[bot]]

/cc @stuartwdouglas

[survivant]

@sberyozkin fyi (https://github.com/keycloak/keycloak-documentation/blob/main/server_admin/topics/export-import.adoc)

the Admin Console as well as exporting most resources.. does not support the export of users.

That was my issue. The admin UI is not a backup/restore solution. Now a have a manual solution to export my config after I'm done playing with the data.

[sberyozkin]

@survivant I see; can you check please if quarkus-keycloak-admin can do it ? (Most likely with keycloak.realm("myRealm").toRepresentation();). I think it has to work via the admin API for Dev Services for Keycloak to support it via the action request from the console

[survivant]

@sberyozkin I checked that few days ago and I think the problem was that the API doesn't return all the contents. Let me explain. getRealmRepresentation will return the root of the Realm, without the users/roles/... we have to do a call to retrieve each of them manually. It's painful to do and you could miss information.