search

p2-inc / keycloak-orgs

Single realm, multi-tenancy for SaaS apps
https://phasetwo.io
Other
417 stars 72 forks source link

Cannot export organizations: NullPointerException #216

Closed kedare closed 7 months ago

kedare commented 7 months ago

Hello.

When trying to export a realm via the kc.sh export --dir out/ CLI, the organization informations are missing from the json export and the following error is logged in the export logs:

...
2024-04-03 14:54:14,032 INFO  [io.phasetwo.keycloak.resources.EventsResourceProviderFactory] (main) initializing event roles following migration
2024-04-03 14:54:14,108 INFO  [io.quarkus] (main) Keycloak 24.0.2 on JVM (powered by Quarkus 3.8.3) started in 4.338s. Listening on:
2024-04-03 14:54:14,108 INFO  [io.quarkus] (main) Profile import_export activated.
2024-04-03 14:54:14,108 INFO  [io.quarkus] (main) Installed features: [agroal, cdi, hibernate-orm, jdbc-postgresql, keycloak, logging-gelf, narayana-jta, reacti
ve-routes, rest-client-reactive, rest-client-reactive-jackson, resteasy-reactive, resteasy-reactive-jackson, smallrye-context-propagation, vertx]
2024-04-03 14:54:14,118 INFO  [io.phasetwo.keycloak.events.HttpSenderEventListenerProviderFactory] (main) Shutting down scheduler
2024-04-03 14:54:14,118 INFO  [io.phasetwo.keycloak.events.WebhookSenderEventListenerProviderFactory] (main) Shutting down scheduler
2024-04-03 14:54:14,119 WARN  [io.phasetwo.keycloak.themes.theme.AttributeThemeProviderFactory] (main) Error removing tmpdir: java.lang.NullPointerException: Ca
nnot invoke "java.io.File.toPath()" because "dir" is null
        at io.phasetwo.keycloak.themes.theme.AttributeThemeProviderFactory.deleteRecursively(AttributeThemeProviderFactory.java:48)
        at io.phasetwo.keycloak.themes.theme.AttributeThemeProviderFactory.close(AttributeThemeProviderFactory.java:41)
        at org.keycloak.services.DefaultKeycloakSessionFactory.close(DefaultKeycloakSessionFactory.java:403)
        at org.keycloak.services.resources.KeycloakApplication.shutdown(KeycloakApplication.java:159)
        at org.keycloak.quarkus.runtime.integration.jaxrs.QuarkusKeycloakApplication.onShutdownEvent(QuarkusKeycloakApplication.java:50)
        at org.keycloak.quarkus.runtime.integration.jaxrs.QuarkusKeycloakApplication_Observer_onShutdownEvent_twTBYFGt8XmSD26Lx0ni0jbf_WY.notify(Unknown Source)
        at io.quarkus.arc.impl.EventImpl$Notifier.notifyObservers(EventImpl.java:346)
        at io.quarkus.arc.impl.EventImpl$Notifier.notify(EventImpl.java:328)
        at io.quarkus.arc.impl.EventImpl.fire(EventImpl.java:82)
        at io.quarkus.arc.runtime.ArcRecorder.fireLifecycleEvent(ArcRecorder.java:155)
        at io.quarkus.arc.runtime.ArcRecorder$2.run(ArcRecorder.java:111)
        at io.quarkus.runtime.StartupContext.runAllInReverseOrder(StartupContext.java:84)
        at io.quarkus.runtime.StartupContext.close(StartupContext.java:73)
        at io.quarkus.runner.ApplicationImpl.doStop(Unknown Source)
        at io.quarkus.runtime.Application.stop(Application.java:208)
        at io.quarkus.runtime.Application.stop(Application.java:155)
        at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:228)
        at io.quarkus.runtime.Quarkus.run(Quarkus.java:71)
        at org.keycloak.quarkus.runtime.KeycloakMain.start(KeycloakMain.java:117)
        at org.keycloak.quarkus.runtime.cli.command.AbstractStartCommand.run(AbstractStartCommand.java:33)
        at org.keycloak.quarkus.runtime.cli.command.AbstractExportImportCommand.run(AbstractExportImportCommand.java:47)
        at picocli.CommandLine.executeUserObject(CommandLine.java:2026)
        at picocli.CommandLine.access$1500(CommandLine.java:148)
        at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2453)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2415)
        at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273)
        at picocli.CommandLine$RunLast.execute(CommandLine.java:2417)
        at picocli.CommandLine.execute(CommandLine.java:2170)
        at org.keycloak.quarkus.runtime.cli.Picocli.parseAndRun(Picocli.java:125)
        at org.keycloak.quarkus.runtime.KeycloakMain.main(KeycloakMain.java:107)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at io.quarkus.bootstrap.runner.QuarkusEntryPoint.doRun(QuarkusEntryPoint.java:62)
        at io.quarkus.bootstrap.runner.QuarkusEntryPoint.main(QuarkusEntryPoint.java:33)

2024-04-03 14:54:14,142 INFO  [io.quarkus] (main) Keycloak stopped in 0.031s

On: quay.io/phasetwo/phasetwo-keycloak:24.0.2

Maybe it's not yet available on this image ? (Looks like the error may be related to something else?)

xgp commented 7 months ago

The exception is a bug in our themes extension. I opened an issue there.

The new export/import functionality should be part of the current image. @rtufisi can you please check if you can reproduce?

xgp commented 7 months ago

Looks like somehow CI didn't run after import/export was merged, so the most recent jar doesn't have it.

rtufisi commented 7 months ago

Hey all,

You are correct @xgp , these changes are not found in the quay.io/phasetwo/phasetwo-keycloak:24.0.2 jar.

The CLI import and export functionality aren't available . The CLI import will work only with standard Keycloak realm files and the CLI export will produce a realm json which doesn't contain the "organizations" The import and export of the organizations can be test only from the Keycloak console import realm and partial export functionalities.

In order to implement this functionality the these legacy service such as DirExportProvider/DirImportProvider and SingleFileImport/ExportProviderneed to be overridden to support the "organizations" key.

xgp commented 7 months ago

This is solved in #222.

@kedare This issue raised a whole host of problems, principally that we didn't understand how different import/export was from the command line and UI. We wrongly assumed they used the same classes, and we could easily override them without too much risk of new Keycloak versions breaking things. We've had too many instances of trying to do things that require upstream not to change, so we decided to go another way and make the organization import/export separate endpoints. Please see the current docs for how this works https://github.com/p2-inc/keycloak-orgs/blob/main/docs/import-export.md

kedare commented 7 months ago

Thank you for the fix.

Maybe it would be interesting to add another import/export command to make it easier to use? That would make the flow easier, I am not sure how this can be automated with the current system.

Looks like we first have to use the import command for everything else and then import via the API using the initial admin user ?

xgp commented 7 months ago

Maybe it would be interesting to add another import/export command to make it easier to use? That would make the flow easier, I am not sure how this can be automated with the current system.

We are considering adding a shell script to our own docker image distribution that will call the Keycloak import/export and then the Organizations import/export in the correct order.

Looks like we first have to use the import command for everything else and then import via the API using the initial admin user ?

:+1: Correct.