search

nebari-dev / nebari

🪴 Nebari - your open source data science platform
https://nebari.dev
BSD 3-Clause "New" or "Revised" License
271 stars 88 forks source link

[ENH] - Expose Backup and Restore for Keycloak API #2519

Open pt247 opened 1 month ago

pt247 commented 1 month ago

Feature description

Backup and Restore RFD

Value and/or benefit

Nebari Admin can access Keycloak API and query user data.

We need to create a backup controller to expose the existing Keycloak API to authenticated users. This can be a REST API using FastAPI.

/api/v1/keycloak: Expose existing Keycload Admin REST API - docs

So for example: GET /api/v1/keycloak/admin/realms/nebari/users should get us a list of all the non-admin users.

Anything else? Related META issue- #2518

pt247 commented 1 month ago

Proposed Design:

Nebari uses Keycloak to manage users and groups.

Keycloak also manages clients for services like Jupyterhub, argo-server-sso, conda_store, etc. Nebari creates These clients automatically, so we don't have to expose them. Nebari also pre-created roles, so we can skip exposing them.

The Nebari API will consolidate responses from the internal Keycloak API and return a consolidated JSON response. This JSON representation should contain enough information to restore the same user in another instance of Keycloak. Assumptions:

  1. The Keycloak on both sides of backup and restore are managed by Nebari and thus expected to have the same default setup with Nebari installation.
  2. The endpoints provide a mechanism to retrieve and restore. The logical sequence of how endpoints will be called is left up to the client application code.
  3. The lists of internal Keycloak endpoints (below) were created using Keycloak REST API docs. While developing this feature, we might have to add/delete a few endpoints.
  4. If a user dependency is not present while restoring a user, for example, its groups, then those groups should be automatically created while creating a user.

With this context, to accurately re-create the Nebari Keycloak, we need the following endpoints:

Endpoints for backup

GET /users (Get all users) ```mermaid sequenceDiagram API Client ->> Nebari API: GET /users Nebari API-->>Keycload API: GET /admin/realms/{realm}/users Keycload API-->>Nebari API: Nebari API-->>Keycload API: GET /admin/realms/{realm}/users/profile Keycload API-->>Nebari API: Nebari API-->>Keycload API: GET /admin/realms/{realm}/users/{id} Keycload API-->>Nebari API: Nebari API-->>Keycload API: GET /admin/realms/{realm}/users/{id}/groups Keycload API-->>Nebari API: Nebari API-->>Keycload API: GET /admin/realms/{realm}/users/{id}/role-mappings Keycload API-->>Nebari API: Nebari API ->> API Client: [{< composit json user representation >}, {...}, ...] ```
GET /users/{id} (Get user details) ```mermaid sequenceDiagram API Client ->> Nebari API: GET /users/{id} Nebari API-->>Keycload API: GET /admin/realms/{realm}/users/{id} Keycload API-->>Nebari API: Nebari API-->>Keycload API: GET /admin/realms/{realm}/users/{id}/groups Keycload API-->>Nebari API: Nebari API-->>Keycload API: GET /admin/realms/{realm}/users/{id}/role-mappings Keycload API-->>Nebari API: Nebari API ->> API Client: {< composit json user representation >} ```

Endpoints for restore

POST /users/ (Create a new user.) ```mermaid sequenceDiagram API Client ->> Nebari API: POST /users/ Nebari API-->>Keycload API: POST /admin/realms/{realm}/users Keycload API-->>Nebari API: Nebari API-->>Keycload API: PUT /admin/realms/{realm}/users/{id}/reset-password Keycload API-->>Nebari API: Nebari API-->>Keycload API: PUT /admin/realms/{realm}/users/{id}/groups/{groupId} Keycload API-->>Nebari API: Nebari API-->>Keycload API: GET /admin/realms/{realm}/users/{id}/role-mappings Keycload API-->>Nebari API: Nebari API ->> API Client: {< composit json user representation >} ```
DELETE /users/{id} (Get user details) ```mermaid sequenceDiagram API Client ->> Nebari API: DELETE /users/{id} Nebari API-->>Keycload API: DELETE /admin/realms/{realm}/users/{id} Keycload API-->>Nebari API: Nebari API ->> API Client: {} ```
viniciusdc commented 1 month ago

That's a nice description of all the concepts thanks, @pt247, Have you started working on this yet? I am considering scheduling a meeting for this as well

pt247 commented 1 month ago

Thanks @viniciusdc

Have you started working on this yet? I am considering scheduling a meeting for this as well

No, I have not started. Meeting only foucsing on Keycloak backup and restore would help.

pt247 commented 4 weeks ago

interesting link: https://github.com/nebari-dev/nebari/compare/develop...viniciusdc:nebari:1784-keycloak-add-multiple-users

pt247 commented 4 weeks ago

@tylergraff I assume you have some experience in backup and restore. I was wondering how you migrate credentials? Also, can you scripts any scripts you use to automate/simi-automate Keycloak migration.

pt247 commented 3 weeks ago

@kalpanachinnappan We are starting work on backup and restore. Keycloak is the first component we are considering. We would like your input on this as well.

pt247 commented 3 weeks ago

As this solution may rely on the Keycloak API, @viniciusdc suggested we might want to compare the current Keycloak API with the latest version of Keycloak.

viniciusdc commented 3 weeks ago

yes, mainly due to more flexibility of the exporting/importing features that the newer Keycloak versions allow (it might help us handle the credentials issue)