viniciusdc
commented
7 months ago I just finished reading @pt247; this looks great! Some considerations below:
- Add Kubernetes CronJobs and ArgoWorkflows as part of your Scope section of the RFD
Keycloak manages user authentication. There is a recommended way of backup and restore recommended Keycloak docs - link
I would suggest doing this differently, as interacting with the kc client is troublesome, and we only care about the users and groups. This could be handled by API requests directly in a moderate way.
nebari backup user-data --backup-location
If we do end up having this structure, I prefer that those commands (user-data, user-creds) are not exposed directly to the user (similar to what nebari render is called right now). The user should only handle this manually if the general backup fails the middle trough.
Scheduled backup of Nebari config: First, we extend the existing Nebari configuration file to provide a backup schedule to the Argo workflow template. The Argo template will encrypt the Nebari config and back it up.
We already saved the kubeconfig as a secret on Kubernetes; we could reuse that as part of this and enable versioning for that secret.
pt247
tylergraff
dcmcand
- Backup and restore - RFD
- Summary
- User benefit
- Design considerations:
- Data protection considerations:
- In the scope of this RFD:
- Out of scope for this RFD:
- Existing backup process
- Backup and Restore strategies
- Backup and restore by component Approach #1
- Example Backup flow:
- Example Restore flow:
- Finer details
- Vertical slices per user migration Approach #2
- Restful Interface Approach #3
- Serializable vs Non-Serializable data
- Design Discussion
- Possible options
- Special note about conda-store
- The simplest approach (Compatible with Approach #1)
- Approach per user (Can be used in Approach #2 and Restful Approach #3)
- Relevant links:
- Unresolved questions:
# Backup and restore - RFD A design proposal for Backup and Restore service in Nebari. ## Summary As Nebari becomes more popular, it's essential to have dependable backup and restore capabilities. Automated scheduled backups are also necessary. Planning is vital since Nebari has several components, including conda-store environments, user profiles in KeyCloak, and user data in the NFS store. ## User benefit 1. Nebari admins will get a straightforward backup process using Nebari CLI. 2. Admins will also be able to define a schedule for automated backup in Nebari config. 3. Nebari upgrades can automatically save the state before providing upgrades. 4. User data and other Nebari components can better protect against accidental deletion. ## Design considerations: We need to look at the development, maintenance, administration, and support requirements to decide on an appropriate strategy for this service. Following is a list of key criteria for the service: 1. **Availability**: Service disruption to perform backup or restore. 2. **Observability**: Visibility of progress, error, and status. 3. **Maintainability**: Ease of building, maintaining and supporting. 4. **Composability**: Backup and restore in small chunks independently. 5. **Security**: Access control to the backup and restore service and the backup itself. 6. **Compatibility**: Forward and, if possible, backwards. 7. **Flexibility**: multiple entry points to the backup and restore, e.g. scheduled API 8. **Scalability**: Scalability should scaled to large deployments. 9. **Feasibility**: developing, maintaining, or computing resources. 10. **Compliance**: with various data protection regulations. 11. **On-prem**: On-prem and air-gapped deployments. ## Data protection considerations: 1. **Encryption at rest and in transit**: We must have data encrypted in motion and at rest to protect against unauthorized access. 2. **Backup location**: Several data protection directives in the US and EU limit where we can store certain data assets. We should design the backup and restore solution with this in mind. 4. **Day zero feature**: Encryption at rest and transit needs to be available in the first version of the backup and restore service. 5. **PoLP** (principle of least privilege): Only authorized users should be able to access the backup and restore service. ## In the scope of this RFD: This Request for Discussion (RFD) aims to establish a high-level strategy for backup and restoration. The goal is to reach a consensus on design choices, API, and a development plan for the backup and restoration of individual components. The implementation details of the identified design will be part of another RFD. The focus of this RFD is to develop a backup and restoration strategy for the following components: - Nebari config - Keycloak - Conda-store - User data in NFS ## Out of scope for this RFD: Following Nebari components are not covered in this document. - Nebari plugins - Loki Logs + prometheus - Nebari migration (for, e.g. from AWS to GCP) - Custom backup schedules (e.g. component specific backup schedules) ## Existing backup process You can find the existing docs for backup on this [page](https://www.nebari.dev/docs/how-tos/manual-backup). ## Backup and Restore strategies There are several approaches to Nebari backup and restore. Some are closer to the current backup and restore, and some are entirely novel approaches. Each of these methods has its own set of advantages and disadvantages. In this section, we will summarise the various approaches suggested in the comments, outline the pros and cons, and briefly describe the implementation. ### Backup and restore by component Approach #1 ```mermaid flowchart TD Backup --> Storage Nebari --> |1. config| Backup Nebari --> |2. Keycloak | Backup Nebari --> |3. Conda Store | Backup Nebari --> |3. User Data | Backup Storage --> Restore1 Restore1 --> |1. config| Nebari1 Restore1 --> |2. Keycloak | Nebari1 Restore1 --> |3. Conda Store | Nebari1 Restore1 --> |3. User Data | Nebari1 ``` This approach aims to automate the current manual backup and restore process. A typical Nebari deployment consists of several components like Keycloak, conda-store, user data and more. #### Example Backup flow: ```mermaid flowchart TD A1[CLI] --> B(Backup workflow) A2[Nebari config.backup.schedule] --> B A3[Argo workflows UI] --> B B --> F(Backup Nebari config) F --> D(Backup Keycloak) D --> C(Backup NFS) D --> E(Backup Conda Store) C --> X(Backup Location) D --> X E --> X F --> X ``` #### Example Restore flow: ```mermaid flowchart TD A[Nebari Restore CLI - Specified backup] --> B(Backup workflow - latest backup) A1[Argo Workflows UI] --> B B --> B1(Restore workflow - Specified backup) B1 --> F(Restore Nebari config) F --> D(Restore Keycloak) D --> C(Restore NFS) D --> E(Restore Conda Store) C --> Z(Validate restore completion) D --> Z E --> Z Z --> |failure| X(Restore workflow - latest backup) Z --> |success| Y(Stop) X --> |success| Y X --> |failure| Y ``` `Note`: Both these workflows are, for example, and must be refined/refactored. Let's look at the pros and cons of this approach: **Pros** 1. Feasibility: We can use tried and tested tools for database dump or Restic to sync files between source and destination. 2. Maintainability: Development of each rach task (say backup conda-store) can happen separately and iteratively. 3. Compatibility: Excellent support is available for tried and tested production-ready tools like pg_dump, Restic, rsync and more. This design can use Nebari component agnostic tools, which means the same solution could work for multiple versions of Nebari, providing backwards and forward compatibility. **Cons** 1. Observability: If the backup fails because of a single failed sub-task, it can result in a whole backup or restore failure. This solution offers little Observability. 2. Composability: The success of individual tasks does not guarantee success of overall success. For example, the solution might find a new user's data for backup without the user being there when Keykloak backs up. 3. Scalability: As user data increases, this design might need to evolve to take incremental snapshots. If the time it takes to back up increases, so do the chances of Nebari state changing. 4. Availability: The solution must implement a maintenance window for the entire Nebari during backup and restore processes. ### Finer details 1. Backup location: This design assumes Nebari has read-write access to the backup location. Nebari will manage the backup location. 2. Local backup: If the backup location is a local directory, the client should have access to read-write to that directory. ### Vertical slices per user migration Approach #2 We could look at nebari from the perspective of the user. Each user has some shared and dedicated state in each Nebari component. | Nebari | Shared | Dedicated | |-------------|----------------------------|---------------------| | Keycloak | Groups, Roles, permissions | User profiles | | Conda store | Shared environments | User environments | | JupyterHub | Shared user data | Dedicated user data | The solution recommends backing or restoring shared resources first. We can then backup/restore users in parallel or any order. User migration workflow ```mermaid flowchart LR rc[Restore user] --> rs[Restore shared state] --> ru[Restore user] s[Storage] -.-> rs s -.-> ru bc[Backup user] --> bs[Migrate shared state] --> bu[Migrate user] bu -.-> Storage bs -.-> Storage ``` Nebari migration overall Backup flowchart ```mermaid flowchart LR nb[Nebari Backup] ==> rs[Backup shared state] rs ==> bu1[Backup user A] & bu2[Backup user B] & bu3[Backup user C] -.-> Storage rs --> | ... | Storage rs --> |Backup user n| Storage ``` Restore flowchart ```mermaid flowchart LR nr[Nebari Restore] ==> rsr[Restore shared state] Storage -.-> ru1[Backup user A] & ru2[Backup user B] & ru3[Backup user C] & ru4[...] & ru5[Backup user N] rsr ==> ru1 & ru2 & ru3 & ru4 & ru5 Storage -.-> rsr ``` Let's look at the pros and cons of this approach: **Pros** 1. Fail fast approach: If all goes well, we will have backed up all users. If not, then there will be two possibilities: 1. Shared state backup/restore failure - which will be immediate or 2. Single User state backup/restore failure - which can be local to a particular user, e.g. another user's backup/restore might still succeed. The likelihood of failure significantly decreases after the initial few users have been successfully migrated. 2. Composable: admins can migrate a single user at a time or a batch of users simultaneously. We could easily create workflows to the backup shared state at a higher or lower cadence than users. 3. Monitoring: The design allows for more granular status and progress monitoring. 4. Availability: Backup of a user should not impact service for other users. **Cons** 1. Maintainability and Compatibility: This design depends on the APIs present in the individual components and our understanding of the state and data within them. However, our understanding and, therefore, the implementation may need to be updated with version upgrades. Hence, this backup/restore solution is only compatible with a limited number of component versions. 2. Feasibility: This design, although more evolved, will also require more building, maintenance, and support. ### Restful Interface Approach #3 The last two designs include the backup and restore functionality in Nebari. The central assumption was that Nebari should be able to back up and restore itself. However, thanks to helpful comments in this RFD, this design challenges this premise and proposes an alternative solution. This design breaks the implementation into two: the interface and the strategy. It argues that Nebari should only provide the interface for importing/exporting data. The backup and restore strategy should be part of the client code. We can extend the interface by providing a Python library. ```mermaid block-beta columns 1 j["Client Script (Backup strategy maintained by Nebari Admin)"] blockArrowId6<[" "]>(updown) L["Nebari backup and restore library (Python package)"] blockArrowId7<[" "]>(updown) D["Nebari Backup and restore REST API"] blockArrowId6<[" "]>(updown) block:ID A["Conda Store REST API"] B["User DATA REST API"] B2["Keycloak REST API"] end ``` The idea is simple: instead of building a backup and restoring _**service**_, we could build a backup and restore _**interface**_. The only job of this interface will be to provide users' state and data to authenticated users outside Nebari. The entire backup and restore logic can be built and maintained outside Nebari. This backup and restore client can then be run from anywhere, providing Admins with flexibility that other designs do not offer. ```mermaid flowchart LR subgraph Backup and restore library Client end Client-->I subgraph Nebari I I[Backup and restore interface API]-->K[Keycloak API] I-->C[Conda store API] I-->J[JupiterHub API] I-->N[User data API] end ``` #### Serializable vs Non-Serializable data An essential requirement for this design is to expose data and state. APIs like Keycloak and conda-store API already provide the bulk of serializable states. However, not all states are serializable, e.g., user data and conda packages. In this case, the design recommends APIs to download location URLs. APIs in Nebari could be completely stateless. Let's look at a few transactions with this proposed API. Serializable data ```mermaid sequenceDiagram Client->>API: GET /users API-->>Keycloak: GET /admin/realms/{realm}/users/ Keycloak-->>API: [A, B, C] API-->>Client: [A, B, C] ``` Non-Serializable data ```mermaid sequenceDiagram Client->>API: GET /users/A/environments API-->>conda-store: GET /api/v1/environment/?namespace={..} conda-store-->>API: [E1, E2, E3 ...] API->>Client: [{envs:[E1, E2]}] Client-->NFS: FTP/Rsync/Restic FETCH Artifact from E1, E2, E3 ... ``` Let's see the pros and cons of this design. **Pros** 1. Flexibility: Nebari admins can write their custom backup strategy based on the organization's needs. 2. Observability: This solution gives Nebari Admins a unique insight into the inner workings of Nebari. It makes it less opaque and, thus more 3. Compliance: The authenticated admins are responsible for enforcing compliance with company policies. 4. Availability: If the client interface is well-developed, this solution can achieve the highest level of availability. 5. Clear division of responsibility: Responsibilities of Component API, Nebari backup, and restore API, client library, and client code. **Cons** 1. Maintainability & Support: This approach moves the complexity of the backup/restore strategy outside Nebari. It now requires Nebari admins to know and understand the inner workings of Nebari. 2. Flexibility: A misconfigured client script can wreak havoc with the Nebari ecosystem. ## Design Discussion ### Possible options Each of the above-discussed designs has its pros and cons. We could also extend the designs.For example,we could extend Approach#2 and#1 via an API toprovide simple interfaces like/users/{uid}/backup/keycloak. Let's look at a few possible options we can vote on. More suggestions welcome. 1. Option#1: Start with [Restful Approach #3](#restful-interface-approach-3) to enable power users in the first iteration. Then extend this to [Sliced Approach #2](#vertical-slices-per-user-migration-approach-2) for normal users. 2. Option#2: Implement [Bulk Backup Approach #1](#backup-and-restore-by-component-approach-1) in the first iteration evolve it to [Sliced Approach #2](#vertical-slices-per-user-migration-approach-2) by exposing an API in second iteration. 3. Option#3: Implement [Bulk Backup Approach #1](#backup-and-restore-by-component-approach-1). 4. Option#4: Implement [Sliced Approach #2](#vertical-slices-per-user-migration-approach-2). 5. Option#5: Implement [Restful Approach #3](#restful-interface-approach-3). ### Special note about conda-store Conda store is one of the more complicated pieces to replicate among the Nebari components. We will need to work with conda-store team to come up with a detailed plan on backup-restore. But, here is a initial analysis based on conda-store docs. >The S3 server is used to store all build artifacts for example logs, docker layers, > and the Conda-Pack tarball. The PostgreSQL database is used for storing all states > on environments and builds along with powering the conda-store web server UI, REST > API, and Docker registry. Redis is used for keeping track of task state and results > along with enabling locks and realtime streaming of logs. #### The simplest approach (Compatible with [Approach #1](#backup-and-restore-by-component-approach-1)) Backup the object storage and dump the database. Restore would be reverse. We might have to ensure that database location entries for artifacts and Conda-pack are pointing to the right location. This might involve simple find and replace operations to the SQL dump. #### Approach per user (Can be used in [Approach #2](#vertical-slices-per-user-migration-approach-2) and [Restful Approach #3](#restful-interface-approach-3))  - Getting the shared state: - Get and SQLDump of entire conda-store - Mark all entries in `environment` as deleted by setting `deleted_on` field. - Get global name spaces, for each - get related `environment`s and reset `deleted_on` to make them available. - for each environment - Get build artifacts to backup `environment` -> `build` -> `build_conda_package_build` -> `conda_package_build` - Backup artifacts from source. - Getting the user state: - Same as getting the shared state except the namespace will be of the given user. Please note: 1. We need to get conda-store team to review this. But it gives a general idea 2. Most of this flow can be done via API except changing environments delete status. 3. We will need to create as separate RFD for Conda store. ## Relevant links: 1. https://www.nebari.dev/docs/how-tos/manual-backup 2. https://www.keycloak.org/server/importExport 3. https://argoproj.github.io/workflows/ 4. https://www.keycloak.org/docs-api/22.0.1/rest-api/index.html#_users 5. https://conda.store/conda-store/references/api ## Unresolved questions: 1. Which design is most suitable? 2. Is there a hybrid design that we can develop iteratively?