step-ca: Backup/restore of badger database

ErikLundJensen commented 2 years ago

What would you like to be added

Backup/restore of the Badger database to keep the ACME registrations when re-installing step-certificates in Kubernetes.

Secondary objective: support migration of database from Badger v1.6 to v2 by backing up existing v1.6 database and restore to v2 database.

Why this is needed

Currently it is not easy to get access to the Badger database inside the step-certificates pod. The step-certificate container must be stopped to be able to take a backup as Badger does not support backup of a database in use. This could be done by stopping the pod and mounting the PVC into another pod, however, that is cumbersome in a GitOps setup.

Suggested solution

Add an init-container that takes a backup of the database. Then it is possible to extract the backup file by using kubectl cp or VolumeSnapshot.

A Kubernetes job or similar cronjob outside of the cluster can then request the pod to restart daily and copy the backup file from the new pod. Note, however, that part will not be part of this solution.

Changes required:

step-certificates Helm chart have to support additional initContainer
an image containing Badger and backup/restore script
documentation of how to configure and operate

Helm values, where badger can be enabled.

ca:
  db:
    backupRestore:
      badger:
        enabled: true
image:
  backupRestoreRepository: step-certificates-badger:latest

where we need to find a proper name for the image or embed the content into step-ca cli image.

Additional functionality

encryption of backup file
support multiple badger versions. Image could contain badger 1.6 and badger 2.x. Init container can then use the ca.db.type in Helm chart to select proper version of badger.

Pull-Request

I have a working setup (without encryption and multi-version support) and I can create a pull-request.

Alternative solution

Enhance Helm chart to support insert of any init-container configuration from values into the Statefulset template. Thereby the backup/restore will not be part of this Helm chart and left to the user to configure.

dopey commented 2 years ago

Hey @ErikLundJensen, we'd love to see the PR you're referencing.

You've made it clear in the issue title that this is specifically for Badger, which is fine, but I'll just mention that distributed databases are supported (mysql, postgres) which would be another workaround for this issue.

ErikLundJensen commented 2 years ago

@dopey I agree that the other databases can be used, however, with added complexity. To keep this chart simple we could just add the option to define initContainers in Helm Chart. Thereby the configuration of backup/restore of Badger inside step-ca could be described in some blog with reference to a configuration sample.

A draft PR has been created to get a picture of how it can be implemented. https://github.com/smallstep/helm-charts/pull/98

smallstep / helm-charts