Enhanced etcd Backup Integration

[davidnuzik]

We want to integrate with Rancher etcd backup and restore. We'll need to include S3 bucket capability and scheduling via the API. on demand snapshots are also required.

[cjellick]

This feature is meant to bring RKE2's etcd backup and restore functionality on par with RKE1.

The current RKE2 snapshot/restore functionality is:

• local disk only
• schedule-based only
• only driven through the rke2 cli
• occurs on every control plane node in the cluster

This feature seeks to enhance that functionality by:

• Supporting backup to S3-compatible object stores
  • We have prior art for this: RKE1 backups and the Rancher2 new (2.5 feature) application level backup/restore.
• Exposing on-demand snapshots through an integration with Rancher2. This means on-demand snapshots WILL NOT be exposed in the rke2 cli. You will only get on-demand snaps through Rancher2 integration.
  • This doesnt mean we are GATING the logic in RKE2, it means the logic isnt exposed through the cli and the entry point is some type of k8s controller triggered by rancher.
• Support etcd restores through the Rancher2 API/UI.
• Support changing the backup schedule through the Rancher API/UI.
  • We get this through Menna's cluster config integration. So, there is no implementation work needed here, but it is part of the "requirements" to finish the backup/restore story.

Scope:

• This feature does creep into Rancher2. You will be writing the APIs in and any new controller logic in Rancher in order to enable this.
• Where possible, we want to "get this for free" in K3s as well. All functionality should be exposed in K3s if possible. If you hit any snags that make it DIFFICULT to do in k3s, we can discuss, but teh goal is to have this functionality for both k3s and rke2 clusters.

Open questions/unknowns:

• RKE1 restore functionality went through a heavy refactor in the past year to make sure that:

  1. The Rancher2 cluster config object was restored as part of the restoration. Here is an example scenario we need to solve for:
     • User takes snapshot, cluster is on k8s 1.17
     • User upgrades cluster from k8s 1.17 to 1.18 in the the Rancher API/UI
     • User rolls back to snapshot.
     • Result: Rancher's cluster config says the cluster is at 1.18, but the cluster is actually at 1.17. This is a problem the two need to match. This example extends to any configuration option exposed in Rancher, not just k8s version. I THINK Luther implemented this for RKE1. Chris Kim also probably knows about this.
  2. Certificates were restored as necessary as part of the backup. This is because if certs were rotated after a snapshot was taken and then you rolled back to that snapshot, the mismatch between the rotated certs and what is in etcd could cause problems. Chris Kim, Luther, Seb, or Hussein might know more. This feature needs to account for that as well.

• Should the S3 backup functionality be exposed in the RKE2 config/cli or just in the Rancher2 ui/api. If it isn't in the CLI, we'd have to resolve conflicts between the two different ways of configuring backups.

  • Craig's answer:
  • What level of S3 API compatibility and support?
  • Craig's answer: follow what we did for the 2.5 backup and restore, which is generally pretty bare bones.

Things I want to see in the design doc/design review:

• How this work can be broken up into smaller deliverable pieces of functionality. Can another dev be given any of those chunks? If your first pass of the design just breaks this down into smaller stories and then you have dedicated design docs for the smaller stories, that's fine.
• An overview of the API and functionality of RKE1 snapshot/restore
• Any changes to APIs and/or config options.
• Identify any other gaps not yet identified between RKE1 and RKE2 snapshot and restores.

[cjellick]

Rough LOE due by EOD 12/18

[rajivml]

I have noticed that the backups you guys are take are not incremental but a full backup every time, is it possible to do an incremental backup as well, so that the incremental backup can be scheduled at one minute interval

[davidnuzik]

@briandowns at first this issue was created thinking we need to see the whole thing thru including the integration-side with Rancher/UI. Seeing as how that's no longer the case I'd like to move this "To Test" is this something you are comfortable with? Basically let's not treat it like an epic or as an issue that must see the whole thing thru. The majority is all in for testing now. Could you move it to test?

[briandowns]

Yeah, I think that's a fair assessment. I'll update.

[ShylajaDevadiga]

Validated using v1.21.1-rc2+rke2r1

• On-demand snapshot creation and restoration of snapshots stored locally and in s3 bucket
• Scheduled snapshot creation and restoration of snapshots stored locally and in s3 bucket
• Snapshot retention
• Snapshot subcommands including list, delete, save, prune

On-demand snapshot stored locally

rke2 etcd-snapshot 

Passing prefix to snapshot name

rke2 etcd-snapshot --name=snapshot1

On-demand snapshot stored in s3 bucket

rke2 etcd-snapshot \
    --s3 \
    --s3-bucket=<bucket-name> \
    --s3-access-key=<redacted> \
    --s3-secret-key=<redacted> \
    --s3-folder=<folder-name> 

Scheduled snapshot stored locally

rke2 server \
--etcd-snapshot-schedule-cron='* * * * *'

Scheduled snapshot stored locally with retention count

rke2 server \
--etcd-snapshot-schedule-cron='* * * * *' \
--etcd-snapshot-retention=2

Scheduled snapshot stored in s3 bucket with retention count

rke2 server \
    --etcd-s3 \
    --etcd-s3-bucket-name=<bucket-name> \
    --etcd-s3-access-key=<redacted> \
    --etcd-s3-secret-key=<redacted> \
    --etcd-snapshot-schedule-cron='* */1 * * *' \
    --etcd-snapshot-retention=2