Strategy for Backup Handling

[danielfoehrKn]

How to categorize this issue?

/area backup /kind enhancement /priority 2

What would you like to be added:

A clear picture of how state generated in components (generated secrets etc.) is backed up. I gave it a high prior, as it has implications on the design of components currently under development (see here)

Why is this needed:

Components like the virtual-garden are designed to

• Generate secrets (CA, TLS serving key pair) that are not explicitly specified in the import configuration
• Generate secrets during first execution and store them as a secret in the Kubernetes cluster.

Generally currently reuses secrets stored in Secret resources in the Cluster. As such the Cluster acts as the "state store".

However, this is insufficient because

• requires the etcd of the Cluster to be backed up
  • Sounds good, but who does that? How do you do that in a managed K8s environment (diff. for each provider)? How on prem?
  • This just shifts the "burden"of taking care of backups to an external component. For the virtual-garden component that's the host cluster that we do not have control over.
• what if someone manually deletes a secret in the Cluster such as the RSA private key specified via --service-account-key-file on the KAPI? The private key is regenerated and all currently service account tokens are invalid.

As a consequence, we should define a clear picture of how state generated in components (generated secrets etc.) is backed up.

Proposal

• Components export all relevant internally generated state

  • relevant is information that cannot be regenerated without breaking / harming the cluster. E.g the RSA private key specified via --service-account-key-file on the KAPI
  • Irrelevant information is for instance the basic-auth credentials on the KAPI that is only used for the liveness probes.

• Components optionally import all the previously exported state

• There is a dedicated landscaper backup component that has the following tasks

  • acts as the last component in the deployment chain
  • imports information from all components and writes them to a configurable backing store (we can start with S3 or Git)
  • acts as the first component in the deployment chain
  • Reads information from the backing store and exports the information for each other component to consume

[danielfoehrKn]

Probably that should not be the concern of the landscaper as a generic installation tooling, but of a Controller/Operator using the landscaper itself. I want to prevent to stuff the landscaper with too many generic features which could be easily solved by a specific operator. I will elaborate on this once the idea is more refined.

Edit: not discussed. Open for solutions. Probably best in the landscaper as generic functionality.

[danielfoehrKn]

We could just use Velero to backup secrets in the landscaper cluster to e.g S3. This way we do not have to build our own, landscaper-specific, backup solution.

[achimweigel]

@In-Ko closed because outdated and/or copied into internal project