Backup checksums

[ncdc]

We need to store checksums of the various files we produce and upload to backup storage. These can then be verified to ensure no tampering.

We may also want/need to add the option to sign the checksum files for additional verification.

[nrb]

@skriss This seems like a security measure to me, guaranteeing the consistency of the data at rest.

[VMmore]

Before closing, we should check what features that restic has around this. This is a common feature of backup software and given that restic is passing the file system we have an opportunity to record the checksums.

For snapshots, I don't think it makes as much sense, we should rely on the snapshot provider to ensure consistency.

[nrb]

I don't think this is for snapshots/restic contents specifically. It's for the JSON Kubernetes manifests, so that an attacker doesn't upload a modified JSON document that has something malicious in it (like an added container image as a sidecar to a valid application).

[skriss]

Removed "Candidate for close" label - we'll need to decide how to prioritize.

[jnaulty]

Describe the problem/challenge you have Integrity of backup object is not guaranteed before restore.

Why is this a Problem? Velero treats object storage as the source of truth. This could be abused by interfering with the object server. A compromise of this could affect the client and the server module if malicious input can trigger potential vulnerabilities in these

Describe the solution you'd like When velero performs a backup of an object, velero acquires a checksum of that object, which it then stores in a custom resource along with the UUID of the object.

When velero performs a restore of an object, it will check the checksum of the object before updating the kubernetes object.

Environment:

• Velero version (use velero version): v1.2.0

[jnaulty]

Info on Backup Objects Based on the documentation for output-file-format, there are two files that would be useful to checksum:

backup artifact

A backup is a gzip-compressed tar file whose name matches the Backup API resource's metadata.name

velero-backup.json

JSON file lists all information about your associated Backup resource, including any default values.

rootBucket/
    backup1234/
        velero-backup.json
        backup1234.tar.gz

A checksum should be done after the creation of each artifact.

Checksum Creation

backup artifact: execute a checksum in the Backup function for the gzipped tar archive https://github.com/vmware-tanzu/velero/blob/dcca3c3d2bd5379a26d72a4737aef124b07eabdc/pkg/backup/backup.go#L201-L280

velero-backup.json execute a checksum for the json file when it is written to disk in persistBackup function: https://github.com/vmware-tanzu/velero/blob/dcca3c3d2bd5379a26d72a4737aef124b07eabdc/pkg/controller/backup_controller.go#L563-L569

The following parts need planning:

• storage of checksum in velero server state (or as a kubernetes resource)
• validation of checksum on restore

[invidian]

storage of checksum in velero server state (or as a kubernetes resource)

I guess the checksum could be stored in Backup CR. And in case of restoring to a different cluster, Velero should not record a checksum from the backup storage. Then user could either provide a valid checksum from the other cluster or accept insecure restore.

Also, alternatively to checksums one could use some sort of signatures, so Velero would sign the backup while it's being created and verify the signature before restoring. This would allow easy way to verify backups if Velero servers share the trusted public keys.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[invidian]

Still important.

[reasonerjt]

I'm not sure if checksum is sufficient to protect the content from tampering as the malicious user may tamper the checksum also, this is not a signature.