Add support for FSB to WebDAV storage via kopia

[Talinx]

Describe the problem/challenge you have

We have the requirement to run everything and to store everything on european cloud providers. We use Hetzner to host our K8s cluster. Hetzner's CSI driver does not support volume snapshots so we want to use FSB to back up volumes.

Velero currently only supports AWS, Azure and GCP for FSB backups via kopia. These are not european cloud providers. For restic the situation is similar.

Describe the solution you'd like

Support for different cloud providers could be added via kopia. Kopia supports more protocols than velero exposes. Support for a different protocol from kopia like WebDAV or SFTP could be added to support arbitrary cloud providers.

Anything else you would like to add:

We need a backup solution and other solutions for K8s do not fit our requirements without modification so we already implemented this:

• Velero build with WebDAV support via kopia for FSB (source code; docker image)
• Object store plugin for WebDAV (source code; docker image)

Environment:

• Velero version (use velero version): 1.13
• Kubernetes version (use kubectl version): 1.29.2
• Kubernetes installer & version: ?
• Cloud provider or hardware configuration: Hetzner
• OS (e.g. from /etc/os-release): OpenSUSE MicroOS

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• :+1: for "The project would be better with this feature added"
• :-1: for "This feature will not enhance the project in a meaningful way"

[weshayutin]

@Talinx best if this is created in https://issues.redhat.com/issues/?filter=12413995 for project = "OADP"

[sseago]

"Velero currently only supports AWS, Azure and GCP for FSB backups via kopia. These are not european cloud providers. For restic the situation is similar." -- note that the AWS provider for object storage uses s3, so any s3-compliant provider should be supported, or are you asking for support for additional object storage providers that do not support S3?

[Talinx]

I didn't know that any S3-compliant provider works, that's good to know! Unfortunately the cloud provider we are using does not support S3.

[sseago]

Also, keep in mind that Kopia is only used for PVC data backup -- it uses the same object store plugin to store kopia data in the BackupStorageLocation that velero uses for storing the backup itself. Supporting additional object stores would need to be done by writing an ObjectStore plugin, since we'll still need to store backup metadata outside of the kopia/PVC context.

[Talinx]

Yes, wrote a plugin already: https://github.com/Talinx/velero-plugin-for-webdav/

[Lyndon-Li]

@Talinx Could you share which cloud provider are you using?

[Talinx]

Hetzner with Storage Box as WebDAV server.

The cluster is setup with this: https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner

(This setup uses SELinux and requires a SELinux policy for FSB backups to work. For this a file called fsb_files_policy.te with the following contents is created on every worker node:)

module fsb_files_policy 1.0;

require {
  type container_file_t;
  type container_t;
  class lnk_file read;
}

#============= container_t ==============
allow container_t container_file_t:lnk_file read;

And applied:

checkmodule -m -o fsb_files_policy.mod fsb_files_policy.te
semodule_package -o fsb_files_policy.pp -m fsb_files_policy.mod
semodule -i fsb_files_policy.pp

[reasonerjt]

@weshayutin I'm assigning this to you to triage if this requires changes in upstream.

[dv29]

@Talinx were you able to backup to hetzner storage box? I'm looking to do the same. if you have any documentation on how you did it, i'd love to take a look. I'm using rook/ceph and want to backup volume as well as RDB object store.

[Talinx]

@dv29 I added some documentation of my setup to the wiki: talinx/velero-plugin-for-webdav/wiki
I hope this helps. If you have any question feel free to contact me.

FYI a few days ago Hetzner opened a beta for S3 object storage. I haven't signed up for the beta but will most likely switch to it once it is out of beta. Object storage requires one less translation layer (my plugin acts kinda like a translation from object storage to WebDAV for velero) and works with mainline velero.