Support a kopia-based backup

[onedr0p]

Describe the feature you'd like to have.

Hi again 👋🏼

Would it be possible to support Kopia? I know you already have support for restic, for data backups, but I figured Kopia would fit in nicely as it's becoming a very popular tool for incremental data backup. In my tests, it is also blazing fast.

https://github.com/kopia/kopia

It is written in Go and has decent APIs

https://pkg.go.dev/github.com/kopia/kopia/repo?utm_source=godoc

What is the value to the end user? (why is it a priority?)

Supporting another way of backing and restoring data.

How will we know we have a good solution? (acceptance criteria)

Additional context

[vh05]

Found below link with Kopia benchmark details while browsing through the history of Kopia. https://www.kasten.io/kubernetes/resources/blog/benchmarking-kopia-architecture-scale-and-performance

[onedr0p]

Yeah those benchmarks are pretty clear that Kopia is a serious contender to Restic :)

[trieded]

hi any plans for this?

[onedr0p]

There's needs to be some considerations made on how to implement this since kopia works a little different than existing movers. For example kopia can support a single "kopia repository" for multiple kubernetes volumes and "kopia snapshots" are the way you backup or restore data.

[onedr0p]

Local Filesystem Commands

This method is for using Kopia on your filesystem, not backing up to s3. Kopia does support s3 but I need to look into how that works. It's probably quite similar.

In the backup and restore examples below the following are defined as:

• /snapshots is the location of the Kopia Repository
• /data is the location of the PVC root filesystem

Note that in the examples below /snapshots is a single Kopia Repository with multiple snapshots across all PVCs that are backed up. This can help with deduplication if all the PVC volumes are in the same Kopia Repository. It could also be architected to support a 1:1 snapshot to Kopia Repository like the restic implementation is currently doing.

At a glance in the Kopia web UI, a single Kopia Repository with multiple snapshots (PVCs) looks like this:

Create backup

[[ ! -f /snapshots/kopia.repository.f ]] && kopia repository create filesystem --path=/snapshots
kopia repo connect filesystem --path=/snapshots --override-hostname=cluster --override-username=root
kopia policy set /data/<namespace/<app>/<claim> --compression=zstd --keep-latest 14 --keep-hourly 0 --keep-daily 7 --keep-weekly 2 --keep-monthly 0 --keep-annual 0
kopia snap create /data/<namespace/<app>/<claim>
kopia repo disconnect

Restore Backup

# Get latest snapshot ID
SNAPSHOT=$(kopia snapshot list /data/<namespace/<app>/<claim> --json | jq --raw-output '.[-1] | .id')

kopia repo connect filesystem --path=/snapshots --override-hostname=cluster --override-username=root
kopia snap restore "${SNAPSHOT}" /data/<namespace/<app>/<claim>
kopia repo disconnect

Maintenance

This can also be set to run after snapshot but it probably wouldn't be wise to have that happen in case it happens several or more times at once.

kopia repo connect filesystem --path=/snapshots --override-hostname=cluster --override-username=root
kopia maintenance set --owner=root@cluster
kopia maintenance run --full
kopia repo disconnect

I hope this can help lend some help on implementing this feature into VolSync.

[JohnStrunk]

hi any plans for this?

We don't have any plans at this time. The issue is simply a lack of dev resources for implementation and ongoing maintenance of a mover that is largely redundant w/ the restic one.

I realize that kopia seems to be gaining favor over restic. Perhaps in the future our priorities will change.

[onedr0p]

Are you all open to PRs to add Kopia support @JohnStrunk ?

[perfectra1n]

I'd be happy to help with the PR as well, if John is open to PRs that add Kopia's support.

[JohnStrunk]

@tesshuflower and I discussed this a bit, and we'd like to find a way that we can take more community contributions to the project. So, if you all are interested in contributing a kopia mover, we're happy to shepherd it through.

By the end of the process, we need to have:

• Mover code (controller + mover parts)
• Documentation
• Tests

Since this will be a fairly substantial piece, starting w/ a design PR is probably a good place to begin.

[DreamingRaven]

Hey guys, just dropping a ping here.

Any news regarding this? I would also be happy to lend a hand with this.

Kopia theoretically seems more appropriate for volumes with a plethora of small files, which I tend to find seems to be the predominant scenario in my volumes. https://cloudcasa.io/blog/comparing-restic-vs-kopia-for-kubernetes-data-movement/