tizbac / pmoxs3backuproxy

Proxy written in golang that will emulate a PBS server and work on one or more S3 buckets
GNU General Public License v3.0
66 stars 4 forks source link

add remote sync support #48

Closed abbbi closed 3 months ago

abbbi commented 3 months ago

First draft to implement some endpoints required for syncing an remote S3 to a local PBS instance. This is how it goes:

1) configure S3 storage in PVE (name "s3store" and schedule some backups) 2) configure the same S3 storage in an PBS instance (fresh install, or existing install) as "remote" with s3store as name

3) Create a new datastore:

proxmox-backup-manager datastore create test2 /tmp/syncme

4) Start an sync from the remote to the local created datastore:

proxmox-backup-manager pull s3store backups test2

pull datastore 'test2' from 's3store/backups'
----
Syncing datastore 'backups', namespace 'Root' into datastore 'test2', namespace 'Root'
Created namespace Root
found 2 groups to sync (out of 2 total)
sync snapshot ct/104/2024-08-20T08:27:02Z
sync archive pct.conf.blob
sync archive root.pxar.didx
[..]
pull datastore 'test2' end
TASK OK

Anyone who feels like testing this is welcome!

abbbi commented 3 months ago

OK, it was a little bit more complicated. The sync process also requests single snapshots with parameters,not just a list of snapshots via the snapshots? api backend.

Now the sync works correctly for me, if there are no objections im going to merge :-)

Its pretty neat to counter-test if all backups are OK by using the proxmox verify function after the sync:

 proxmox-backup-manager verify test2  --ignore-verified false | tail -n 1
TASK OK

It would probably make for a good test with the current implementation of the garbage collector, too, like

1) expire backups 2) run experimental GC 3) sync to PBS 4) run verify

:)

Testing welcome!