search

tizbac / pmoxs3backuproxy

Proxy written in golang that will emulate a PBS server and work on one or more S3 buckets
GNU General Public License v3.0
64 stars 4 forks source link

add support for dynamic indexes #32

Closed abbbi closed 3 months ago

abbbi commented 3 months ago

hi,

this is just POC implementation of dynamic index. It works for me with a simle proxmox-backup client backup and dump:

proxmox-backup-client backup root.pxar:/tmp/source --repository SBQMMH19KWC0L521X6DY@pbs@127.0.0.1:backups
Starting backup: host/cefix/2024-08-17T20:23:26Z
Client name: cefix
Starting backup protocol: Sat Aug 17 22:23:26 2024
No previous manifest available.
Upload directory '/tmp/source' to 'SBQMMH19KWC0L521X6DY@pbs@127.0.0.1:8007:backups' as root.pxar.didx
failed to open file: ".pxarexclude": access denied
failed to open file: ".pxarexclude": access denied
root.pxar: had to backup 13.931 MiB of 13.931 MiB (compressed 2.644 MiB) in 0.09s
root.pxar: average backup speed: 161.344 MiB/s
Uploaded backup catalog (88.975 KiB)
Duration: 0.10s
End Time: Sat Aug 17 22:23:26 2024

dumping works

proxmox-backup-client catalog dump "host/cefix/2024-08-17T20:23:26Z" --repository SBQMMH19KWC0L521X6DY@pbs@127.0.0.1:backups |& head -3
d "./root.pxar.didx"
f "./root.pxar.didx/data" 7 2024-08-17T20:06:33Z
d "./root.pxar.didx/etc"

but restore currently fails with:

Error: wrong checksum for file 'root.pxar.didx'

i compared the checksum that is passed with csum parameter and an own sha256 checksum of the index im creating and they match.. i currently fail to see whats wrong.

abbbi commented 3 months ago

that needs some more work, the offsets in the root didx are wrong. original code does some chunk lookup to calculate the offsets right, but i currently fail to see how this all works, ..