Closed bmailhe closed 1 year ago
Trying to download give me also an error:
# clickhouse-backup -c /etc/clickhouse-backup/config.yml download rZhV-FULL-full_orderbooks_v1-snapshot_v1-221206130904
2022/12/13 13:21:00.426602 info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2022/12/13 13:21:00.428874 info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2022/12/13 13:21:00.429022 info SELECT value FROM `system`.`build_options` where name='VERSION_INTEGER' logger=clickhouse
2022/12/13 13:21:00.430854 info SELECT * FROM system.disks; logger=clickhouse
2022/12/13 13:21:00.433589 info SELECT max(toInt64(bytes_on_disk * 1.02)) AS max_file_size FROM system.parts logger=clickhouse
2022/12/13 13:21:00.438443 info SELECT count() AS is_macros_exists FROM system.tables WHERE database='system' AND name='macros' logger=clickhouse
2022/12/13 13:21:00.442009 info SELECT * FROM system.macros logger=clickhouse
2022/12/13 13:21:02.567258 info done backup=rZhV-FULL-full_orderbooks_v1-snapshot_v1-221206130904 duration=997ms logger=backuper operation=download size=11.33KiB table_metadata=full_orderbooks_v1.snapshot_v1
2022/12/13 13:21:02.567444 info clickhouse connection closed logger=clickhouse
2022/12/13 13:21:02.567472 error one of Download go-routine return error: /var/lib/clickhouse/backup/rZhV-FULL-full_orderbooks_v1-snapshot_v1-221206130904/shadow/full_orderbooks_v1/snapshot_v1/default/201904_0_137_12 not found after download backup
If I download the backup using rclone sync
and then restore it, it works.
could you share
clickhouse-backup version
clickhouse-backup print-config
without sensitive credentials?
could you also share
clickhouse-backup list remote
?
Version
clickhouse-backup --version
Version: 2.1.2
Git Commit: 376c8dd499c3f8c4a0f873670345c8b53ef5c44a
Build Date: 2022-10-23
List remote
$ clickhouse-backup list remote
G8tJ-FULL-full_orderbooks_v1-snapshot_v2-221205160449 3.12TiB 06/12/2022 09:47:14 remote , regular
rZhV-FULL-full_orderbooks_v1-snapshot_v1-221206130904 7.72TiB 06/12/2022 15:56:23 remote , regular
rZhV-PARTIAL-001-full_orderbooks_v1-snapshot_v1-221212090143 111.82GiB 12/12/2022 09:20:37 remote +rZhV-FULL-full_orderbooks_v1-snapshot_v1-221206130904 tar, regular
G8tJ-PARTIAL-001-full_orderbooks_v1-snapshot_v2-221212130038 3.12TiB 12/12/2022 13:01:52 remote +G8tJ-FULL-full_orderbooks_v1-snapshot_v2-221205160449 tar, regular
FYI the "FULL" backups are uploaded using rclone
, I think the problem comes from the fact some directories are not compressed with tar ?
FYI the "FULL" backups are uploaded using rclone, I think the problem comes from the fact some directories are not compressed with tar ?
yes, this is root reason
for compression_format: tar
clickhouse-backup expect backup_name/db/table/disk_name/part_name.tar files
also during download expect "Files" section in backup_name/metadata/db/table_name.json which save only when you use clickhouse-backup
if you want improve speed for upload \ download to s3 and memory usage
you can integrate rclone with remote_storage: custom
inside clickhouse-backup, but not sure how to create incremental backup for rclone
Moreover, if you'd like, you could integrate restic
instead of rclone
with remote_storage: custom
look for detail https://github.com/Altinity/clickhouse-backup/tree/master/test/integration/restic and https://github.com/Altinity/clickhouse-backup/blob/master/test/integration/config-custom-restic.yml
Thanks for the explanation.
I did more tests with upload_command
and download_command
set with rclone
and it works !
custom:
upload_command: "rclone sync --fast-list --b2-disable-checksum "
download_command: "rclone sync --fast-list --b2-disable-checksum "
then
# create and upload a backup
clickhouse-backup create full-backup
clickhouse-backup upload full-backup
# add data
# create a partial backup
clickhouse-backup create incr-backup
clickhouse-backup upload --diff-from-remote=full-backup incr-backup
# restore from remote
clickhouse-backup delete local full-backup
clickhouse-backup delete local incr-backup
clickhouse-backup restore_remote incr-backup --rm
check remote backup sizes via your S3 backed UI
not sure for --diff-from-remote
will work properly for rclone wihtout changed "custom" section
moreover, if you will store local backups it will allocate disk space when you run mutation ALTER TABLE ... DELETE / UPDATE
or after any background parts merge
fill free to make Pull Request with examples for rclone integration
Steps to reproduce: On a cluster
G8tJ-FULL-full_orderbooks_v1-snapshot_v2-221205160449
G8tJ-PARTIAL-001-full_orderbooks_v1-snapshot_v2-221212130038
based on previous one (upload --diff-from-remote=G8tJ-FULL-full_orderbooks_v1-snapshot_v2-221205160449 G8tJ-PARTIAL-001-full_orderbooks_v1-snapshot_v2-221212130038
)On another cluster
Trying with an another diff backup, it fails with the parent backup:
Am I doing something wrong ?