search

Altinity / clickhouse-backup

Tool for easy backup and restore for ClickHouse® using object storage for backup files.
https://altinity.com
Other
1.25k stars 225 forks source link

Way of compression changed from one archive to archive per table #314

Closed Philippians413 closed 2 years ago

Philippians413 commented 2 years ago

After upgrade from version 0.6.0 to 1.2.2 and switched from ftp to sftp upload we get an interesting situation. After ftp backup and transfer to ftp server we got backup in one archive For example

2021/11/30 00:00:27 Upload backup '2021-11-30T00-00-19'
 348.67 MiB / 348.67 MiB  100.00% 12s01s
2021/11/30 00:00:40   Done.

On backup server:

2021-11-30T00-00-19.tar.gz

But after switching to sftp and version 1.2.2 we got next situation:

...
2021/11/29 23:05:36  info done                      backup=2021-11-29T23-00-25 operation=create table=AA.TEST23
2021/11/29 23:05:36  info done                      backup=2021-11-29T23-00-25 duration=5m10.743s operation=create
2021/11/29 23:05:46  info done                      backup=2021-11-29T16-05-43 duration=10.134s location=local operation=delete
2021/11/29 23:05:47  info done                      backup=2021-11-29T23-00-25 duration=135ms operation=upload size=449.83KiB table=AA.TEST1
2021/11/29 23:05:47  info done                      backup=2021-11-29T23-00-25 duration=62ms operation=upload size=116.17KiB table=AA.TEST2
2021/11/29 23:07:51  info done                      backup=2021-11-29T23-00-25 duration=2m3.457s operation=upload size=455.60MiB table=AA.TEST3
2021/11/29 23:07:51  info done                      backup=2021-11-29T23-00-25 duration=41ms operation=upload size=10.96KiB table=AA.TEST4
2021/11/29 23:08:17  info done                      backup=2021-11-29T23-00-25 duration=26.235s operation=upload size=126.23MiB table=AA.TEST5
...
2021/11/29 23:10:38  info done                      backup=2021-11-29T23-00-25 duration=16ms operation=upload size=1.50KiB table=AA.TEST6
2021/11/29 23:10:38  info done                      backup=2021-11-29T23-00-25 duration=4m51.813s operation=upload size=1.20GiB

And on backup server:

# cd 2021-11-29T23-00-25/
# ls -R
.:
metadata  metadata.json  shadow

./metadata:
AA  BB  CC  

./metadata/AA:
TEST1.json  TEST2.json TEST3.json ...

./metadata/BB:
TEST1.json  TEST2.json TEST3.json ...

./metadata/CC:
TEST1.json  TEST2.json TEST3.json ...

./shadow:
AA  BB  CC

./shadow/AA:
TEST1  TEST2 TEST3 ...

./shadow/AA/TEST1:
default_1.tar

./shadow/AA/TEST2:
default_1.tar

./shadow/AA/TEST3:
default_1.tar

...

Could you help with it, please? We want to have 1 archive for all metadata and tables

Slach commented 2 years ago

@Philippians413 new 1.x format is one archive per table. 1.x also supports format one archive per backup for download and restore

this format allows us better upload \ download speed via multiple parallel table upload / download

On 2.x version we plan to change backup format one archive per part it allow us better handling incremental backups.

Philippians413 commented 2 years ago

Thanks @Slach

Philippians413 commented 2 years ago

1.x also supports format one archive per backup for download and restore

@Slach How can I enable one archive per backup for 1.x?

Slach commented 2 years ago

How can I enable one archive per backup for 1.x?

When you run 1.x version clickhouse-backup download for backup created from 0.6.x version clickhouse-backup upload it' detects automatic