search

Altinity / clickhouse-backup

Tool for easy backup and restore for ClickHouse® using object storage for backup files.
https://altinity.com
Other
1.29k stars 227 forks source link

clickhouse-backup restore fails #293

Closed smpa0321 closed 2 years ago

smpa0321 commented 3 years ago

restore works partially and it fails to attach few partitions as in below log

2021/10/27 13:52:28 error can't attach partitions for table '<>': code: 233, message: Detached part "20211023_1_1_0" not found

restore same id second time we get below error error can't restore 'prom.alarms': error during filepath.Walk for partition '20211023_1_1_0': failed to create hard link '/clickhouse1/backup/2021-10-27T10-17-09/shadow/<>/<>/default/20211023_1_1_0/<>.bin' -> '/clickhouse1/data/<>/<>/detached/20211023_1_1_0/<>.bin': link /clickhouse1/backup/2021-10-27T10-17-09/shadow/prom/alarms/default/20211023_1_1_0/<>.bin /clickhouse1/data/<>/<>/detached/20211023_1_1_0/<>.bin: file exists

We have 2 clickhouse server(replica ch1 and ch2) PV volumes are shared and we are able to backup data . in order to test restore we backed up data and then deleted data from table and tried restore.

Please point in case we are missing something

Slach commented 3 years ago

PV volumes are shared

Do you mean PV shared between ch1 and ch2 and you mount the same /var/lib/clickhouse on two servers?

smpa0321 commented 3 years ago

ch1 has pv1 and ch2 has pv2 clickhouse-backup is installed on same compute node as ch2 and shares the pv2 .(the folder is /clickhouse1/ this has data/shadow and rest of directories) The backup is created at /clickhouse1/backup/ While trying to restore we see below error

2021/11/10 07:53:44 debug SELECT FROM system.tables WHERE is_temporary = 0 SETTINGS show_table_uuid_in_table_create_query_if_not_nil=1 2021/11/10 07:53:44 debug SELECT FROM system.disks; 2021/11/10 07:53:44 debug dstDataPaths=map[default:/clickhouse1/data/prom/alarms/] disks=[{default /clickhouse1/ local}] tableDataPaths=[/clickhouse1/data/prom/alarms/] 2021/11/10 07:53:44 debug detachedParentDir=/clickhouse1/data/prom/alarms/detached 2021/11/10 07:53:44 debug detachedPath=/clickhouse1/data/prom/alarms/detached/20211015_0_0_0 2021/11/10 07:53:44 debug SELECT * FROM system.disks; 2021/11/10 07:53:44 debug detachedPath=/clickhouse1/data/prom/alarms/detached/20211019_0_0_0 2021/11/10 07:53:44 debug detachedPath=/clickhouse1/data/prom/alarms/detached/20211021_0_0_0 2021/11/10 07:53:44 debug detachedPath=/clickhouse1/data/prom/alarms/detached/20211022_0_0_0 2021/11/10 07:53:44 debug detachedPath=/clickhouse1/data/prom/alarms/detached/20211024_16_16_0 2021/11/10 07:53:44 debug detachedPath=/clickhouse1/data/prom/alarms/detached/20211026_0_0_0 2021/11/10 07:53:44 debug detachedPath=/clickhouse1/data/prom/alarms/detached/20211027_16_16_0 2021/11/10 07:53:44 debug detachedPath=/clickhouse1/data/prom/alarms/detached/20211027_17_17_0 2021/11/10 07:53:44 debug copied data to 'detached' backup=2021-11-10T07-41-13 operation=restore table=prom.alarms 2021/11/10 07:53:44 debug ALTER TABLE prom.alarms ATTACH PART '20211015_0_0_0' 2021/11/10 07:53:44 error can't attach partitions for table 'prom.alarms': code: 233, message: Detached part "20211015_0_0_0" not found

2021/11/10 07:53:44 debug ALTER TABLE prom.alarms ATTACH PART '20211015_0_0_0' 2021/11/10 07:53:44 error can't attach partitions for table 'prom.alarms': code: 233, message: Detached part "20211015_0_0_0" not found

Slach commented 3 years ago

you should have the same disk schema

could you check SELECT * FROM system.disks on source host ch1 where backup was created? and compare path field with SELECT * FROM system.disks on desitnation host where you try to restart?

Slach commented 2 years ago

I close issue cause inactive feel free add comment or create separate issue if needed