Open tman5 opened 6 months ago
Could you share results of the following command?
kubectl exec -n <your-namespace> chi-clickhouse-replicated-0-0 -c clickhouse-pod -- clickhouse-client -q "SELECT * FROM system.backup_actions"
list 2024-03-09 03:00:03 2024-03-09 03:00:04 success
list 2024-03-09 03:00:04 2024-03-09 03:00:04 success
list 2024-03-09 03:00:04 2024-03-09 03:00:04 success
create chi-clickhouse-replicated-0-0-increment-2024-03-09-03-00-01 2024-03-09 03:00:04 2024-03-09 03:00:05 success
upload --diff-from-remote=chi-clickhouse-replicated-0-0-increment-2024-03-08-03-00-01 chi-clickhouse-replicated-0-0-increment-2024-03-09-03-00-01 2024-03-09 03:00:06 2024-03-09 03:00:07 success
delete local chi-clickhouse-replicated-0-0-increment-2024-03-09-03-00-01 2024-03-09 03:00:11 2024-03-09 03:00:11 success
list 2024-03-10 03:00:02 2024-03-10 03:00:03 success
list 2024-03-10 03:00:03 2024-03-10 03:00:03 success
list 2024-03-10 03:00:03 2024-03-10 03:00:04 success
create chi-clickhouse-replicated-0-0-increment-2024-03-10-03-00-01 2024-03-10 03:00:04 2024-03-10 03:00:04 success
upload --diff-from-remote=chi-clickhouse-replicated-0-0-increment-2024-03-09-03-00-01 chi-clickhouse-replicated-0-0-increment-2024-03-10-03-00-01 2024-03-10 03:00:05 2024-03-10 03:00:05 success
delete local chi-clickhouse-replicated-0-0-increment-2024-03-10-03-00-01 2024-03-10 03:00:10 2024-03-10 03:00:11 success
list 2024-03-11 03:00:02 2024-03-11 03:00:03 success
list 2024-03-11 03:00:03 2024-03-11 03:00:03 success
list 2024-03-11 03:00:03 2024-03-11 03:00:03 success
create chi-clickhouse-replicated-0-0-full-2024-03-11-03-00-01 2024-03-11 03:00:03 2024-03-11 03:00:04 success
upload chi-clickhouse-replicated-0-0-full-2024-03-11-03-00-01 2024-03-11 03:00:05 2024-03-11 03:00:05 success
delete local chi-clickhouse-replicated-0-0-full-2024-03-11-03-00-01 2024-03-11 03:00:10 2024-03-11 03:00:10 success
list 2024-03-12 03:00:05 2024-03-12 03:00:05 success
list 2024-03-12 03:00:05 2024-03-12 03:00:05 success
list 2024-03-12 03:00:05 2024-03-12 03:00:05 success
create chi-clickhouse-replicated-0-0-increment-2024-03-12-03-00-02 2024-03-12 03:00:05 2024-03-12 03:00:06 success
upload --diff-from-remote=chi-clickhouse-replicated-0-0-full-2024-03-11-03-00-01 chi-clickhouse-replicated-0-0-increment-2024-03-12-03-00-02 2024-03-12 03:00:07 2024-03-12 03:00:08 success
delete local chi-clickhouse-replicated-0-0-increment-2024-03-12-03-00-02 2024-03-12 03:00:12 2024-03-12 03:00:12 success
list 2024-03-13 03:00:03 2024-03-13 03:00:04 success
list 2024-03-13 03:00:04 2024-03-13 03:00:04 success
list 2024-03-13 03:00:04 2024-03-13 03:00:04 success
create chi-clickhouse-replicated-0-0-increment-2024-03-13-03-00-01 2024-03-13 03:00:04 2024-03-13 03:00:05 success
upload --diff-from-remote=chi-clickhouse-replicated-0-0-increment-2024-03-12-03-00-02 chi-clickhouse-replicated-0-0-increment-2024-03-13-03-00-01 2024-03-13 03:00:06 2024-03-13 03:00:06 success
delete local chi-clickhouse-replicated-0-0-increment-2024-03-13-03-00-01 2024-03-13 03:00:11 2024-03-13 03:00:11 success
list 2024-03-14 03:00:06 2024-03-14 03:00:06 success
list 2024-03-14 03:00:06 2024-03-14 03:00:06 success
list 2024-03-14 03:00:06 2024-03-14 03:00:07 success
create chi-clickhouse-replicated-0-0-increment-2024-03-14-03-00-04 2024-03-14 03:00:07 2024-03-14 03:00:07 success
upload --diff-from-remote=chi-clickhouse-replicated-0-0-increment-2024-03-13-03-00-01 chi-clickhouse-replicated-0-0-increment-2024-03-14-03-00-04 2024-03-14 03:00:08 2024-03-14 03:00:09 success
delete local chi-clickhouse-replicated-0-0-increment-2024-03-14-03-00-04 2024-03-14 03:00:14 2024-03-14 03:00:14 success
list 2024-03-15 03:00:02 2024-03-15 03:00:03 success
list 2024-03-15 03:00:03 2024-03-15 03:00:04 success
list 2024-03-15 03:00:04 2024-03-15 03:00:04 success
create chi-clickhouse-replicated-0-0-full-2024-03-15-03-00-01 2024-03-15 03:00:04 2024-03-15 03:00:04 success
upload chi-clickhouse-replicated-0-0-full-2024-03-15-03-00-01 2024-03-15 03:00:05 2024-03-15 03:00:06 success
delete local chi-clickhouse-replicated-0-0-full-2024-03-15-03-00-01 2024-03-15 03:00:11 2024-03-15 03:00:11 success
list 2024-03-16 03:00:02 2024-03-16 03:00:02 success
list 2024-03-16 03:00:02 2024-03-16 03:00:03 success
list 2024-03-16 03:00:03 2024-03-16 03:00:03 success
create chi-clickhouse-replicated-0-0-increment-2024-03-16-03-00-01 2024-03-16 03:00:03 2024-03-16 03:00:03 success
upload --diff-from-remote=chi-clickhouse-replicated-0-0-full-2024-03-15-03-00-01 chi-clickhouse-replicated-0-0-increment-2024-03-16-03-00-01 2024-03-16 03:00:04 2024-03-16 03:00:05 success
delete local chi-clickhouse-replicated-0-0-increment-2024-03-16-03-00-01 2024-03-16 03:00:10 2024-03-16 03:00:10 success
list 2024-03-17 03:00:02 2024-03-17 03:00:02 success
list 2024-03-17 03:00:03 2024-03-17 03:00:03 success
list 2024-03-17 03:00:03 2024-03-17 03:00:03 success
create chi-clickhouse-replicated-0-0-increment-2024-03-17-03-00-01 2024-03-17 03:00:03 2024-03-17 03:00:03 success
upload --diff-from-remote=chi-clickhouse-replicated-0-0-increment-2024-03-16-03-00-01 chi-clickhouse-replicated-0-0-increment-2024-03-17-03-00-01 2024-03-17 03:00:05 2024-03-17 03:00:05 success
delete local chi-clickhouse-replicated-0-0-increment-2024-03-17-03-00-01 2024-03-17 03:00:10 2024-03-17 03:00:10 success
list 2024-03-18 03:00:03 2024-03-18 03:00:04 success
list 2024-03-18 03:00:04 2024-03-18 03:00:05 success
list 2024-03-18 03:00:05 2024-03-18 03:00:05 success
create chi-clickhouse-replicated-0-0-full-2024-03-18-03-00-02 2024-03-18 03:00:05 2024-03-18 03:00:05 success
upload chi-clickhouse-replicated-0-0-full-2024-03-18-03-00-02 2024-03-18 03:00:06 2024-03-18 03:00:07 success
delete local chi-clickhouse-replicated-0-0-full-2024-03-18-03-00-02 2024-03-18 03:00:12 2024-03-18 03:00:12 success
Is SELECT * FROM system.backup_list
contains chi-clickhouse-replicated-0-0-full-2024-03-18-03-00-02
currently?
According to logs you successfully upload incremental backup
2024/03/14 03:00:08.716007 info /backup/actions call: upload --diff-from-remote=chi-clickhouse-replicated-0-0-increment-2024-03-13-03-00-01 chi-clickhouse-replicated-0-0-increment-2024-03-14-03-00-04 logger=server
and delete old full backup
2024/03/14 03:00:09.173829 info done backup=chi-clickhouse-replicated-0-0-full-2024-03-11-03-00-01 duration=94ms location=remote logger=s3 operation=RemoveOldBackupsRemote
weird why your incemental backup have 0B size
do you have any data in your tables?
@tman5 any news from your side?
It looks like the data wasn't in there. But now that the data has been put back it's showing a 29.79kb backup so it looks like it's working now. Thanks for your help
Running clickhouse-backup is only producing incremental backups of 0B in size
Configs
Logs: