Altinity / clickhouse-backup

Tool for easy backup and restore for ClickHouse® using object storage for backup files.
https://altinity.com
Other
1.29k stars 227 forks source link

Option to create a local incremental backup based on a full local backup #1054

Closed lesandie closed 16 hours ago

lesandie commented 16 hours ago

We can create an incremental that will diff from a remote full backup like this:

clickhouse-backup create --tables=keylogger.\* --diff-from-remote keylogger_full keylogger_inc1

https://github.com/Altinity/clickhouse-backup#cli-command---create

The keylogger_full was already uploaded to s3.

Also what you can do is create a remote/upload incremental backup based on a local one:

clickhouse-backup create_remote/upload --tables=keylogger.\* --diff-from keylogger_full keylogger_inc1

https://github.com/Altinity/clickhouse-backup#cli-command---create_remote

What we don't have is to create an incremental local backup from a full local backup like this:

clickhouse-backup create --tables=keylogger.\* --diff-from keylogger_full keylogger_inc1

@Slach do you think this is a good idea?. There are some use cases that could benefit from this.

Slach commented 16 hours ago

just for search engines

--diff-from and --diff-from-remote actually parameters for upload command, not for create cause increment calculates only during upload

we added --diff-from-remote parameter into create command for use_embedded_backup: true and for object disk data parts upload

create --diff-from have no sense because local backup is just a set of hard links and don't allocate disk space until the original data files deleted