RESTORE: Support backups created by clickhouse-backup

[vitlibar]

ClickHouse supports built-in commands BACKUP and RESTORE to make backups and to restore from them. There is also an external utility clickhouse-backup which can do mostly the same. However the formats of such backups are different.

For example, for a single table mydb.mytable containing two parts:

• BACKUP TABLE mydb.mytable TO Disk('backups', 'backup1') generates the following folder as a backup:

  ├── .backup
  ├── data
  │   └── mydb
  │       └── mytable
  │           ├── all_1_1_0
  │           │   ├── checksums.txt
  │           │   ├── columns.txt
  │           │   ├── count.txt
  │           │   ├── data.bin
  │           │   ├── data.cmrk3
  │           │   ├── default_compression_codec.txt
  │           │   ├── metadata_version.txt
  │           │   └── serialization.json
  │           └── all_2_2_0
  │               ├── checksums.txt
  │               ├── columns.txt
  │               ├── count.txt
  │               ├── data.bin
  │               ├── data.cmrk3
  │               ├── default_compression_codec.txt
  │               ├── metadata_version.txt
  │               └── serialization.json
  └── metadata
  ├── mydb
  │   └── mytable.sql
  └── mydb.sql

• clickhouse-backup create backup1 generates the following folder as a backup:

  ├── metadata
  │   └── mydb
  │       └── mytable.json
  ├── metadata.json
  └── shadow
  └── mydb
      └── mytable
          └── default
              ├── all_1_1_0
              │   ├── checksums.txt
              │   ├── columns.txt
              │   ├── count.txt
              │   ├── data.bin
              │   ├── data.cmrk3
              │   ├── default_compression_codec.txt
              │   └── serialization.json
              └── all_2_2_0
                  ├── checksums.txt
                  ├── columns.txt
                  ├── count.txt
                  ├── data.bin
                  ├── data.cmrk3
                  ├── default_compression_codec.txt
                  └── serialization.json

It seems useful to add some support of clickhouse-backup backups to the RESTORE command, so RESTORE could restore from a backup generated by the clickhouse-backup utility.

[UnamedRus]

I see "problem", that backups still needs to be downloaded by clickhouse-backup and unpacked from remote storage. (format on remote storage is different)

So, if you already used this tool to download them, you can use the same tool for restoring of them.

And implementation of "download+restore" will be more complicated and it doesn't answer question what if backup stored on storage which is not supported by embedded backup ((S)FTP for example).

May be, if embedded backup got some features/feature parity with clickhouse-backup, it would be better goal.

[Slach]

better will implement BACKUP LIST and BACKUP DELETE SQL commands in this case clickhouse-backup could be used as just wrapper around BACKUP/RESTORE commands

we create umbrella backup issue and issues about BACKUP DELETE and BACKUP LIST was closed as "not planned"