Altinity / clickhouse-backup

Tool for easy backup and restore for ClickHouse® using object storage for backup files.
https://altinity.com
Other
1.26k stars 225 forks source link

Backup not working using azblob with use_embedded_backup_restore flag as true #1031

Closed hermeswaldemarin closed 2 hours ago

hermeswaldemarin commented 4 hours ago

The backup for azure is not working like expected when using the use_embedded_backup_restore: true.

This is the env variable I'm using.

AZBLOB_ACCOUNT_NAME=myaccount AZBLOB_ENDPOINT_SUFFIX=core.windows.net AZBLOB_ACCOUNT_KEY=my_access_key AZBLOB_CONTAINER=my-backup-bucket

In this method, used by clickhouse-backup to access the container, the BlobEndpoint is constructed correctly azblob.go#L94

But when it sends the backup command to clickhouse the BlobEndpoint constructed is different and we receive an error by clickhouse in the command. backuper.go#L352

This is how the command is created to send to clickhouse.

BACKUP TABLEmyschema.json_layoutsTO AzureBlobStorage('DefaultEndpointsProtocol=https;AccountName=myaccount;AccountKey=my_access_key;BlobEndpoint=https://core.windows.net/myaccount;','my-backup-bucket','/chi-clickhouse-clickhouse-0-1-full-2024-10-22-15-48-46/') SETTINGS allow_azure_native_copy=1

image

the Blob endpoint is wrong, should be BlobEndpoint=https://myaccount.blob.core.windows.net; instead.

BACKUP TABLE myschema.json_layouts TO AzureBlobStorage('DefaultEndpointsProtocol=https;AccountName=myaccount;AccountKey=my_access_key;BlobEndpoint=https://myaccount.blob.core.windows.net;', 'my-backup-bucket', '/chi-clickhouse-clickhouse-0-1-full-2024-10-22-15-48-46/') SETTINGS allow_azure_native_copy = 1

I also tried omitting the BlobEndpoint in the config and it also works. This command bellow also works...

BACKUP TABLE myschema.json_layouts TO AzureBlobStorage('DefaultEndpointsProtocol=https;AccountName=myaccount;AccountKey=my_access_key', 'my-backup-bucket', '/chi-clickhouse-clickhouse-0-1-full-2024-10-22-15-48-46/') SETTINGS allow_azure_native_copy = 1

hermeswaldemarin commented 3 hours ago

Clickhouse Version: 24.8.4.13 clickhouse-backup Version: 2.6.0