search

Altinity / clickhouse-backup

Tool for easy backup and restore for ClickHouse® using object storage for backup files.
https://altinity.com
Other
1.25k stars 225 forks source link

Download from Azure Blob storage times out in 60 seconds #467

Closed mxalis closed 2 years ago

mxalis commented 2 years ago

When downloading backups from Azure Blob storage, the default go library uses a 60 seconds timeout. This will cause downloads to fail (and therefore backup restore) if either your connection is slow or the file is large.

Can be fixed by adding the TryTimeout option to https://github.com/AlexAkulov/clickhouse-backup/blob/820961e12158a39772b370dd8460f408c4833f70/pkg/new_storage/azblob.go#L99

eg. increasing it to 15 minutes

s.Container = azblob.NewServiceURL(*u, azblob.NewPipeline(credential, azblob.PipelineOptions{
                                Retry: azblob.RetryOptions{
                                TryTimeout: 15 * time.Minute,
                                },
                          })).NewContainerURL(s.Config.Container)

It would be great to have the timeout configurable with a reasonable default (eg 5 mins). Unfortunately I am a golang noob and cannot help you with a pull request

Discussion relating to the timeout: https://github.com/Azure/azure-storage-blob-go/issues/60

k0t3n commented 2 years ago

Faced same problem. @Slach is it possible to apply fix for v1 version?

Slach commented 2 years ago

@k0t3n check https://github.com/Altinity/clickhouse-backup/releases/tag/v1.5.0

k0t3n commented 2 years ago

@Slach works like a charm, thanks!