search

Altinity / clickhouse-backup

Tool for easy backup and restore for ClickHouse® using object storage for backup files.
https://altinity.com
Other
1.29k stars 227 forks source link

Multipart Upload to Minio Fails in Newest Minio Release #925

Closed tman5 closed 6 months ago

tman5 commented 6 months ago

We recently upgraded our minio to version 2024-05-01 and on our prod clusters we're seeing backup upload failures. This does not happen on our dev/test clusters, but their data upload is much smaller, so only larger uploads. Minio is behind an haproxy load balancer

clickhouse-backup version 2.5.8

Logs are attached from the backup process clickhouse_backup_logs_5_17_24.txt

Slach commented 6 months ago

issue is not related to clickhouse-backup

2024/05/16 04:10:39.582969 error UploadCompressedStream 
return error: upload multipart failed, 
upload id: NjYxOGNjY2YtOGY3MC00NzMzLTg3MTYtNDIzY2Q3NzYyYmExLjA0MmMxMzlkLTRlM2QtNGE4MC1hNTZlLTEyM2EzMmVjMjY5MQ, cause: operation error S3: CompleteMultipartUpload, 
exceeded maximum number of attempts, 3, 
https response error StatusCode: 504, RequestID: , 
HostID: , api error GatewayTimeout: Gateway Timeout logger=uploadTableData

GatewayTimeout: Gateway Timeout means your haproxy can't pass http request to minio

and we tried it 3 times with 30 seconds pause between

check your minio and haproxy logs