search

Altinity / clickhouse-backup

Tool for easy backup and restore for ClickHouse® using object storage for backup files.
https://altinity.com
Other
1.25k stars 225 forks source link

Upload to GCS error #517

Closed mmolinac closed 1 year ago

mmolinac commented 2 years ago

Hi, It's only recently we're getting errors when uploading files to GCS, like this:

2022/09/17 01:11:21.636318 error one of upload go-routine return error: one of uploadTableData go-routine return error: can't check uploaded file: key not found

or this other one:

2022/09/20 04:10:51.581880  warn can't close writer: googleapi: Error 503: We encountered an internal error. Please try again., backendError
2022/09/20 04:10:51.644658 error can't acquire semaphore during Upload data parts: context canceled
2022/09/20 04:10:51.798495  warn can't close writer: googleapi: Error 503: We encountered an internal error. Please try again., backendError
2022/09/20 04:10:52.714160  warn can't close writer: googleapi: Error 503: We encountered an internal error. Please try again., backendError
2022/09/20 04:10:53.604868  warn can't close writer: googleapi: Error 503: We encountered an internal error. Please try again., backendError
2022/09/20 04:10:53.667549 error can't acquire semaphore during Upload data parts: context canceled
2022/09/20 04:10:54.037670  warn can't close writer: googleapi: Error 503: We encountered an internal error. Please try again., backendError
2022/09/20 04:10:54.114429 error can't acquire semaphore during Upload data parts: context canceled
2022/09/20 04:10:54.511411  warn can't close writer: googleapi: got HTTP response code 503 with body: Service Unavailable

I've seen that there's a fixed retry count for S3, as mentioned by @AlexAkulov in https://github.com/AlexAkulov/clickhouse-backup/issues/36#issuecomment-531590247 .

I can't see if there's something similar in https://github.com/AlexAkulov/clickhouse-backup/blob/master/pkg/storage/gcs.go .

Is the app trying several times already or does it stop after first error?

Regards, Manuel

Slach commented 2 years ago

it should stop after first error now we try to add storage independent retry mechanism in 2.1.0

Slach commented 1 year ago

implements in altinity/clickhouse-backup:master duplicated with https://github.com/AlexAkulov/clickhouse-backup/issues/397