search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.86k stars 3.77k forks source link

disaster-recovery: Backup failure due to error from GCS as Received 429 `"Too many requests."` #127895

Open nameisbhaskar opened 1 month ago

nameisbhaskar commented 1 month ago

The backup failure is because GCS could not handle the number of request at that specific point in time. This is because if the rate-limiting on the GCS side. But, in this particular scenario, there can be an exponential retry as the rate may eventually come down.

Job reference - https://drt-chaos-0002.drt.crdb.io:26258/#/jobs/989603761182736390.

Error: 

unable to list files in gcs bucket: Get "https://storage.googleapis.com/storage/v1/b/cockroachdb-backup-testing/o?alt=json&delimiter=&endOffset=&includeTrailingDelimiter=false&pageToken=&prefix=roachprod-scheduled-backups%2Fdrt-chaos%2Fsystem%2F1714752607933605000%2Fmetadata%2Flatest&prettyPrint=false&projection=full&startOffset=&versions=false": compute: Received 429 `"Too many requests."`

Jira issue: CRDB-40657

blathers-crl[bot] commented 1 month ago

cc @cockroachdb/disaster-recovery