search

dchaley / deepcell-imaging

Tools & guidance to scale DeepCell imaging on Google Cloud Batch
7 stars 2 forks source link

Add retry / backoff logic for BigQuery upload #190

Closed dchaley closed 3 months ago

dchaley commented 3 months ago

We got this error when running 10 items in parallel:

google.api_core.exceptions.Forbidden: 403 Exceeded rate limits: too many table update operations for this table. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas; reason: rateLimitExceeded, location: table.write, message: Exceeded rate limits: too many table update operations for this table. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas

We need some kind of retry logic with backoff to get around the rate limit.