AlexsLemonade / refinebio

Refine.bio harmonizes petabytes of publicly available biological data into ready-to-use datasets for cancer researchers and AI/ML scientists.
https://www.refine.bio/
Other
129 stars 19 forks source link

Slow down batch job deregistering process #3376

Closed arkid15r closed 1 year ago

arkid15r commented 1 year ago

It looks like the production deploy process failed because of AWS API rate limit.

Traceback (most recent call last):
  File "deregister_batch_job_definitions.py", line 28, in <module>
    response = batch.describe_job_definitions(**data)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/botocore/client.py", line 386, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/botocore/client.py", line 705, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (TooManyRequestsException) when calling the DescribeJobDefinitions operation (reached max retries: 4): Too Many Requests

In order to slow down our request rate we are going to add some sleeps to the terraform script to prevent being throttled.