test_concurrent_portal_db_crud occasionally fails IT with SlowDown error

Issue originally addressed by https://github.com/DataBiosphere/azul/issues/2399 (PR https://github.com/DataBiosphere/azul/pull/2602) occasionally still occurs when running IT locally.

test_concurrent_portal_db_crud (integration_test.PortalRegistrationIntegrationTest)
Use multithreading to simulate multiple users simultaneously modifying ... 2022-06-30 18:23:33,597 INFO    ThreadPoolExecutor-7_3: Found credentials in environment variables.
2022-06-30 18:23:33,599 INFO    ThreadPoolExecutor-7_2: Found credentials in environment variables.
2022-06-30 18:23:33,600 INFO    ThreadPoolExecutor-7_1: Found credentials in environment variables.
2022-06-30 18:23:33,601 INFO    ThreadPoolExecutor-7_0: Found credentials in environment variables.
ERROR

======================================================================
ERROR: test_concurrent_portal_db_crud (integration_test.PortalRegistrationIntegrationTest)
Use multithreading to simulate multiple users simultaneously modifying
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/daniel/repo/azul1/test/azul_test_case.py", line 175, in wrapped
    return method(*args, **kwargs)
  File "/Users/daniel/repo/azul1/test/integration_test.py", line 1270, in test_concurrent_portal_db_crud
    self.assertTrue(all(f.result() is None for f in futures))
  File "/Users/daniel/repo/azul1/test/integration_test.py", line 1270, in <genexpr>
    self.assertTrue(all(f.result() is None for f in futures))
  File "/Users/daniel/.pyenv/versions/3.9.12/lib/python3.9/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/Users/daniel/.pyenv/versions/3.9.12/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/Users/daniel/.pyenv/versions/3.9.12/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/daniel/repo/azul1/test/integration_test.py", line 1265, in run
    self.portal_service._crud(lambda db: [*db, mock_entry])
  File "/Users/daniel/repo/azul1/src/azul/portal_service.py", line 185, in _crud
    self._write_db(new_db, version)
  File "/Users/daniel/repo/azul1/src/azul/portal_service.py", line 238, in _write_db
    response = self.client.put_object(Bucket=self.bucket,
  File "/Users/daniel/repo/azul1/.venv/lib/python3.9/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/daniel/repo/azul1/.venv/lib/python3.9/site-packages/botocore/client.py", line 676, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (SlowDown) when calling the PutObject operation (reached max retries: 4): Please reduce your request rate.

DataBiosphere / azul

test_concurrent_portal_db_crud occasionally fails IT with SlowDown error #4285