scottwernervt / cloudstorage

Unified cloud storage API for storage services.
http://cloudstorage.readthedocs.io/en/latest/
MIT License
93 stars 27 forks source link

Tests: Google Driver #33

Open scottwernervt opened 5 years ago

scottwernervt commented 5 years ago

Google driver tests still randomly fail due to API rate limiting even after introducing delays. Need to rework how we use fixtures for containers and blobs.

Ideas:

micimize commented 5 years ago

This just a travis cli / testing issue, right? It shouldn't effect end users?

scottwernervt commented 5 years ago

Yes, it is only a testing issue and does NOT effect end users.

habibutsu commented 3 years ago

You can try make following improvements and it will useful not only for tests

--- a/src/cloudstorage/drivers/google.py
+++ b/src/cloudstorage/drivers/google.py
@@ -8,6 +8,9 @@ from datetime import datetime, timedelta
 from http import HTTPStatus
 from typing import Any, Dict, Iterable, List  # noqa: F401

+from urllib3.util.retry import Retry
+from requests.adapters import HTTPAdapter
+
 # noinspection PyPackageRequirements
 from google.auth.exceptions import GoogleAuthError

@@ -110,6 +113,13 @@ class GoogleStorageDriver(Driver):

         self._client = storage.Client()

+        retries_strategy = Retry(
+            total=5,
+            backoff_factor=0.1,
+            status_forcelist=[408, 429])
+        http = self._client._http
+        http.mount('https://', HTTPAdapter(max_retries=retries_strategy))
+
     def __iter__(self) -> Iterable[Container]:
         for bucket in self.client.list_buckets():
             yield self._make_container(bucket)