We have been seeing some failed tests due to quota issues. This PR attempts to address that by breaking up the error handler logic into two separate cases: 1) the log isn't found yet 2) a quota error was returned from the API
For 1), the test sleeps for 10 seconds and then tries again, up to 3 times
For 2), the test assumes the read_requests_per_minute quota is likely the issue. It goes into backoff mode, pausing for a random time between 1-5 minutes. Because it's a randomized backoff, parallel-running tests should start at different times, avoiding hitting the quota again when they start back up.
This PR also reduces the inner max_tries variable from 6 to 3, which should reduce quota use when a test fails. (Note that the @RetryErrors decorator will also retry the whole block 2 times, bringing the total number of tries on a failure to 6)
We have been seeing some failed tests due to quota issues. This PR attempts to address that by breaking up the error handler logic into two separate cases: 1) the log isn't found yet 2) a quota error was returned from the API
For 1), the test sleeps for 10 seconds and then tries again, up to 3 times For 2), the test assumes the read_requests_per_minute quota is likely the issue. It goes into backoff mode, pausing for a random time between 1-5 minutes. Because it's a randomized backoff, parallel-running tests should start at different times, avoiding hitting the quota again when they start back up.
This PR also reduces the inner
max_triesvariable from 6 to 3, which should reduce quota use when a test fails. (Note that the@RetryErrorsdecorator will also retry the whole block 2 times, bringing the total number of tries on a failure to 6)See https://github.com/googleapis/python-logging/pull/445 to see the environment tests running against the python-logging codebase