Closed flaky-bot[bot] closed 2 months ago
Looks like this issue is flaky. :worried:
I'm going to leave this open and stop commenting.
A human should fix and close this.
When run at the same commit (30c71f231f60558037ca388bebf87cd523f4c88a), this test passed in one build (Build Status, Sponge) and failed in another build (Build Status, Sponge).
Seems transient, closing for now.
Oops! Looks like this issue is still flaky. It failed again. :grimacing:
I reopened the issue, but a human will need to close it again.
commit: e4983870f225963294b10fb26f419fcdf1bdc169 buildURL: Build Status, Sponge status: failed
target = functools.partial(>, retry=None) predicate = .if_exception_type_predicate at 0x7f030523dee0> sleep_generator = timeout = 90, on_error = None, kwargs = {} deadline = datetime.datetime(2023, 8, 24, 12, 16, 2, 982611) last_exc = _OperationNotComplete(), sleep = 7.585074742603595 next_attempt_time = datetime.datetime(2023, 8, 24, 12, 16, 4, 557774) def retry_target( target, predicate, sleep_generator, timeout=None, on_error=None, **kwargs ): """Call a function and retry if it fails. This is the lowest-level retry helper. Generally, you'll use the higher-level retry helper :class:`Retry`. Args: target(Callable): The function to call and retry. This must be a nullary function - apply arguments with `functools.partial`. predicate (Callable[Exception]): A callable used to determine if an exception raised by the target should be considered retryable. It should return True to retry or False otherwise. sleep_generator (Iterable[float]): An infinite iterator that determines how long to sleep between retries. timeout (float): How long to keep retrying the target. on_error (Callable[Exception]): A function to call while processing a retryable exception. Any error raised by this function will *not* be caught. deadline (float): DEPRECATED: use ``timeout`` instead. For backward compatibility, if specified it will override ``timeout`` parameter. Returns: Any: the return value of the target function. Raises: google.api_core.RetryError: If the deadline is exceeded while retrying. ValueError: If the sleep generator stops yielding values. Exception: If the target raises a method that isn't retryable. """ timeout = kwargs.get("deadline", timeout) if timeout is not None: deadline = datetime_helpers.utcnow() + datetime.timedelta(seconds=timeout) else: deadline = None last_exc = None for sleep in sleep_generator: try: > return target() .nox/py-3-8/lib/python3.8/site-packages/google/api_core/retry.py:191: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = retry = None def _done_or_raise(self, retry=None): """Check if the future is done and raise if it's not.""" if not self.done(retry=retry): > raise _OperationNotComplete() E google.api_core.future.polling._OperationNotComplete .nox/py-3-8/lib/python3.8/site-packages/google/api_core/future/polling.py:120: _OperationNotComplete The above exception was the direct cause of the following exception: self = timeout = 90, retry = None polling = def _blocking_poll(self, timeout=_DEFAULT_VALUE, retry=None, polling=None): """Poll and wait for the Future to be resolved.""" if self._result_set: return polling = polling or self._polling if timeout is not PollingFuture._DEFAULT_VALUE: polling = polling.with_timeout(timeout) try: > polling(self._done_or_raise)(retry=retry) .nox/py-3-8/lib/python3.8/site-packages/google/api_core/future/polling.py:137: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = (), kwargs = {'retry': None} target = functools.partial( >, retry=None) sleep_generator = @functools.wraps(func) def retry_wrapped_func(*args, **kwargs): """A wrapper that calls target function with retry.""" target = functools.partial(func, *args, **kwargs) sleep_generator = exponential_sleep_generator( self._initial, self._maximum, multiplier=self._multiplier ) > return retry_target( target, self._predicate, sleep_generator, self._timeout, on_error=on_error, ) .nox/py-3-8/lib/python3.8/site-packages/google/api_core/retry.py:349: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ target = functools.partial( >, retry=None) predicate = .if_exception_type_predicate at 0x7f030523dee0> sleep_generator = timeout = 90, on_error = None, kwargs = {} deadline = datetime.datetime(2023, 8, 24, 12, 16, 2, 982611) last_exc = _OperationNotComplete(), sleep = 7.585074742603595 next_attempt_time = datetime.datetime(2023, 8, 24, 12, 16, 4, 557774) def retry_target( target, predicate, sleep_generator, timeout=None, on_error=None, **kwargs ): """Call a function and retry if it fails. This is the lowest-level retry helper. Generally, you'll use the higher-level retry helper :class:`Retry`. Args: target(Callable): The function to call and retry. This must be a nullary function - apply arguments with `functools.partial`. predicate (Callable[Exception]): A callable used to determine if an exception raised by the target should be considered retryable. It should return True to retry or False otherwise. sleep_generator (Iterable[float]): An infinite iterator that determines how long to sleep between retries. timeout (float): How long to keep retrying the target. on_error (Callable[Exception]): A function to call while processing a retryable exception. Any error raised by this function will *not* be caught. deadline (float): DEPRECATED: use ``timeout`` instead. For backward compatibility, if specified it will override ``timeout`` parameter. Returns: Any: the return value of the target function. Raises: google.api_core.RetryError: If the deadline is exceeded while retrying. ValueError: If the sleep generator stops yielding values. Exception: If the target raises a method that isn't retryable. """ timeout = kwargs.get("deadline", timeout) if timeout is not None: deadline = datetime_helpers.utcnow() + datetime.timedelta(seconds=timeout) else: deadline = None last_exc = None for sleep in sleep_generator: try: return target() # pylint: disable=broad-except # This function explicitly must deal with broad exceptions. except Exception as exc: if not predicate(exc): raise last_exc = exc if on_error is not None: on_error(exc) if deadline is not None: next_attempt_time = datetime_helpers.utcnow() + datetime.timedelta( seconds=sleep ) if deadline < next_attempt_time: > raise exceptions.RetryError( "Deadline of {:.1f}s exceeded while calling target function".format( timeout ), last_exc, ) from last_exc E google.api_core.exceptions.RetryError: Deadline of 90.0s exceeded while calling target function, last exception: .nox/py-3-8/lib/python3.8/site-packages/google/api_core/retry.py:207: RetryError During handling of the above exception, another exception occurred: def test_create_and_delete_glossary() -> None: sys.path.insert(1, "../") from google.cloud import translate_v3 as translate languages = ["fr", "en"] glossary_name = f"test-glossary-{uuid.uuid4()}" glossary_uri = "gs://cloud-samples-data/translation/bistro_glossary.csv" # create_glossary will raise an exception if creation fails > created_glossary_name = hybrid_tutorial.create_glossary( languages, PROJECT_ID, glossary_name, glossary_uri ) hybrid_tutorial_test.py:49: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ hybrid_tutorial.py:113: in create_glossary operation.result(timeout=90) .nox/py-3-8/lib/python3.8/site-packages/google/api_core/future/polling.py:256: in result self._blocking_poll(timeout=timeout, retry=retry, polling=polling) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = timeout = 90, retry = None polling = def _blocking_poll(self, timeout=_DEFAULT_VALUE, retry=None, polling=None): """Poll and wait for the Future to be resolved.""" if self._result_set: return polling = polling or self._polling if timeout is not PollingFuture._DEFAULT_VALUE: polling = polling.with_timeout(timeout) try: polling(self._done_or_raise)(retry=retry) except exceptions.RetryError: > raise concurrent.futures.TimeoutError( f"Operation did not complete within the designated timeout of " f"{polling.timeout} seconds." ) E concurrent.futures._base.TimeoutError: Operation did not complete within the designated timeout of 90 seconds. .nox/py-3-8/lib/python3.8/site-packages/google/api_core/future/polling.py:139: TimeoutError
Retry failed. If it fails again for retry issue then I'll look into this. Has not failed in recent runs.
Oops! Looks like this issue is still flaky. It failed again. :grimacing:
I reopened the issue, but a human will need to close it again.
commit: 7a74b96f48e99f6bc28a9f5fcb279c76066d2131 buildURL: Build Status, Sponge status: failed
target = functools.partial(>, retry=None) predicate = .if_exception_type_predicate at 0x7ff0cb164280> sleep_generator = timeout = 180, on_error = None, kwargs = {} deadline = datetime.datetime(2023, 9, 13, 12, 20, 13, 696211) last_exc = _OperationNotComplete(), sleep = 15.296538479788133 next_attempt_time = datetime.datetime(2023, 9, 13, 12, 20, 24, 987424) def retry_target( target, predicate, sleep_generator, timeout=None, on_error=None, **kwargs ): """Call a function and retry if it fails. This is the lowest-level retry helper. Generally, you'll use the higher-level retry helper :class:`Retry`. Args: target(Callable): The function to call and retry. This must be a nullary function - apply arguments with `functools.partial`. predicate (Callable[Exception]): A callable used to determine if an exception raised by the target should be considered retryable. It should return True to retry or False otherwise. sleep_generator (Iterable[float]): An infinite iterator that determines how long to sleep between retries. timeout (float): How long to keep retrying the target. on_error (Callable[Exception]): A function to call while processing a retryable exception. Any error raised by this function will *not* be caught. deadline (float): DEPRECATED: use ``timeout`` instead. For backward compatibility, if specified it will override ``timeout`` parameter. Returns: Any: the return value of the target function. Raises: google.api_core.RetryError: If the deadline is exceeded while retrying. ValueError: If the sleep generator stops yielding values. Exception: If the target raises a method that isn't retryable. """ timeout = kwargs.get("deadline", timeout) if timeout is not None: deadline = datetime_helpers.utcnow() + datetime.timedelta(seconds=timeout) else: deadline = None last_exc = None for sleep in sleep_generator: try: > return target() .nox/py-3-8/lib/python3.8/site-packages/google/api_core/retry.py:191: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = retry = None def _done_or_raise(self, retry=None): """Check if the future is done and raise if it's not.""" if not self.done(retry=retry): > raise _OperationNotComplete() E google.api_core.future.polling._OperationNotComplete .nox/py-3-8/lib/python3.8/site-packages/google/api_core/future/polling.py:120: _OperationNotComplete The above exception was the direct cause of the following exception: self = timeout = 180, retry = None polling = def _blocking_poll(self, timeout=_DEFAULT_VALUE, retry=None, polling=None): """Poll and wait for the Future to be resolved.""" if self._result_set: return polling = polling or self._polling if timeout is not PollingFuture._DEFAULT_VALUE: polling = polling.with_timeout(timeout) try: > polling(self._done_or_raise)(retry=retry) .nox/py-3-8/lib/python3.8/site-packages/google/api_core/future/polling.py:137: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = (), kwargs = {'retry': None} target = functools.partial( >, retry=None) sleep_generator = @functools.wraps(func) def retry_wrapped_func(*args, **kwargs): """A wrapper that calls target function with retry.""" target = functools.partial(func, *args, **kwargs) sleep_generator = exponential_sleep_generator( self._initial, self._maximum, multiplier=self._multiplier ) > return retry_target( target, self._predicate, sleep_generator, self._timeout, on_error=on_error, ) .nox/py-3-8/lib/python3.8/site-packages/google/api_core/retry.py:349: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ target = functools.partial( >, retry=None) predicate = .if_exception_type_predicate at 0x7ff0cb164280> sleep_generator = timeout = 180, on_error = None, kwargs = {} deadline = datetime.datetime(2023, 9, 13, 12, 20, 13, 696211) last_exc = _OperationNotComplete(), sleep = 15.296538479788133 next_attempt_time = datetime.datetime(2023, 9, 13, 12, 20, 24, 987424) def retry_target( target, predicate, sleep_generator, timeout=None, on_error=None, **kwargs ): """Call a function and retry if it fails. This is the lowest-level retry helper. Generally, you'll use the higher-level retry helper :class:`Retry`. Args: target(Callable): The function to call and retry. This must be a nullary function - apply arguments with `functools.partial`. predicate (Callable[Exception]): A callable used to determine if an exception raised by the target should be considered retryable. It should return True to retry or False otherwise. sleep_generator (Iterable[float]): An infinite iterator that determines how long to sleep between retries. timeout (float): How long to keep retrying the target. on_error (Callable[Exception]): A function to call while processing a retryable exception. Any error raised by this function will *not* be caught. deadline (float): DEPRECATED: use ``timeout`` instead. For backward compatibility, if specified it will override ``timeout`` parameter. Returns: Any: the return value of the target function. Raises: google.api_core.RetryError: If the deadline is exceeded while retrying. ValueError: If the sleep generator stops yielding values. Exception: If the target raises a method that isn't retryable. """ timeout = kwargs.get("deadline", timeout) if timeout is not None: deadline = datetime_helpers.utcnow() + datetime.timedelta(seconds=timeout) else: deadline = None last_exc = None for sleep in sleep_generator: try: return target() # pylint: disable=broad-except # This function explicitly must deal with broad exceptions. except Exception as exc: if not predicate(exc): raise last_exc = exc if on_error is not None: on_error(exc) if deadline is not None: next_attempt_time = datetime_helpers.utcnow() + datetime.timedelta( seconds=sleep ) if deadline < next_attempt_time: > raise exceptions.RetryError( "Deadline of {:.1f}s exceeded while calling target function".format( timeout ), last_exc, ) from last_exc E google.api_core.exceptions.RetryError: Deadline of 180.0s exceeded while calling target function, last exception: .nox/py-3-8/lib/python3.8/site-packages/google/api_core/retry.py:207: RetryError During handling of the above exception, another exception occurred: def test_create_and_delete_glossary() -> None: sys.path.insert(1, "../") from google.cloud import translate_v3 as translate languages = ["fr", "en"] glossary_name = f"test-glossary-{uuid.uuid4()}" glossary_uri = "gs://cloud-samples-data/translation/bistro_glossary.csv" # create_glossary will raise an exception if creation fails created_glossary_name = hybrid_tutorial.create_glossary( languages, PROJECT_ID, glossary_name, glossary_uri ) # Delete glossary so that future tests will pass # delete_glossary will raise an exception if deletion fails client = translate.TranslationServiceClient() name = client.glossary_path(PROJECT_ID, "us-central1", created_glossary_name) operation = client.delete_glossary(name=name) > result = operation.result(timeout=180) hybrid_tutorial_test.py:60: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ .nox/py-3-8/lib/python3.8/site-packages/google/api_core/future/polling.py:256: in result self._blocking_poll(timeout=timeout, retry=retry, polling=polling) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = timeout = 180, retry = None polling = def _blocking_poll(self, timeout=_DEFAULT_VALUE, retry=None, polling=None): """Poll and wait for the Future to be resolved.""" if self._result_set: return polling = polling or self._polling if timeout is not PollingFuture._DEFAULT_VALUE: polling = polling.with_timeout(timeout) try: polling(self._done_or_raise)(retry=retry) except exceptions.RetryError: > raise concurrent.futures.TimeoutError( f"Operation did not complete within the designated timeout of " f"{polling.timeout} seconds." ) E concurrent.futures._base.TimeoutError: Operation did not complete within the designated timeout of 180 seconds. .nox/py-3-8/lib/python3.8/site-packages/google/api_core/future/polling.py:139: TimeoutError
Ivan, can you help take a look at this flaky test?
Oops! Looks like this issue is still flaky. It failed again. :grimacing:
I reopened the issue, but a human will need to close it again.
commit: 25dcfc949c5a829fc967758c2b94b64780dc3b1c buildURL: Build Status, Sponge status: failed
closing unaddressed flaky-bot:flaky
issues, see go/flakybot-bankruptcy
Note: #8552 was also for this test, but it was closed more than 10 days ago. So, I didn't mark it flaky.
commit: 30c71f231f60558037ca388bebf87cd523f4c88a buildURL: Build Status, Sponge status: failed
Test output