Cannot set polling timeout in BatchControllerClient.create_batch

[dhpikolo]

Looks like with google_api_core version==2.11.0, BatchControllerClient cannot control the polling timeout.

use case: I have a dataproc batches of longer duration (>900secs) being launched from my local machine and I run into this error.

Environment details

• OS type and version: macOS Monterey + v12.3.1
• Python version: 3.7.10
• pip version: 22.2.2
• google-cloud-dataproc version: 5.0.3

Steps to reproduce

1. Use BatchControllerClient with default options
2. Submit a job on dataproc (Job time should be > 900secs)

   Code example

from google.api_core.client_options import ClientOptions
from google.cloud.dataproc_v1 import  BatchControllerClient

gcp_region = "us-central1"
client_options = ClientOptions(
                api_endpoint=f"{gcp_region}-dataproc.googleapis.com:443"
            )
client = BatchControllerClient(client_options=client_options)
client.create_batch(
            request=request,
            retry=None,  
            timeout=None,
            metadata=(),
)

Stack trace

File "/opt/conda/lib/python3.9/site-packages/google/api_core/retry.py", line 191, in retry_target
  return target()
 File "/opt/conda/lib/python3.9/site-packages/google/api_core/future/polling.py", line 120, in _done_or_raise
     raise _OperationNotComplete()
google.api_core.future.polling._OperationNotComplete

 - The above exception was the direct cause of the following exception:

Traceback (most recent call last):
    File "/opt/conda/lib/python3.9/site-packages/google/api_core/future/polling.py", line 137, in _blocking_poll
     polling(self._done_or_raise)(retry=retry)
   File "/opt/conda/lib/python3.9/site-packages/google/api_core/retry.py", line 349, in retry_wrapped_func
     return retry_target(
   File "/opt/conda/lib/python3.9/site-packages/google/api_core/retry.py", line 207, in retry_target
     raise exceptions.RetryError(
 google.api_core.exceptions.RetryError: Deadline of 900.0s exceeded while calling target function, last exception:

Making sure to follow these steps will guarantee the quickest resolution possible.

Thanks!

[dhpikolo]

I have tried to reproduce the error with a smaller timeout value and expected a RetryError

client.create_batch(
            request=request,
            retry=None,  
            timeout=10,
            metadata=(),
)

This did not had any effect on the polling timeout.

[atulep]

Hi @dhpikolo, timeout you're passing in your snippet only affects create_batch method, which creates an Operation object. So, essentially you're setting timeout for creating an Operation object. What you want to do is set timeout for the result of the operation. So, something like this:

operation = clien.create_batch(request)
response = operation.result(timeout=None)  # infinite timeout (not recommended)

Can you try this?

[parthea]

I'm going to close this issue as a response was provided in https://github.com/googleapis/python-dataproc/issues/505#issuecomment-1378013402