googleapis / python-pubsub

Apache License 2.0
392 stars 205 forks source link

feat: add use_topic_schema for Cloud Storage Subscriptions #1154

Closed gcf-owl-bot[bot] closed 4 months ago

gcf-owl-bot[bot] commented 7 months ago

PiperOrigin-RevId: 649277844

Source-Link: https://github.com/googleapis/googleapis/commit/fd7efa2da3860e813485e63661d3bdd21fc9ba82

Source-Link: https://github.com/googleapis/googleapis-gen/commit/50be251329d8db5b555626ebd4886721f547d3cc Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiNTBiZTI1MTMyOWQ4ZGI1YjU1NTYyNmViZDQ4ODY3MjFmNTQ3ZDNjYyJ9

BEGIN_NESTED_COMMIT feat: add use_topic_schema for Cloud Storage Subscriptions PiperOrigin-RevId: 643385701

BEGIN_COMMIT_OVERRIDE feat: add use_topic_schema for Cloud Storage Subscriptions feat: add service_account_email for export subscriptions END_COMMIT_OVERRIDE

Source-Link: https://github.com/googleapis/googleapis/commit/f29a82a206156fd665eaa01cb7987170ac0250f0

Source-Link: https://github.com/googleapis/googleapis-gen/commit/f8f3f60ad2ab4e81fe024368ad0430900174e82b Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiZjhmM2Y2MGFkMmFiNGU4MWZlMDI0MzY4YWQwNDMwOTAwMTc0ZTgyYiJ9 END_NESTED_COMMIT BEGIN_NESTED_COMMIT chore: Update gapic-generator-python to v1.18.0 PiperOrigin-RevId: 638650618

Source-Link: https://github.com/googleapis/googleapis/commit/6330f0389afdd04235c59898cc44f715b077aa25

Source-Link: https://github.com/googleapis/googleapis-gen/commit/44fa4f1979dc45c1778fd7caf13f8e61c6d1cae8 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiNDRmYTRmMTk3OWRjNDVjMTc3OGZkN2NhZjEzZjhlNjFjNmQxY2FlOCJ9 END_NESTED_COMMIT BEGIN_NESTED_COMMIT feat: add service_account_email for export subscriptions PiperOrigin-RevId: 638641437

Source-Link: https://github.com/googleapis/googleapis/commit/a3016a877a58aee3bef1e93871542ef79993daf7

Source-Link: https://github.com/googleapis/googleapis-gen/commit/2001c330aee62a9d4e8a9852d8e5bf2662818ad4 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiMjAwMWMzMzBhZWU2MmE5ZDRlOGE5ODUyZDhlNWJmMjY2MjgxOGFkNCJ9 END_NESTED_COMMIT BEGIN_NESTED_COMMIT chore: Update gapic-generator-python to v1.17.1 PiperOrigin-RevId: 629071173

Source-Link: https://github.com/googleapis/googleapis/commit/4afa392105cc62e965631d15b772ff68454ecf1c

Source-Link: https://github.com/googleapis/googleapis-gen/commit/16dbbb4d0457db5e61ac9f99b0d52a46154455ac Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiMTZkYmJiNGQwNDU3ZGI1ZTYxYWM5Zjk5YjBkNTJhNDYxNTQ0NTVhYyJ9 END_NESTED_COMMIT

mukund-ananthu commented 5 months ago

cc: @parthea Could we rope in the code generator library owners for GAPIC for them to fix the code generator libraries to ensure the code doesn't break unit tests like they are now:

self = <google.api_core.grpc_helpers_async._WrappedUnaryUnaryCall object at 0x7fea08ad9e50>

    def __await__(self):
        try:
>           response = yield from self._call.__await__()

.nox/unit-3-8/lib/python3.8/site-packages/google/api_core/grpc_helpers_async.py:81: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <_AioCall of RPC that terminated with:
    status = The request is missing a valid API key.
    details = "StatusCode.PERMIS...essage:"The request is missing a valid API key.", grpc_status:7, created_time:"2024-07-02T17:[44](https://github.com/googleapis/python-pubsub/actions/runs/9765604292/job/26956648919?pr=1154#step:5:45):25.648799599+00:00"}"
>

    def __await__(self) -> Generator[Any, None, ResponseType]:
        """Wait till the ongoing RPC request finishes."""
        try:
            response = yield from self._call_response
        except asyncio.CancelledError:
            # Even if we caught all other CancelledError, there is still
            # this corner case. If the application cancels immediately after
            # the Call object is created, we will observe this
            # `CancelledError`.
            if not self.cancelled():
                self.cancel()
            raise

        # NOTE(lidiz) If we raise RpcError in the task, and users doesn't
        # 'await' on it. AsyncIO will log 'Task exception was never retrieved'.
        # Instead, if we move the exception raising here, the spam stops.
        # Unfortunately, there can only be one 'yield from' in '__await__'. So,
        # we need to access the private instance variable.
        if response is cygrpc.EOF:
            if self._cython_call.is_locally_cancelled():
                raise asyncio.CancelledError()
            else:
>               raise _create_rpc_error(
                    self._cython_call._initial_metadata,
                    self._cython_call._status,
                )
E               grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
E                   status = StatusCode.PERMISSION_DENIED
E                   details = "The request is missing a valid API key."
E                   debug_error_string = "UNKNOWN:Error received from peer ipv4:172.253.122.95:443 {grpc_message:"The request is missing a valid API key.", grpc_status:7, created_time:"2024-07-02T17:44:25.6[48](https://github.com/googleapis/python-pubsub/actions/runs/9765604292/job/26956648919?pr=1154#step:5:49)799599+00:00"}"
E               >

.nox/unit-3-8/lib/python3.8/site-packages/grpc/aio/_call.py:318: AioRpcError

The above exception was the direct cause of the following exception:

creds = <google.auth.credentials.AnonymousCredentials object at 0x7fea0667f340>

    @pytest.mark.asyncio
    async def test_sync_pull_warning_if_return_immediately_async(creds):
        from google.pubsub_v1.services.subscriber.async_client import SubscriberAsyncClient

        client = SubscriberAsyncClient(credentials=creds)
        subscription_path = "projects/foo/subscriptions/bar"

        patcher = mock.patch(
            "google.pubsub_v1.services.subscriber.async_client.gapic_v1.method_async.wrap_method",
            new=mock.AsyncMock,
        )

        with patcher, pytest.warns(
            DeprecationWarning,
            match="The return_immediately flag is deprecated and should be set to False",
        ) as warned:
>           await client.pull(subscription=subscription_path, return_immediately=True)

tests/unit/pubsub_v1/subscriber/test_subscriber_client.py:311: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
google/pubsub_v1/services/subscriber/async_client.py:1289: in pull
    response = await rpc(
.nox/unit-3-8/lib/python3.8/site-packages/google/api_core/retry_async.py:223: in retry_wrapped_func
    return await retry_target(
.nox/unit-3-8/lib/python3.8/site-packages/google/api_core/retry_async.py:121: in retry_target
    return await asyncio.wait_for(
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/asyncio/tasks.py:[49](https://github.com/googleapis/python-pubsub/actions/runs/9765604292/job/26956648919?pr=1154#step:5:50)4: in wait_for
    return fut.result()
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/asyncio/tasks.py:695: in _wrap_awaitable
    return (yield from awaitable.__await__())
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <google.api_core.grpc_helpers_async._WrappedUnaryUnaryCall object at 0x7fea08ad9e[50](https://github.com/googleapis/python-pubsub/actions/runs/9765604292/job/26956648919?pr=1154#step:5:51)>

    def __await__(self):
        try:
            response = yield from self._call.__await__()
            return response
        except grpc.RpcError as rpc_error:
>           raise exceptions.from_grpc_error(rpc_error) from rpc_error
E           google.api_core.exceptions.PermissionDenied: 403 The request is missing a valid API key.

.nox/unit-3-8/lib/python3.8/site-packages/google/api_core/grpc_helpers_async.py:84: PermissionDenied
- generated xml file: /home/runner/work/python-pubsub/python-pubsub/unit_3.8_sponge_log.xml -
=========================== short test summary info ============================
FAILED tests/unit/pubsub_v1/subscriber/test_subscriber_client.py::test_sync_pull_warning_if_return_immediately_async - google.api_core.exceptions.PermissionDenied: 403 The request is missing a valid API key.
1 failed, 16[52](https://github.com/googleapis/python-pubsub/actions/runs/9765604292/job/26956648919?pr=1154#step:5:53) passed in 21.49s
nox > Command py.test --quiet --junitxml=unit_3.8_sponge_log.xml --cov=google/cloud --cov=tests/unit --cov-append --cov-config=.coveragerc --cov-report= --cov-fail-under=0 tests/unit failed with exit code 1
nox > Session unit-3.8 failed.
parthea commented 5 months ago

Following up on https://github.com/googleapis/python-pubsub/pull/1154#issuecomment-2203948313, the test that is failing is part of the handwritten code. We can follow up on a long term solution via https://github.com/googleapis/gapic-generator-python/issues/2053. In the meantime, the following change fixes the issue.

(py3112env) partheniou@partheniou-vm-3:~/git/python-pubsub$ git diff
diff --git a/tests/unit/pubsub_v1/subscriber/test_subscriber_client.py b/tests/unit/pubsub_v1/subscriber/test_subscriber_client.py
index a09d85b..5cd2cae 100644
--- a/tests/unit/pubsub_v1/subscriber/test_subscriber_client.py
+++ b/tests/unit/pubsub_v1/subscriber/test_subscriber_client.py
@@ -298,10 +298,8 @@ async def test_sync_pull_warning_if_return_immediately_async(creds):

     client = SubscriberAsyncClient(credentials=creds)
     subscription_path = "projects/foo/subscriptions/bar"
-
-    patcher = mock.patch(
-        "google.pubsub_v1.services.subscriber.async_client.gapic_v1.method_async.wrap_method",
-        new=mock.AsyncMock,
+    patcher = mock.patch.object(
+        type(client.transport.pull), "__call__", new_callable=mock.AsyncMock,
     )

     with patcher, pytest.warns(
snippet-bot[bot] commented 4 months ago

No region tags are edited in this PR.

This comment is generated by snippet-bot. If you find problems with this result, please file an issue at: https://github.com/googleapis/repo-automation-bots/issues. To update this comment, add snippet-bot:force-run label or use the checkbox below: