search

googleads / google-ads-python

Google Ads API Client Library for Python
Apache License 2.0
521 stars 480 forks source link

Stream removed exception downloading RSAs #663

Closed nev7n closed 2 years ago

nev7n commented 2 years ago

Describe the bug: Initially reported via the Google Ads Support forum, who asked that we raise the issue here:

Upon checking on my end, I can see that your API request and response is getting large message size. Since you’re using the Python client library, it appears that the issue “Stream removed” resides on the ​​Python gRPC implementation side. Note that the Google Ads API client libraries use gRPC for generating requests and responses. For further assistance, we recommend you to reach out to the Python client library owner by filing an issue on the issue tracker on the Issue tab of your client library on the github

Getting an exception trying to download around 422,000 RSA ads, this was working until recently when we consistently started seeing "Stream removed" errors. We've worked around the issue by splitting the download into multiple downloads one Campaign at a time, but still posting here in case it can be fixed or recommendations for maximum download sizes added to documentation.

Steps to Reproduce: Use query similar to:

SELECT campaign.name,ad_group.name,ad_group_ad.ad.responsive_search_ad.headlines,ad_group_ad.ad.responsive_search_ad.descriptions,ad_group_ad.ad.responsive_search_ad.path1,ad_group_ad.ad.responsive_search_ad.path2,ad_group_ad.ad.final_urls,ad_group_ad.labels,ad_group_ad.ad.id,ad_group_ad.status FROM ad_group_ad WHERE ad_group_ad.ad.type = \'RESPONSIVE_SEARCH_AD\' AND ad_group_ad.status IN (\'ENABLED\', \'PAUSED\') AND ad_group.status != \'REMOVED\' AND ad_group_ad.ad.system_managed_resource_source != \'AD_VARIATIONS\' AND campaign.id IN (909310504,938819651,938819654,938819657,938819660,1409202860,1409202863,1409202866,1409202869,1446309069,1650512859,6484816965,6484897424,6484965589,6484965592,6510996159,6510996162,6510996165,6510996168,6510996171,6511086910,6511086913,6511086916,6511086919,6511086922,6511086925,6511086928,6511086931,6511086934,6511086937,6511086940,6511218375,6511218378,6511218381,6511218384,6511218387,6511218390,6511218393,6511218396,6511218399,6511218402,6511218405,6511218885,6511218888,6511218891,6511218894,6511218897,6511218900,6511219578,6511219581,6511219584,6511219587,6511219590,6511219623,6511219626,6511219629,6511219632,6511219635,6511219638,6767253058,6767253061,6767253064,6767253067,6767253070,6767253073,6767253076,6767253079,6767253082,6767253085,6767253088,6767253259,6767253262,6767253265,6767253268,6767253271,9725674020,9725674023,9725674026,9725674029,12297942867,12298126684,12298131250,12298135177,12349394320,13959390682,13959425503,14307696953,14330278995,14333601074,14805434528,14805434531,16072015620,16077624235) AND segments.date >= 20220604 AND segments.date <= 20220704 PARAMETERS omit_unselected_resource_names = true"

to download Responsive Search Ads from an account with 422k ads.

Code snippet:

        ga_service = self.client.get_service("GoogleAdsService")

        columns_joined = ",".join(columns)
        query = f"SELECT {columns_joined} FROM {resource}"
        if len(predicates)>0:
            predicates_joined = " AND ".join(predicates)
            query += f" WHERE {predicates_joined}"
        query += " PARAMETERS omit_unselected_resource_names = true"
        search_request = self.client.get_type("SearchGoogleAdsStreamRequest")
        search_request.customer_id = self.account_id
        search_request.query = query

        stream = retry_with_backoff(ga_service.search_stream, search_request)
        for batch in stream:
            for row in batch.results:
                yield row

The ads start streaming but after a while an exception is thrown while reading the results.

Expected behavior: Ads are downloaded.

Client library version and API version: Client library version: google-ads==17.0.0 Google Ads API version: v10

attrs==21.4.0
bingads==13.0.13
blinker==1.4
cached-property==1.5.2
cachetools==4.2.4
certifi==2022.6.15
charset-normalizer==2.1.0
future==0.18.2
google-ads==17.0.0
google-api-core==2.8.2
google-auth==1.35.0
google-auth-oauthlib==0.5.2
googleads==21.0.0
googleapis-common-protos==1.56.3
grpcio==1.46.3
grpcio-status==1.46.3
idna==3.3
isodate==0.6.1
jsonpickle==2.2.0
lxml==4.9.1
mock==4.0.3
numpy==1.22.3
oauthlib==3.2.0
pandas==1.4.2
platformdirs==2.5.2
proto-plus==1.19.6
protobuf==3.20.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
python-dateutil==2.8.2
pytz==2022.1
PyYAML==5.4.1
raygun4py==4.3.2
requests==2.28.1
requests-file==1.5.1
requests-oauthlib==1.3.1
requests-toolbelt==0.9.1
rsa==4.8
six==1.16.0
suds-community==1.1.2
urllib3==1.26.9
xmltodict==0.13.0
zeep==4.1.0

Request/Response Logs:

2022-07-04T05:49:41.839+01:00 | Request made: ClientCustomerId: REDACTED, Host: googleads.googleapis.com, Method: /google.ads.googleads.v11.services.GoogleAdsService/Search, RequestId: uFsfXv0OjRqg2B8ma0rzGg, IsFault: False, FaultMessage: None
-- | --
  | 2022-07-04T06:51:19.823+01:00 | INFO:google.ads.googleads.client:Request
  | 2022-07-04T06:51:19.823+01:00 | -------
  | 2022-07-04T06:51:19.823+01:00 | Method: /google.ads.googleads.v11.services.GoogleAdsService/SearchStream
  | 2022-07-04T06:51:19.823+01:00 | Host: googleads.googleapis.com
  | 2022-07-04T06:51:19.823+01:00 | Headers: {
  | 2022-07-04T06:51:19.823+01:00 | "developer-token": "REDACTED",
  | 2022-07-04T06:51:19.823+01:00 | "login-customer-id": "REDACTED",
  | 2022-07-04T06:51:19.823+01:00 | "x-goog-api-client": "gl-python/3.8.13 grpc/1.46.3 gax/2.8.2 gccl/17.0.0",
  | 2022-07-04T06:51:19.823+01:00 | "x-goog-request-params": "customer_id=REDACTED"
  | 2022-07-04T06:51:19.823+01:00 | }
  | 2022-07-04T06:51:19.823+01:00 | Request: customer_id: "REDACTED"
  | 2022-07-04T06:51:19.823+01:00 | query: "SELECT campaign.name,ad_group.name,ad_group_ad.ad.responsive_search_ad.headlines,ad_group_ad.ad.responsive_search_ad.descriptions,ad_group_ad.ad.responsive_search_ad.path1,ad_group_ad.ad.responsive_search_ad.path2,ad_group_ad.ad.final_urls,ad_group_ad.labels,ad_group_ad.ad.id,ad_group_ad.status FROM ad_group_ad WHERE ad_group_ad.ad.type = \'RESPONSIVE_SEARCH_AD\' AND ad_group_ad.status IN (\'ENABLED\', \'PAUSED\') AND ad_group.status != \'REMOVED\' AND ad_group_ad.ad.system_managed_resource_source != \'AD_VARIATIONS\' AND campaign.id IN (909310504,938819651,938819654,938819657,938819660,1409202860,1409202863,1409202866,1409202869,1446309069,1650512859,6484816965,6484897424,6484965589,6484965592,6510996159,6510996162,6510996165,6510996168,6510996171,6511086910,6511086913,6511086916,6511086919,6511086922,6511086925,6511086928,6511086931,6511086934,6511086937,6511086940,6511218375,6511218378,6511218381,6511218384,6511218387,6511218390,6511218393,6511218396,6511218399,6511218402,6511218405,6511218885,6511218888,6511218891,6511218894,6511218897,6511218900,6511219578,6511219581,6511219584,6511219587,6511219590,6511219623,6511219626,6511219629,6511219632,6511219635,6511219638,6767253058,6767253061,6767253064,6767253067,6767253070,6767253073,6767253076,6767253079,6767253082,6767253085,6767253088,6767253259,6767253262,6767253265,6767253268,6767253271,9725674020,9725674023,9725674026,9725674029,12297942867,12298126684,12298131250,12298135177,12349394320,13959390682,13959425503,14307696953,14330278995,14333601074,14805434528,14805434531,16072015620,16077624235) AND segments.date >= 20220604 AND segments.date <= 20220704 PARAMETERS omit_unselected_resource_names = true"
  | 2022-07-04T06:51:19.823+01:00 | Response
  | 2022-07-04T06:51:19.823+01:00 | -------
  | 2022-07-04T06:51:19.823+01:00 | Headers: {}
  | 2022-07-04T06:51:19.823+01:00 | Fault: {
  | 2022-07-04T06:51:19.823+01:00 | "created": "@1656913879.822245365",
  | 2022-07-04T06:51:19.823+01:00 | "description": "Error received from peer ipv4:142.250.73.234:443",
  | 2022-07-04T06:51:19.823+01:00 | "file": "src/core/lib/surface/call.cc",
  | 2022-07-04T06:51:19.823+01:00 | "file_line": 952,
  | 2022-07-04T06:51:19.823+01:00 | "grpc_message": "Stream removed",
  | 2022-07-04T06:51:19.823+01:00 | "grpc_status": 2
  | 2022-07-04T06:51:19.823+01:00 | }
  | 2022-07-04T06:51:19.823+01:00 | Request
  | 2022-07-04T06:51:19.823+01:00 | -------
  | 2022-07-04T06:51:19.823+01:00 | Method: /google.ads.googleads.v11.services.GoogleAdsService/SearchStream
  | 2022-07-04T06:51:19.823+01:00 | Host: googleads.googleapis.com
  | 2022-07-04T06:51:19.823+01:00 | Headers: {
  | 2022-07-04T06:51:19.823+01:00 | "developer-token": "REDACTED",
  | 2022-07-04T06:51:19.823+01:00 | "login-customer-id": "REDACTED",
  | 2022-07-04T06:51:19.823+01:00 | "x-goog-api-client": "gl-python/3.8.13 grpc/1.46.3 gax/2.8.2 gccl/17.0.0",
  | 2022-07-04T06:51:19.823+01:00 | "x-goog-request-params": "customer_id=REDACTED"
  | 2022-07-04T06:51:19.823+01:00 | }
  | 2022-07-04T06:51:19.823+01:00 | Request: customer_id: "REDACTED"
  | 2022-07-04T06:51:19.823+01:00 | query: "SELECT campaign.name,ad_group.name,ad_group_ad.ad.responsive_search_ad.headlines,ad_group_ad.ad.responsive_search_ad.descriptions,ad_group_ad.ad.responsive_search_ad.path1,ad_group_ad.ad.responsive_search_ad.path2,ad_group_ad.ad.final_urls,ad_group_ad.labels,ad_group_ad.ad.id,ad_group_ad.status FROM ad_group_ad WHERE ad_group_ad.ad.type = \'RESPONSIVE_SEARCH_AD\' AND ad_group_ad.status IN (\'ENABLED\', \'PAUSED\') AND ad_group.status != \'REMOVED\' AND ad_group_ad.ad.system_managed_resource_source != \'AD_VARIATIONS\' AND campaign.id IN (909310504,938819651,938819654,938819657,938819660,1409202860,1409202863,1409202866,1409202869,1446309069,1650512859,6484816965,6484897424,6484965589,6484965592,6510996159,6510996162,6510996165,6510996168,6510996171,6511086910,6511086913,6511086916,6511086919,6511086922,6511086925,6511086928,6511086931,6511086934,6511086937,6511086940,6511218375,6511218378,6511218381,6511218384,6511218387,6511218390,6511218393,6511218396,6511218399,6511218402,6511218405,6511218885,6511218888,6511218891,6511218894,6511218897,6511218900,6511219578,6511219581,6511219584,6511219587,6511219590,6511219623,6511219626,6511219629,6511219632,6511219635,6511219638,6767253058,6767253061,6767253064,6767253067,6767253070,6767253073,6767253076,6767253079,6767253082,6767253085,6767253088,6767253259,6767253262,6767253265,6767253268,6767253271,9725674020,9725674023,9725674026,9725674029,12297942867,12298126684,12298131250,12298135177,12349394320,13959390682,13959425503,14307696953,14330278995,14333601074,14805434528,14805434531,16072015620,16077624235) AND segments.date >= 20220604 AND segments.date <= 20220704 PARAMETERS omit_unselected_resource_names = true"
  | 2022-07-04T06:51:19.823+01:00 | Response
  | 2022-07-04T06:51:19.823+01:00 | -------
  | 2022-07-04T06:51:19.823+01:00 | Headers: {}
  | 2022-07-04T06:51:19.823+01:00 | Fault: {
  | 2022-07-04T06:51:19.823+01:00 | "created": "@1656913879.822245365",
  | 2022-07-04T06:51:19.823+01:00 | "description": "Error received from peer ipv4:142.250.73.234:443",
  | 2022-07-04T06:51:19.823+01:00 | "file": "src/core/lib/surface/call.cc",
  | 2022-07-04T06:51:19.823+01:00 | "file_line": 952,
  | 2022-07-04T06:51:19.823+01:00 | "grpc_message": "Stream removed",
  | 2022-07-04T06:51:19.823+01:00 | "grpc_status": 2
  | 2022-07-04T06:51:19.823+01:00 | }
  | 2022-07-04T06:51:19.823+01:00 | WARNING:google.ads.googleads.client:Request made: ClientCustomerId: REDACTED, Host: googleads.googleapis.com, Method: /google.ads.googleads.v11.services.GoogleAdsService/SearchStream, RequestId: None, IsFault: True, FaultMessage: Stream removed
  | 2022-07-04T06:51:19.823+01:00 | Request made: ClientCustomerId: REDACTED, Host: googleads.googleapis.com, Method: /google.ads.googleads.v11.services.GoogleAdsService/SearchStream, RequestId: None, IsFault: True, FaultMessage: Stream removed
  | 2022-07-04T06:51:20.591+01:00 | E0704 05:51:20.590935400 14 fork_posix.cc:76] Other threads are currently calling into gRPC, skipping fork() handlers
  | 2022-07-04T06:51:20.953+01:00 | ERROR:root:Exception in GetResponsiveSearchAds
  | 2022-07-04T06:51:20.953+01:00 | Traceback (most recent call last):
  | 2022-07-04T06:51:20.953+01:00 | File "/usr/local/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 90, in __next__
  | 2022-07-04T06:51:20.953+01:00 | return next(self._wrapped)
  | 2022-07-04T06:51:20.953+01:00 | File "/usr/local/lib/python3.8/site-packages/google/ads/googleads/interceptors/response_wrappers.py", line 108, in __next__
  | 2022-07-04T06:51:20.953+01:00 | raise e
  | 2022-07-04T06:51:20.953+01:00 | File "/usr/local/lib/python3.8/site-packages/google/ads/googleads/interceptors/response_wrappers.py", line 105, in __next__
  | 2022-07-04T06:51:20.953+01:00 | self._failure_handler(self._underlay_call)
  | 2022-07-04T06:51:20.953+01:00 | File "/usr/local/lib/python3.8/site-packages/google/ads/googleads/interceptors/exception_interceptor.py", line 71, in _handle_grpc_failure
  | 2022-07-04T06:51:20.953+01:00 | raise self._get_error_from_response(response)
  | 2022-07-04T06:51:20.953+01:00 | File "/usr/local/lib/python3.8/site-packages/google/ads/googleads/interceptors/response_wrappers.py", line 88, in __next__
  | 2022-07-04T06:51:20.953+01:00 | message = next(self._underlay_call)
  | 2022-07-04T06:51:20.953+01:00 | File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 426, in __next__
  | 2022-07-04T06:51:20.953+01:00 | return self._next()
  | 2022-07-04T06:51:20.953+01:00 | File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 642, in _next
  | 2022-07-04T06:51:20.953+01:00 | return self._next_response()
  | 2022-07-04T06:51:20.953+01:00 | File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 617, in _next_response
  | 2022-07-04T06:51:20.953+01:00 | raise self
  | 2022-07-04T06:51:20.953+01:00 | grpc._channel._SingleThreadedRendezvous: <_SingleThreadedRendezvous of RPC that terminated with:
  | 2022-07-04T06:51:20.953+01:00 | status = StatusCode.UNKNOWN
  | 2022-07-04T06:51:20.953+01:00 | details = "Stream removed"
  | 2022-07-04T06:51:20.953+01:00 | debug_error_string = "{"created":"@1656913879.822245365","description":"Error received from peer ipv4:142.250.73.234:443","file":"src/core/lib/surface/call.cc","file_line":952,"grpc_message":"Stream removed","grpc_status":2}"
  | 2022-07-04T06:51:20.953+01:00 | >
  | 2022-07-04T06:51:20.953+01:00 | The above exception was the direct cause of the following exception:
  | 2022-07-04T06:51:20.953+01:00 | Traceback (most recent call last):
  | 2022-07-04T06:51:20.953+01:00 | File "/app/service/service.py", line 604, in GetResponsiveSearchAds
  | 2022-07-04T06:51:20.953+01:00 | for ad in self.GetAdwordsRSA(request):
  | 2022-07-04T06:51:20.953+01:00 | File "/app/service/service.py", line 654, in GetAdwordsRSA
  | 2022-07-04T06:51:20.953+01:00 | for row in generator:
  | 2022-07-04T06:51:20.953+01:00 | File "/app/adwords/reports.py", line 104, in stream_report_data
  | 2022-07-04T06:51:20.953+01:00 | for batch in stream:
  | 2022-07-04T06:51:20.953+01:00 | File "/usr/local/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 93, in __next__
  | 2022-07-04T06:51:20.953+01:00 | raise exceptions.from_grpc_error(exc) from exc
  | 2022-07-04T06:51:20.953+01:00 | google.api_core.exceptions.Unknown: None Stream removed
  | 2022-07-04T06:51:20.953+01:00 | Exception in GetResponsiveSearchAds
  | 2022-07-04T06:51:20.953+01:00 | Traceback (most recent call last):
  | 2022-07-04T06:51:20.953+01:00 | File "/usr/local/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 90, in __next__
  | 2022-07-04T06:51:20.953+01:00 | return next(self._wrapped)
  | 2022-07-04T06:51:20.953+01:00 | File "/usr/local/lib/python3.8/site-packages/google/ads/googleads/interceptors/response_wrappers.py", line 108, in __next__
  | 2022-07-04T06:51:20.953+01:00 | raise e
  | 2022-07-04T06:51:20.953+01:00 | File "/usr/local/lib/python3.8/site-packages/google/ads/googleads/interceptors/response_wrappers.py", line 105, in __next__
  | 2022-07-04T06:51:20.953+01:00 | self._failure_handler(self._underlay_call)
  | 2022-07-04T06:51:20.953+01:00 | File "/usr/local/lib/python3.8/site-packages/google/ads/googleads/interceptors/exception_interceptor.py", line 71, in _handle_grpc_failure
  | 2022-07-04T06:51:20.953+01:00 | raise self._get_error_from_response(response)
  | 2022-07-04T06:51:20.953+01:00 | File "/usr/local/lib/python3.8/site-packages/google/ads/googleads/interceptors/response_wrappers.py", line 88, in __next__
  | 2022-07-04T06:51:20.953+01:00 | message = next(self._underlay_call)
  | 2022-07-04T06:51:20.953+01:00 | File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 426, in __next__
  | 2022-07-04T06:51:20.953+01:00 | return self._next()
  | 2022-07-04T06:51:20.953+01:00 | File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 642, in _next
  | 2022-07-04T06:51:20.953+01:00 | return self._next_response()
  | 2022-07-04T06:51:20.953+01:00 | File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 617, in _next_response
  | 2022-07-04T06:51:20.953+01:00 | raise self
  | 2022-07-04T06:51:20.953+01:00 | grpc._channel._SingleThreadedRendezvous: <_SingleThreadedRendezvous of RPC that terminated with:
  | 2022-07-04T06:51:20.953+01:00 | status = StatusCode.UNKNOWN
  | 2022-07-04T06:51:20.953+01:00 | details = "Stream removed"
  | 2022-07-04T06:51:20.953+01:00 | debug_error_string = "{"created":"@1656913879.822245365","description":"Error received from peer ipv4:142.250.73.234:443","file":"src/core/lib/surface/call.cc","file_line":952,"grpc_message":"Stream removed","grpc_status":2}"
  | 2022-07-04T06:51:20.953+01:00 | >
  | 2022-07-04T06:51:20.953+01:00 | The above exception was the direct cause of the following exception:
  | 2022-07-04T06:51:20.953+01:00 | Traceback (most recent call last):
  | 2022-07-04T06:51:20.953+01:00 | File "/app/service/service.py", line 604, in GetResponsiveSearchAds
  | 2022-07-04T06:51:20.953+01:00 | for ad in self.GetAdwordsRSA(request):
  | 2022-07-04T06:51:20.953+01:00 | File "/app/service/service.py", line 654, in GetAdwordsRSA
  | 2022-07-04T06:51:20.953+01:00 | for row in generator:
  | 2022-07-04T06:51:20.953+01:00 | File "/app/adwords/reports.py", line 104, in stream_report_data
  | 2022-07-04T06:51:20.953+01:00 | for batch in stream:
  | 2022-07-04T06:51:20.953+01:00 | File "/usr/local/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 93, in __next__
  | 2022-07-04T06:51:20.953+01:00 | raise exceptions.from_grpc_error(exc) from exc
  | 2022-07-04T06:51:20.953+01:00 | google.api_core.exceptions.Unknown: None Stream removed
  | 2022-07-04T06:51:20.954+01:00 | Exception iterating responses: None Stream removed
  | 2022-07-04T06:51:20.954+01:00 | Traceback (most recent call last):
  | 2022-07-04T06:51:20.954+01:00 | File "/usr/local/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 90, in __next__
  | 2022-07-04T06:51:20.954+01:00 | return next(self._wrapped)
  | 2022-07-04T06:51:20.954+01:00 | File "/usr/local/lib/python3.8/site-packages/google/ads/googleads/interceptors/response_wrappers.py", line 108, in __next__
  | 2022-07-04T06:51:20.954+01:00 | raise e
  | 2022-07-04T06:51:20.954+01:00 | File "/usr/local/lib/python3.8/site-packages/google/ads/googleads/interceptors/response_wrappers.py", line 105, in __next__
  | 2022-07-04T06:51:20.954+01:00 | self._failure_handler(self._underlay_call)
  | 2022-07-04T06:51:20.954+01:00 | File "/usr/local/lib/python3.8/site-packages/google/ads/googleads/interceptors/exception_interceptor.py", line 71, in _handle_grpc_failure
  | 2022-07-04T06:51:20.954+01:00 | raise self._get_error_from_response(response)
  | 2022-07-04T06:51:20.954+01:00 | File "/usr/local/lib/python3.8/site-packages/google/ads/googleads/interceptors/response_wrappers.py", line 88, in __next__
  | 2022-07-04T06:51:20.954+01:00 | message = next(self._underlay_call)
  | 2022-07-04T06:51:20.954+01:00 | File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 426, in __next__
  | 2022-07-04T06:51:20.954+01:00 | return self._next()
  | 2022-07-04T06:51:20.954+01:00 | File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 642, in _next
  | 2022-07-04T06:51:20.954+01:00 | return self._next_response()
  | 2022-07-04T06:51:20.954+01:00 | File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 617, in _next_response
  | 2022-07-04T06:51:20.954+01:00 | raise self
  | 2022-07-04T06:51:20.954+01:00 | grpc._channel._SingleThreadedRendezvous: <_SingleThreadedRendezvous of RPC that terminated with:
  | 2022-07-04T06:51:20.954+01:00 | status = StatusCode.UNKNOWN
  | 2022-07-04T06:51:20.954+01:00 | details = "Stream removed"
  | 2022-07-04T06:51:20.954+01:00 | debug_error_string = "{"created":"@1656913879.822245365","description":"Error received from peer ipv4:142.250.73.234:443","file":"src/core/lib/surface/call.cc","file_line":952,"grpc_message":"Stream removed","grpc_status":2}"
  | 2022-07-04T06:51:20.954+01:00 | >
  | 2022-07-04T06:51:20.954+01:00 | The above exception was the direct cause of the following exception:
  | 2022-07-04T06:51:20.954+01:00 | Traceback (most recent call last):
  | 2022-07-04T06:51:20.954+01:00 | File "/usr/local/lib/python3.8/site-packages/grpc/_server.py", line 461, in _take_response_from_response_iterator
  | 2022-07-04T06:51:20.954+01:00 | return next(response_iterator), True
  | 2022-07-04T06:51:20.954+01:00 | File "/app/service/service.py", line 604, in GetResponsiveSearchAds
  | 2022-07-04T06:51:20.954+01:00 | for ad in self.GetAdwordsRSA(request):
  | 2022-07-04T06:51:20.954+01:00 | File "/app/service/service.py", line 654, in GetAdwordsRSA
  | 2022-07-04T06:51:20.954+01:00 | for row in generator:
  | 2022-07-04T06:51:20.954+01:00 | File "/app/adwords/reports.py", line 104, in stream_report_data
  | 2022-07-04T06:51:20.954+01:00 | for batch in stream:
  | 2022-07-04T06:51:20.954+01:00 | File "/usr/local/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 93, in __next__
  | 2022-07-04T06:51:20.954+01:00 | raise exceptions.from_grpc_error(exc) from exc
  | 2022-07-04T06:51:20.954+01:00 | google.api_core.exceptions.Unknown: None Stream removed
  | 2022-07-04T06:51:20.954+01:00 | ERROR:grpc._server:Exception iterating responses: None Stream removed
  | 2022-07-04T06:51:20.954+01:00 | Traceback (most recent call last):
  | 2022-07-04T06:51:20.954+01:00 | File "/usr/local/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 90, in __next__
  | 2022-07-04T06:51:20.954+01:00 | return next(self._wrapped)
  | 2022-07-04T06:51:20.954+01:00 | File "/usr/local/lib/python3.8/site-packages/google/ads/googleads/interceptors/response_wrappers.py", line 108, in __next__
  | 2022-07-04T06:51:20.954+01:00 | raise e
  | 2022-07-04T06:51:20.954+01:00 | File "/usr/local/lib/python3.8/site-packages/google/ads/googleads/interceptors/response_wrappers.py", line 105, in __next__
  | 2022-07-04T06:51:20.954+01:00 | self._failure_handler(self._underlay_call)
  | 2022-07-04T06:51:20.954+01:00 | File "/usr/local/lib/python3.8/site-packages/google/ads/googleads/interceptors/exception_interceptor.py", line 71, in _handle_grpc_failure
  | 2022-07-04T06:51:20.954+01:00 | raise self._get_error_from_response(response)
  | 2022-07-04T06:51:20.954+01:00 | File "/usr/local/lib/python3.8/site-packages/google/ads/googleads/interceptors/response_wrappers.py", line 88, in __next__
  | 2022-07-04T06:51:20.954+01:00 | message = next(self._underlay_call)
  | 2022-07-04T06:51:20.954+01:00 | File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 426, in __next__
  | 2022-07-04T06:51:20.954+01:00 | return self._next()
  | 2022-07-04T06:51:20.954+01:00 | File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 642, in _next
  | 2022-07-04T06:51:20.954+01:00 | return self._next_response()
  | 2022-07-04T06:51:20.954+01:00 | File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 617, in _next_response
  | 2022-07-04T06:51:20.954+01:00 | raise self
  | 2022-07-04T06:51:20.954+01:00 | grpc._channel._SingleThreadedRendezvous: <_SingleThreadedRendezvous of RPC that terminated with:
  | 2022-07-04T06:51:20.954+01:00 | status = StatusCode.UNKNOWN
  | 2022-07-04T06:51:20.954+01:00 | details = "Stream removed"
  | 2022-07-04T06:51:20.954+01:00 | debug_error_string = "{"created":"@1656913879.822245365","description":"Error received from peer ipv4:142.250.73.234:443","file":"src/core/lib/surface/call.cc","file_line":952,"grpc_message":"Stream removed","grpc_status":2}"
  | 2022-07-04T06:51:20.954+01:00 | >
  | 2022-07-04T06:51:20.954+01:00 | The above exception was the direct cause of the following exception:
  | 2022-07-04T06:51:20.954+01:00 | Traceback (most recent call last):
  | 2022-07-04T06:51:20.954+01:00 | File "/usr/local/lib/python3.8/site-packages/grpc/_server.py", line 461, in _take_response_from_response_iterator
  | 2022-07-04T06:51:20.954+01:00 | return next(response_iterator), True
  | 2022-07-04T06:51:20.954+01:00 | File "/app/service/service.py", line 604, in GetResponsiveSearchAds
  | 2022-07-04T06:51:20.954+01:00 | for ad in self.GetAdwordsRSA(request):
  | 2022-07-04T06:51:20.954+01:00 | File "/app/service/service.py", line 654, in GetAdwordsRSA
  | 2022-07-04T06:51:20.954+01:00 | for row in generator:
  | 2022-07-04T06:51:20.954+01:00 | File "/app/adwords/reports.py", line 104, in stream_report_data
  | 2022-07-04T06:51:20.954+01:00 | for batch in stream:
  | 2022-07-04T06:51:20.954+01:00 | File "/usr/local/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 93, in __next__
  | 2022-07-04T06:51:20.954+01:00 | raise exceptions.from_grpc_error(exc) from exc
  | 2022-07-04T06:51:20.954+01:00 | google.api_core.exceptions.Unknown: None Stream removed

Anything else we should know about your project / environment: Python version 3.8.13

BenRKarl commented 2 years ago

@nev7n - this is likely due to a timeout on the server side. Do you know about how long the stream stays open before this error is encountered? I believe there is a 60 minute timeout on our end that may cause this. In general the only way to fix this is to retry the request, but if it's happening consistently at around the 60 minute mark I would try splitting your request into smaller ones and aggregating the results on the client side if necessary.

nev7n commented 2 years ago

@BenRKarl - yes the stream was being closed after 60 minutes so sounds like it was hitting the 60 minute server side timeout, the stream of data seems to have hung sending data a while before that so I think it's possibly hit some either server side size/timeout limit before that.

In any case we worked around the issue by downloading the RSA ads one campaign at a time, and given it seems to be caused by server side limits I think this is unlikely to be an issue with the google ads python library.