Closed BBalazsJurgen closed 2 years ago
BenRKarl
commented
2 years ago Hi @BBalazsJurgen thanks for sharing all of this information, and sorry that you're running into this problem. As you mentioned we typically don't recommend making requests with the python library using threading and/or multiprocessing because of how the streaming functionality works at low levels. My recommendation in this case would be to implement some retry logic anytime you encounter one of the above errors.
zachliu
commented
2 years ago @BenRKarl in our case (we are using python's multiprocessing pool with ~300 workers), the api stops responding and the program hangs forever after a few retries :crying_cat_face:
could you elaborate on how the streaming functionality works at low levels?
multiprocessing is pretty much the only arrow in our quiver to battle against querying a large # of clients
we'd like to understand the reason behind it
Dear Support,
We have been directed to you by the central Google Ads API group.
Describe the bug: The precedent thread is available here: https://groups.google.com/g/adwords-api/c/Q-DvgmZI4K0 In short, the Google Ads API currently does not support returning aggregate data on the client level, segmented by campaign type (advertising_channel_type). The only option is to query at the campaign level and ask for SUMMARY_ROW_ONLY. This way however, we need several SearchStream requests, one for each campaign type, and with different time periods needed, we quickly end up with (currently) 21 SearchStream requests per client account. With multiple clients, you can see this racking up into the thousands of requests very quickly.
With advice from the Google Ads API team, we went with the many-many requests, as we had no other option. However, doing everything in a linear fashion, waiting for one request to finish and starting the other would obviously take massive hours to finish. Therefore, we decided to opt for using Threading and make simultaneous calls. This works amazingly well for manageable call counts, but in this case, sometimes we get errors. I know, making thousands of simultaneous requests is a bad idea; but we are open to alternatives, yet we had been advised that the mass requests way is the right way.
Do note that this error does not occur reliably, all the time. Sometimes, we have no issue making thousands of requests and everyhting works like a charm. However, sometimes we get the following error.
Steps to Reproduce: Use Threading to send thousands of simultaneous SearchStream requests.
Expected behavior: Thousands of simultaneous SearchStream answers.
Client library version and API version: Client library version: 17.0.0 Google Ads API version: v11
Request/Response Logs: NOTE that the full request/response logs would be WAY TOO LONG. However, the notable part seems to be the following: `report = pd.DataFrame(gafunctions.ads_get_report(cid, query, 1, summary_row_setting)) File "*\venv\lib\site-packages\pt_tools\gafunctions.py", line 41, in ads_get_report File "*\venv\lib\site-packages\google\api_core\grpc_helpers.py", line 144, in error_remapped_callable response = ga_service.search_stream(request=request) raise exceptions.from_grpc_error(exc) from exc google.api_core.exceptions.ServiceUnavailable: 503 failed to connect to all addresses response = ga_service.search_stream(request=request) for data in response: File "*\venv\lib\site-packages\pt_tools\gafunctions.py", line 100, in _getdata response = ga_service.search_stream(customer_id=customer_id, query=query) File "*\venv\lib\site-packages\google\ads\googleads\v11\services\services\google_ads_service\client.py", line 3426, in search_stream File "***\venv\lib\site-packages\google\ads\googleads\v11\services\services\google_ads_service\client.py", line 3426, in search_stream
File "*\venv\lib\site-packages\google\ads\googleads\v11\services\services\google_ads_service\client.py", line 3426, in search_stream raise exceptions.from_grpc_error(exc) from exc google.api_core.exceptions.ServiceUnavailable: 503 failed to connect to all addresses response = rpc( File "*\venv\lib\site-packages\google\api_core\gapic_v1\method.py", line 154, in call return wrapped_func(args, kwargs) File "\\venv\lib\site-packages\google\api_core\grpc_helpers.py", line 144, in error_remapped_callable response = rpc( File "*\venv\lib\site-packages\google\api_core\gapic_v1\method.py", line 154, in call return wrapped_func(*args, *kwargs) File "\\venv\lib\site-packages\google\api_core\grpc_helpers.py", line 144, in error_remapped_callable raise exceptions.from_grpc_error(exc) from exc google.api_core.exceptions.ServiceUnavailable: 503 failed to connect to all addresses raise exceptions.from_grpc_error(exc) from exc google.api_core.exceptions.ServiceUnavailable: 503 failed to connect to all addresses response = rpc( File "*\venv\lib\site-packages\google\api_core\gapic_v1\method.py", line 154, in call return wrapped_func(*args, *kwargs) File "\\venv\lib\site-packages\google\api_core\grpc_helpers.py", line 144, in error_remapped_callable raise exceptions.from_grpc_error(exc) from exc google.api_core.exceptions.ServiceUnavailable: 503 failed to connect to all addresses`
Please be so kind as to assist us in this matter.