search

googleads / google-ads-java

Google Ads API Client Library for Java
Apache License 2.0
170 stars 177 forks source link

High Rate of Deadline Exceeded and Internal Error #671

Closed PeterLavetsky closed 2 years ago

PeterLavetsky commented 2 years ago

Overnight we started seeing an elevated rate of DEADLINE_EXCEEDED errors when querying SearchStream and now we're seeing an extremely high rate of "Internal error occurred"

./deadline-exceeded-test.sh 
/efs/api/google/requests/new-google-requests-summary.2022-09-06.log 103
/efs/api/google/requests/new-google-requests-summary.2022-09-07.log 155
/efs/api/google/requests/new-google-requests-summary.2022-09-08.log 649
./internal-error-test.sh 
/efs/api/google/requests/new-google-requests-summary.2022-09-06.log 67
/efs/api/google/requests/new-google-requests-summary.2022-09-07.log 449
/efs/api/google/requests/new-google-requests-summary.2022-09-08.log 10579
traceroute to googleads.googleapis.com (172.217.0.170), 50 hops max, 60 byte packets
 1  ip-172-31-67-206.us-east-2.compute.internal (172.31.67.206)  0.126 ms  0.108 ms  0.101 ms
 2  ec2-52-15-0-41.us-east-2.compute.amazonaws.com (52.15.0.41)  7.318 ms ec2-52-15-0-53.us-east-2.compute.amazonaws.com (52.15.0.53)  6.556 ms ec2-52-15-0-45.us-east-2.compute.amazonaws.com (52.15.0.45)  8.382 ms
 3  240.1.236.33 (240.1.236.33)  0.954 ms 100.65.27.128 (100.65.27.128)  4.780 ms 100.65.24.144 (100.65.24.144)  2.218 ms
 4  100.66.12.8 (100.66.12.8)  3.368 ms 100.66.12.68 (100.66.12.68)  3.799 ms 100.66.12.238 (100.66.12.238)  7.478 ms
 5  100.66.14.140 (100.66.14.140)  15.989 ms 100.66.15.8 (100.66.15.8)  19.229 ms  19.202 ms
 6  241.0.12.71 (241.0.12.71)  0.794 ms 241.0.12.74 (241.0.12.74)  0.784 ms 241.0.12.68 (241.0.12.68)  0.940 ms
 7  108.166.244.19 (108.166.244.19)  0.827 ms 108.166.244.22 (108.166.244.22)  1.195 ms 108.166.244.29 (108.166.244.29)  0.924 ms
 8  242.0.79.253 (242.0.79.253)  1.776 ms 242.0.79.125 (242.0.79.125)  1.703 ms 242.0.79.101 (242.0.79.101)  0.815 ms
 9  15.230.134.203 (15.230.134.203)  2.959 ms 15.230.134.219 (15.230.134.219)  2.780 ms 52.95.3.127 (52.95.3.127)  1.755 ms
10  52.95.1.166 (52.95.1.166)  3.191 ms 52.95.1.194 (52.95.1.194)  5.407 ms 15.230.39.180 (15.230.39.180)  2.583 ms
11  52.95.1.233 (52.95.1.233)  3.242 ms 52.95.2.99 (52.95.2.99)  2.182 ms 52.95.2.29 (52.95.2.29)  9.112 ms
12  100.91.177.207 (100.91.177.207)  12.156 ms 100.91.176.189 (100.91.176.189)  11.912 ms 100.91.177.111 (100.91.177.111)  11.722 ms
13  100.100.8.97 (100.100.8.97)  12.143 ms 100.100.8.91 (100.100.8.91)  12.232 ms 100.100.8.81 (100.100.8.81)  12.238 ms
14  100.100.94.70 (100.100.94.70)  12.156 ms 100.100.93.136 (100.100.93.136)  12.335 ms 100.100.80.136 (100.100.80.136)  12.311 ms
15  100.100.81.163 (100.100.81.163)  14.644 ms 100.100.78.99 (100.100.78.99)  12.759 ms 100.100.76.131 (100.100.76.131)  12.204 ms
16  100.100.34.40 (100.100.34.40)  13.322 ms 100.100.4.0 (100.100.4.0)  12.066 ms 100.100.4.2 (100.100.4.2)  11.927 ms
17  99.83.113.93 (99.83.113.93)  13.970 ms  13.962 ms 99.83.65.1 (99.83.65.1)  12.113 ms
18  * 108.170.246.67 (108.170.246.67)  12.543 ms *
19  142.251.53.2 (142.251.53.2)  12.394 ms * 142.251.64.102 (142.251.64.102)  12.203 ms
20  108.170.240.112 (108.170.240.112)  12.703 ms 108.170.246.66 (108.170.246.66)  12.628 ms 108.170.246.3 (108.170.246.3)  12.705 ms
21  142.251.49.77 (142.251.49.77)  13.374 ms 108.170.232.199 (108.170.232.199)  14.782 ms *
22  142.251.234.37 (142.251.234.37)  18.029 ms * 142.251.49.95 (142.251.49.95)  13.271 ms
23  142.251.64.253 (142.251.64.253)  18.219 ms 142.251.64.255 (142.251.64.255)  19.809 ms *
24  142.251.234.41 (142.251.234.41)  18.715 ms 72.14.232.108 (72.14.232.108)  18.388 ms 142.251.234.68 (142.251.234.68)  19.304 ms
25  ord38s42-in-f10.1e100.net (172.217.0.170)  18.513 ms 108.170.243.174 (108.170.243.174)  20.021 ms mia09s16-in-f10.1e100.net (172.217.0.170)  18.654 ms
Request
-------
MethodName: google.ads.googleads.v11.services.GoogleAdsService/SearchStream
Endpoint: googleads.googleapis.com:443
Headers: {developer-token=REDACTED, login-customer-id=5864512560, x-goog-api-client=gl-java/11.0.15 gccl/21.0.0 gapic/21.0.0 gax/2.16.0 grpc/1.45.1 pb/3.19.4}
Body: customer_id: "9292695725"
query: "select accessible_bidding_strategy.id, accessible_bidding_strategy.maximize_conversion_value.target_roas, accessible_bidding_strategy.maximize_conversions.target_cpa_micros, accessible_bidding_strategy.name, accessible_bidding_strategy.owner_customer_id, accessible_bidding_strategy.owner_descriptive_name, accessible_bidding_strategy.resource_name, accessible_bidding_strategy.target_cpa.target_cpa_micros, accessible_bidding_strategy.target_impression_share.cpc_bid_ceiling_micros, accessible_bidding_strategy.target_impression_share.location, accessible_bidding_strategy.target_impression_share.location_fraction_micros, accessible_bidding_strategy.target_roas.target_roas, accessible_bidding_strategy.target_spend.cpc_bid_ceiling_micros, accessible_bidding_strategy.target_spend.target_spend_micros, accessible_bidding_strategy.type, bidding_strategy.currency_code, bidding_strategy.effective_currency_code, bidding_strategy.enhanced_cpc, bidding_strategy.id, bidding_strategy.maximize_conversion_value.cpc_bid_ceiling_micros, bidding_strategy.maximize_conversion_value.cpc_bid_floor_micros, bidding_strategy.maximize_conversion_value.target_roas, bidding_strategy.maximize_conversions.cpc_bid_ceiling_micros, bidding_strategy.maximize_conversions.cpc_bid_floor_micros, bidding_strategy.maximize_conversions.target_cpa_micros, bidding_strategy.name, bidding_strategy.resource_name, bidding_strategy.status, bidding_strategy.target_cpa.cpc_bid_ceiling_micros, bidding_strategy.target_cpa.cpc_bid_floor_micros, bidding_strategy.target_cpa.target_cpa_micros, bidding_strategy.target_impression_share.cpc_bid_ceiling_micros, bidding_strategy.target_impression_share.location, bidding_strategy.target_impression_share.location_fraction_micros, bidding_strategy.target_roas.cpc_bid_ceiling_micros, bidding_strategy.target_roas.cpc_bid_floor_micros, bidding_strategy.target_roas.target_roas, bidding_strategy.target_spend.cpc_bid_ceiling_micros, bidding_strategy.target_spend.target_spend_micros, bidding_strategy.type, campaign.accessible_bidding_strategy, campaign.ad_serving_optimization_status, campaign.advertising_channel_sub_type, campaign.advertising_channel_type, campaign.app_campaign_setting.app_id, campaign.app_campaign_setting.app_store, campaign.app_campaign_setting.bidding_strategy_goal_type, campaign.audience_setting.use_audience_grouped, campaign.base_campaign, campaign.bidding_strategy, campaign.bidding_strategy_system_status, campaign.bidding_strategy_type, campaign.campaign_budget, campaign.campaign_group, campaign.commission.commission_rate_micros, campaign.dynamic_search_ads_setting.domain_name, campaign.dynamic_search_ads_setting.feeds, campaign.dynamic_search_ads_setting.language_code, campaign.dynamic_search_ads_setting.use_supplied_urls_only, campaign.end_date, campaign.excluded_parent_asset_field_types, campaign.experiment_type, campaign.final_url_suffix, campaign.frequency_caps, campaign.geo_target_type_setting.negative_geo_target_type, campaign.geo_target_type_setting.positive_geo_target_type, campaign.hotel_setting.hotel_center_id, campaign.id, campaign.local_campaign_setting.location_source_type, campaign.local_services_campaign_settings.category_bids, campaign.manual_cpa, campaign.manual_cpc.enhanced_cpc_enabled, campaign.manual_cpm, campaign.manual_cpv, campaign.maximize_conversion_value.target_roas, campaign.maximize_conversions.target_cpa_micros, campaign.name, campaign.network_settings.target_content_network, campaign.network_settings.target_google_search, campaign.network_settings.target_partner_search_network, campaign.network_settings.target_search_network, campaign.optimization_goal_setting.optimization_goal_types, campaign.optimization_score, campaign.payment_mode, campaign.percent_cpc.cpc_bid_ceiling_micros, campaign.percent_cpc.enhanced_cpc_enabled, campaign.performance_max_upgrade.performance_max_campaign, campaign.performance_max_upgrade.pre_upgrade_campaign, campaign.performance_max_upgrade.status, campaign.real_time_bidding_setting.opt_in, campaign.resource_name, campaign.selective_optimization.conversion_actions, campaign.serving_status, campaign.shopping_setting.campaign_priority, campaign.shopping_setting.enable_local, campaign.shopping_setting.feed_label, campaign.shopping_setting.merchant_id, campaign.shopping_setting.sales_country, campaign.shopping_setting.use_vehicle_inventory, campaign.start_date, campaign.status, campaign.target_cpa.cpc_bid_ceiling_micros, campaign.target_cpa.cpc_bid_floor_micros, campaign.target_cpa.target_cpa_micros, campaign.target_cpm, campaign.target_impression_share.cpc_bid_ceiling_micros, campaign.target_impression_share.location, campaign.target_impression_share.location_fraction_micros, campaign.target_roas.cpc_bid_ceiling_micros, campaign.target_roas.cpc_bid_floor_micros, campaign.target_roas.target_roas, campaign.target_spend.cpc_bid_ceiling_micros, campaign.target_spend.target_spend_micros, campaign.targeting_setting.target_restrictions, campaign.tracking_setting.tracking_url, campaign.tracking_url_template, campaign.url_custom_parameters, campaign.url_expansion_opt_out, campaign.vanity_pharma.vanity_pharma_display_url_mode, campaign.vanity_pharma.vanity_pharma_text, campaign.video_brand_safety_suitability, campaign_budget.amount_micros, campaign_budget.delivery_method, campaign_budget.explicitly_shared, campaign_budget.has_recommended_budget, campaign_budget.id, campaign_budget.name, campaign_budget.period, campaign_budget.recommended_budget_amount_micros, campaign_budget.recommended_budget_estimated_change_weekly_clicks, campaign_budget.recommended_budget_estimated_change_weekly_cost_micros, campaign_budget.recommended_budget_estimated_change_weekly_interactions, campaign_budget.recommended_budget_estimated_change_weekly_views, campaign_budget.reference_count, campaign_budget.resource_name, campaign_budget.status, campaign_budget.total_amount_micros, campaign_budget.type, campaign_group.id, campaign_group.name, campaign_group.resource_name, campaign_group.status from campaign where campaign.id   =   \"17796259637\" "

Response
--------
Headers: Metadata(content-type=application/grpc,request-id=_vBEkOHCQh0U_eja49is-g,date=Thu, 08 Sep 2022 21:45:50 GMT,alt-svc=h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43")
Body: null
Failure message: errors {
  error_code {
    internal_error: INTERNAL_ERROR
  }
  message: "An internal error has occurred."
}
request_id: "_vBEkOHCQh0U_eja49is-g"

Status: Status{code=INTERNAL, description=Internal error encountered., cause=null}.
request-id=_vBEkOHCQh0U_eja49is-g

The Internal Errors rate started elevating after 4pm EST

Pete

PeterLavetsky commented 2 years ago

Many others reporting this on the forum

https://groups.google.com/g/adwords-api/c/Zcyg7c1X8jY
AnashOommen commented 2 years ago

I think this was an issue that has been fixed at our end.

On Thu, Sep 8, 2022, 6:07 PM Peter Lavetsky @.***> wrote:

Many others reporting this on the forum

https://groups.google.com/g/adwords-api/c/Zcyg7c1X8jY

— Reply to this email directly, view it on GitHub https://github.com/googleads/google-ads-java/issues/671#issuecomment-1241285733, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABCOJCBRE5CXOQCJBNRXW7TV5JPTBANCNFSM6AAAAAAQIE3I3Q . You are receiving this because you are subscribed to this thread.Message ID: @.***>

PeterLavetsky commented 2 years ago

Good to hear Anash! We're no longer seeing the elevated failure rates from before.

Thanks Pete

PeterLavetsky commented 2 years ago

We're starting to see these numbers elevate again:

./deadline-exceeded-test.sh 
/efs/api/google/requests/new-google-requests-summary.2022-09-12.log 73
/efs/api/google/requests/new-google-requests-summary.2022-09-13.log 192
/efs/api/google/requests/new-google-requests-summary.2022-09-14.log 994
./internal-error-test.sh 
/efs/api/google/requests/new-google-requests-summary.2022-09-12.log 14
/efs/api/google/requests/new-google-requests-summary.2022-09-13.log 138
/efs/api/google/requests/new-google-requests-summary.2022-09-14.log 544
traceroute to googleads.googleapis.com (142.250.191.234), 50 hops max, 60 byte packets
 1  ip-172-31-130-116.us-east-2.compute.internal (172.31.130.116)  0.120 ms  0.110 ms  0.123 ms
 2  ec2-52-15-0-183.us-east-2.compute.amazonaws.com (52.15.0.183)  8.616 ms ec2-52-15-0-161.us-east-2.compute.amazonaws.com (52.15.0.161)  40.989 ms ec2-52-15-0-187.us-east-2.compute.amazonaws.com (52.15.0.187)  2.709 ms
 3  100.65.24.48 (100.65.24.48)  14.815 ms 100.65.30.16 (100.65.30.16)  7.087 ms 100.65.28.144 (100.65.28.144)  8.693 ms
 4  100.66.13.120 (100.66.13.120)  1.701 ms 100.66.12.200 (100.66.12.200)  38.767 ms 100.66.12.134 (100.66.12.134)  5.486 ms
 5  100.66.14.78 (100.66.14.78)  38.082 ms 100.66.14.8 (100.66.14.8)  17.437 ms 100.66.14.32 (100.66.14.32)  162.246 ms
 6  241.0.12.200 (241.0.12.200)  0.666 ms 241.0.12.205 (241.0.12.205)  0.700 ms 241.0.12.202 (241.0.12.202)  0.736 ms
 7  108.166.252.59 (108.166.252.59)  0.634 ms 108.166.252.53 (108.166.252.53)  0.752 ms 108.166.252.48 (108.166.252.48)  0.787 ms
 8  108.166.252.50 (108.166.252.50)  0.776 ms 108.166.252.52 (108.166.252.52)  0.669 ms 108.166.252.50 (108.166.252.50)  0.675 ms
 9  242.0.107.65 (242.0.107.65)  1.229 ms 242.0.106.77 (242.0.106.77)  1.302 ms 242.0.106.205 (242.0.106.205)  1.298 ms
10  15.230.134.116 (15.230.134.116)  2.635 ms 15.230.134.148 (15.230.134.148)  3.372 ms 15.230.134.124 (15.230.134.124)  2.966 ms
11  15.230.140.95 (15.230.140.95)  1.411 ms 52.95.1.142 (52.95.1.142)  1.918 ms 15.230.140.131 (15.230.140.131)  1.350 ms
12  52.93.239.82 (52.93.239.82)  1.126 ms 52.95.2.1 (52.95.2.1)  1.502 ms 52.93.239.100 (52.93.239.100)  4.611 ms
13  100.91.177.197 (100.91.177.197)  14.115 ms 100.91.176.199 (100.91.176.199)  11.614 ms 100.91.177.39 (100.91.177.39)  16.294 ms
14  100.100.6.51 (100.100.6.51)  11.637 ms 100.100.8.3 (100.100.8.3)  12.646 ms 100.100.8.119 (100.100.8.119)  11.611 ms
15  100.100.93.200 (100.100.93.200)  11.347 ms 100.100.78.70 (100.100.78.70)  11.588 ms 100.100.73.134 (100.100.73.134)  11.452 ms
16  100.100.88.29 (100.100.88.29)  12.192 ms 100.100.72.221 (100.100.72.221)  11.631 ms 100.100.77.163 (100.100.77.163)  11.464 ms
17  100.100.28.108 (100.100.28.108)  11.576 ms 100.100.4.4 (100.100.4.4)  11.485 ms 100.100.28.2 (100.100.28.2)  14.864 ms
18  100.95.7.33 (100.95.7.33)  11.420 ms  11.290 ms 99.83.65.3 (99.83.65.3)  11.357 ms
19  108.170.246.49 (108.170.246.49)  12.027 ms 100.100.4.26 (100.100.4.26)  11.378 ms 100.100.4.24 (100.100.4.24)  11.199 ms
20  142.251.67.234 (142.251.67.234)  11.638 ms 108.170.232.199 (108.170.232.199)  12.735 ms 99.83.113.93 (99.83.113.93)  12.669 ms
21  * * 108.170.246.66 (108.170.246.66)  12.579 ms
22  216.239.50.93 (216.239.50.93)  12.536 ms 216.239.35.162 (216.239.35.162)  12.523 ms 142.251.64.249 (142.251.64.249)  19.683 ms
23  108.170.246.67 (108.170.246.67)  12.603 ms 108.170.246.2 (108.170.246.2)  11.534 ms 172.253.71.187 (172.253.71.187)  12.930 ms
24  216.239.48.101 (216.239.48.101)  12.564 ms 142.251.65.3 (142.251.65.3)  18.983 ms 142.251.78.159 (142.251.78.159)  18.328 ms
25  * 142.250.209.75 (142.250.209.75)  13.272 ms 142.251.234.162 (142.251.234.162)  18.431 ms
26  * 108.170.243.174 (108.170.243.174)  19.876 ms *
27  209.85.241.125 (209.85.241.125)  17.802 ms 142.251.60.17 (142.251.60.17)  17.944 ms ord38s32-in-f10.1e100.net (142.250.191.234)  17.985 ms
 5  100.66.14.78 (100.66.14.78)  38.082 ms 100.66.14.8 (100.66.14.8)  17.437 ms 100.66.14.32 (100.66.14.32)  162.246 ms

Pete

jradcliff commented 2 years ago

Hi Pete,

We're looking into this and I'll get back to you. Thanks for highlighting it.

-Josh

PeterLavetsky commented 2 years ago

Hey Josh,

The errors seem to have backed off, but over the last two days we've seen very anomalous mutate durations for two specific calls:

CampaignBudgetServiceClient mutateCampaignBudgets CampaignServiceClient mutateCampaigns

Here are the last couple days worth of average job times ( in seconds ) for mutateCampaignBudgets jobs ... these times are not strictly the mutate call, but I know from additional metric collection that the majority of the time is on the campaignBudgetServiceClient.mutateCampaignBudgets( request ) call

2022-09-14  42.42
2022-09-13  8.8
2022-09-12  8.82
2022-09-11  7.3
2022-09-10  6.35
2022-09-09  7.41

Same caveats for campaignServiceClient.mutateCampaigns( request )

2022-09-14  154.06
2022-09-13  38.04
2022-09-12  32.47
2022-09-11  27.55
2022-09-10  25.71
2022-09-09  30

Pete

jradcliff commented 2 years ago

Hi Pete,

The issues with GoogleAdsService.searchStream were due to a temporary problem that was resolved within 24 hours. Are you still seeing issues with streaming requests or the mutate requests you mentioned?

Thanks

PeterLavetsky commented 2 years ago

Hi Josh,

Things seem back to normal.

Thanks Pete

jradcliff commented 2 years ago

Thanks for confirming. Will close this one again then.

PeterLavetsky commented 2 years ago

Josh,

It gives me no pleasure to report that soon after our last correspondence this issue has come back up, with an additional kicker of getting errors of RESOURCE_EXHAUSTED:

Caused by: com.google.ads.googleads.v11.errors.GoogleAdsException: errors {
error_code {
quota_error: RESOURCE_EXHAUSTED
}
message: "Too many requests. Retry in 900 seconds."
}
request_id: "ATmfNRHuaclFOBdipYLP9Q"

We have not changed any code, nor significantly increased our request load.

GoogleAdsServiceClient searchStreamCallable requests started seeing a noticeable slowdown starting around 5pm EST on 9/26, soon after we closed this issue down. Around 2am EST on 9/27 the response times were about 400% slower and at 4am EST this morning the search response times were about 2,000% slower than what we normally enjoy.

The following hours are UTC and the times noted are in seconds.

Day.                Hour Average Job Time
2022-09-26  8   1.9725
2022-09-26  9   1.932
2022-09-26  10  2.9016
2022-09-26  11  1.8385
2022-09-26  12  2.1983
2022-09-26  13  2.5466
2022-09-26  14  2.674
2022-09-26  15  1.9535
2022-09-26  16  1.6244
2022-09-26  17  1.4612
2022-09-26  18  1.4646
2022-09-26  19  1.3422
2022-09-26  20  1.5366
2022-09-26  21  3.6096
2022-09-26  22  4.02
2022-09-26  23  3.0292
2022-09-27  0   2.4258
2022-09-27  1   4.9145
2022-09-27  2   4.6897
2022-09-27  3   6.3486
2022-09-27  4   4.0788
2022-09-27  5   4.1629
2022-09-27  6   10.0844
2022-09-27  7   24.1455
2022-09-27  8   43.9109
./resource-exhausted-test.sh 
/efs/api/google/requests/new-google-requests-summary.2022-09-25.log 0
/efs/api/google/requests/new-google-requests-summary.2022-09-26.log 0
/efs/api/google/requests/new-google-requests-summary.2022-09-27.log 496
./deadline-exceeded-test.sh 
/efs/api/google/requests/new-google-requests-summary.2022-09-25.log 40
/efs/api/google/requests/new-google-requests-summary.2022-09-26.log 37
/efs/api/google/requests/new-google-requests-summary.2022-09-27.log 404

We're certainly not here to report every minor fluctuation in service response, but these do look to be significantly anomalous and we ( and I assume many others ) are currently being severely impacted by a degraded search service.

One specific GoogleAdsServiceClient searchStreamCallable heavy job that routinely takes under 2 hours to complete is now entering it's 5th hour with about 25% of its workload still to be completed.

As always, any additional details you may request I am happy to provide and any guidance you may offer will be gratefully accepted.

Thanks Pete

traceroute to googleads.googleapis.com (142.250.191.170), 50 hops max, 60 byte packets
 1  ip-172-31-130-116.us-east-2.compute.internal (172.31.130.116)  0.108 ms  0.098 ms  0.104 ms
 2  ec2-52-15-0-169.us-east-2.compute.amazonaws.com (52.15.0.169)  5.412 ms ec2-52-15-0-191.us-east-2.compute.amazonaws.com (52.15.0.191)  46.487 ms ec2-52-15-0-167.us-east-2.compute.amazonaws.com (52.15.0.167)  3.923 ms
 3  100.65.24.0 (100.65.24.0)  2.172 ms 100.65.30.32 (100.65.30.32)  10.553 ms  10.540 ms
 4  100.66.13.28 (100.66.13.28)  63.058 ms 100.66.12.192 (100.66.12.192)  7.355 ms 100.66.13.26 (100.66.13.26)  1.087 ms
 5  100.66.15.136 (100.66.15.136)  12.929 ms 100.66.15.166 (100.66.15.166)  55.441 ms 100.66.15.140 (100.66.15.140)  22.519 ms
 6  241.0.12.199 (241.0.12.199)  0.704 ms 241.0.12.193 (241.0.12.193)  0.698 ms 241.0.12.195 (241.0.12.195)  0.672 ms
 7  108.166.252.50 (108.166.252.50)  0.654 ms 108.166.252.62 (108.166.252.62)  0.713 ms 108.166.252.60 (108.166.252.60)  0.642 ms
 8  108.166.252.59 (108.166.252.59)  0.754 ms 108.166.252.53 (108.166.252.53)  0.645 ms 108.166.252.49 (108.166.252.49)  0.747 ms
 9  242.0.107.217 (242.0.107.217)  0.822 ms 242.0.106.73 (242.0.106.73)  1.102 ms 242.0.107.221 (242.0.107.221)  1.057 ms
10  15.230.135.107 (15.230.135.107)  3.655 ms  12.295 ms 15.230.135.81 (15.230.135.81)  1.552 ms
11  15.230.39.252 (15.230.39.252)  1.203 ms 52.95.0.232 (52.95.0.232)  7.213 ms 52.95.0.254 (52.95.0.254)  2.002 ms
12  52.93.239.112 (52.93.239.112)  6.899 ms 52.95.2.183 (52.95.2.183)  10.295 ms 52.95.2.127 (52.95.2.127)  10.133 ms
13  100.91.177.189 (100.91.177.189)  11.407 ms 100.91.177.221 (100.91.177.221)  11.288 ms 100.91.177.241 (100.91.177.241)  11.186 ms
14  100.100.6.27 (100.100.6.27)  11.601 ms 100.100.6.101 (100.100.6.101)  11.286 ms 100.100.6.41 (100.100.6.41)  11.590 ms
15  100.100.73.70 (100.100.73.70)  11.528 ms 100.100.80.136 (100.100.80.136)  11.526 ms 100.100.78.8 (100.100.78.8)  22.043 ms
16  100.100.94.35 (100.100.94.35)  11.481 ms 100.100.69.93 (100.100.69.93)  38.977 ms 100.100.77.157 (100.100.77.157)  11.685 ms
17  100.100.4.16 (100.100.4.16)  11.238 ms 100.100.4.10 (100.100.4.10)  11.524 ms 100.100.2.32 (100.100.2.32)  11.288 ms
18  99.83.65.3 (99.83.65.3)  11.221 ms 99.83.113.93 (99.83.113.93)  12.408 ms 99.83.68.209 (99.83.68.209)  12.021 ms
19  * * *
20  142.251.77.60 (142.251.77.60)  11.317 ms 99.83.65.3 (99.83.65.3)  11.487 ms 108.170.240.97 (108.170.240.97)  12.560 ms
21  * * *
22  142.251.49.77 (142.251.49.77)  12.991 ms 142.251.49.192 (142.251.49.192)  12.900 ms 216.239.48.101 (216.239.48.101)  12.655 ms
23  142.251.49.199 (142.251.49.199)  13.318 ms 172.253.67.43 (172.253.67.43)  13.336 ms *
24  142.251.64.249 (142.251.64.249)  19.210 ms 142.251.65.3 (142.251.65.3)  18.253 ms 142.251.65.7 (142.251.65.7)  18.985 ms
25  172.253.74.193 (172.253.74.193)  12.259 ms 142.250.209.75 (142.250.209.75)  77.467 ms 209.85.250.34 (209.85.250.34)  18.313 ms
26  108.170.243.174 (108.170.243.174)  18.195 ms  18.053 ms 108.170.243.193 (108.170.243.193)  17.312 ms
27  209.85.249.137 (209.85.249.137)  18.937 ms 142.251.60.9 (142.251.60.9)  18.035 ms 216.239.40.188 (216.239.40.188)  17.422 ms
28  108.170.243.174 (108.170.243.174)  17.887 ms ord38s30-in-f10.1e100.net (142.250.191.170)  17.127 ms  17.126 ms
jradcliff commented 2 years ago

Hi Pete,

The root cause of the latest issue you highlighted has been fixed and traffic should be back to normal now. Appreciate you sharing the results of your monitoring as well. I'm working with the team to improve monitoring on our side (based on what we discovered as a result of the most recent problem) so we can be alerted to these issues sooner.

Thanks, Josh

PeterLavetsky commented 1 year ago

Hi All,

Just wanted to highlight that we're currently getting a very high rate of Deadline Exceeded exceptions

google-requests-summary.2023-06-19.log 12447
google-requests-summary.2023-06-18.log 10
google-requests-summary.2023-06-17.log 349
google-requests-summary.2023-06-16.log 91
google-requests-summary.2023-06-15.log 56
traceroute to googleads.googleapis.com (142.250.191.202), 50 hops max, 60 byte packets
 1  ip-172-31-67-206.us-east-2.compute.internal (172.31.67.206)  0.129 ms  0.116 ms  0.119 ms
 2  ec2-52-15-0-51.us-east-2.compute.amazonaws.com (52.15.0.51)  9.925 ms ec2-52-15-0-31.us-east-2.compute.amazonaws.com (52.15.0.31)  3.526 ms ec2-52-15-0-39.us-east-2.compute.amazonaws.com (52.15.0.39)  5.206 ms
 3  100.65.30.128 (100.65.30.128)  30.440 ms 100.65.25.128 (100.65.25.128)  4.696 ms 100.65.31.160 (100.65.31.160)  5.844 ms
 4  100.66.13.254 (100.66.13.254)  7.660 ms 100.66.13.44 (100.66.13.44)  4.232 ms 100.66.12.160 (100.66.12.160)  53.682 ms
 5  100.66.14.234 (100.66.14.234)  19.706 ms 100.66.14.36 (100.66.14.36)  15.285 ms 100.66.14.230 (100.66.14.230)  47.163 ms
 6  241.0.12.70 (241.0.12.70)  0.788 ms 241.0.12.74 (241.0.12.74)  0.756 ms 241.0.12.75 (241.0.12.75)  0.747 ms
 7  242.0.76.39 (242.0.76.39)  1.390 ms 242.0.77.39 (242.0.77.39)  2.196 ms 242.0.76.161 (242.0.76.161)  1.457 ms
 8  240.0.236.12 (240.0.236.12)  12.261 ms 240.0.236.34 (240.0.236.34)  12.290 ms 240.0.236.33 (240.0.236.33)  12.554 ms
 9  240.0.184.2 (240.0.184.2)  12.218 ms 240.0.184.6 (240.0.184.6)  12.185 ms 240.0.184.4 (240.0.184.4)  12.657 ms
10  100.100.36.98 (100.100.36.98)  11.935 ms 100.100.36.106 (100.100.36.106)  12.182 ms 100.100.34.110 (100.100.34.110)  18.098 ms
11  99.82.180.131 (99.82.180.131)  13.281 ms * *
12  * * *
13  142.251.70.84 (142.251.70.84)  13.046 ms 142.251.67.234 (142.251.67.234)  12.650 ms 108.170.246.33 (108.170.246.33)  14.032 ms
14  108.170.246.3 (108.170.246.3)  14.570 ms  13.528 ms 108.170.246.49 (108.170.246.49)  12.438 ms
15  142.251.49.75 (142.251.49.75)  15.045 ms * *
16  172.253.71.185 (172.253.71.185)  14.478 ms 142.250.236.137 (142.250.236.137)  14.216 ms  14.396 ms
17  * 142.251.64.249 (142.251.64.249)  20.519 ms *
18  209.85.241.125 (209.85.241.125)  18.404 ms  19.463 ms 209.85.250.145 (209.85.250.145)  19.822 ms
19  108.170.243.174 (108.170.243.174)  19.251 ms 108.170.243.193 (108.170.243.193)  18.455 ms 108.170.243.174 (108.170.243.174)  19.544 ms
20  142.251.60.15 (142.251.60.15)  19.505 ms 142.251.60.13 (142.251.60.13)  19.945 ms  19.645 ms
21  ord38s31-in-f10.1e100.net (142.250.191.202)  19.411 ms  18.886 ms  19.663 ms

Intending to ride it out but raising for awareness here if there's anything that can be done to expedite the solution.

Thanks Pete