UpCloud - google_api_connection_fail

jamesozzie commented 2 years ago

Bug Description

Check for any possible block at host level from UpCloud servers. One user in the support forums is encountering the below on sites hosted on a specific server, while not on others.

“Your site may not be ready for Site Kit
Looks like your site is having a technical issue with requesting data from Google services.
To get more help, ask a question on our support forum and include the text of the original error message:
google_api_connection_fail”

The same user who opened the related support topic, who manages or supports multiple sites reported the below:

I actually just received a confirmation from Upcloud that the IPv6 block in FI-HEL1 is currently blocked by Google. Can you try and escalate the issue on your end? At least the range 2a04:3540:1000:310:: /64 seems to be affected.

There is another report from a user in the support forums, who's site seems to be hosted on the same platform. They encounter the same error.

Steps to reproduce

Go to '...'
Click on '....'
Scroll down to '....'
See error

Screenshots

Additional Context

PHP Version:
OS: [e.g. iOS]
Browser: [e.g. chrome, safari]
Plugin Version: [e.g. 22]
Device: [e.g. iPhone6]

Do not alter or remove anything below. The following sections will be managed by moderators only.

Acceptance criteria

Implementation Brief

Test Coverage

QA Brief

Changelog entry

jamesozzie commented 2 years ago

I've been testing this on a trial server with UpCloud, where I can reproduce the same error. In my case I selected the same FI-HEL1 location indicated by one user. I also see various errors within my Site Health status, likely as a result of configurations at host level. In my trial mode it's not possible to disable the Firewall.

https://user-images.githubusercontent.com/41326532/199546632-1709431b-2e45-480f-abb0-49e54c655461.mp4

If I ignore the pre flight error and attempt set up, by clicking on the "Sign in with Google" button I encounter the below (https://*****mysiteexample****.com/wp-admin/index.php?action=googlesitekit_proxy_setup_start&nonce=017dc8560d):

PHP error logs:

Notes on this:

No browser console errors or invalid XHR response from the screen with the initial pre flight error
If checking the PHP error logs there are errors that appear (screenshot above)
Testing ongoing, this time after upgrading to allow Firewall configurations

Browser console errors

No related browser console errors on the initial screen with the error. The below appear if clicking on "Sign in with Google", on the screen in with the "The request to the authentication proxy has failed with an error: request_failed [Get help](https://sitekit.withgoogle.com/support?error_id=request_to_auth_proxy_failed). notice" ``` GET https://vodkio.com/wp-admin/index.php?action=googlesitekit_proxy_setup_start&nonce=017dc8560d 500 (anonymous) @ /wp-content/plugins/…3817bdd502b033.js:1 (anonymous) @ /wp-content/plugins/…11b13d18cd1a54.js:7 (anonymous) @ /wp-content/plugins/…11b13d18cd1a54.js:7 n @ /wp-content/plugins/…11b13d18cd1a54.js:7 (anonymous) @ /wp-content/plugins/…11b13d18cd1a54.js:7 t.any @ /wp-content/plugins/…11b13d18cd1a54.js:7 (anonymous) @ /wp-content/plugins/…11b13d18cd1a54.js:7 n @ /wp-content/plugins/…11b13d18cd1a54.js:7 r @ /wp-content/plugins/…11b13d18cd1a54.js:7 (anonymous) @ /wp-content/plugins/…11b13d18cd1a54.js:7 n @ /wp-content/plugins/…11b13d18cd1a54.js:7 (anonymous) @ /wp-content/plugins/…11b13d18cd1a54.js:7 a @ /wp-content/plugins/…11b13d18cd1a54.js:7 e @ /wp-content/plugins/…11b13d18cd1a54.js:7 (anonymous) @ /wp-content/plugins/…11b13d18cd1a54.js:7 (anonymous) @ /wp-content/plugins/…11b13d18cd1a54.js:7 (anonymous) @ /wp-content/plugins/…11b13d18cd1a54.js:7 (anonymous) @ /wp-content/plugins/…11b13d18cd1a54.js:7 (anonymous) @ /wp-content/plugins/…11b13d18cd1a54.js:7 (anonymous) @ /wp-content/plugins/…11b13d18cd1a54.js:7 (anonymous) @ /wp-content/plugins/…19313d0cf9ea0.js:32 s @ /wp-content/plugins/…11b13d18cd1a54.js:7 (anonymous) @ /wp-content/plugins/…11b13d18cd1a54.js:7 forEach.e. @ /wp-content/plugins/…11b13d18cd1a54.js:7 n @ /wp-content/plugins/…11b13d18cd1a54.js:1 _next @ /wp-content/plugins/…11b13d18cd1a54.js:1 Promise.then (async) n @ /wp-content/plugins/…11b13d18cd1a54.js:1 _next @ /wp-content/plugins/…11b13d18cd1a54.js:1 ```

SH info for imacted site

### wp-core ### version: 6.1 site_language: en_US user_language: en_US timezone: +00:00 permalink: /%year%/%monthnum%/%day%/%postname%/ https_status: true multisite: false user_registration: 0 blog_public: 1 default_comment_status: open environment_type: production user_count: 1 dotorg_communication: true ### wp-paths-sizes ### wordpress_path: /var/www/vhosts/vodkio.com/httpdocs wordpress_size: 49.44 MB (51839975 bytes) uploads_path: /var/www/vhosts/vodkio.com/httpdocs/wp-content/uploads uploads_size: 0.00 B (0 bytes) themes_path: /var/www/vhosts/vodkio.com/httpdocs/wp-content/themes themes_size: 12.35 MB (12951100 bytes) plugins_path: /var/www/vhosts/vodkio.com/httpdocs/wp-content/plugins plugins_size: 10.90 MB (11432210 bytes) database_size: 6.61 MB (6930432 bytes) total_size: 79.30 MB (83153717 bytes) ### wp-dropins (1) ### db.php: true ### wp-active-theme ### name: Twenty Twenty-Three (twentytwentythree) version: 1.0 author: the WordPress team author_website: https://wordpress.org parent_theme: none theme_features: core-block-patterns, post-thumbnails, responsive-embeds, editor-styles, html5, automatic-feed-links, block-templates, widgets-block-editor theme_path: /var/www/vhosts/vodkio.com/httpdocs/wp-content/themes/twentytwentythree auto_update: Disabled ### wp-themes-inactive (2) ### Twenty Twenty-One: version: 1.6, author: the WordPress team (latest version: 1.7), Auto-updates disabled Twenty Twenty-Two: version: 1.3, author: the WordPress team, Auto-updates disabled ### wp-plugins-active (3) ### Log HTTP Requests: version: 1.3.2, author: FacetWP, LLC, Auto-updates disabled Query Monitor: version: 3.10.1, author: John Blackbourn, Auto-updates disabled Site Kit by Google: version: 1.86.0, author: Google, Auto-updates disabled ### wp-plugins-inactive (2) ### Akismet Anti-Spam: version: 5.0.1, author: Automattic, Auto-updates disabled Hello Dolly: version: 1.7.2, author: Matt Mullenweg, Auto-updates disabled ### wp-media ### image_editor: WP_Image_Editor_Imagick imagick_module_version: 1690 imagemagick_version: ImageMagick 6.9.10-68 Q16 x86_64 2020-04-01 https://imagemagick.org imagick_version: 3.4.4 file_uploads: File uploads is turned off post_max_size: 8M upload_max_filesize: 2M max_effective_size: 2 MB max_file_uploads: 20 imagick_limits: imagick::RESOURCETYPE_AREA: 2 GB imagick::RESOURCETYPE_DISK: 9.2233720368548E+18 imagick::RESOURCETYPE_FILE: 6144 imagick::RESOURCETYPE_MAP: 2 GB imagick::RESOURCETYPE_MEMORY: 908 MB imagick::RESOURCETYPE_THREAD: 1 imagemagick_file_formats: 3FR, 3G2, 3GP, AAI, AI, ART, ARW, AVI, AVS, BGR, BGRA, BGRO, BMP, BMP2, BMP3, BRF, CAL, CALS, CANVAS, CAPTION, CIN, CIP, CLIP, CMYK, CMYKA, CR2, CRW, CUR, CUT, DATA, DCM, DCR, DCX, DDS, DFONT, DNG, DOT, DPX, DXT1, DXT5, EPDF, EPI, EPS, EPS2, EPS3, EPSF, EPSI, EPT, EPT2, EPT3, ERF, EXR, FAX, FILE, FITS, FRACTAL, FTP, FTS, G3, G4, GIF, GIF87, GRADIENT, GRAY, GRAYA, GROUP4, GV, H, HALD, HDR, HISTOGRAM, HRZ, HTM, HTML, HTTP, HTTPS, ICB, ICO, ICON, IIQ, INFO, INLINE, IPL, ISOBRL, ISOBRL6, JNG, JNX, JPE, JPEG, JPG, JPS, JSON, K25, KDC, LABEL, M2V, M4V, MAC, MAGICK, MAP, MASK, MAT, MATTE, MEF, MIFF, MKV, MNG, MONO, MOV, MP4, MPC, MPEG, MPG, MRW, MSL, MSVG, MTV, MVG, NEF, NRW, NULL, ORF, OTB, OTF, PAL, PALM, PAM, PANGO, PATTERN, PBM, PCD, PCDS, PCL, PCT, PCX, PDB, PDF, PDFA, PEF, PES, PFA, PFB, PFM, PGM, PGX, PICON, PICT, PIX, PJPEG, PLASMA, PNG, PNG00, PNG24, PNG32, PNG48, PNG64, PNG8, PNM, PPM, PREVIEW, PS, PS2, PS3, PSB, PSD, PTIF, PWP, RADIAL-GRADIENT, RAF, RAS, RAW, RGB, RGBA, RGBO, RGF, RLA, RLE, RMF, RW2, SCR, SCT, SFW, SGI, SHTML, SIX, SIXEL, SPARSE-COLOR, SR2, SRF, STEGANO, SUN, SVG, SVGZ, TEXT, TGA, THUMBNAIL, TIFF, TIFF64, TILE, TIM, TTC, TTF, TXT, UBRL, UBRL6, UIL, UYVY, VDA, VICAR, VID, VIFF, VIPS, VST, WBMP, WMF, WMV, WMZ, WPG, X, X3F, XBM, XC, XCF, XPM, XPS, XV, XWD, YCbCr, YCbCrA, YUV gd_version: bundled (2.1.0 compatible) gd_formats: GIF, JPEG, PNG, WebP, BMP, XPM ghostscript_version: not available ### wp-server ### server_architecture: Linux 3.10.0-1127.19.1.el7.x86_64 x86_64 httpd_software: Apache php_version: 7.4.11 64bit php_sapi: fpm-fcgi max_input_variables: 1000 time_limit: 30 memory_limit: 128M admin_memory_limit: 256M max_input_time: 60 upload_max_filesize: 2M php_post_max_size: 8M curl_version: 7.29.0 NSS/3.44 suhosin: false imagick_availability: true pretty_permalinks: true htaccess_extra_rules: false ### wp-database ### extension: mysqli server_version: 5.5.65-MariaDB client_version: mysqlnd 7.4.11 max_allowed_packet: 1048576 max_connections: 151 ### wp-constants ### WP_HOME: undefined WP_SITEURL: undefined WP_CONTENT_DIR: /var/www/vhosts/vodkio.com/httpdocs/wp-content WP_PLUGIN_DIR: /var/www/vhosts/vodkio.com/httpdocs/wp-content/plugins WP_MEMORY_LIMIT: 40M WP_MAX_MEMORY_LIMIT: 256M WP_DEBUG: false WP_DEBUG_DISPLAY: true WP_DEBUG_LOG: true SCRIPT_DEBUG: false WP_CACHE: false CONCATENATE_SCRIPTS: undefined COMPRESS_SCRIPTS: undefined COMPRESS_CSS: undefined WP_ENVIRONMENT_TYPE: Undefined DB_CHARSET: utf8 DB_COLLATE: undefined ### wp-filesystem ### wordpress: writable wp-content: writable uploads: writable plugins: writable themes: writable ### google-site-kit ### version: 1.86.0 php_version: 7.4.11 wp_version: 6.1 reference_url: https://vodkio.com amp_mode: no site_status: not-connected user_status: not authenticated verification_status: not-verified connected_user_count: none active_modules: site-verification, search-console, pagespeed-insights recoverable_modules: none required_scopes: openid: ⭕ https://www.googleapis.com/auth/userinfo.profile: ⭕ https://www.googleapis.com/auth/userinfo.email: ⭕ https://www.googleapis.com/auth/siteverification: ⭕ https://www.googleapis.com/auth/webmasters: ⭕ capabilities: googlesitekit_authenticate: ✅ googlesitekit_setup: ✅ googlesitekit_view_posts_insights: ⭕ googlesitekit_view_dashboard: ⭕ googlesitekit_manage_options: ⭕ googlesitekit_view_splash: ✅ googlesitekit_view_authenticated_dashboard: ⭕ googlesitekit_view_wp_dashboard_widget: ⭕ googlesitekit_view_admin_bar_menu: ⭕ enabled_features: adsenseSetupV2: ⭕ dashboardSharing: ⭕ ga4ActivationBanner: ⭕ ideaHubModule: ⭕ twgModule: ⭕ userInput: ⭕ search_console_property: none

jamesozzie commented 2 years ago

The same occurs with the Firewall at host level disabled:

On the VPS solution the same also occurs with ModSecurity and IP Addressing banning disabled

I also encounter 403 error when trying to communicate with https://sitekit.withgoogle.com via cURL

jamesozzie commented 2 years ago

@adamsilverstein I know you're performing some internal checks. I'm happy to share a login to the host, the VPS or the WP site if needed.

mxbclang commented 1 year ago

This is a possible code emergency as we've had other reports recently that we're looking into. Will update soon!

adamdunnage commented 1 year ago

Update: This has been happening due to a block on some users IPv6 addresses. The blocks were intentional and is due to the IP being part of a block that also includes IPs from a banned region.

Instructions on how to unblock these IPs were passed on to known hosting providers shared by affected users. We have now been getting confirmation from users that this error is no longer showing and they are able to complete Site Kit setup.

This can take some time for the block to clear on the users side so some may need to allow some time for this to take effect. We will continue to monitor this situation over the coming weeks.

webdock-io commented 1 year ago

We see this happening on a number of our IPv6 ranges both in Finland and in Canada. We see the ranges flip-flop from blocked to unblocked from time to time. Just yesterday we had some unblocked ranges which are blocked today.

We started seeing this start of December, so this has been going on for a long time now.

We have followed the instructions from @adamdunnage and added feeds to the Google ISP portal which have been ingested correctly, but still no effect.

I am not sure if anyone has brought this up, but our clear feeling is that these are automated blocks of users on ipv6 blocks which Google perceives are coming from Iran. As geolocation is not as good for ipv6 as it is for ipv4 Google is being rather aggressive in their blocks of ipv6 ranges. Given that theory, it is odd that adding feeds to the ISP portal is not working for us.

This is clearly still an on-going issue for multiple providers, we have identified Upcloud, us (Webdock) and Linode being affected from looking at various issues posted by users online.

adamdunnage commented 1 year ago

@webdock-io Thanks for dropping a comment here. I have followed up with you via our email chain and have passed on the information you have shared to the team to investigate further.

As mentioned on the email chain this could be a case of waiting a little longer for these to take effect but rest assured we are looking into this and the information you have shared has been passed on to the relevant team for review.

We will share any updates on this matter as we get them. Thanks for your patience with this.

detra commented 1 year ago

I think google servers for youtube are continually blocking the webdock IPv6 IP

I'm on webdock (@webdock-io) and for youtube to work on my sites [runcloud], I have to use a workaround for the IPv6 IP of webdock to 'avert' the api call to an IPv4 IP address only call - via an global type edit in functions.php (Child Theme):

`/- youtube fix /

add_action( 'http_api_curl', function( $curl_handle ) { curl_setopt( $curl_handle, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_V4 ); });`

jamesozzie commented 1 year ago

Many thanks for sharing what works for you @detra, much appreciated. This may be useful for others also impacted by this. I use Webdock myself and I can confirm that I also ran into the same issue previously. In my case temporarily deactivating IPv6 did allow Site Kit set up to complete.

Note also that these IPv6 blocks can be caused by one site on the same IPv6 range coming from a region where Google services are blocks. It doesn't necessarily have to be the same IP, it can occur with IP's within a range. We are investigating this reports as they come in, while also reaching out to impacted hosting providers with workarounds.

aaemnnosttv commented 1 year ago

Closing this out as this is a hosting / networking issue.

google / site-kit-wp