search

google / site-kit-wp

Site Kit is a one-stop solution for WordPress users to use everything Google has to offer to make them successful on the web.
https://sitekit.withgoogle.com
Apache License 2.0
1.25k stars 291 forks source link

Site verification fails with roots.io's Bedrock environment #1523

Closed jamesozzie closed 3 years ago

jamesozzie commented 4 years ago

Bug Description

As reported in a WordPress support topic a standard Roots.io Bedrock setup results in a site verification failure.

When using Bedrock the default WordPress directory structure is modified, placing all WP files into a /web directory. The index.php in the root directly reflects this as below: require( dirname( __FILE__ ) . '/wp/wp-blog-header.php' );

The default admin URL with a Bedrock install is set to /wp/wp-admin.

A solution to #1474 may also apply to this issue.

Steps to reproduce

  1. Setup site as described following this guide
  2. Install Site Kit and follow setup steps
  3. Verification fails

Additional Context

Do not alter or remove anything below. The following sections will be managed by moderators only.

Acceptance criteria

Implementation Brief

QA Brief

Changelog entry

Sidlegionair commented 4 years ago

Hi @jamesozzie,

I'm the OP over at wordpress.org, I don't suspect the issue is with bedrock's folder structure because we have multiple sites running the same setup (just a branch of same repository with some small mods) and working just fine, the only difference seems to be the absense of Varnish and different webhosting server on the problematic website. In fact I just verified another website, and it went through just fine.

### wp-core ###

version: 5.4.1
site_language: nl_NL
user_language: nl_NL
timezone: +00:00
permalink: /%postname%/
https_status: true
user_registration: 0
default_comment_status: closed
multisite: false
user_count: 100
dotorg_communication: true

### wp-paths-sizes ###

wordpress_path: /home/username/domains/mywebsite.com/web/wp
wordpress_size: 43,46 MB (45570251 bytes)
uploads_path: /home/username/domains/mywebsite.com/web/app/uploads
uploads_size: 881,50 MB (924323191 bytes)
themes_path: /home/username/domains/mywebsite.com/web/app/themes
themes_size: 7,95 MB (8338881 bytes)
plugins_path: /home/username/domains/mywebsite.com/web/app/plugins
plugins_size: 131,38 MB (137766251 bytes)
database_size: 217,05 MB (227590144 bytes)
total_size: 1,25 GB (1343588718 bytes)

### wp-dropins (1) ###

advanced-cache.php: true

### wp-active-theme ###

name: Hello (unicorn/resources)
version: 1.0.0
author: OneTap.Online
author_website: https://onetap.online
parent_theme: none
theme_features: woocommerce, soil-clean-up, soil-jquery-cdn, soil-nav-walker, soil-nice-search, soil-relative-urls, custom-logo, title-tag, menus, post-thumbnails, html5, customize-selective-refresh-widgets, editor-style, widgets
theme_path: /home/username/domains/mywebsite.com/web/app/themes/unicorn/resources

### wp-themes-inactive (3) ###

Twenty Nineteen: version: 1.5, author: het WordPress team
Twenty Seventeen: version: 2.3, author: het WordPress team
Twenty Twenty: version: 1.2, author: the WordPress team

### wp-mu-plugins (3) ###

Bedrock Autoloader: version: 1.0.0, author: Roots
Disallow Indexing: version: 1.0.0, author: Roots
Register Theme Directory: version: 1.0.0, author: Roots

### wp-plugins-active (47) ###

Ad Inserter: version: 2.6.8, author: Igor Funa
Ads.txt Manager: version: 1.3.0, author: 10up
Advanced Access Manager: version: 6.5.0, author: Vasyl Martyniuk <vasyl@vasyltech.com>
BuddyPress: version: 5.2.0, author: The BuddyPress Community
BuddyPress to WordPress Full Sync: version: 0.3.4, author: OneTap Online
Cookie Notice: version: 1.3.1, author: dFactory
Crocoblock Wizard: version: 1.0.1, author: Crocoblock
Dashboard Welcome for Elementor: version: 1.0.5, author: IdeaBox Creations
Duplicate Post: version: 3.2.4, author: Enrico Battocchi
DynamicConditions: version: 1.4.6, author: RTO GmbH
Edit Flow: version: 0.9.6, author: Daniel Bachhuber, Scott Bressler, Mohammad Jangda, Automattic, and others
Ele Custom Skin: version: 2.1.0, author: Dudaster.com
Elementor: version: 2.9.8, author: Elementor.com
Elementor BuddyPress Widgets: version: 0.0.1, author: Floris Boers
Elementor Contact Form DB: version: 1.5, author: Sean Barton - Tortoise IT
Elementor Pro: version: 2.9.3, author: Elementor.com (latest version: 2.9.4)
Envato Elements: version: 1.1.9, author: Envato
Header Footer Code Manager: version: 1.1.7, author: 99robots
JetBlog For Elementor: version: 2.2.4, author: Crocoblock (latest version: 2.2.5)
JetElements Dynamic Data Addon: version: 1.1.2, author: Zemez
JetEngine: version: 2.3.3, author: Crocoblock (latest version: 2.3.5)
Jet Fix Framework URL: version: 1.0.0, author: Zemez
JetMenu: version: 2.0.4, author: Crocoblock
JetStyleManager: version: 1.0.0-beta, author: Crocoblock
JetThemeCore: version: 1.1.23, author: Crocoblock
JetTricks: version: 1.2.8, author: Crocoblock
Loco Translate: version: 2.3.3, author: Tim Whitlock
Make Column Clickable Elementor: version: 1.3.1, author: Fernando Acosta
MemberPress + BuddyPress Integration: version: 1.1.5, author: Caseproof, LLC
MemberPress Developer Tools: version: 1.1.34, author: Caseproof, LLC
MemberPress Downloads: version: 1.0.7, author: Caseproof, LLC
MemberPress Importer: version: 1.6.2, author: Caseproof, LLC (latest version: 1.6.4)
MemberPress Plus: version: 1.8.8, author: Caseproof, LLC
Mollie Forms: version: 2.3.2, author: Wobbie.nl
Post SMTP: version: 2.0.11, author: Yehuda Hassine
Pronamic Pay: version: 6.1.2, author: Pronamic
Redirection: version: 4.7.1, author: John Godley
Redis Object Cache: version: 1.5.7, author: Till Krüss
Regenerate Thumbnails: version: 3.1.3, author: Alex Mills (Viper007Bond)
Safe SVG: version: 1.9.8, author: Daryll Doyle (latest version: 1.9.9)
Site Kit by Google: version: 1.7.1, author: Google
Stream: version: 3.4.3, author: XWP
Warp iMagick - Image Compressor: version: 1.0.6, author: Dragan Đurić
WP Post Modules for Elementor: version: 1.5.3, author: SaurabhSharma
WP Rocket: version: 3.5.4, author: WP Media
WP User Avatar: version: 2.2.6, author: ProfilePress
Yoast SEO: version: 14.0.4, author: Team Yoast

### wp-plugins-inactive (6) ###

Developersuite Install Manager: version: 1.0.0, author: Floris Boers
GA Google Analytics: version: 20200325, author: Jeff Starr
Limit Login Attempts Reloaded: version: 2.12.3, author: WPChef
Polylang: version: 2.7.2, author: WP SYNTEX
Polylang Connect for Elementor: version: 1.0.0, author: David Decker - DECKERWEB
Two Factor: version: 0.5.2, author: Plugin Contributors (latest version: 0.6.0)

### wp-media ###

image_editor: Warp_Image_Editor_Imagick
imagick_module_version: 1684
imagemagick_version: ImageMagick 6.9.4-10 Q16 x86_64 2017-05-23 http://www.imagemagick.org
gd_version: bundled (2.1.0 compatible)
ghostscript_version: 9.07

### wp-server ###

server_architecture: Linux 3.10.0-962.3.2.lve1.5.25.10.el7.x86_64 x86_64
httpd_software: Apache/2
php_version: 7.3.6 64bit
php_sapi: litespeed
max_input_variables: 1000
time_limit: 120
memory_limit: 1024M
max_input_time: -1
upload_max_size: 2048M
php_post_max_size: 2048M
curl_version: 7.62.0 OpenSSL/1.0.2k
suhosin: false
imagick_availability: true

### wp-database ###

extension: mysqli
server_version: 10.2.24-MariaDB-cll-lve
client_version: 10.2.24-MariaDB

### wp-constants ###

WP_HOME: https://mywebsite.com
WP_SITEURL: https://mywebsite.com/wp
WP_CONTENT_DIR: /home/username/domains/mywebsite.com/web/app
WP_PLUGIN_DIR: /home/username/domains/mywebsite.com/web/app/plugins
WP_MAX_MEMORY_LIMIT: 512M
WP_DEBUG: false
WP_DEBUG_DISPLAY: false
WP_DEBUG_LOG: false
SCRIPT_DEBUG: false
WP_CACHE: true
CONCATENATE_SCRIPTS: undefined
COMPRESS_SCRIPTS: undefined
COMPRESS_CSS: undefined
WP_LOCAL_DEV: undefined
DB_CHARSET: utf8mb4
DB_COLLATE: undefined

### wp-filesystem ###

wordpress: writable
wp-content: writable
uploads: writable
plugins: writable
themes: writable
mu-plugins: writable

### pronamic-pay ###

license_status: Geldig
next_license_check: vr 8 mei 2020 om 10:19
time: 07-05-2020 16:15
openssl_version: OpenSSL 1.0.2k  26 Jan 2017
active_plugin_integrations: MemberPress

### buddypress ###

version: 5.2.0
active_components: Leden van de community, Uitgebreide profielen, Accountinstellingen, Privéberichten, Meldingen, Gebruikers groepen
template_pack: BuddyPress-Nouveau 5.2.0
! hide-loggedout-adminbar: Nee
! bp-disable-account-deletion: Ja
! bp-disable-avatar-uploads: Ja
! bp-disable-cover-image-uploads: Ja
! bp-disable-profile-sync: Ja
! bp_restrict_group_creation: Ja
! bp-disable-group-avatar-uploads: Ja
! bp-disable-group-cover-image-uploads: Ja

### google-site-kit ###

version: 1.7.1
php_version: 7.3.6
wp_version: 5.4.1
reference_url: https://mywebsite.com
amp_mode: no
site_status: connected-site
user_status: authenticated
active_modules: site-verification, search-console, adsense, analytics
required_scopes: 
    openid: ✅
    https://www.googleapis.com/auth/userinfo.profile: ✅
    https://www.googleapis.com/auth/userinfo.email: ✅
    https://www.googleapis.com/auth/siteverification: ✅
    https://www.googleapis.com/auth/webmasters: ✅
    https://www.googleapis.com/auth/adsense: ✅
    https://www.googleapis.com/auth/analytics: ✅
    https://www.googleapis.com/auth/analytics.readonly: ✅
    https://www.googleapis.com/auth/analytics.manage.users: ✅
    https://www.googleapis.com/auth/analytics.edit: ✅
search_console_property: https://mywebsite.com/
adsense_account_id: pub-498•••••••••••••
adsense_client_id: ca-pub-498•••••••••••••
adsense_account_status: account-connected
adsense_use_snippet: no
analytics_account_id: 1416•••••
analytics_property_id: UA-2754••••••
analytics_profile_id: 8441••••
analytics_use_snippet: yes
jamesozzie commented 4 years ago

@Sidlegionair Thanks for the input, very useful. We'll continue to test from our side.

ernee commented 4 years ago

Possible related topic: https://wordpress.org/support/topic/redirects-to-google-login-after-logging-in-and-accepting-permissions-to-account/

cole10up commented 4 years ago

Tested

Ran through a series of tests to install bedrock - Exhausted efforts to mimic with roots.io in the environment. This is more of a L2 issue. Sending over to L2.

ernee commented 4 years ago

Adding additional info here: the user in the initial topic clarified that they do not get redirected back to the dashboard when attempting setup, but are instead looped back/ redirected to the proxy with the following URL in this response: https://wordpress.org/support/topic/infinite-verification-loop/#post-12936405

Site Health info for the site with the issue here: https://wordpress.org/support/topic/infinite-verification-loop/#post-12797811

ernee commented 4 years ago

another report: https://wordpress.org/support/topic/plugin-setup-stuck-in-a-loop-without-any-error-notice/

(Site Health sent via form)

brandomeniconi commented 4 years ago

If there is something I can do to help please let me know. We have more than 100 sites on bedrock and we will to upgrade all of them to SiteKit, but we are stuck to this bug. In few sites we are able to make it work, but every one we are trying now it's failing to authenticate.

Thanks!

aaemnnosttv commented 4 years ago

Hi @brandomeniconi - are there any common differences between the sites that are not able to authenticate and the sites that work? As someone with so many sites this would help a lot as it seems that Bedrock itself is compatible with Site Kit as far as I can tell.

brandomeniconi commented 4 years ago

Hi @aaemnnosttv I've sent some SiteHealts reports to @ernee via this topic https://wordpress.org/support/topic/plugin-setup-stuck-in-a-loop-without-any-error-notice/ of working and non-working sites. I have a wide range of combos of WP setups (multisite, single site, etc) and they all share the same environment (Google App Engine).

This is our deploy procedure that's actually failing to activate SiteKit: 1) Develop the site code on Docker-based local environment 2) Deploy it to App Engine on a staging 4th level domain (sitename.staging.example.com) 3) Setup WP, plugins, theme and insert contents 4) Connect the final 1st level domain to App Engine (this requires search console domain verification) 5) Switch WP to the final domain 6) Setup SiteKit with the same Google account for all websites

Some of the sites where SiteKit is working, are ones that are already been published (step 5) in the past, then SiteKit has been installed directly when the site was already on the final domain. Maybe that's a difference.

Also be aware that Berock usually uses 2 different URLs for siteurl and home WP options: siteurl: https://example.com/wp home: https://example.com/ and that is some that can maybe disrupt the OAuth redirect/return URL.

Unfortunately I cannot debug myself easily the live sites because I don't have shell access on GAE and it install the plugin automatically via composer. But I can try to force some debug lines and output them in Cloud Logging if required.

If there is anything I can do please ask because this bug is delaying the publishing of many sites and I'm thrilling to resolve it 😉

aaemnnosttv commented 4 years ago

@brandomeniconi thanks for the detailed reply. I've tested a standard Bedrock install (with home and siteURLs like you mentioned) a few times now and have yet to replicate the issue you're experiencing, so it's possible it's not Bedrock-specific.

One thing I noticed is that the URL for the file verification method returned a 404 on the site you referenced in the linked support ticket: image

We do not actually place a file there but serve it via PHP if requested. It should also fallback to using a meta tag if that fails though, but that may not work if you have full-page caching on the front-end.

If it's only the site verification step that is a problem, a temporary workaround would be to manually verify your user using any of the supported methods image

This is not an ideal solution, I'm only suggesting it as a means of unblocking you quickly.

Let me know if that helps?

brandomeniconi commented 4 years ago

Hi @aaemnnosttv , I don't think the domain verification is a problem because by using App Engine we must verify the domain in search console before applying it to the site. We use the same Google Account. Something I forgot to mention is that we use Cloudflare on all our sites, but the issue is happening also when it's completely disabled (DNS pointing to GAE directly) so I don't think it will be affecting this.

I suspect that the redirect is caused by this line here: https://github.com/google/site-kit-wp/blob/3590c624a075e9d9cb28bc9bf6ee313dd7aba516/includes/Core/Authentication/Clients/OAuth_Client.php#L665

I will do further investigation and let me know if I need to do something else.

brandomeniconi commented 4 years ago

I confirm that is shouldn't be a domain verification failure, here is a Search Console result of one of our staging domains:

google_verification

The HTML verification fails because the Google App Engine configuration treats .html URLs as static file and doesn't pipe those to the WP's php entry script.

Here is another one which says that is already verified even before I tried to authenticate SiteKit:

google_verification2

They both fail SiteKit authentication.

brandomeniconi commented 4 years ago

I've also logged all the WP incoming and outgoing requests in Google Cloud Logging and the authentication secrets seems to be issued. Does the plugin has some internal validation?

google_sitekit_log2

google_sitekit_log

Sorry for flooding this issue 😸 .

This is also our Bedrock/AppEngine boilerplate if you want to give it a try: https://github.com/silverbackstudio/wp-website

aaemnnosttv commented 4 years ago

If your Google user is already a verified owner for the domain/site then that step should already be completed without going through the flow between the authentication service and your site to facilitate it.

Looking back at your gif of the loop, you aren't getting to that step anyways which is really odd.

This is also our Bedrock/AppEngine boilerplate if you want to give it a try: https://github.com/silverbackstudio/wp-website

This is great, I'll give this a shot 👍

jamesozzie commented 4 years ago

Another reported issue on the below WordPress topic: https://wordpress.org/support/topic/unable-to-complete-setup/#post-13179635

jamesozzie commented 4 years ago

Additional findings on this issue below: https://wordpress.org/support/topic/unable-to-complete-setup/#post-13207950

User attempted site verification in Search Console manually before trying to setup Site Kit. Unfortunately the setup loop issue remains, presumably at the "Proceed" button stage, directing the user back.

Sidlegionair commented 3 years ago

@jamesozzie Any progress?

jamesozzie commented 3 years ago

@Sidlegionair No update on this just yet, but the team are aware of this and will be working on it.

aaemnnosttv commented 3 years ago

After giving this another try with a fresh site, I'm still unable to reproduce any problem setting up Site Kit on Bedrock.

It certainly may be possible to have trouble completing the Site Kit-facilitated site verification process though for a number of reasons such as improper web server configuration. It does not seem however that Bedrock itself is responsible for this as this has worked as expected every time I have tested it. This leads me to conclude that other factors are likely causing problems with site verification and that using Bedrock is one thing that some of these sites have in common.

As a result, I'm going to close this out for now but happy to reopen in the future given more information about how to reproduce this.

Sidlegionair commented 3 years ago

@aaemnnosttv I'm willing to give more information to get this fixed, what would you need? As long as this issue isn't fixed (or a workaround has not yet been found) I cannot reliably deploy the plugin to all of my customers websites.

aaemnnosttv commented 3 years ago

@Sidlegionair we have yet to identify any incompatibility with Bedrock specifically or be able to reproduce the reported problem.

As you mentioned previously, the plugin works fine for you in some environments with Bedrock and not in others, so it seems Bedrock is not the source of the conflict.

Please try again with the latest version of the plugin and if you're still having trouble, please reach out via support and we'll do our best to help 👍