Implement Search Console updates to rely on latest API version and display fresh data

[felixarntz]

Search Console recently released a big API update relevant for Site Kit: Related to #2284 (where we actually incorrectly assumed this was already in place), we can now display "fresh data" in Search Console reports via the API.

We should migrate from using the legacy API https://www.googleapis.com/webmasters/v3/ to https://searchconsole.googleapis.com/ as the new functionality is only available on the latter. Other than that though, the new API is fully backward-compatible, in fact it uses the same URLs (including "legacy prefix" /webmasters/v3/ within the new API) to achieve that.

We can easily take care of this migration by updating the Google API client services library to the latest version and relying on the SearchConsole class instead of the Webmasters class. We can then set the new dataState argument via the new request class to include fresh data in the API responses. In combination with that change, we can now finally change the day offset to 1 day before, from currently 2 days before.

---

Do not alter or remove anything below. The following sections will be managed by moderators only.

Acceptance criteria

• The Search Console module should rely on the new SearchConsole API instead of the legacy Webmasters API.
  • The webmasters service of the module should accordingly be renamed to searchconsole.
  • Since we're fully using the PHP client services library, this shouldn't require any actual migration code other than exchanging the PHP classes relied upon (and included in the bundle).
• The GET:searchanalytics REST datapoint should be updated to always pass the dataState argument with value "all" to the Google API request.
• The Search Console day offset for parsing a date range to actual dates should be updated from 2 to just 1 day delay, in both PHP and JavaScript.

Implementation Brief

• Use composer to update google/apiclient-services to version 0.156 with composer update google/apiclient-services
  • Update all dependancy use statements in the file includes/Modules/Search_Console.php that are prefixed with Google_Service_Webmasters to Google_Service_SearchConsole (Which is an extended version of the existing class) and update all uses of these classes in the rest of the file.
  • Update all instance of the old module name webmasters in values and function names to searchconsole in the includes/Modules/Search_Console.php + test
• Add a new line to create_search_analytics_data_request function within includes/Modules/Search_Console.php before $filters = array() to always set the dataState to "all" like so: $request->setDataState('all');
  • Update the offset value passed to $this->parse_date_range( at line 203 from 2 // Offset. back to 1.
  • Update the offset value passed DATE_RANGE_OFFSET in assets/js/modules/search-console/datastore/constants.js value back to1.

Test Coverage

Visual Regression Changes

QA Brief

• Install this version of the plugin then update your packages using composer with the following commands:

  10updocker shell
  // ...
  cd wp-content/plugins/google-site-kit
  composer install

  • You should then be able to match the data returned via the API to the requests made before the API class was updated.
  • The Search Console data should always include values up to 1 day ago (previously it was 2 days ago). Use e.g. the time-based charts to verify.

Changelog entry

• Rely on new Search Console "fresh data" feature to display metrics as recent as 1 day ago, improving on the previous offset of 2 days.

[benbowler]

IB ready for review.

[benbowler]

Added an estimate.

[benbowler]

Updated to IB slightly to mention the assets/js/modules/search-console/datastore/constants.js DATE_RANGE_OFFSET value.

[felixarntz]

@benbowler IB ✅ , except one thing:

Update the offset value passed DATE_RANGE_OFFSET in assets/js/modules/search-console/datastore/constants.js value back to2.

That should be set to 1 similar to how it is in PHP. Can you update that? I'll move over to Execution Backlog since that doesn't need another review 👍

[benbowler]

After this is merged other devs may have to run composer install within docker to update their local third-party directory using phpscoper as this directory is ignored by git.

[benbowler]

@felixarntz I just noticed that the dataState property is only available on master branch not on the v0.156 release as of yet. I've tested this working with the dev-master branch of the repository in composer.json.

I would normally avoid using dev-master for live projects but the Google PHP API Client Services README says:

If you are currently using the v1-master branch of the client library, but want to use the latest API services, you can do so by requiring this library directly into your project via the same composer command: composer require google/apiclient-services:dev-master.

[benbowler]

I've continued as such and created a PR and written a QA Brief.

[felixarntz]

@eclarke1 @fhollis FYI this is awaiting an external update of the Google API PHP client library we're using. The work on this issue is basically completed, but it can only be merged and QAd once that library has released an update.

[felixarntz]

@benbowler I've added an extra bullet point to the QA Brief here, I think it's important to mention the exact user-facing change here: Now data is shown for 1 day ago, previously it was a different offset.

[wpdarren]

QA Update: Pass ✅

• The date on the graphs within search console is different as expected. All of the elements within the graph were checked to make sure the same date was applied.

1.24.0

1.25.0