RCA of the 3.16.0 release

johnbillion commented 4 months ago

I'm performing a root cause analysis of why the 3.16.0 release was broken so I can hopefully prevent it happening again. I'll update this issue as I go.

Problem

Version 3.16.0 was released on April 22nd. Users who updated to this version immediately reported a fatal error that caused their sites to go down.

Stack trace from a WP-CLI request:

PHP Fatal error: Uncaught Error: Failed opening required 'wp-content/plugins/query-monitor/vendor/composer/../symfony/deprecation-contracts/function.php' (include_path='.:/usr/share/php') in wp-content/plugins/query-monitor/vendor/composer/autoload_real.php:42
Stack trace:
0 wp-content/plugins/query-monitor/vendor/composer/autoload_real.php(46): {closure}()
1 wp-content/plugins/query-monitor/vendor/autoload.php(25): ComposerAutoloaderInitad25d53a23099da4b0886b4f6754360e::getLoader()
2 wp-content/plugins/query-monitor/query-monitor.php(56): require_once('…')
3 wp-settings.php(517): include_once('…')
4 phar:///usr/local/bin/wp/vendor/wp-cli/wp-cli/php/WP_CLI/Runner.php(1374): require('…')
5 phar:///usr/local/bin/wp/vendor/wp-cli/wp-cli/php/WP_CLI/Runner.php(1293): WP_CLI\Runner->load_wordpress()
6 phar:///usr/local/bin/wp/vendor/wp-cli/wp-cli/php/WP_CLI/Bootstrap/LaunchRunner.php(28): WP_CLI\Runner->start()
7 phar:///usr/local/bin/wp/vendor/wp-cli/wp-cli/php/bootstrap.php(83): WP_CLI\Bootstrap\LaunchRunner->process()
8 phar:///usr/local/bin/wp/vendor/wp-cli/wp-cli/php/wp-cli.php(32): WP_CLI\bootstrap()
9 phar:///usr/local/bin/wp/php/boot-phar.php(20): include('…')
10 /usr/local/bin/wp(4): include('…')
11 {main}
thrown in wp-content/plugins/query-monitor/vendor/composer/autoload_real.php on line 42

References

Notes

The tag and release zip on GitHub for 3.16 are as expected
Here are the changes between 3.15.0 and 3.16.0
Here are the changes for plugin-infrastructure between the two releases
The problem is with the deployment to wordpress.org

Root cause

The plugin at version 3.16.0 as shipped to wordpress.org triggered a fatal error

Why?

The vendor/composer/autoload_real.php was attempting to load a vendor/symfony/deprecation-contracts/function.php PHP file that does not (and should not) exist

Why?

The vendor/composer/autoload_real.php file contains unexpected calls to require for a bunch of files that do not exist in the build

Compare version 3.15 to version 3.16

Main change is the addition of:

$filesToLoad = \Composer\Autoload\ComposerStaticInitad25d53a23099da4b0886b4f6754360e::$files;
$requireFile = \Closure::bind(static function ($fileIdentifier, $file) {
  if (empty($GLOBALS['__composer_autoload_files'][$fileIdentifier])) {
      $GLOBALS['__composer_autoload_files'][$fileIdentifier] = true;

      require $file;
  }
}, null, null);
foreach ($filesToLoad as $fileIdentifier => $file) {
  $requireFile($fileIdentifier, $file);
}

When composer dump-autoload --no-dev is run, the $filesToLoad code isn't present. It's only present when the autoloader is created with dev dependencies.

Why?

At the point where the Composer autoload files were last generated within the deploy-tag.yml or reusable-deploy-tag.yml workflow, Composer thought those files were needed

Why?

composer install is called within the reusable-deploy-tag.yml workflow, which appears to be the cause of the regenerated autoloader.

Why?

The above is the cause of the regenerated autoloader but I've been unable to determine why this didn't affect version 3.15 which was released after the introduction of this call to composer install. The logs for GitHub Actions workflow runs are only retained for 90 days.

Additional issues

The adjustment to the setClassMapAuthoritative line (which should happen here) was also not present in the shipped release. See https://github.com/johnbillion/wp-crontrol/issues/91 for the history.

Counter measures

[ ] Need to increase my confidence in the reliability of the build that happens in the GitHub Actions workflows
[ ] Need to run the automated acceptance tests against a version of the plugin which is built in exactly the same manner as when the plugin is built for the deployment to wordpress.org
[x] Need to be able to dry-run the build and deployment workflows and inspect the output

Kilbourne commented 3 months ago

Not yet tested but could the problem be there isn't

          composer run build-vendor
          git add -f vendor/autoload.php
          git add -f vendor/composer/*

in reusable-deploy-tag.yml?

Because it seems that workflow installs also dev deps, which pollute the autoloader with not required files

johnbillion commented 3 months ago

I've identified the root cause as being the call to composer install which was added in https://github.com/johnbillion/plugin-infrastructure/commit/13b30ba69e3e0b1e13319eadb773a23ffb47a6dc but I wasn't able to determine why 3.15 wasn't affected. This change was made prior to Query Monitor 3.15 being released. Alas the GitHub Actions run logs are only retained for 90 days so I've exhausted everything that I can investigate.

Improvements that have been made as a result:

johnbillion / query-monitor