Set up an Integration Test environment

[raaaahman]

For now, the project only contains unit tests, which doesn't require a full WordPress environment to be triggered. Sooner or later, we will need to connect this project with WordPress itself, so we may add integration tests as well.

• Unit tests verify that the components of our projects behave in the way we expect. Any change on the project may make these tests fail, so we could correct before pursuing the changes
• Integration tests the project interfaces correctly with external systems (database, REST endpoints, filesystem,etc. which will all be provided by the WordPress Core in this project)

The easiest way to set up unit tests for a WordPress plugin is to use the WP CLI scaffold plugin-tests. Although this command is intended to set up a new project, while we already have a code base.

[raaaahman]

Wp-env for integration testing

The Gutenberg team has released their environment as en external tool: wp-env

This packages creates a Docker Composer environment with WordPress installed. It contains the following services.

Troubleshooting

Composer

The composer service use a PHP version that I don't know how to locate. Here's some commands and their output:

wp-env run composer 'composer install'
ℹ Starting 'composer install' on the composer container. 

Installing dependencies from lock file (including require-dev)
Verifying lock file contents can be installed on current platform.
Your lock file does not contain a compatible set of packages. Please run composer update.

  Problem 1
    - phar-io/manifest is locked to version 1.0.3 and an update of this package was not requested.
    - phar-io/manifest 1.0.3 requires php ^5.6 || ^7.0 -> your php version (8.0.1) does not satisfy that requirement.
  Problem 2
    - phar-io/version is locked to version 2.0.1 and an update of this package was not requested.
    - phar-io/version 2.0.1 requires php ^5.6 || ^7.0 -> your php version (8.0.1) does not satisfy that requirement.
  Problem 3
    - phpunit/php-code-coverage is locked to version 6.1.4 and an update of this package was not requested.
    - phpunit/php-code-coverage 6.1.4 requires php ^7.1 -> your php version (8.0.1) does not satisfy that requirement.
  Problem 4
    - phpunit/phpunit is locked to version 7.5.20 and an update of this package was not requested.
    - phpunit/phpunit 7.5.20 requires php ^7.1 -> your php version (8.0.1) does not satisfy that requirement.
  Problem 5
    - sebastian/global-state is locked to version 2.0.0 and an update of this package was not requested.
    - sebastian/global-state 2.0.0 requires php ^7.0 -> your php version (8.0.1) does not satisfy that requirement.
  Problem 6
    - phpunit/phpunit 7.5.20 requires php ^7.1 -> your php version (8.0.1) does not satisfy that requirement.
    - johnkary/phpunit-speedtrap v3.2.0 requires phpunit/phpunit ^7.0 || ^8.0 || ^9.0 -> satisfiable by phpunit/phpunit[7.5.20].
    - johnkary/phpunit-speedtrap is locked to version v3.2.0 and an update of this package was not requested.

✖ Command failed with exit code 2
Command failed with exit code 2

wp-env run cli bash
ℹ Starting 'bash' on the cli container. Exit bash with ctrl-d.

Starting e417f1f061675ba01bf64eaaa6eea9fb_mysql_1_f0bf38c01fc8 ... done
Starting e417f1f061675ba01bf64eaaa6eea9fb_wordpress_1_6aa44a4e588c ... done
bash-5.0$ php --version
PHP 7.2.34 (cli) (built: Dec 17 2020 10:26:11) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies
bash-5.0$ exit
✔ Ran `bash` in 'cli'. (in 32s 327ms)

ℹ Starting 'bash' on the tests-cli container. Exit bash with ctrl-d.

Starting e417f1f061675ba01bf64eaaa6eea9fb_mysql_1_f0bf38c01fc8 ... done
Starting e417f1f061675ba01bf64eaaa6eea9fb_tests-wordpress_1_21ef860dc414 ... donbash-5.0$ php --version
PHP 7.2.34 (cli) (built: Dec 17 2020 10:26:11) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies
bash-5.0$ exit
✔ Ran `bash` in 'tests-cli'. (in 19s 703ms)
sylvain@andev-d:~/development/wordpress-plugins

The workaround is to use a local Composer to install PHP dependencies inside the working directory, which is mounted in the container (although that means that each developer MUST have an up-to-date version of Composer installed locally):

composer install

PHPUnit

The phpunit service use its own wp-phpunit binary that I don't know how to configure. The problem is that it cannot use our dependency to JohnKary/SpeedTrapListener which results in a fatal error:

wp-env run phpunit 'phpunit -c /var/www/html/wp-content/plugins/wp-notify'
ℹ Starting 'phpunit -c /var/www/html/wp-content/plugins/wp-notify' on the phpunit container. 

Starting e417f1f061675ba01bf64eaaa6eea9fb_mysql_1_f0bf38c01fc8 ... done
Starting e417f1f061675ba01bf64eaaa6eea9fb_tests-wordpress_1_21ef860dc414 ... donClass "JohnKary\PHPUnit\Listener\SpeedTrapListener" does not exist
✖ Command failed with exit code 2
Command failed with exit code 2

The actual workaround is to open a bash session inside the containers and pass the command from there:

wp-env run cli bash
bash-5.0$ vendor/bin/phpunit

Notes

To test this, you need a test branch that merges tool/wp-env and upgrade-phpunit.

[swissspidy]

The composer service use a PHP version that I don't know how to locate.

By default the docker images use PHP 8, but you can override this by specifying phpVersion in .wp-env.json. You might be getting mixed results in your testing due to older containers, as this default changed only a few days ago.

Once you fix that you can try running PHPUnit from the container again.

[talldan]

@swissspidy I'm actually starting to get the impression that the composer image being updated to PHP 8 is the cause of the recent issues rather than changes to the WordPress image (see https://github.com/composer/docker/pull/154).

When I check my local env, I'm still showing WordPress images using PHP 7.4, even after completely nuking and rebuilding.

[swissspidy]

Ah, interesting! Yeah that seems likely.

[swissspidy]

That means there’s not really something you can do about it (except for filing an issue on the Gutenberg repo perhaps) besides making your code PHP 8 compatible.

[raaaahman]

That means there’s not really something you can do about it (except for filing an issue on the Gutenberg repo perhaps) besides making your code PHP 8 compatible.

Indeed, that's something we should definitely do. Also, I didn't mention it in the opening post, but I can run the tests with the default PHP 8.0 version and the phpunit service, given that I remove JohnKary/SpeedTrapListener from our composer dependencies. I think I will settle for this solution and pursue this discussion on the Gutenberg repository like @talldan suggested.

Thank you both!