Develop black box PHPUnit tests for WordPress TRAC #13459

bobbingwide commented 2 years ago

In #1 I've developed PHPUnit tests that run in situ under oik-batch, with wordpress-develop-tests.

Most core developers don't expect to have to use oik-batch. It would be better for other contributors if the tests could be run as part of the core PHPUnit tests. This will be the first time for ages that I've tried to develop PHPUnit tests for core.

Requirement

PHPUnit tests that can be run in a wordpress-develop environment
Almost exactly the same tests for core as for In Situ

Proposed solution

Implement Workflow 4: PHPUnit PHAR file without composer.

[x] Clone the latest c:\github\WordPress\wordpress-develop to c:\apache\htdocs\wordpress-develop
[x] Clone Yoast/PHPUnit-Polyfills to c:\github\Yoast\PHPUnit-Polyfills
[x] Create a database called empty
[x] Update wp-tests-config.php to set 'WP_TESTS_PHPUNIT_POLYFILLS_PATH' and DB info
[x] Create a phpempty.bat file to invoke PHPUnit
[x] Demonstrate that the core tests run.
[x] Add the tests already developed.
[x] Run the tests
[ ] Extend the tests

bobbingwide commented 2 years ago

Run the tests

To attempt to access the posts in the same way that an end user does the tests include calls to wp_remote_get(). The site therefore has to be set up as a working site that you can actually log into. Target site: https://s.b/wordpress-develop/src/

Installation and configuration steps

Installation as above into c:\apache\htdocs\wordpress-develop.
Install npm and build the runtime code using npm run dev or npm run build.
Run WordPress installation to create c:\apache\htdocs\wordpress-develop\wp-config.php.
Update c:\apache\htdocs\wordpress-develop\wp-tests-config.php:

define( 'WP_DEFAULT_THEME', 'twentytwentytwo' );  
define( 'WP_DEBUG',  true );
define( 'WP_TESTS_DOMAIN', 'https://s.b/wordpress-develop/src/' );
define( 'WP_TESTS_PHPUNIT_POLYFILLS_PATH', 'C:/github/Yoast/PHPUnit-Polyfills' );

Changing the default theme constant didn't appear to take any affect in the front end.
I found I had to copy a theme into the default folder; I chose twentytwentytwo.
Distraction: Appearance Themes now says it needs updating. It's found the original default theme, rather than TT2.
I also had to set the environment variable WP_TESTS_SKIP_INSTALL=1 before calling PHPunit

@echo Run PHPUnit against WordPress core
rem See https://make.wordpress.org/core/handbook/testing/automated-testing/phpunit/#setup
cd \apache\htdocs\wordpress-develop
echo This is the one that empties databases. Press Enter to continue.
pause
set WP_TESTS_SKIP_INSTALL=1
php c:\apache\htdocs\phpLibraries\phpunit\phpunit-9.5.5.phar "--verbose" "--disallow-test-output" "--log-junit=phpunit.json" %*

Copy the test-issue-13459.php into tests/phpunit/tests
Change BW_UnitTestCase to WP_UnitTestCase.
Run the tests using

phpempty --filter Tests_issue_13459 --stop-on-failure --stop-on-error

bobbingwide commented 2 years ago

Run the tests using

Note: WordPress develop is currently: WordPress 6.0-alpha-52448-src

I get different results depending on the initial value for the site's permalink structure. The tests fail early when the permalink structure isn't set... default plain.

Initial permalink structure	first fails at...
( plain )	See 1) Tests_issue_13459::test_permalink_structures
/%year%/%monthnum%/%day%/%postname%/ ( day and name )	See 1) Tests_issue_13459::test_permalink_structures_page_post below

First failure for plain

1) Tests_issue_13459::test_permalink_structures
 Permalink: `https://s.b/wordpress-develop/src/2022/03/29/duplicate-title-trac-13459/` 
Structure: /%year%/%monthnum%/%day%/%postname%/ 
Failed asserting that 404 matches expected 200.

First failure for Day and Name

1) Tests_issue_13459::test_permalink_structures_page_post
Structure: /%postname%/
Failed asserting that two strings are equal.
--- Expected
+++ Actual
@@ @@
-'<p class="duplicate-title-trac-13459">duplicate title TRAC 13459:post:publish:617:/%postname%/</p>'
+'<p class="duplicate-title-trac-13459">duplicate title TRAC 13459:page:publish:615:/%postname%/</p>'

Distraction: Theme without comments.php message produced.

bobbingwide commented 2 years ago

It helps if I refer to the correct TRAC ticket.

bobbingwide commented 2 years ago

The tests fail early when the permalink structure isn't set... default plain.

I tracked down the first part of this problem with the installation. The .htaccess file didn't contain the rewrite logic I'd expect to see for a subdirectory install.

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
RewriteBase /wordpress-develop/src/
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /wordpress-develop/src/index.php [L]
</IfModule>

Once I'd added the above then the requests made with a permalink structure were correctly served.

BTW: I'd already noted that the .htaccess file wasn't normal, but for some reason I ignored this problem. During the problem determination:

I activated oik-bwtrace
implemented the In Situ version of the tests to the "empty" environment
Added some tests to confirm the round trip was working.
I'd also changed the parent class on the tests In Situ tests back to BW_UnitTestCase.

But after resetting it to WP_UnitTestCase the In Situ tests failed immediately.

Fetching by permalink:Fetching: https://s.b/wordpress-develop/src/2022/03/29/new-post/
1) Tests_issue_13459::test_round_trip_day_and_name
Failed asserting that false is greater than 0.

C:\apache\htdocs\wordpress\wp-content\plugins\dupes\tests\test-issue-13459.php:261
C:\apache\htdocs\wordpress\wp-content\plugins\dupes\tests\test-issue-13459.php:344

bobbingwide commented 2 years ago

But after resetting it to WP_UnitTestCase the In Situ tests failed immediately.

Meanwhile the tests run by phpempty produce the same results as I got in other environments with the In Situ tests. This indicates that the version of WP_UnitTestCase class in oik-batch needs updating.

bobbingwide / dupes