Packages: offer standardized way to set up WordPress / run WordPress PHPUnit tests within a package

[jeherve]

Until now, we've relied on 2 packages to help us with unit tests inside our PHP packages:

• https://github.com/Automattic/wordbless
• https://github.com/Brain-WP/BrainMonkey

While those 2 options work, they do not offer the full WordPress core PHPUnit test library that is included with WordPress' development environment: https://github.com/WordPress/wordpress-develop/tree/master/tests/phpunit

That's sometimes problematic, for example when wanting to develop tests against functions that use WP_Query for example: #19579

It would be nice to come up with a solution that we can agree on, and that we would rely on for most packages. Maybe something like https://github.com/wp-phpunit/wp-phpunit

[jeherve]

cc @leogermani for opinions, as WorDBless' creator. :)

[anomiex]

and that we would rely on for most packages

Let's try to keep the testing environment as simple as is feasible.

• If a package doesn't actually depend on WordPress, it really should just use basic PHPUnit functions. Don't bring in WordPress stuff if we don't need it.
• BrainMonkey is the next step up. It provides mocks for a few WP functions, and lets you mock any others you might need. If you're only using a few WordPress functions, don't go for more complex options.
• WorDBless is the next step. It pulls in all the WordPress code, and makes a lot of it work well enough for testing. Ideally we'd add functionality here rather than pulling in something even more complicated.
• This PR is apparently because WorDBless doesn't support some things?

Keep in mind that it's possible to use both WorDBless and BrainMonkey so the latter can potentially mock things the former doesn't handle, see instructions in monorepo.md.

Maybe something like https://github.com/wp-phpunit/wp-phpunit

I note that has the same complication that we have with the Jetpack plugin's tests, in that it currently refuses to work with any version of PHPUnit that supports PHP 8. Our CI setup does some kludging to make that work when using WordPress's built-in testing stuff for plugins. I'd rather we avoid extending that to packages too.

If we do go a route that needs that sort of kludging, I'd rather we separate "plugin-packages" from normal packages and use the same kludging for both. The ideal might be for the "plugin-packages" to be such that a plugin or mu-plugin can be created automatically within the CI setup (and within the JP Docker env), and then they can be tested just like plugins are already.

[leogermani]

cc @leogermani for opinions, as WorDBless' creator. :)

In general, I like the approach Brad suggests. It keeps things small and simple, only adding complexity when needed. Even though it leaves some room for confusion (each developer taking a different approach when writing tests)

It's worth noting two things. One is that WorDBless (or any other alternative) will never be able to fully mock WordPress. WP_Query in particular is a central piece in WP core and we'll never be able to fully mock it because it's basically a SQL query builder and we need a SQL server to test it...

The second thing is that the whole point of using these mocking in packages is to avoid having to set up a whole environment for them to run the tests. The idea, if I get it correctly, is that packages should have their tests independent of jetpack-the-plugin. In other words, if we have a copy of the package outside the context of the monorepo, we should be able to easily run the tests.

BUT, if we are relying on and investing in our monorepo, is providing a full test environment for packages in our Docker set up an alternative?

If we did that, we wouldn't need any of those mockings, and writing tests would be much simpler. On the other hand, tests would be slower and it would be difficult to run them outside the monorepo...