Dynamic Test Cases

[wandersonwhcr]

Hi!

I'm developing an app where the user can install or remove modules via an admin interface with composer. How can I create dynamic testsuites?

Example:

I have a composer root package where I install modules. I'm developing a module called A. Module A needs the module B. Composer installs modules A and B. Now, I want to install module C, for some unknown reason.

I can run phpunit ., but I want to run only the A testsuite.

Can I create an "dynamic" run, creating a bootstrap to search for installed modules and create testsuites programmatically?

[marcioAlmada]

Hi,

Couldn't you just enforce that each module should have a test @group? Then you could run: phpunit --group ModuleA. Just an idea :)

[sun]

We're facing the same issue in Drupal, which can be extended via modules, too. (Not via Composer, but that doesn't matter.)

As @marcioAlmada said, one (cumbersome) way to resolve this problem is to create and adopt a test authoring policy, which instructs developers to always add a @group tag whose value is their module's name, so that everyone can run the tests of any module via

$ phpunit --group ModuleA

However, this approach fully relies on correct @group assignments, which have to be added and maintained manually by all developers of all modules. There is currently no way to validate or enforce @group tags.

The @group approach also allows integration tests between A, B, C to be discovered and executed by specifying secondary groups (more than one @group).

Unless I'm mistaken, an alternative way is to use --filter with the module's namespace instead:

$ phpunit --filter "Vendor\\ModuleA"

I'm currently playing with the idea of writing a custom TestSuite class, which would not only enforce @group tags, but also perform the actual test file discovery (which is deeply nested in our case, so the phpunit.xml.dist config isn't really sufficient).

Ideally, I'd also like to make that class introduce a new command line parameter, so that you'd be able to specify --module ModuleA, but unfortunately, PHPUnit does not allow for custom CLI options at this point (see #634).

[marcioAlmada]

@sun

Yep, custom cli options would provide a much a better solution IMMO too.

Speaking strictly about what we have available at moment, the namespaces + --filter idea you just commented is much better than @group enforcement because you can avoid test name collisions between modules so everyone should be using namespaced tests in a situation like this anyway.

[wandersonwhcr]

omg! thank you for the quick response!

I'll try this when I'm at home, but I think it's the answer. I'll return the result.

[sun]

In any case, I think this is a very common problem space for application frameworks that are extensible through a module/plugin/component/bundle/extension mechanism (e.g., Drupal, Joomla, WordPress, SymfonyCMF, etc.pp.).

Each module/extension may ship with pure unit tests on its own, but more advanced integration/acceptance tests typically require a centrally coordinated/maintained testing framework of the larger application framework; e.g., in the form of base classes.

I will definitely look into this topic in the next weeks for Drupal. My hope is to possibly come up with something that could potentially be added to PHPUnit itself, so that users facing the same challenge can work together.

[whatthejeff]

I will definitely look into this topic in the next weeks for Drupal. My hope is to possibly come up with something that could potentially be added to PHPUnit itself, so that users facing the same challenge can work together.

Looking forward to this!

[wandersonwhcr]

With these files:

<?php
namespace A;
use PHPUnit_Framework_TestCase as TestCase;
// APPLICATION_PATH/modules/A/tests/Test.php
class Test extends TestCase
{
    public function testDumb()
    {
        $this->assertTrue(true);
    }
}

<?php
namespace B;
use PHPUnit_Framework_TestCase as TestCase;
// APPLICATION_PATH/modules/B/tests/Test.php
class Test extends TestCase
{
    public function testDumb()
    {
        $this->assertTrue(false);
    }
}

And with --filter option, everything works fine ;)

$ phpunit --filter 'A\\' --debug .
Starting test 'A\Test::testDumb'.
.
OK (1 test, 1 assertion)

$ phpunit --filter 'B\\' --debug .
Starting test 'B\Test::testDumb'.
F
There was 1 failure:
1) B\Test::testDumb
Failed asserting that false is true.
FAILURES!
Tests: 1, Assertions: 1, Failures: 1.

But when I use the --filter 'B\\', every class with B\ will be used, like A\B\AnotherTest::testDumb().

Now, I'll try with groups.

[wandersonwhcr]

OK I create some tests with groups and I think the --filter approach is better with modules.

But, I have some other issue: installation tests.

I can't run installation tests with normal tests, because one will break another. Example: a Test to check if a database search is OK and a Test to check if the table for database search is OK. All these tests must be ordered and controlled. Imagine one thousand tests like this; doomed?

But, if we use groups.

<?php
namespace A;
use PHPUnit_Framework_TestCase as TestCase;
// APPLICATION_PATH/modules/A/tests/Test.php
class Test extends TestCase
{
    /**
     * @group Install
     */
    public function testInstallation()
    {
        $this->assertTrue(true);
    }
}

<?php
namespace B;
use PHPUnit_Framework_TestCase as TestCase;
// APPLICATION_PATH/modules/B/tests/Test.php
class Test extends TestCase
{
    public function testBusinessLogic()
    {
        $this->assertTrue(false);
    }
}

We can run "labeled" tests.

$ phpunit --group Install --debug .
Starting test 'A\Test::testInstallation'.
.
OK (1 test, 1 assertion)

$ phpunit --exclude-group Install --debug .
Starting test 'B\Test::testBusinessLogic'.
F
There was 1 failure:
1) B\Test::testBusinessLogic
Failed asserting that false is true.
FAILURES!
Tests: 1, Assertions: 1, Failures: 1.

I can config my CI to only execute tests that don't match the "filter" with --exclude-group Install.

These two approaches are interesting. I'll use both.

Any other suggestions?

[wandersonwhcr]

[off-topic][suggestion]

Ah, Continuous Integration. I remember other problem with modules and PHPUnit. When I push to a module repository, a Git hook calls my CI to build the "root application" with the module. So, composer installs all the dependency and PHPUnit is called.

<testsuites>
    <testsuite name="Application Module Test">
        <directory>.</directory>
    </testsuite>
</testsuites>

I don't know if Drupal uses this approach, but it works for me, @sun . :)

[marcioAlmada]

These two approaches are interesting. I'll use both. Any other suggestions?

@wandersonwhcr Can't think of any other taxonomy available that could help you to classify|modularize tests. The only thing I can suggest is to allow --filter to receive a regex so anchors could be used to avoid namespace ambiguities:

--filter "#^B\\#"

[wandersonwhcr]

@marcioAlmada

I check and we can use REGEXP too.

https://github.com/sebastianbergmann/phpunit/blob/master/src/Runner/Filter/Test.php#L86

phpunit --filter '^A' --debug phpunit --filter 'T.st' --debug

:+1:

[marcioAlmada]

Cool :D

[whatthejeff]

@marcioAlmada actually had a hand in that :)

[marcioAlmada]

Nope, never messed with --filter. Didn't even know it accepted regexp until now :)

[whatthejeff]

Look closely at the highlighted line in the linked code :)

[marcioAlmada]

woot, git never forgets xD

[wandersonwhcr]

How we say here: UTSL (use the source, Luke) XD

So, I think I can close my issue now. Hey @sun , what do you think? We can open other issue for "unit tests with modules" and reference this too. :)

[sun]

@wandersonwhcr Sure. :) I'll create one after doing some more investigations. Glad to hear it works for you now.

[wandersonwhcr]

Thank you all! You're amazing ;)