Encourage PHPUnit testing in plugins

[stianlik]

Are there any guidelines for testing plugins using PHPUnit? In some cases it is sufficient to create simple unit test for a class without dependecies on the Elgg engine, however, we should be able to bootstrap Elgg and the given plugin if necessary.

Now that core tests are moving to PHPUnit as described in #2330, it makes sense that plugin authors should do the same. I'm considering adding some documentation for this. Currently, I use the following configuration to exploit the bootstrap.php file already included in engine:

MY_PLUGIN/phpunit.xml

<?xml version="1.0" encoding="UTF-8"?>
<phpunit bootstrap="tests/bootstrap.php" colors="true">
    <testsuites>
        <testsuite name="My plugin tests">
            <directory>tests</directory>
        </testsuite>
    </testsuites>
</phpunit>

MY_PLUGIN/tests/bootstrap.php:

$engine = dirname(__FILE__) . '/../../../engine';
$plugin = dirname(__FILE__) . '/..';

require_once "$engine/tests/phpunit/bootstrap.php";

// Configuration
global $CONFIG;
$CONFIG->cookies = array(
    'session' => array(),
);

// Function stubs
// TODO Replace with inclusion of the file configuration.php 
// once engine does not redefine elgg_get_site_url()

function elgg_get_root_path() {
    return '';
}

function elgg_get_config() {
    return '';
}

// Autoloader
_elgg_services()->autoloadManager->addClasses("$plugin/classes");

This makes it possible to run all tests for a given plugin by running phpunit from mod/MY_PLUGIN, which is very convenient when you have separate projects / repositories for each plugin.

It would be great if we could include some usable bootstrap for testing plugins in the engine to make it easier for plugin authors to start testing. The preceding example is far from complete, but I think it would be useful to have some options here. For instance, I would like to test saving and retrieving custom entities.

To execute database tests fast, we can do something similar to WordPress, begin transaction at the start of test and rollback in the tear down, see WP_UnitTestCase. Maybe create an abstract ElggEntityTestCase class to simplify entity testing.

[hypeJunction]

It would be nice to have the option to use predefined bootstrap in plugins. My recent attempts to use PHPUnit in plugins was not very successful. Most test cases were relying on testing database entries and hooks and how they work in the global scope. I went with SimpleTest because it was much easier to benchmark the performance of a plugin in relation to the core, and other enabled plugins.

[ewinslow]

Example of how I'm doing unit-testing in plugins:

https://github.com/ewinslow/elgg-evan/blob/master/tests/bootstrap.php

You'll notice that the plugin's composer.json requires elgg/elgg explicitly. All classes are available at that point. I don't even bother trying to test global state stuff.

This is where I think we should be headed.

[stianlik]

I like your approach @ewinslow, the setup is simple and tests easy to understand. Still, I think it is useful to be able to write integration tests in PHPUnit. Some situations where this can come in handy:

1. Verify that registered entities are instantiated correctly by Elgg (i.e. classnames)
2. Making sure queries are correct (i.e. elgg_get_entities_by_metadata() is easy to get wrong when you're adding a couple of arguments)
3. Verifying that you trigger required hooks
4. Verify that you remembered to do something when some event occured.

In many cases, we rely on typing strings, which is error prone, i.e. attaching a hook to 'entity:url' can be misspelled without any warning. Testing that you retrieve the correct entities in a search goes a long way in avoiding misspelled search arguments.

Example hook verification:

    public function test_getURL_shouldTriggerGetUrlHook()
    {
        $entity = new ElggObject;
        $entity->getURL();
        $this->assertHookTriggered('entity:url');
    }

Example testing entity class:

    public function test_entity_shouldBeAssignedMyEntityClass()
    {
        $entity = new ElggObject();
        $guid = $entity->save();
        _elgg_invalidate_cache_for_entity($guid);
        $this->assertInstanceOf('MyEntity', get_entity($guid));
    }

Example test that a logger plugin logs group joins to verify that we listened to the 'join', 'group' hook:

    public function test_entity_shouldLogGroupJoins()
    {
        $user = new ElggUser;
        $group = new ElggGroup;
        $group->join($user);
        $this->assertEquals(
            $user->getGUID() . 'joined ' . $group->getGUID(),
            $this->loggerPlugin->getLastMessage()
        );
    }

[ewinslow]

Agreed that plugins should be able to do integration tests for all the configuration they set up.

We've talked about the typos issue before. Steve had some thoughts on providing API so that you can get "compile" time errors rather than needing to write tests to check for this.

Bit I can see the need for a shorter term solution too. Unfortunately I just don't have any guidance at the moment. The rampant global state makes it really cumbersome.

On Tue, Oct 28, 2014, 12:03 PM Stian Liknes notifications@github.com wrote:

I like your approach @ewinslow https://github.com/ewinslow, the setup is simple and tests easy to understand. Still, I think it is useful to be able to write integration tests in PHPUnit. Some situations where this can come in handy:

1. Verify that registered entities are instantiated correctly by Elgg (i.e. classnames)
2. Making sure queries are correct (i.e. elgg_get_entities_by_metadata() is easy to get wrong when you're adding a couple of arguments)
3. Verifying that you trigger required hooks
4. Verify that you remembered to do something when some event occured.

In many cases, we rely on typing strings, which is error prone, i.e. attaching a hook to 'entity:url' can be misspelled without any warning. Testing that you retrieve the correct entities in a search goes a long way in avoiding misspelled search arguments.

Example hook verification:

public function test_getURL_shouldTriggerGetUrlHook()
{
    $entity = new ElggObject;
    $entity->getURL();
    $this->assertHookTriggered('entity:url');
}

Example testing entity class:

public function test_save_ShouldBeAssignedMyEntityClass()
{
    $entity = new ElggObject();
    $guid = $entity->save();
    _elgg_invalidate_cache_for_entity($guid);
    $this->assertInstanceOf('MyEntity', get_entity($guid));
}

— Reply to this email directly or view it on GitHub https://github.com/Elgg/Elgg/issues/7408#issuecomment-60812586.

[stianlik]

I like the idea of "compile time" errors, could save much debugging time in some cases.

Testing with global state is not ideal. I have done some experimentation, and came up with a setup that allows me to run tests with the Elgg API and a test database:

1. Create a PHPUnit bootstrap that reads the standard engine/start.php and use eval() to execute a modified version to allow custom configuration.
2. Create a test configuration MY_PLUGIN/tests/settings.php that are loaded after engine/settings.php and before Elgg library is included (mainly to specify login for test database).
3. Create PHPUnit tests that use the Elgg API.
4. Create some helpers to make testing more comfortable

It is definitely hacky, but it works without having to modify the core and will give you a test environment very close to the actual web site. If we were to include the bootstrap, utility class and a default test config in the core repository, all plugin authors need to do is to specify the bootstrap in phpunit.xml and start writing tests. We could default to a database named {dbname}_test for plugins that does not provide a custom configuration.

The biggest downside is that bootstrap.php is sensitive to changes in engine/start.php.

Test setup

Boilerplate bootstrap used to load the engine and test configuration.

MY_PLUGIN/tests/bootstrap.php:

<?php

$engine_dir = dirname(__FILE__) . '/../../../engine';
$test_dir = dirname(__FILE__);

$search = array(
    '<?php',
    'include_once("$engine_dir/settings.php");',
    '__FILE__'
);

$replace =  array(
    '',

    // Allows tests to override settings before Elgg components start working,
    // typically used to specify a test database to avoid data corruption.
    'include_once("$engine_dir/settings.php");' . "include_once('$test_dir/settings.php');",

    "'$engine_dir/start.php'"
);

$engine_bootstrap = file_get_contents("$engine_dir/start.php");
$start = str_replace($search, $replace, $engine_bootstrap);
eval($start);

Test configuration used to selectively override site configuration.

MY_PLUGIN/tests/settings.php:

<?php
global $CONFIG;
$CONFIG->dbname = 'test';
$CONFIG->dbuser = 'test';
$CONFIG->dbpass = 'test';

Test utilities

Utility class that provides a standard methods for saving entities, avoiding cache issues and cleanup after each test.

MY_PLUGIN/tests/ElggEntityTestCase.php:

<?php

abstract class ElggEntityTestCase extends PHPUnit_Framework_TestCase
{
    /**
     * @var ElggEntity[]
     */
    private $entities = array();

    protected function tearDown()
    {
        parent::tearDown();
        elgg_set_ignore_access(true);
        foreach ($this->entities as $entity) {
            $entity->delete();
        }
        $this->entities = array();
        elgg_set_ignore_access(false);
    }

    /**
     * Save an entity by calling $entity->save(). All entities saved through
     * this method will be deleted in @see tearDown() so that subsequent tests
     * are unaffected. Furthermore, the entity cache is cleared after each save
     * operation so that @see get_entity() always will fetch data from the actual
     * database.
     * 
     * @param ElggEntity $entity
     * @return bool|int Return value from $entity->save()
     */
    protected function save(ElggEntity $entity)
    {
        $this->entities[] = $entity;
        $guid = $entity->save();
        $this->clearCache();
        return $guid;
    }

    /**
     * Login user. If already logged in, @see logout() is 
     * performed first.
     * 
     * @param string $username
     * @return \ElggEntityTestCase
     */
    protected function login($username)
    {
        if (elgg_is_logged_in()) {
            $this->logout();
        }
        login(get_user_by_username($username));
        return $this;
    }

    /**
     * Logout user and clear cache.
     * 
     * @return \ElggEntityTestCase
     */
    protected function logout()
    {
        logout();
        $this->clearCache();
        return $this;
    }

    /**
     * @global array $ENTITY_CACHE
     * @return \ElggEntityTestCase
     */
    protected function clearCache()
    {
        global $ENTITY_CACHE;
        $ENTITY_CACHE = array();
        _elgg_get_metadata_cache()->flush();
        return $this;
    }

}

Integration tests

The end result is that plugin authors easily can test functionality that depends on the Elgg engine.

MY_PLUGIN/tests/CompanyTest.php

<?php

class CompanyTest extends ElggEntityTest
{
    private $users = array();

    public function setUp()
    {
        parent::setUp();

        // Create test users
        elgg_set_ignore_access(true);
        $this->users['testuser'] = get_user(register_user('testuser', 'password', 'Test User', 'testuser@test.no'));
        $this->users['testuser2'] = get_user(register_user('testuser2', 'password', 'Test User', 'testuser2@test.no'));
        elgg_set_ignore_access(false);
    }

    public function tearDown()
    {
        parent::tearDown();

        // Delete test users
        elgg_set_ignore_access(true);
        foreach ($this->users as $user) {
            $user->delete();
        }
        elgg_set_ignore_access(false);
    }

    public function test_initializeAttributes_shouldRegisterSubtype()
    {
        $company = new Company;
        $this->assertEquals('company', $company->subtype);
    }

    public function test_save_shouldDefaultPermissionsToLoggedInUsers()
    {
        // Login as user 'testuser' and save a new entity
        $this->login('testuser');
        $company = new Company;
        $guid = $this->save($company);

        // Verify that user 'testuser' can retrieve entity from database
        $this->assertNotEmpty(get_entity($guid), 'entity can be retrieved by owner');

        // Login as user 'testuser2' and verify that entity can be retrieved from database
        $this->login('testuser2');
        $this->assertNotEmpty(get_entity($guid), 'entity can be retrieved by authenticated');

        // Logout and verify that guests cannot access entity
        $this->logout();
        $this->assertEmpty(get_entity($guid), 'entity cannot be retrieved by guests');
    }

}

[ewinslow]

Seems like something you can already start doing?

On Sat, Nov 1, 2014, 11:00 AM Stian Liknes notifications@github.com wrote:

I like the idea of "compile time" errors, could save much debugging time in some cases.

Testing with global state is not ideal. I have done some experimentation, and came up with a setup that allows me to run tests with the entire Elgg API and a test database:

1. Create a PHPUnit bootstrap that reads the standard engine/start.php and use eval() to execute a modified version to allow custom configuration.
2. Create a test configuration MY_PLUGIN/tests/settings.php that are loaded after engine/settings.php and before Elgg library is included (mainly to specify login for test database).
3. Create PHPUnit tests that use the Elgg API.
4. Create some helpers to make testing more comfortable

It is definitely hacky, but it works without having to modify the core and will give you a test environment very close to the actual web site. If we were to include the bootstrap, utility class and a default test config in the core repository, all plugin authors need to do is to specify the bootstrap in phpunit.xml and start writing tests. We could default to a database named {dbname}_test for plugins that does not provide a custom configuration. Test setup

Boilerplate bootstrap used to load the engine and test configuration.

MY_PLUGIN/tests/bootstrap.php:

<?php$engine_dir = dirname(FILE) . '/../../../engine';$test_dir = dirname(FILE);$search = array( '<?php', 'include_once("$engine_dir/settings.php");', 'FILE');$replace = array( '', // Allows tests to override settings before Elgg components start working, // typically used to specify a test database to avoid data corruption. 'include_once("$engine_dir/settings.php");' . "include_once('$test_dir/settings.php');", "'$engine_dir/start.php'");$engine_bootstrap = file_get_contents("$engine_dir/start.php");$start = str_replace($search, $replace, $engine_bootstrap);eval($start);

Test configuration used to selectively override site configuration.

MY_PLUGIN/tests/settings.php:

<?phpglobal $CONFIG;$CONFIG->dbname = 'test';$CONFIG->dbuser = 'test';$CONFIG->dbpass = 'test';

Test utilities

Utility class that provides a standard methods for saving entities, avoiding cache issues and cleanup after each test.

MY_PLUGIN/tests/ElggEntityTestCase.php:

<?phpabstract class ElggEntityTestCase extends PHPUnit_FrameworkTestCase{ /* \ @var ElggEntity[] / private $entities = array(); protected function tearDown() { parent::tearDown(); elgg_set_ignore_access(true); foreach ($this->entities as $entity) { $entity->delete(); } $this->entities = array(); elgg_set_ignoreaccess(false); } /* * Save an entity by calling $entity->save(). All entities saved through * this method will be deleted in @see tearDown() so that subsequent tests * are unaffected. Furthermore, the entity cache is cleared after each save * operation so that @see getentity() always will fetch data from the actual * database. * * @param ElggEntity $entity * @return bool|int Return value from $entity->save() / protected function save(ElggEntity $entity) { $this->entities[] = $entity; $guid = $entity->save(); $this->clearCache(); return $guid; } /* * Login user. If already logged in, @see logout() is * performed first. * * @param string $username * @return \ElggEntityTestCase / protected function login($username) { if (elgg_is_logged_in()) { $this->logout(); } login(get_user_byusername($username)); return $this; } /* * Logout user and clear cache. * * @return \ElggEntityTestCase / protected function logout() { logout(); $this->clearCache(); return $this; } /* * @global array $ENTITY_CACHE * @return \ElggEntityTestCase */ protected function clearCache() { global $ENTITY_CACHE; $ENTITY_CACHE = array(); _elgg_get_metadata_cache()->flush(); return $this; }}

Integration tests

The end result is that plugin authors easily can test functionality that depends on the Elgg engine.

MY_PLUGIN/tests/CompanyTest.php

<?phpclass CompanyTest extends ElggEntityTest{ private $users = array(); public function setUp() { parent::setUp(); // Create test users elgg_set_ignore_access(true); $this->users['testuser'] = get_user(register_user('testuser', 'password', 'Test User', 'testuser@test.no')); $this->users['testuser2'] = get_user(register_user('testuser2', 'password', 'Test User', 'testuser2@test.no')); elgg_set_ignore_access(false); } public function tearDown() { parent::tearDown(); // Delete test users elgg_set_ignore_access(true); foreach ($this->users as $user) { $user->delete(); } elgg_set_ignore_access(false); } public function test_initializeAttributes_shouldRegisterSubtype() { $company = new Company; $this->assertEquals('company', $company->subtype); } public function test_save_shouldDefaultPermissionsToLoggedInUsers() { // Login as user 'testuser' and save a new entity $this->login('testuser'); $company = new Company; $guid = $this->save($company); // Verify that user 'testuser' can retrieve entity from database $this->assertNotEmpty(get_entity($guid), 'entity can be retrieved by owner'); // Login as user 'testuser2' and verify that entity can be retrieved from database $this->login('testuser2'); $this->assertNotEmpty(get_entity($guid), 'entity can be retrieved by authenticated'); // Logout and verify that guests cannot access entity $this->logout(); $this->assertEmpty(get_entity($guid), 'entity cannot be retrieved by guests'); }}

— Reply to this email directly or view it on GitHub https://github.com/Elgg/Elgg/issues/7408#issuecomment-61377003.

[stianlik]

Yes, I'm using this setup in a project and it seems to work pretty well. I'm considering to add the bootstrap / utility in a branch of Elgg 1.9 so that I could get started testing quicker in my next project. Is this something that could be useful to have in the repository?

[ewinslow]

I think the most we'd do in core is refactor the start code so you can do this setup without eval. I'd really prefer to just push towards less global state though. Was just dealing with our simple tests yesterday and they are incomprehensible when they fail, and it feels like they all fail together, so it really doesn't help find the problem any quicker.

[stianlik]

Created a PR with modified start code and some documentation.

[jdalsem]

With a seperate plugin for this: https://github.com/stianlik/elgg-phpunit what is needed regarding this issue to keep this open?

[stianlik]

No need to keep it open, seems like my last comment "unclosed" the issue.