Explore solutions for multi back-end unit tests

[stoewer]

Explore solutions for multi back-end unit tests

[stoewer]

Just found a stackoverflow question about parameterized tests with cpp-unit:

https://stackoverflow.com/questions/290099/parameterizing-a-test-using-cppunit

[jgrewe]

looks interesting but introduces a further, I will try it the next days

[stoewer]

I just tested two different ways how unit tests can be executed for different back-ends. Sadly both variants only work if the back-ends to test are already known at compile time. I implemented them in a small toy project.

Here is the class under test I used in my small toy example:

class Foo {
    std::string foo;
public:

    Foo(const std::string foo);
    std::string get_foo() const;
};

The first version I tried uses inheritance in order to create test suites that have their own setUp() and tearDown() methods. Here is how this looks like:

class TestFooInherit : public CPPUNIT_NS::TestFixture {

protected:
    Foo foo;

public:
    TestFooInherit()
            : foo("")
    {}

    void test_foo() {
        CPPUNIT_ASSERT(foo.get_foo().size() > 0);
        std::cout << std::endl << foo.get_foo() << std::endl;
    }
};

class TestFooInherit1 : public TestFooInherit {

    CPPUNIT_TEST_SUITE(TestFooInherit1);
        CPPUNIT_TEST(test_foo);
    CPPUNIT_TEST_SUITE_END();

public:
    void setUp() {
        foo = Foo("blafasel");
    }

    void tearDown() {}
};

class TestFooInherit2 : public TestFooInherit {

    CPPUNIT_TEST_SUITE(TestFooInherit2);
        CPPUNIT_TEST(test_foo);
    CPPUNIT_TEST_SUITE_END();

public:
    void setUp() {
        foo = Foo("somethingelse");
    }

    void tearDown() {}
};

Since the actual test suite is defined in the child classes this method allows to disable tests of unimplemented featured during the development of a back-end.

Here is the other version that works with templates:

struct Param1 {
    std::string param = "balfasel";  // this would be our back-end name in NIX
};

struct Param2 {
    std::string param = "somethingelse";
};

template<class PARAM>
class TestFooTemplate : public CPPUNIT_NS::TestFixture, public PARAM {

    CPPUNIT_TEST_SUITE(TestFooTemplate<PARAM>);
        CPPUNIT_TEST(test_foo);
    CPPUNIT_TEST_SUITE_END();

protected:
    Foo foo;

public:
    TestFooTemplate()
            : PARAM(), foo("")
    {}

    void setUp() {
        foo = Foo(this->param);
    }

    void tearDown() {}

    void test_foo() {
        CPPUNIT_ASSERT(foo.get_foo().size() > 0);
        std::cout << std::endl << foo.get_foo() << std::endl;
    }
};

This variant requires only a very small amount of additional code, but it has also some disadvantages:

• It is not possible to disable single tests (which can also be seen as an advantage)
• Since the test class is now a template all methods have to be implemented in the header

This is how the test suite registration looks like in the runner:

CPPUNIT_TEST_SUITE_REGISTRATION(TestFooInherit1); 
CPPUNIT_TEST_SUITE_REGISTRATION(TestFooInherit2);
CPPUNIT_TEST_SUITE_REGISTRATION(TestFooTemplate<Param1>);
CPPUNIT_TEST_SUITE_REGISTRATION(TestFooTemplate<Param2>);

[stoewer]

Btw. the test suite registration is just one part. We also have to find a solution for adding the tests with add_test to ctest (or we have to put e.g. template specializations in separate files and name them accordingly).

[jgrewe]

If I understand correctly, the inheritance version requires to override the setUp and tearDown methods and to set up the testSuite. Imho, this is an effort one can live with. Plus it grants some more flexibility (and risk) by leaving out tests. This, in my opinion, is crucial for the development phase of a backend. The question for me is here if the solution, beside being more elegant, is better than the way I chose so far. The effort is similar, it is not "out of the box"...

The second approach saves this effort but is inflexible during development phase. I am afraid, that this does not really help.

[stoewer]

If I understand correctly, the inheritance version requires to override the setUp and tearDown methods and to set up the testSuite. Imho, this is an effort one can live with. Plus it grants some more flexibility (and risk) by leaving out tests. This, in my opinion, is crucial for the development phase of a backend.

I also think that this is mostly an advantage.

The question for me is here if the solution, beside being more elegant, is better than the way I chose so far.

It is indeed similar. But I think using inheritance is nicer than duplicating all fixtures (e.g. file <-> file_fs, file_other <-> file_other_fs etc.)

The second approach saves this effort but is inflexible during development phase. I am afraid, that this does not really help.

Agreed. If there are no other objections I'll try to apply the inheritance version to all our front-end tests.

[gicmo]

Seems like the second approach is the way to go. Only one thing: if we were to adopt a different unit testing framework (like gtest, which is more modern, has clion support, etcpp ) now would be a good time I guess.

[gicmo]

Args, seems I got confused about which approach to call first and second. I meant the inheritance one is the way to go - if we stick we CppUnit.