Displaying of passed tests using -p flag

[MariusDe]

Hi, first thank you for developing lest, which I find a really clean & easy to use testing framework. In today's world of increasingly complex software, it is refreshing to see cleanly designed testing frameworks like lest that are so easy to integrate, learn and use .. that one can simply focus on the tests! Please keep developing and maintaining this framework, I think it has a lot of potential.

I just started to run my first tests (using lest 1.24.3) and noticed something questionable (from the user perspective) when using -p flag. When all modules (added using lest_MODULE()) pass, the tests are displayed on the console. But when one of the modules fail (in my case I have two modules, the 1st module fails and the 2nd passes), the output shows the failed module and "1 out of 2 selected tests failed.". So the 2nd module (which passed) was not printed on console, as one would expect when using -p. Is this the expected behavior for -p in the above use-case?

Cheers, Marius

[martinmoene]

Thanks for the praise ;)

With option -p (--pass) all assertions are expected to be reported.

I'm unable to replicate the effect you describe. Perhaps it's easiest to share your code, so I can have a look at it?

cheers, Martin

[MariusDe]

Hi Martin, my reply comes a bit late, but only tonight I had time. Here is the sample I extracted from my existing code (I hope it compiles in this form). So after playing with it a bit more, the use-case looks simpler: -p prints correctly until a failing check, after which passed checks will no longer be printed.

////////////// test.cpp
#include "main.h"

// note: this module is executed after another one, which is here omitted as it is not relevant
const lest::test 2_Module[] =
{
    CASE( "2 Module test" )
    {
            EXPECT(0 == 1); // always printed on console
            EXPECT(2 == 2); // not printed when -p used and a previous failure already encountered
    },

    CASE( "Another dummy test" )
    {
      static int x = 1;
      EXPECT(x == 1); // not printed when -p is used, as 0 == 1 check above failed. 
    },
};

lest_MODULE( getSpecs(), 2_Module );

////////////// main.cpp

lest::tests& getSpecs()  // this function shall be in main.h
{
    static lest::tests tests;
    return tests;
}

int main( int argc, char * argv[] )
{
  return lest::run( getSpecs(), lest::texts( argv + 1, argv + argc ), std::cout );
}

In case you will still not manage to reproduce this, I will share some environment details (MinGW / gcc + Eclipse on Win 7).

Cheers, Marius

[martinmoene]

It's esssentially as in example 12-module-1.cpp and 12-module-2.cpp.

Note: 2_Module => Module2.

// main.h

#ifndef MAIN_H_INCLUDED
#define MAIN_H_INCLUDED

#include "lest.hpp"

lest::tests& getSpecs();

#endif // MAIN_H_INCLUDED

// main.cpp

#include "main.h"

lest::tests& getSpecs() 
{
    static lest::tests tests;
    return tests;
}

int main( int argc, char * argv[] )
{
#if 0
    return lest::run( getSpecs(), lest::texts( argv + 1, argv + argc ), std::cout );
#else
    // I'd preferr:
    return lest::run( getSpecs(), argc, argv, std::cout );
#endif
}

#if 0
cl -nologo -W3 -EHsc -I. -Femain.exe main.cpp test.cpp && main --pass
g++ -Wall -Wextra -std=c++11 -I. -o main.exe main.cpp test.cpp && main --pass
#endif

// test.cpp

#include "main.h"

const lest::test Module2[] =
{
    CASE( "2 Module test" )
    {
        EXPECT(0 == 1); 
        EXPECT(2 == 2); 
    },

    CASE( "Another dummy test" )
    {
        static int x = 1;
        EXPECT(x == 1);
    },
};

lest_MODULE( getSpecs(), Module2 );

This works for me and prints:

prompt>g++ -Wall -Wextra -std=c++11 -I. -o main.exe main.cpp test.cpp && main --pass
test.cpp:10: failed: 2 Module test: 0 == 1 for 0 == 1
test.cpp:17: passed: Another dummy test: x == 1 for 1 == 1
1 out of 2 selected tests failed.

cheers, Martin

[MariusDe]

Martin, thanks, I think what I see may be environment dependent. So in another day I will take the above sample files and reproduce this with your commands (but with gcc 4.8.1 on Windows).

Cheers, Marius

[MariusDe]

Martin, I took the above and tried to reproduced the issue without success outside of the IDE. Sorry, this is an issue which is not related to lest framework. From my side you may close this issue.

Cheers, Marius

[martinmoene]

Ok, thanks & cheers, Martin