SKIPm doesn't print anything

[bebehei]

I just used SKIPm to skip a test in a special case. But in the case of a skipped test, I assumed that SKIPm will actually print out its message parameter.

I wanted to use it, to skip a special test but also not doing it silently, so I chose SKIPm over SKIP.

But, sad news, SKIPm does not print out anything.

Here's a small reproducer:

// example.c
#include <stdlib.h>
#include <stdio.h>
#include <assert.h>

#include "greatest.h"

TEST standalone_test(void) {
        SKIPm("help me");
}

GREATEST_MAIN_DEFS();

int main(int argc, char **argv) {
    GREATEST_MAIN_BEGIN();
    RUN_TEST(standalone_test);
    GREATEST_MAIN_END();
}

Actual Output

[bebe:~/greatest] > gcc -o skipm ./example.c
[bebe:~/greatest] > ./skipm
s
Total: 1 test (90 ticks, 0.000 sec), 0 assertions
Pass: 0, fail: 0, skip: 1.

Desired/Expected output

[bebe:~/greatest] > gcc -o skipm ./example.c
[bebe:~/greatest] > ./skipm
skipped standalone_test: help me
Total: 1 test (90 ticks, 0.000 sec), 0 assertions
Pass: 0, fail: 0, skip: 1.

[silentbicycle]

It printed an s above -- greatest only prints the full message for failures by default (not SKIPm or PASSm), it just prints an s. If you run the above with verbosity of > 0 (using -v or greatest_set_verbosity(1);) it should, though:

$ ./skipm 
s
Total: 1 test (37 ticks, 0.000 sec), 0 assertions
Pass: 0, fail: 0, skip: 1.
$ ./skipm -v
SKIP standalone_test: help me (0 ticks, 0.000 sec)

Total: 1 test (47 ticks, 0.000 sec), 0 assertions
Pass: 0, fail: 0, skip: 1.

I think of a SKIP as distinct from PASS but less severe than a FAIL, so it traces them out as s rather than ., but only adds more detail on request. It also doesn't affect the return status of the CLI test runner. I tend to use SKIPs either as an informal todo list during early active development, or to appease tests I know will be temporarily broken while in the middle of refactoring, as well as skipping tests in specific environments (such as your musl+valgrind case).

I see what you mean about expecting a "message" form to always print the message -- I'm going to post another version of greatest (1.4.1) soon, and I'll think about how to rephrase the documentation to make that distinction clearer. It's the 'm' message variant of SKIP in that you can set a custom message -- with SKIP();, it would just print SKIP testname (no further details) with -v. PASSm lets you set a custom pass, in case there are multiple ways for a test to pass (which I haven't used much), and FAILm lets you set a custom failure message, if the default one generated based on ASSERT_EQ or whatever could use extra context.

Does that distinction make sense? Is there a way you'd suggest describing them to make it clearer?

[bebehei]

Thanks for your rapid answer.

Well, I've missed the -v switch. That might work already.

Does that distinction make sense?

In some terms yes, but not really to me yet. What really confuses me:

	FAIL	FAILm	PASS	PASSm	SKIP	SKIPm
show the message	done automatically	done automatically	use -v	use -v	use -v	use -v
mute the message	impossible	impossible	do not use -v	do not use -v	do not use -v	do not use -v

You've got there a differentiation between positive and negative results. But IMO the m of (SKIP|PASS|FAIL)m actually shows the intention to print the message. Otherwise I could have used the plain (SKIP|PASS|FAIL).

And this isn't clear. I'd love to see some docs explaining exactly this table above, that the m parts of positive results are only printed, if verbose mode is activated.

[silentbicycle]

It may be worth adding an option to make particular SKIPs always print their message, either by a flag (SKIPm(true, "this always prints");) or a function that overrides the default behavior in the table above (greatest_always_print_result(1);) until the current test completes. The former would break reverse compatibility pretty badly; since the latter is only an additive interface chance, it could go in 1.5.0. Thoughts?

Otherwise: before the 1.4.1 release, I'll update the docs to make it clearer that only FAIL always prints its message, and that the m message forms allow you to set a non-default message, but don't impact verbosity/messaging behavior otherwise.

It probably wouldn't hurt to have a new subsection about verbosity levels and message formats. I use greatest_get_verbosity() quite a bit (I'm currently working on a succinct data structure library that prints extra stats with -v -v and hexdumps the packed forms with -v -v -v, for example), but it doesn't get much emphasis in the README.

[silentbicycle]

I added comments to the documentation to clarify this. I'm going to think about whether there should be a function to toggle always printing SKIP custom messages, but if so, that will have to wait to 1.5.0.

[soul4soul]

Independently I came with with using multiple -v's to add more verbosity for custom debug print outs too. Initially I was using a debug flag to indicate to print out more messages until I found that greatest_get_verbosity() was available. That was a great upgrade to the test app since recompiling was no longer necessary to get debug messages.

In practice developers here general use no -v's or two -v's for informal testing and debugging. Formal testing is always run with a single -v.

I don't particularly see the need for more message control but I guess customization isn't such a bad thing. The documentation can at least be updated to discuss the availability of greatest_get_verbosity().

Lastly I only skip tests that physically can't be run for one reason or another such as the environment can't support the test case.

[silentbicycle]

You're right -- greatest_get_verbosity() isn't mentioned in the readme. It really should be!

I make heavy use of custom verbosity levels.