Assert collections vs assert stacks

[schwernbot]

From: @schwern Date: Saturday Sep 11, 2010 at 01:41 GMT Orig: https://github.com/Test-More/test-more/issues/63

There are three basic classes of asserts one might write.

The simplest is an assert which calls another assert. It takes the result of the assert, possibly modifies it, and sends it on. This is what most folks will do.

sub is {
    my($have, $want, $name) = @_;

    my $ok = ok( $have eq $want, $name );
    $ok->add_diagnostics( have => $have, want => $want );
    return $ok;
}

There's the assert which may contain other independent asserts, like Test::Warn and Test::Exception. The asserts called are unattached to the assert they're being called from.

no_warnings_ok {
    ok( $obj->foo );
    ok( $obj->bar );
};

There's the assert which is really just a collection of asserts and doesn't really have an assert of its own. The trick here is you still want the internal asserts to report from where the wrapper was called, not inside the wrapper.

sub sanity_check_ok {
    my $obj = shift;
    ok $obj->id, "has an id";
    ok $ok->name, "has a name";
    ok !$ok->errors, "it has no errors";
}

Finally there's the assert which happens to contain a bunch of asserts but wants to act like one assert. In this case, if any one assert fails the whole thing fails.

sub file_contents_ok {
    my($have_file, $want_file, $name) = @_;

    my $have = eval { $have_file->slurp };
    # yeah, I know I shouldn't check $@, makes the example easier
    ok !$@, "slurp $have_file";

    my $want = eval { $want_file->slurp };
    ok !$@, "slurp $want_file";

    my $ok = is $have, $want, $name;
    $ok->add_diagnostics(
        have_file => $have_file,
        want_file => $want_file
    );

    return $ok;
}

Some would say the above should throw an exception if the slurps fail. Maybe, but for the purposes of this example we're not. What comes out is a single result object that contains the three internal asserts as siblings. The parent result is the cross product of the siblings. The formatter can then decide if it wants to render that as a single item or multiple items or as one test made up of subtests. The parent result fails if any of the asserts failed, but takes its name and diagnostics from the returned result.

Each of these would require different wrappers and be declared with different functions in TB2::Module.

[schwernbot]

From: @pdl Date: Saturday Sep 15, 2012 at 19:07 GMT Orig: https://github.com/Test-More/test-more/issues/63#issuecomment-8587812

I have another one, although perhaps I'm abusing the word 'assert'...

I'd like to write a test which allows one of several different things to be true. I want to be able to include diagnostic information in the same way as code that only tests one thing. So as a rough example:

sub matches_any{
    my ($have, $want, $reason) = @_;
    return ok (
        grep{
            matches($have, $_);
        } @$want,
        $reason
    );
}

... but you could potentially also want to do weirder things like matching at least 4 but no more than 7 of 11 tests.

The point being asserts are true/false but also occupy a space on the bad/good/neutral spectrum: The end-user/module developer ultimately may need diags on which ones match and which ones don't in the same way as a sub-assert, but with the exception that 'doesn't match' doesn't mean that the parent assert is broken. Where do those diags fit into TB2?