Following sections sometimes get executed after an unexpected exception is thrown

[cschreib]

Sections allow executing the same test case multiple time with only some of the logic varying. When an unexpected exceptions is thrown during one of these executions, we have two choices:

• either stop the test case completely and do not execute the remaining sections, or
• only stop the current execution, and let following sections execute in the next runs.

Currently, snitch is doing sometimes one and sometimes the other. It depends on whether the test case has had a chance to explore and discover (but not execute) all sections before the unexpected exception is thrown.

For example:

TEST_CAST("test") {
    SECTION("section 1") {
        // some stuff that doesn't throw
    }
    SECTION("section 2") {
        throw std::runtime_error("bad section 2");
    }
    SECTION("section 3") {
        throw std::runtime_error("bad section 3");
    }
}

will execute all sections. This is because on first entry we only execute "section 1", then discover (but skip over) "section 2" and "section 3". The test case now knows that is has three sections to execute, and will do so. Therefore, it runs the test again and "section 2" fails by throwing, and it runs it one more time to execute "section 3" as well.

However if we remove the first non-throwing section like so:

TEST_CAST("test") {
    SECTION("section 2") {
        throw std::runtime_error("bad section 2");
    }
    SECTION("section 3") {
        throw std::runtime_error("bad section 3");
    }
}

then only "section 2" is executed. This is because "section 3" was never discovered; we threw before we could get there. The test running never realizes that "section 3" exists, and does not try to run the test case again, thinking "section 2" was the last.

This is not a consistent behavior, and should be fixed. The README says (and snitch tests also expect) that we should stop the whole test case when an exception is thrown, so I believe the correct solution is we should fix the first example so that "section 3" does not run.