gyrovorbis / libgimbal

C17-based extended standard library, cross-language type system, and unit testing framework targeting Sega Dreamcast, Sony PSP and PSVita, Windows, Mac, Linux, Android, iOS, and WebAssembly.
http://libgimbal.elysianshadows.com
MIT License
73 stars 4 forks source link

Add signals for catching failed GblTestScenarios and GblTestSuites #12

Closed gyrovorbis closed 4 months ago

gyrovorbis commented 4 months ago

Thanks to @suicvne for raising this issue:

It is often desirable for the user to be able to capture or filter over all of the failing test cases when executing a series of GblTestSuites from a GblTestScenario. You can currently do this by:

1) Overriding GblTestSuiteClass::pFnCaseRun() to capture the result code. 2) Iterating over the test suites within a test scenario and logging the failing issue:

for(int i = 0; i < GblObject_childCount(GBL_OBJECT(pScenario)); ++i) {
    GblTestSuite* pSuite = GblObject_findChildByIndex(GBL_OBJECT(pScenario), i);
    if(pSuite->casesFailed) {
        printf("%s has reested with code %s at %s:%s:%zu\n",
               GblObject_name(GBL_OBJECT(pSuite)), gblResultString(pSuite->failingIssue.result),
               pSuite->failingIssue.sourceLocation.pFile, pSuite->failingIssue.sourceLocation.pFunc,
               pSuite->failingIssue.sourceLocation.line);
    }
}

Unfortuantely there are a few shortcomings: 1) There is no way to retrieve the name of or function pointer to the exact test case that failed. 2) This would be SO MUCH MORE PRISTINE from an API perspective using the signal/slots mechanism to emit a signal when a test case fails.

gyrovorbis commented 4 months ago

Alighty, should be done now.

I'm now storing just the GBL_RESULT code from every test case, which can be queried at any time (will return GBL_RESULT_UNKNOWN if the test hasn't even run yet), using GblTestSuite_caseResult(); however, I didn't think it was RAM prudent to store the actual GblCallRecord with source context and error message for every single test case, as that would get pretty big for Dreamcast and PSP, especially when you have a buttload of tests...

Instead, the GblTestSuite stores the most recent test case's GblCallRecord, which means you can query for it while the tests are executing within a callback connected to the new signals I've just added to GblTestScenario... Note that when a test case fails, the GblTestSuite stops executing and skips over all remaining cases, so this message also serves as the persistent "failing error message" for a failed GblTestSuite.

Anyway, here's how to actually do what you want... You can now connect a callback to GblTestScenario's new signals that will allow you to log or store failing test case information as the test is executing:

static void GblTestScenario_onCaseEnded_(GblTestScenario* pSelf, GblTestSuite* pSuite, size_t caseIndex) {
    const GBL_RESULT result = GblTestSuite_caseResult(pSuite, caseIndex);

    printf("[%s]: %s.%s.%s\n",
               gblResultString(result),
               GblObject_name(GBL_OBJECT(pSelf)),
               GblTestSuite_name(pSuite),
               GblTestSuite_caseName(pSuite, caseIndex));

    // If we failed the test case, add extra context information           
    if(GBL_RESULT_ERROR(result))
        printf("\t'%s' from %s @ %s:%zu\n",
                   pSuite->lastRecord.message,
                   pSuite->lastRecord.srcLocation.pFile,
                   pSuite->lastRecord.srcLocation.pFunc,
                   pSuite->lastRecord.srcLocation.line);
}

int main(int argc, char* argv[]) {
    GblTestScenario* pScenario = GblTestScenario_create("My Scenario");

    // Connect the "caseEnded" signal to our callback to get notified every time a test case runs
    GBL_CONNECT(pScenario, "caseEnded", GblTestScenario_onCaseEnded_);

    GblTestScenario_addSuite(pScenario, GblTestSuite_create(MY_TEST_SUITE_TYPE));

    const GBL_RESULT result = GblTestScenario_run(pScenario);

    GblTestScenario_unref(pScenario);

    return GBL_RESULT_ERROR(result)? EXIT_FAILURE : EXIT_SUCCESS;
}

Note that I've also added the following signals to GblTestScenario which can let you trigger custom logic at different times during test execution: