MithrilJS / ospec

Noiseless testing framework
MIT License
48 stars 13 forks source link

re-think the results output #55

Open pygy opened 1 year ago

pygy commented 1 year ago

We currently shove every kind of problems into the results array as assertions, even timeouts and bail outs.

The latter two should be tagged as such IMO.

Also, we shouldn't format the output on error, but in the reporter (with a possible exception for custom assertions).

Here's the current format:


{
    pass: null,
    message: "Incomplete assertion in the test definition starting at...",
    error: $$_testOrHook.error,
    task: $$_testOrHook,
    timeoutLimbo: $$_timedOutAndPendingResolution === 0,
    // Deprecated
    context: ($$_timedOutAndPendingResolution === 0 ? "" : "??? ") + $$_testOrHook.context,
    testError: $$_testOrHook.error
}

We should aim for something like this:

{
    kind: "assertion" | "error" | "timeout",
        pass: boolean,
    message?: {value, methodName, reference} | string
        error?: Error // or maybe just a stack trace?
    task: Task
}

This is a prerequisite for #31.

The "incomplete assertion" checks should run on task finalization time, and on timeout, and cause a hard error to be thrown.

Likewise, syntax errors while parsing a test file should cause hard errors, not bailouts.

At long last, it would be useful to be able to bulk-add results, such that we can parallelize running the tests and merge results.