Open cgay opened 3 years ago
Don't display "0 benchmarks", and if there are no tests (because it's a benchmark suite) then don't display "0 tests". The two cases can be symmetric.
None of this makes sense:
$ ../_build/bin/time-test-suite --test test-load-tzif-version-1
Running test test-load-tzif-version-1: NOT IMPLEMENTED in 0.000822s and 68KiB
test-load-tzif-version-1 not implemented
Ran 0 checks: PASSED
Ran 1 test: PASSED (1 not implemented)
NOT IMPLEMENTED in 0.000822 seconds
I still find this output for failed checks to be pretty bad:
FAILED: $expected-failure = test-results.result-status [#"expected-failure" and #"unexpected-success" are not =.]
I think better would be
FAILED: #"expected-failure" ~= #"unexpected-success" for assert-equal($expected-failure, test-results.result-status)
Or even better, a multi-line display:
FAILED: assert-equal($expected-failure, test-results.result-status)
Expected: #"expected-failure"
Got: #"unexpected-success"
Current assert-equal
output now looks like this:
FAILED: lox-test-suite
FAILED: test-eval-precedence
FAILED: 7 = %eval("2*(3+4)/3;")
want: 7
got: 4.6666667d0
FAILED: test-eval-program
FAILED: #[1, 2, 3, 4] = #[1, 2, 5, 4]
want: #[1, 2, 3, 4]
got: #[1, 2, 5, 4]
detail: element 2 is the first mismatch
I'd like to see several improvements to Testworks output:
--progress failures
I suppose. i.e., don't display pass, not implemented, expected to fail.test-output:testworks
and should by default go totest-temp-directory()/test-name.log
and there can be an option to also send it to the console.--test-output=all
, for example...not sure of name.make check
(--report json --report-file foo.json
) I still want the final count of passing/failing tests to be shown on the terminal.--load
so they can be associated with their .so file and we can execute them in that same order and display the .so filename before any progress output for that set of tests. (The output frommake check
is currently pretty random and hard to know what tests belong to what library.)should become something like this
Those are my thoughts. We'll see how it develops.