dylan-lang / testworks

Test framework for Dylan
http://dylan-lang.github.io/testworks/
5 stars 5 forks source link

Improve test run output #139

Open cgay opened 3 years ago

cgay commented 3 years ago

I'd like to see several improvements to Testworks output:

Those are my thoughts. We'll see how it develops.

housel commented 3 years ago

Don't display "0 benchmarks", and if there are no tests (because it's a benchmark suite) then don't display "0 tests". The two cases can be symmetric.

cgay commented 2 years ago

None of this makes sense:

$ ../_build/bin/time-test-suite --test test-load-tzif-version-1
Running test test-load-tzif-version-1: NOT IMPLEMENTED in 0.000822s and 68KiB

test-load-tzif-version-1 not implemented

Ran 0 checks: PASSED
Ran 1 test: PASSED (1 not implemented)
NOT IMPLEMENTED in 0.000822 seconds
cgay commented 1 year ago

I still find this output for failed checks to be pretty bad:

FAILED: $expected-failure = test-results.result-status [#"expected-failure" and #"unexpected-success" are not =.]

I think better would be

FAILED: #"expected-failure" ~= #"unexpected-success" for assert-equal($expected-failure, test-results.result-status)

Or even better, a multi-line display:

FAILED: assert-equal($expected-failure, test-results.result-status)
    Expected: #"expected-failure"
    Got: #"unexpected-success"
cgay commented 3 months ago

Current assert-equal output now looks like this:

FAILED: lox-test-suite
  FAILED: test-eval-precedence
    FAILED: 7 = %eval("2*(3+4)/3;")
      want: 7
      got:  4.6666667d0
  FAILED: test-eval-program
    FAILED: #[1, 2, 3, 4] = #[1, 2, 5, 4]
      want: #[1, 2, 3, 4]
      got:  #[1, 2, 5, 4]
      detail: element 2 is the first mismatch