search

Shinmera / parachute

An extensible and cross-compatible testing framework.
https://shinmera.github.io/parachute
zlib License
97 stars 9 forks source link

Failed test isn't shown #33

Closed TatriX closed 3 years ago

TatriX commented 3 years ago

Hi! When running my test suite I'm getting the following results:

(run)
 #| ... skip passed tests ... #|
  0.120 ✔ SERVER-TESTS::USERS/SET-PASSWORD
        ？ SERVER-TESTS::ADMIN/USERS
        ？   SKIP: TODO
  0.000 ー     #| snip |#
  0.000 ✔   SKIP: TODO
  0.000 ✔ SERVER-TESTS::ADMIN/USERS
        ？ SERVER-TESTS::USERS/INVITES
WARNING: Unhandled error when evaluating SERVER-TESTS::USERS/INVITES:
  Request failed. Status = 500

  0.097 ✘ SERVER-TESTS::USERS/INVITES

;; Summary:
Passed:     6
Failed:     0
Skipped:    1

;; Failures:
   0/   0 tests failed in SERVER-TESTS::USERS/INVITES

Number of Failed tests should be 1, rather then 0. Same with Failures. I know it's hard to debug without a simple test case, but perhaps it's still possible? Otherwise I'll try to find a way to reproduce the problem with a short test case.

Thanks!

Shinmera commented 3 years ago

That looks like an error outside of an is, etc. form, which is considered an error in how the test itself is written, rather than a failure of what you're testing, and thus doesn't show in the report. You should pack your failing form in a finish form to capture it as a test if you want its failure to be reported.

TatriX commented 3 years ago

Ah, I see, thanks! Perhaps this deserves a mention in a readme/docs.