Shinmera / parachute

An extensible and cross-compatible testing framework.
https://shinmera.github.io/parachute
zlib License
94 stars 9 forks source link

Add notes about aborted tests at the end of plain report #12

Closed phoe closed 1 year ago

phoe commented 4 years ago

Adding my two cents to the recent discussion about unexpected errors in tests.

Let's assume this following Lisp code that uses Parachute:

(defgeneric frobnicate (a b)
  (:documentation "blah blah"))

(defmethod frobnacite (a b)
  (+ a b))

(define-test basic-frobnicate
  (is = (frobnicate 2 2) 4))

If someone can spot the typo in the DEFMETHOD then they can infer that this test is going to fail:

SET> (parachute:test 'riichi-evaluator.test::basic-frobnicate)

        ? RIICHI-EVALUATOR.TEST::BASIC-FROBNICATE
WARNING:
   Unhandled error when evaluating RIICHI-EVALUATOR.TEST::BASIC-FROBNICATE:
  There is no applicable method for the generic function
    #<STANDARD-GENERIC-FUNCTION RIICHI-EVALUATOR.TEST::FROBNICATE (0)>
  when called with arguments
    (2 2).
See also:
  The ANSI Standard, Section 7.6.6

  0.004 ✘ RIICHI-EVALUATOR.TEST::BASIC-FROBNICATE

;; Summary:
Passed:     0
Failed:     0
Skipped:    0

;; Failures:
   0/   0 tests failed in RIICHI-EVALUATOR.TEST::BASIC-FROBNICATE
#<PARACHUTE:PLAIN 1 results>

The summary is particularly confusing:

;; Summary:
Passed:     0
Failed:     0
Skipped:    0

;; Failures:
   0/   0 tests failed in RIICHI-EVALUATOR.TEST::BASIC-FROBNICATE

The summary reads as if nothing has gone wrong in the test. This might be an issue if someone wants to skip the whole test output and look at the summary.

I suggest adding the following lines at the end if any of the tests have failed due to unexpected errors:

   1 test aborted due to an unexpected error:
     RIICHI-EVALUATOR.TEST::BASIC-FROBNICATE

So in result, we get the following output that should ultimately be clear:

;; Summary:
Passed:     0
Failed:     0
Skipped:    0

;; Failures:
   0/   0 tests failed in RIICHI-EVALUATOR.TEST::BASIC-FROBNICATE
   1 test aborted due to an unexpected error:
     RIICHI-EVALUATOR.TEST::BASIC-FROBNICATE
phoe commented 4 years ago

A fix is to (is = 4 (...) instead of (is = (...) 4) - ensure the order of arguments.

Shinmera commented 1 year ago

You can bind *silence-plain-compilation-errors-p* to NIL to trip the debugger when errors occur, and the system will now provide a more useful description of the failure if you don't.