(ql:quickload :parachute)
To load "parachute":
Load 3 ASDF systems:
asdf documentation-utils form-fiddle
Install 1 Quicklisp release:
parachute
; Fetching #<URL "http://beta.quicklisp.org/archive/parachute/2019-03-07/parachute-20190307-git.tgz">
; 45.61KB
==================================================
46,702 bytes in 0.09 seconds (495.73KB/sec)
; Loading "parachute"
[package parachute]...............
(:PARACHUTE)
CL-USER> (in-package :parachute)
#<PACKAGE "PARACHUTE">
PARACHUTE> (define-test failing-tests
:description "Expecting all tests to fail"
(true nil)
(false 1)
(fail 1)
(is = 1 2)
(isnt = 1 1)
(is-values (values 1 1)
(= 0)
(= 2))
(isnt-values (values 1 1)
(= 1)
(= 1))
(of-type 'string 1)
(finish (error "generic issue")))
FAILING-TESTS
PARACHUTE> (test 'failing-tests)
? PARACHUTE::FAILING-TESTS
0.000 ✘ (true ())
0.000 ✘ (false 1)
0.000 ✘ (fail 1 error)
0.000 ✘ (is = 1 2)
0.000 ✘ (is = 1 1)
0.002 ✔ (is-values (values 1 1) (= 0) (= 2))
0.000 ✔ (isnt-values (values 1 1) (= 1) (= 1))
0.000 ✘ (of-type 'string 1)
0.003 ✘ (finish (error "generic issue"))
0.042 ✘ PARACHUTE::FAILING-TESTS
;; Summary:
Passed: 2
Failed: 7
Skipped: 0
;; Failures:
7/ 9 tests failed in PARACHUTE::FAILING-TESTS
Expecting all tests to fail
The test form ()
evaluated to ()
when t
was expected to be equal under GEQ.
The test form 1
evaluated to 1
when ()
was expected to be equal under GEQ.
The test form (capture-error 1)
evaluated to ()
when error
was expected to be equal under TYPEP.
The test form 2
evaluated to 2
when 1
was expected to be equal under =.
The test form 1
evaluated to 1
when 1
was expected to be unequal under =.
The test form 1
evaluated to 1
when string
was expected to be equal under TYPEP.
The test form (finish (error "generic issue"))
exited by generic issue
Expected result
I was expecting (isnt = 1 1) to get reported as
0.000 ✘ (isnt = 1 1)
instead of
0.000 ✘ (is = 1 1)
The is-values and isnt-values tests should be failing.
Skipped tests
Skipping all default test forms using the following definition:
I was expecting the (finish (error "generic issue")) to get reported as:
0.000 ー (finish (error "generic issue"))
instead of
0.000 ✔ (finish (error "generic issue"))
and have the Summary reflect that the test is being skipped.
Potential Fixes
It looks like in eval-in-context for finishing-result, when the test is skipped the (status result) is getting overridden with :passed.
Regarding the reporting of (isnt ...) as (is ...), I think it's sufficient to update the :expression value of the make-instance form in the isnt macro definition.
Regarding the is-values passing when expecting a failure, I think it's getting the :passed status, as the default status assigned at the :around method on eval-in-context for the result class. Heres a trace for that test:
I see that multiple-value-comparison-result inherits from both multiple-value-result and comparison-result, however, it appears that the eval-in-context implementation for comparison-result is not getting called on multiple-value-comparison-result instances, contrary to what one might think by seeing a specialization on value-expected-p for multiple-value-comparison-result.
Thank you for writing this library!
Here are a couple of issues present in the latest version from quicklisp regarding failing and skipped tests.
Failing tests
Trying out all default test forms using the following definition:
Actual result
Expected result
I was expecting
(isnt = 1 1)
to get reported asinstead of
The
is-values
andisnt-values
tests should be failing.Skipped tests
Skipping all default test forms using the following definition:
Actual result
Expected result
I was expecting the
(finish (error "generic issue"))
to get reported as:instead of
and have the Summary reflect that the test is being skipped.
Potential Fixes
It looks like in
eval-in-context
forfinishing-result
, when the test is skipped the(status result)
is getting overridden with:passed
.Regarding the reporting of
(isnt ...)
as(is ...)
, I think it's sufficient to update the:expression
value of themake-instance
form in theisnt
macro definition.Regarding the
is-values
passing when expecting a failure, I think it's getting the:passed
status, as the default status assigned at the:around
method oneval-in-context
for theresult
class. Heres a trace for that test:I see that
multiple-value-comparison-result
inherits from bothmultiple-value-result
andcomparison-result
, however, it appears that theeval-in-context
implementation forcomparison-result
is not getting called onmultiple-value-comparison-result
instances, contrary to what one might think by seeing a specialization onvalue-expected-p
formultiple-value-comparison-result
.