EVAL-IN-CONTEXT :AROUND methods on test classes do not work

[phoe]

If I try to define an :around method on EVAL-IN-CONTEXT on a test class (or instance), I would expect that the code will be executed around the test. However, it is not the case.

Example definition:

(define-test sql-select-dummy :parent sql)

(defmethod parachute:eval-in-context :around
    (context (test (eql (parachute:find-test 'sql-select-dummy))))
  (print "Installing dummy data.")
  (with-test-db () (uninstall) (install) (install-dummy-data))
  (let ((*dummy-data* t))
    (call-next-method))
  (print "Cleanup.")
  (with-test-db () (uninstall) (install)))

Fragment of the test running:

SQL/TEST> (test 'sql-select-dummy)
             ？ GATEWAY.SQL/TEST::SQL-SELECT-DUMMY

"Installing dummy data." 
"Cleanup."              ？   GATEWAY.SQL/TEST::PLAYER-SELECT-DUMMY
             #     Phase "????"
????   0.000 ✘     (is = id i)
????   0.000 ✘     (is string= login (format () "player~D" i))
????   0.000 ✘     (is string= email (format () "player~D@gate.way" i))
????   0.000 ✘     (is string= name (format () "Player ~D" i))
(...)

Trace fragment:

  10: (PARACHUTE:EVAL-IN-CONTEXT #<PARACHUTE:PLAIN 1 results> #<PARACHUTE:TEST GATEWAY.SQL/TEST::SQL-SELECT-DUMMY>)
    11: ((METHOD PARACHUTE:EVAL-IN-CONTEXT :AROUND (T (EQL #<PARACHUTE:TEST GATEWAY.SQL/TEST::SQL-SELECT-DUMMY>))) #<PARACHUTE:PLAIN 1 results> #<PARACHUTE:TEST GATEWAY.SQL/TEST::SQL-SELECT-DUMMY>)
"Installing dummy data."
      12: ((METHOD PARACHUTE:EVAL-IN-CONTEXT :AROUND (T T)) #<PARACHUTE:PLAIN 1 results> #<PARACHUTE:TEST GATEWAY.SQL/TEST::SQL-SELECT-DUMMY>)
        13: ((METHOD PARACHUTE:EVAL-IN-CONTEXT (T PARACHUTE:TEST)) #<PARACHUTE:PLAIN 1 results> #<PARACHUTE:TEST GATEWAY.SQL/TEST::SQL-SELECT-DUMMY>)
        13: (METHOD PARACHUTE:EVAL-IN-CONTEXT (T PARACHUTE:TEST)) returned NIL
      12: (METHOD PARACHUTE:EVAL-IN-CONTEXT :AROUND (T T)) returned NIL
"Cleanup."
     11: (METHOD PARACHUTE:EVAL-IN-CONTEXT :AROUND (T (EQL #<PARACHUTE:TEST GATEWAY.SQL/TEST::SQL-SELECT-DUMMY>))) returned NIL
  10: PARACHUTE:EVAL-IN-CONTEXT returned NIL

[Shinmera]

I can't reproduce this.

(parachute:define-test foo
  (parachute:is = 5 5)
  (parachute:is eq "a" "b"))

(defmethod parachute:eval-in-context :around (context (test (eql (parachute:find-test 'foo))))
  (print "before")
  (call-next-method)
  (print "after"))

(trace parachute:eval-in-context)
(parachute:test 'foo)

Output:

  0: (PARACHUTE:EVAL-IN-CONTEXT #<PARACHUTE:PLAIN 1 results> #<PARACHUTE:TEST-RESULT :UNKNOWN COMMON-LISP-USER::FOO>)
        ？ COMMON-LISP-USER::FOO
    1: (PARACHUTE:EVAL-IN-CONTEXT #<PARACHUTE:PLAIN 1 results> #<PARACHUTE:TEST COMMON-LISP-USER::FOO>)

"before"       2: (PARACHUTE:EVAL-IN-CONTEXT #<PARACHUTE:PLAIN 2 results> #<PARACHUTE:COMPARISON-RESULT :UNKNOWN (is = 5 5)>)
  0.000 ✔   (is = 5 5)
      2: PARACHUTE:EVAL-IN-CONTEXT returned "(is = 5 5)"
      2: (PARACHUTE:EVAL-IN-CONTEXT #<PARACHUTE:PLAIN 3 results> #<PARACHUTE:COMPARISON-RESULT :UNKNOWN (is eq "a" "b")>)
  0.000 ✘   (is eq "a" "b")
      2: PARACHUTE:EVAL-IN-CONTEXT returned "(is eq \"a\" \"b\")"

"after"     1: PARACHUTE:EVAL-IN-CONTEXT returned "after"
  0.012 ✘ COMMON-LISP-USER::FOO
  0: PARACHUTE:EVAL-IN-CONTEXT returned "COMMON-LISP-USER::FOO"

;; Summary:
Passed:     1
Failed:     1
Skipped:    0

;; Failures:
   1/   2 tests failed in COMMON-LISP-USER::FOO
The test form   "b"
evaluated to    "b"
when            "a"
was expected to be equal under EQ.

[phoe]

Oh right - I forgot to mention it's for test suites.

(parachute:define-test foo)

(parachute:define-test bar
  :parent foo
  (parachute:is = 5 5)
  (parachute:is eq "a" "b"))

(defmethod parachute:eval-in-context :around (context (test (eql (parachute:find-test 'foo))))
  (print "before")
  (call-next-method)
  (print "after"))

(trace parachute:eval-in-context)
(parachute:test 'foo)

  0: (PARACHUTE:EVAL-IN-CONTEXT #<PARACHUTE:PLAIN 1 results> #<PARACHUTE:TEST-RESULT :UNKNOWN GATEWAY.SQL/TEST::FOO>)
             ？ GATEWAY.SQL/TEST::FOO
    1: (PARACHUTE:EVAL-IN-CONTEXT #<PARACHUTE:PLAIN 1 results> #<PARACHUTE:TEST GATEWAY.SQL/TEST::FOO>)

"before" 
"after"     1: PARACHUTE:EVAL-IN-CONTEXT returned "after"
    1: (PARACHUTE:EVAL-IN-CONTEXT #<PARACHUTE:PLAIN 2 results> #<PARACHUTE:TEST-RESULT :UNKNOWN GATEWAY.SQL/TEST::BAR>)
             ？   GATEWAY.SQL/TEST::BAR
      2: (PARACHUTE:EVAL-IN-CONTEXT #<PARACHUTE:PLAIN 2 results> #<PARACHUTE:TEST GATEWAY.SQL/TEST::BAR>)
        3: (PARACHUTE:EVAL-IN-CONTEXT #<PARACHUTE:PLAIN 3 results> #<PARACHUTE:COMPARISON-RESULT :UNKNOWN (is = 5 5)>)
       0.000 ✔     (is = 5 5)
        3: PARACHUTE:EVAL-IN-CONTEXT returned "(is = 5 5)"
        3: (PARACHUTE:EVAL-IN-CONTEXT #<PARACHUTE:PLAIN 4 results> #<PARACHUTE:COMPARISON-RESULT :UNKNOWN (is eq "a" "b")>)
       0.000 ✘     (is eq "a" "b")
        3: PARACHUTE:EVAL-IN-CONTEXT returned "(is eq \"a\" \"b\")"
      2: PARACHUTE:EVAL-IN-CONTEXT returned NIL
       0.008 ✘   GATEWAY.SQL/TEST::BAR
    1: PARACHUTE:EVAL-IN-CONTEXT returned "GATEWAY.SQL/TEST::BAR"
       0.008 ✘ GATEWAY.SQL/TEST::FOO
  0: PARACHUTE:EVAL-IN-CONTEXT returned "GATEWAY.SQL/TEST::FOO"

;; Summary:
Passed:     1
Failed:     1
Skipped:    0

;; Failures:
   1/   1 tests failed in GATEWAY.SQL/TEST::FOO
   1/   2 tests failed in GATEWAY.SQL/TEST::BAR

   1/   2 tests failed in GATEWAY.SQL/TEST::BAR
The test form   "b"
evaluated to    "b"
when            "a"
was expected to be equal under EQ.

[Shinmera]

Actually, the situation is that you're expecting test children to be run within the test itself, which is not a valid expectation.

[phoe]

Got it. Let me jump back up and describe what I want and not what is going on.

I want to be able to evaluate some dynamically scoped code around all children of a particular test. Something like:

(defvar *frob* nil)

(parachute:define-test foo)

(parachute:define-test bar
  :parent foo
  (print *frob*))

(defmethod parachute:somehow-eval-around-children
    :around (context (test (eql (parachute:find-test 'foo))))
  (let ((*frob* t))
    (call-next-method)))

I expect T to be printed and not NIL.

[Shinmera]

Should be solved by 79ac98e

[phoe]

T gets printed, so yes, I have dynamic scope around children now. Thanks!