tdrhq / slite

a SLIme-based TEst runner for FiveAM and Parachute Tests
Apache License 2.0
50 stars 4 forks source link

Add parachute support #1

Open TatriX opened 3 years ago

TatriX commented 3 years ago

Hi! Please consider adding parachute support. Thanks!

tdrhq commented 3 years ago

Sure, let me take a look at it today, I might reach out if I have parachute specific questions since I haven't used it before

tdrhq commented 3 years ago

@TatriX Done! Try it out and let me know how it goes. I've updated the README with Parachute specific information. The only feature that doesn't work would be jumping to test definition. That would need a patch to Parachute, and even then would only work on Lispworks.

tdrhq commented 3 years ago

Things to look out for: In translating the results model from parachute to fiveam, I might've made some assumptions about the hierarchy of tests. Watch out for slite showing all green but parachute saying a test is failing. In this case, let me know what the hierarchy of the tests look like.

TatriX commented 3 years ago

When I'm trying to load slite.el I'm getting an error:

add-hook: Symbol’s function definition is void: case

from this code I think:

(add-hook (case (slite--slime-impl)
            (:sly
             'sly-compilation-finished-hook)
            (:slime
             'slime-compilation-finished-hook))
          'slite--compilation-finished)
TatriX commented 3 years ago

I think this is due cl lib being deprecated since Emacs 27 and probably it is remove in Emacs 28. Most likely you want to (require 'cl-lib) and use (cl-case ...) as well as add cl- prefix to other functions as needed.

tdrhq commented 3 years ago

@TatriX Fixed: https://github.com/tdrhq/slite/commit/7ed03625eced85457b69675872e8bcbfd1a3db34

TatriX commented 3 years ago

Nice! Now I do

(asdf:load-system :slite/parachute)
;; C-c v
(parachute:test :foo)

and getting:

Waiting for test results...
[sly] `sly-eval-async' errored: (void-function every)
tdrhq commented 3 years ago

Oops, git pull and try one more time: https://github.com/tdrhq/slite/commit/9a3bb24b9f066afd77ad5b7012f6b338e18f053a

TatriX commented 3 years ago

define-test name can be a symbol or a string: (define-test "A more descriptive name, perhaps")

It seems this insn't handled right now:

The value
  "registered user should be able create an invite"
is not of type
  SYMBOL
   [Condition of type TYPE-ERROR]

Restarts:
 0: [*ABORT] Return to SLY's top level.
 1: [ABORT] abort thread (#<THREAD "slynk-worker" RUNNING {10058211C3}>)

Backtrace:
 0: ((:METHOD SLITE::GET-TEST-CASE-DETAILS (T)) #<TEST SERVER-TESTS::registered user should be able create an invite>) [fast-method]
 1: ((:METHOD SLITE::PROCESS-RESULTS (T)) #<PLAIN 14, FAILED results>) [fast-method]
 2: (SB-INT:SIMPLE-EVAL-IN-LEXENV (SLITE::PROCESS-RESULTS (EVAL (READ-FROM-STRING "(parachute:test 'server-tests)"))) #<NULL-LEXENV>)
 3: (EVAL (SLITE::PROCESS-RESULTS (EVAL (READ-FROM-STRING "(parachute:test 'server-tests)"))))
 --more--
tdrhq commented 3 years ago

@TatriX ah, let me think through this one

tdrhq commented 3 years ago

(keeping the issue opened for now)

tdrhq commented 3 years ago

@TatriX Okay done, try this one more time: https://github.com/tdrhq/slite/commit/766704b2179ee8a258e411f96aa4e4ecb57caddb

tdrhq commented 3 years ago

^ the above change will make the test results load, but rerunning in debugger appears to have broken with that change, will fix that in a second

tdrhq commented 3 years ago

Re-run in debugger fixed too https://github.com/tdrhq/slite/commit/df92f133f6a00d81bb88e9f2ce6d2e64065ca95c

TatriX commented 3 years ago

Now I'm getting:

[sly] `sly-eval-async' errored: (no-catch --cl-block-nil-- Failed: "(TRUE (SIGNIN (CURRENT-USER)))")
tdrhq commented 3 years ago

@TatriX at which point do you get this error, just when running the tests? Or was that when rerunning in the debugger?

tdrhq commented 3 years ago

Also, what was the CL expression you passed to M-x slite-run?

TatriX commented 3 years ago

I guess it will be easier to debug on my side. I used C-c V with (parachute:test :my-package)

tdrhq commented 3 years ago

@TatriX yeah, your debugging would be helpful in this situation :) In particular, here's what I'm looking for, if you run (parachute:test :my-package) in a repl, does it signal an error or does it just show you the failures? Is there some global parachute default that you've set that might cause failures to become conditions? (for instance, fiveam has fiveam:on-failure set to :signal or something like that, couldn't find anything obvious for parachute except for the specific report class)

The second issue is, even if that condition did get signalled, it should've just opened up a SLY debugger. Do you have any configuration that might prevent SLY from opening a debugger? For instance, if I run M-x slite-run (parachute:test 'foobar :report 'parachute:interactive) with a failing test, it still opens a debugger for me.