search

fukamachi / prove

Yet another unit testing framework for Common Lisp
218 stars 23 forks source link

:file vs :test-file issue #31

Open slyrus opened 7 years ago

slyrus commented 7 years ago

If I declare my file to be a :file in the .asd file, everything seems fines (although the reporting says All 0 files passed, which was a clue that maybe something wasn't right). If I declare the file as a :test-file, it doesn't get tested when I try to test the system via (asdf:test-system 'cl-bio-align). If, OTOH, I try to run the tests a bit more manually by doing: (prove-asdf:run (asdf:find-system "cl-bio-align-test")) then things seem to work. Any ideas?

slyrus commented 7 years ago

Apparently things work fine (as a :test-file) as long as the hierarchy is:

...
:components
((:module :test :components ((:test-file "cl-bio-align-test"))))
...

But if I have another level of nesting, e.g.:

((:module :test :components ((:module :align :components ((:test-file "cl-bio-align-test"))))))

then things break. It seems as though the :test-file is being ignored if it's not at the top-level of the module.

slyrus commented 7 years ago

One more thought... After running things a bunch of times, I get a very polluted prove-asdf::system-test-files hashtable. Might this be part of the problem?

fukamachi commented 7 years ago

Add :perform directive to defsystem in reference to README.

slyrus commented 7 years ago

Not sure I follow. I have one of those: https://github.com/slyrus/cl-bio/blob/master/cl-bio-align-test.asd#L18 but the problem persists.

fukamachi commented 7 years ago

Try (asdf:test-system :cl-bio-align-test).

Symbolics commented 7 years ago

I'm having exactly the same problem described at the start of this issue. "All 0 files passed".

Slyrus, by chance are you running CCL? I've tried this a few ways, but have not got it work. The file with the test forms just doesn't seem to run.

slyrus commented 7 years ago

No, I'm using SBCL.

Symbolics commented 7 years ago

Slyrus, did you manage to fix the issue? Here is a summary of what I'm seeing. First, the .asd:

(defsystem "lisp-stat-tests"
  :description "Unit tests for CLS"
  :depends-on ("lisp-stat" "prove")
  :defsystem-depends-on (:prove-asdf)
  :pathname "tests/"
  :components ((:test-file "variables-test"))
  :perform (test-op :after (op c)
;           (symbol-call :lisp-stat-tests :run-test-system c)
;           (funcall (intern #.(string :run-test-system) :prove-asdf) c)
            (funcall (intern #.(string :run) :prove) c)
            ))

As you can see, I've been experimenting with various test-ops; all produce the same result:

CL-USER> (asdf:test-system :lisp-stat-tests)
To load "lisp-stat-tests":
  Load 1 ASDF system:
    lisp-stat-tests
; Loading "lisp-stat-tests"

Summary:
  All 0 files passed.

It is clear that the results of variables-test.lisp is not being executed, because if I put trivial failure tests in there, I get the same result. If I rename it, asdf complains, so the file location seems correct. However if I call prove on the test file directly, it works:

CL-USER> (prove:run #P"s:/src.sct/lisp-stat/tests/variables-test.lisp" :reporter :fiveam)
...
 Did 3 checks.
    Pass: 3 (100%)
    Fail: 0 (  0%)
T
(#P"s:/src.sct/lisp-stat/tests/variables-test.lisp")

Any way to trace what's going on in ASDF or Prove to find out where & why this isn't being executed?

Symbolics commented 7 years ago

A bit more data. Taking a cue from Slyrus's initial description of the problem, I changed from keyword :test-file to :file in -tests.asd. With this (asdf:test-system :main-system) worked, once, but not in a subsequent run. Here is the sequence:

(asdf:test-system :main-system) -> Worked once after changing from :test-file to :file. No idea why. (asdf:test-system :main-system-tests) evaluated right after the above, failed (asdf:test-system :main-system), run right after the above, failed, despite working less than 30 seconds earlier.

It does look like something is being corrupted somewhere.

Has anyone got this working reliably?

Symbolics commented 7 years ago

A little more data: making a change in the -test.lisp file makes it work, once. Subsequent runs fail to execute the tests until another change is made in the file.

Even in the case where the tests are run, the system still reports:

CL-USER> (asdf:test-system :lisp-stat)
1..3

  ✓ 4 is not in the list (1 2 3) 

  ✓ 4 is expected to be 4 

  ✓ 1 is not expected to be #\1 

✓ 3 tests completed (0ms)
To load "lisp-stat-tests":
  Load 1 ASDF system:
    lisp-stat-tests
; Loading "lisp-stat-tests"

Summary:
  All 0 files passed.
T
CL-USER>

That "All 0 files passed." can't be right.

Symbolics commented 7 years ago

A final comment: this bug appears to be ASDF not loading the test file, probably because of the belief that it is already loaded. If any change is made to the test file, the test file is executed, once. Subsequent runs fail to run/load the test file.

This code has a learning curve I cannot afford to climb right now; hopefully these comments will point someone who can fix it in the right direction.