search

CoNarrative / precept

A declarative programming framework
MIT License
657 stars 33 forks source link

Tests will sometimes fail on CircleCI but show as passed #90

Open alex-dixon opened 7 years ago

alex-dixon commented 7 years ago

https://circleci.com/gh/CoNarrative/precept/295?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link

Previously the test runner setup will show a failure if any of our tests fail, despite the last result (of the test runner itself) showing 0 tests, 0 failures. Because this appears to happen inconsistently we should revisit the test runner setup. We might want to consider CLJS tests #61 in conjunction with this, perhaps borrowing from Clara. The test runner setup came about due to inconsistencies between running tests with macroexpand on CircleCI vs locally (they would pass locally but fail on CircleCI).

timgilbert commented 6 years ago

I wonder if this is because you're using lein do syntax, and lein's exit code reflects the lein uberjar task, not the lein test one? Possibly splitting this into two - run: lines in the circle config would help.

alex-dixon commented 6 years ago

Thanks! That makes perfect sense.

That may ultimately be causing a false positive but it looks like we get one before that.

Running lein test :only precept.test-runner with a failing test:

... other tests...
Testing precept.listeners-test

FAIL in (listeners-state-transitions) (listeners_test.cljc:161)
I fail
expected: (= 0 1)
  actual: (not (= 0 1))

Ran 2 tests containing 23 assertions.
1 failures, 0 errors.

lein test precept.test-runner

Ran 0 tests containing 0 assertions.
0 failures, 0 errors.
$ echo $?                            
0

I added this test runner namespace to get the test output to agree with what I was seeing in the REPL for macro tests that assert the equality of what our DSL expands to against Clara's. Running lein test alone gives failures for these. The main thing they have in common is that they call macroexpand. The expansion doesn't appear to take place the same way with lein test. E.g.

FAIL in (rule-test) (rule_test.cljc:290)
With :test op
expected: (= (macroexpand (quote (rule my-rule [?fact <- [_ :my-attribute _ ?t]] [:test (> ?t ?fact)] => (insert! "RHS")))) 
(clojure.core/seq (clojure.core/concat (clojure.core/list (quote do)) (clojure.core/list (macroexpand (quote (defrule my-rule [?fact <- :my-attribute (= ?t (:t this))] [:test (> ?t ?fact)] => (insert! "RHS"))))))))
  actual: (not (= (rule my-rule [?fact <- [_ :my-attribute _ ?t]] [:test (> ?t ?fact)] => (insert! "RHS")) 
(do (defrule my-rule [?fact <- :my-attribute (= ?t (:t this))] [:test (> ?t ?fact)] => (insert! "RHS")))))