search

cognitect-labs / test-runner

A test runner for clojure.test
Eclipse Public License 2.0
288 stars 32 forks source link

Report a non-zero exit code when tests fail #9

Closed plexus closed 6 years ago

plexus commented 6 years ago

When tests fail, the process should exit with a non-zero exit code. This is especially important for people using this in CI.

Current behavior

$ clj -Atest

Running tests in #{"test"}

Testing foo

FAIL in (foo-test) (foo.clj:5)
expected: false
  actual: false

Ran 1 tests containing 1 assertions.
1 failures, 0 errors.

$ echo $?
0

Expected

$ echo $?
1
arichiardi commented 6 years ago

Confirm this does not happen, probably feature request number one for CI.

joshkh commented 5 years ago

I recently stumbled across this problem when attempting to use test-runner, post merge of the exit code PR (latest SHA: 028a6d41ac9ac5d5c405dfc38e4da6b4cc1255d5). It looks like it's still (not) failing.

Example project using the latest test-runner commit (master as of December 2018):

$ git clone https://github.com/cambioscience/cognitect-test-runner
$ clj -Atest

Testing ghost.test-runner

FAIL in (test-will-fail) (test_runner.clj:5)
expected: (= true false)
  actual: (not (= true false))

Ran 1 tests containing 1 assertions.
1 failures, 0 errors.
$ echo $?
0

I've created a branch of the example project above which explicitly moves (System/exit ...) to the project's local test runner. It fails appropriately, along with some CircleCI integration:

https://github.com/cambioscience/cognitect-test-runner/tree/working-exit-code

(defn -main []
  (let [{:keys [fail error]} (t/run-tests 'ghost.test-runner)]
    (System/exit (if (zero? (+ fail error)) 0 1))))