Closed plexus closed 6 years ago
Confirm this does not happen, probably feature request number one for CI.
I recently stumbled across this problem when attempting to use test-runner
, post merge of the exit code PR (latest SHA: 028a6d41ac9ac5d5c405dfc38e4da6b4cc1255d5). It looks like it's still (not) failing.
Example project using the latest test-runner
commit (master as of December 2018):
$ git clone https://github.com/cambioscience/cognitect-test-runner
$ clj -Atest
Testing ghost.test-runner
FAIL in (test-will-fail) (test_runner.clj:5)
expected: (= true false)
actual: (not (= true false))
Ran 1 tests containing 1 assertions.
1 failures, 0 errors.
$ echo $?
0
I've created a branch of the example project above which explicitly moves (System/exit ...)
to the project's local test runner. It fails appropriately, along with some CircleCI integration:
https://github.com/cambioscience/cognitect-test-runner/tree/working-exit-code
(defn -main []
(let [{:keys [fail error]} (t/run-tests 'ghost.test-runner)]
(System/exit (if (zero? (+ fail error)) 0 1))))
When tests fail, the process should exit with a non-zero exit code. This is especially important for people using this in CI.
Current behavior
Expected