“boot test-cljs” exits with status 0 even after a Doo error

[bdesham]

Even if Doo exits with an error, it seems that boot test-cljs exits with a status code of 0. This is a problem (for example) in CI/CD systems because it makes it seem like the tests have run and passed even when they haven’t.

To reproduce this problem, make sure you don’t have phantomjs installed. In an otherwise empty directory, make a build.boot file with the following contents:

(set-env! :dependencies '[[crisptrutski/boot-cljs-test "0.3.4" :scope "test"]])

(require '[crisptrutski.boot-cljs-test :refer [test-cljs]])

(deftask testing
  []
  (merge-env! :resource-paths #{"test"}))

Then, running boot testing test-cljs will give

Adding: ([adzerk/boot-cljs "2.1.2"] [org.clojure/clojurescript "1.7.228"] [doo "0.1.7"]) to :dependencies
Retrieving clojure-1.7.0.jar from https://repo1.maven.org/maven2/ (3748k)
Retrieving google-closure-library-0.0-20151016-61277aea.jar from https://repo1.maven.org/maven2/ (5219k)
Retrieving closure-compiler-v20151216.jar from https://repo1.maven.org/maven2/ (6384k)
Retrieving clojurescript-1.7.228.jar from https://repo1.maven.org/maven2/ (656k)
Retrieving google-closure-library-third-party-0.0-20151016-61277aea.jar from https://repo1.maven.org/maven2/ (76k)
Retrieving tools.reader-1.0.0-alpha1.jar from https://repo1.maven.org/maven2/ (43k)
Classpath conflict: org.clojure/clojure version 1.8.0 already loaded, NOT loading version 1.7.0
Compiling ClojureScript...
• cljs_test/generated_test_suite.js

;; ======================================================================
;; Testing with Phantom:

[doo] Failed to run command:  #error {
 :cause "error=2, No such file or directory"
 :via
 [{:type java.io.IOException
   :message "Cannot run program \"phantomjs\" (in directory \"/Users/bdesham/.boot/cache/tmp/private/var/folders/pt/ct3y0mx90y5270s8sy0_sq200000gn/T/tmp.qh8ztICHh1/w8u/doo9u2\"): error=2, No such file or directory"
   :at [java.lang.ProcessBuilder start "ProcessBuilder.java" 1048]}
  {:type java.io.IOException
   :message "error=2, No such file or directory"
   :at [java.lang.UNIXProcess forkAndExec "UNIXProcess.java" -2]}]
 :trace
 [[java.lang.UNIXProcess forkAndExec "UNIXProcess.java" -2]
  [java.lang.UNIXProcess <init> "UNIXProcess.java" 247]
  [java.lang.ProcessImpl start "ProcessImpl.java" 134]
  [java.lang.ProcessBuilder start "ProcessBuilder.java" 1029]
  [java.lang.Runtime exec "Runtime.java" 620]
  [doo.shell$exec_BANG_ invokeStatic "shell.clj" 49]
  [doo.shell$exec_BANG_ invoke "shell.clj" 43]
  [doo.shell$sh invokeStatic "shell.clj" 76]
  [doo.shell$sh invoke "shell.clj" 71]
  [doo.core$run_script invokeStatic "core.clj" 268]
  [doo.core$run_script invoke "core.clj" 236]
  [clojure.lang.Var invoke "Var.java" 388]
  [crisptrutski.boot_cljs_test$run_tests_BANG_ invokeStatic "boot_cljs_test.clj" 209]
  [crisptrutski.boot_cljs_test$run_tests_BANG_ invoke "boot_cljs_test.clj" 168]
  [crisptrutski.boot_cljs_test$_run_cljs_tests$fn__298$fn__299 invoke "boot_cljs_test.clj" 218]
  [adzerk.boot_cljs$eval1173$fn__1174$fn__1179$fn__1180 invoke "boot_cljs.clj" 250]
  [adzerk.boot_cljs$eval1118$fn__1119$fn__1124$fn__1125 invoke "boot_cljs.clj" 169]
  [crisptrutski.boot_cljs_test$_prep_cljs_tests$fn__237$fn__238 invoke "boot_cljs_test.clj" 153]
  [crisptrutski.boot_cljs_test$eval378$fn__379$fn__384$fn__385 invoke "boot_cljs_test.clj" 239]
  [boot.core$run_tasks invoke "core.clj" 1021]
  [boot.core$boot$fn__933 invoke "core.clj" 1031]
  [clojure.core$binding_conveyor_fn$fn__4676 invoke "core.clj" 1938]
  [clojure.lang.AFn call "AFn.java" 18]
  [java.util.concurrent.FutureTask run "FutureTask.java" 266]
  [java.util.concurrent.ThreadPoolExecutor runWorker "ThreadPoolExecutor.java" 1149]
  [java.util.concurrent.ThreadPoolExecutor$Worker run "ThreadPoolExecutor.java" 624]
  [java.lang.Thread run "Thread.java" 748]]}
We tried running phantomjs but we couldn't find it your system. Try:

     phantomjs -v 

If it doesn't work you need to install phantomjs, see https://github.com/bensu/doo#setting-up-environments

If it does work, file an issue and we'll sort it together!

The resulting status code (which you can check with echo $?) is zero, but it should be something nonzero.

[sinistral]

Try passing --exit? to test-cljs.

[bdesham]

Thanks for the pointer—that flag does make test-cljs work the way I expected.

I would respectfully argue that the default behavior is wrong. It’s a universal convention that command-line tools should exit with a nonzero code when an error occurs. CI/CD systems in particular are the place in which it’s maybe most important for automated-testing tools to work as expected, and most of these systems use nothing but the exit code to determine whether the tests passed or failed. That this behavior must be opted into is very surprising and has the potential to cause a lot of undetected errors down the road.

[Deraen]

In Boot the failure exit code can be set with throwing exception. Process can't be directly closed, as this would break composition with other tasks. Previously Boot-cljs-test couldn't (or didn't want to) throw exceptions, as this would cause long stack trace, hiding the real message about test failure. In recent versions the exception stacktrace can be hidden, so Boot-cljs-test could now throw exception always and remove the --exit? option.

Example of boot.util/omit-stacktrace? option: https://github.com/metosin/boot-alt-test/blob/master/src/metosin/boot_alt_test.clj#L53