Open bdesham opened 6 years ago
Try passing --exit?
to test-cljs
.
Thanks for the pointer—that flag does make test-cljs work the way I expected.
I would respectfully argue that the default behavior is wrong. It’s a universal convention that command-line tools should exit with a nonzero code when an error occurs. CI/CD systems in particular are the place in which it’s maybe most important for automated-testing tools to work as expected, and most of these systems use nothing but the exit code to determine whether the tests passed or failed. That this behavior must be opted into is very surprising and has the potential to cause a lot of undetected errors down the road.
In Boot the failure exit code can be set with throwing exception. Process can't be directly closed, as this would break composition with other tasks. Previously Boot-cljs-test couldn't (or didn't want to) throw exceptions, as this would cause long stack trace, hiding the real message about test failure. In recent versions the exception stacktrace can be hidden, so Boot-cljs-test could now throw exception always and remove the --exit?
option.
Example of boot.util/omit-stacktrace?
option: https://github.com/metosin/boot-alt-test/blob/master/src/metosin/boot_alt_test.clj#L53
Even if Doo exits with an error, it seems that
boot test-cljs
exits with a status code of 0. This is a problem (for example) in CI/CD systems because it makes it seem like the tests have run and passed even when they haven’t.To reproduce this problem, make sure you don’t have phantomjs installed. In an otherwise empty directory, make a build.boot file with the following contents:
Then, running
boot testing test-cljs
will giveThe resulting status code (which you can check with
echo $?
) is zero, but it should be something nonzero.