Open mjmeintjes opened 5 years ago
Please provide as much context as possible when reporting issues
Thanks
tests.edn:
{:tests [{:id :all
:test-paths ["src" "test"]}]
:watch? true
:plugins [:kaocha.plugin/notifier]
:reporter kaocha.report/documentation}
Command line flags:
bin/kaocha --reporter kaocha.report/documentation --fail-fast --no-randomize
Running in CLI.
Not sure what is failing - cannot figure out what the exception means, whether it is an error in my code or an error in kaocha. i.e what does the following mean?
Fatal error in test run #error {
:cause throw+: #:kaocha{:fail-fast true}
:data #:kaocha{:fail-fast true}
This has to do with the fail-fast
mechanism, kaocha generates an exception to halt execution, rather than continuing the current test var. This is normally handled by the leaf test types, but it seems in this case the exception is being raised outside of that scope. Looking at the line numbers it seems to be in a pre-test hook.
(t/do-report {:type :error
:kaocha/testable {:kaocha.testable/id :kaocha/watch}
:message (str "Failed reloading ns: " (::error-ns test-plan))
:actual error})
So that's probably the thing that triggered this. You have a namespace that isn't reloading cleanly.
We should fix this by having *fail-fast?*
be false outside of the run phase. The fail-fast reporter should also add the original error to the exception data so it's more clear where issues like these originate.
Thanks for the info. The issue that caused the problem seems to be fixed now on my side - not sure what caused it, but getting all my tests to pass sorted it out.
However, it might be worth keeping this issue open as the current error is unfriendly and probably worth fixing.
Also, unrelated, but thanks for Kaocha, it works really well and looks like a well thought out system. Just started using it today.
This definitely needs fixing, it might not be super common as it's really a combination of watch+fail-fast+load errors, but it will affects plugins/hooks more generally.
I'm pretty busy right now with other stuff, but I'll probably make more time for Kaocha again coming fall.
I hit something like this using greenlight-kaocha. it only failed in CI (could not reproduce locally) and fail-fast masked the failing assertion so it was hard to find the culprit. when I saw this bug I tried disabling fail-fast and then I could see the problem.
I'm inclined to leave this issue open because it was never really addressed (although it's possible we fixed it inadvertently), even though it appears to be a very rare bug.
After a test failure, I edit the test, and then I get the following output:
lambdaisland/kaocha {:mvn/version "0.0-529"}