Prevent loading of namespaces that are skipped

[plexus]

Via @borkdude on Clojurians Slack

When using kaocha, is there an equivalent to -n from the cognitect test runner or :only from leiningen, from the command line? nm, got it: https://cljdoc.org/d/lambdaisland/kaocha/1.0.632/doc/6-focusing-and-skipping#on-a-namespace I'm trying to port this:

clojure -A:test -n babashka.pods.jvm-test
;;=>
clojure -A:test:kaocha --focus babashka.pods.jvm-testclojure -A:sci:test -n babashka.pods.sci-test
;;=>
clojure -A:sci:test:kaocha --focus babashka.pods.sci-test

This one:

clojure -A:test:kaocha --focus babashka.pods.jvm-test

doesn't work, since the sci library is (deliberately!) not available on the classpath. The code should run without the library, which is part of my test. With the Cognitest test runner or leiningen that works, but with kaocha it doesn't:

$ clojure -A:test:kaocha --focus babashka.pods.jvm-test
WARNING: :focus [:babashka.pods.jvm-test] did not match any tests.
[E]
Randomized with --seed 2027872620ERROR in unit (babashka/pods/sci.clj:1)
Exception: clojure.lang.Compiler$CompilerException: Syntax error compiling at (babashka/pods/sci.clj:1:1).
#:clojure.error{:phase :compile-syntax-check, :line 1, :column 1, :source "babashka/pods/sci.clj"}

So my proposal would be that focus should not load files other than it needs.

[plexus]

To provide some background here, a Kaocha test runs consists of two phases, load and run. Focusing/skipping happens after load. Why? Because we need to know which tests there are before we can mark them in the test plan to be skipped.

Keep in mind that we support several different types of tests, clojure.test, cljs.test, Cucumber, ... These don't all have the concept of a namespace, so "namespace" is not a concept in Kaocha. Instead we have :kaocha.testable/id, which in the case of a clojure.test test namespace is the name of the namespace. So when you put in --focus babashka.pods.jvm-test you're telling Kaocha to focus on the test with the id :babashka.pods.jvm-test.

So the kaocha.type/clojure.test test type will perform its load step, the result is a test-plan, then the focus/skip plugin updates that test-plan by marking tests as skipped. This is a nice orthogonal design in that we can implement more test types and get focus/skip for free. It is of no concern of the test type.

Except that for the above feature this is something we'll have to do per test type (starting with clojure.test). I think the right thing to do is to provide an affordance so a test type can check if a certain test id at a certain place in the test-plan hierarchy would be skipped based on the user's input. If it is able to figure out the test id without actually loading the test then it can use this to skip the loading altogether.

[robhanlon22]

A parallel manifestation of this is that :once fixtures in the kaocha.type/clojure.test type are still run even if all of the tests in the namespace are skipped. As far as I can tell, there isn't an easy way to work around this outside of switching all of the :once fixtures to be :each fixtures, which won't be run if a test is skipped. However, this issue doesn't necessarily have to be solved by skipping the entire namespace, but supporting skipping at the namespace level would probably be the cleanest solution to the problem.

[arichiardi]

What we see here is that also :each fixtures` and if you rely on metadata for your suites you will have a hard time using them.

For instance skipping :integration tests that have use-fixtures will still run database connection/initialization code.