Ability to run test namespaces one by one

[vemv]

Hey there,

I use karma-chome-headless to perform integration tests. This is, I load the whole app and black-box test it, Selenium style.

My setup for this is a bit hacky but it works. Or it mostly does!

I have 3 existing test namespaces (that's my whole suite) that are fine. I implemented a 4th one, which only passes if doo-tests only includes that test.

IOW:

(doo-tests 'a 'b 'c) -> always works (doo-tests 'd) -> always works (doo-tests 'a 'b 'c 'd) -> never works

So, a/b/c are uncompatible with d.

Regardless of the exact cause of the error, which I don't want to bore you with, I'd just like a way that test namepaces are run "one by one", sequentially: one test, one karma execution.

That way one would guarantee that each test has a 100% clean/isolated state.

I realise that it might result in a slower suite, but I'm willing to take that cost.

Cheers - Victor

[bensu]

Hi @vemv,

Can you please check if there is anything in Karma that would help us get that behavior?

[vemv]

I will!

In any case, the suggested strategy seems rather universal (karma-agnostic).

Pseudo code. Instead of

(karma a b c d)

one would do

(karma a)
(karma b)
(karma c)
(karma d)

My point is, this might be achievable regardless of what Karma may have.

I realise my request may be a bit whimsical, but who knows, maybe for the maintainers this would be super easy to implement. Ultimately I can give it a shot.

A related missing feature in doo, and a less strange one, is that one cannot run just one file (testing namespace), right?

If I want to run just one testing namespace (why? focus, performance, debugging), I have to comment out the rest, which is tedious.

Cheers - Victor

[ransomw]

here is a concrete e2e_test.clj example analogous to what i surmise the issue is about: there are several deftests on the .clj side, each corresponding to a different test .cljs test namespace.

the proliferation of deftests and namespaces (smallish for this project, but potentially large for more complex applications) resulted from, midway thru implementing tests according to the end-to-end example on the wiki, realizing i wanted the server-side test fixture (i.e. database setup and teardown) to run in between some Clojurescript tests.

the suggested strategy seems rather universal (karma-agnostic)

indeed, the desired behavior for my use-case (although not so high priority) would be an opt to doo.core/run-script or doo.runner/doo-tests (or both? haven't thought about it much) that would have old 'Scoob do a use-fixtures-like trick: setup/teardown specified in Clojure could be paired with tests written in Clojurescript.

that way, there'd a single e2e_runner.cljs file with, for example,

(doo-tests
 'emcg.act-test
 'emcg.add-mcg-res-test
)

rather than a separate e2e_runner_add_mcg_res.cljs.

perhaps cljs.test/report could be helpful in this regard.

idk if this description is what's meant by the above pseudocode. hopefully it can serve as a point of departure for clarifying what the issue at stake actually is.

---

also, it occurs to me that the desired behavior described here may well be possible with the current doo implementation, and it's just a matter of digging into the docs and source in more depth (?)

[vemv]

@ransomw cool code! I never thought of calling cljs/build and doo from clojure as part of the test suite.

However we are talking about different things. I was talking about test namespaces rather than individual tests.

I originally reported the issue talking about 'tests' (not test namespaces) which I reckon was confusing. Sorry.

My need, in short is that the Karma .html page is loaded from scratch for each testing namespace (which of course can contain N deftests), for guaranteeing clean state.

Looking at @ransomw's example again, maybe I can repeatedly pass doo/run-script with a different :main option each time. In any case that'd be a lot of work and a workaround, whereas -I suspect- doo would have it easier and cleaner.

Cheers - Victor