search

ocaml-batteries-team / batteries-included

Batteries Included project
http://ocaml-batteries-team.github.com/batteries-included/hdoc2/
Other
514 stars 106 forks source link

Ability to run tests for a given module separately #138

Closed gasche closed 13 years ago

gasche commented 13 years ago

With the increasing number of tests (which is good !), running "make test" is longer and longer and I find this is becoming an issue in my development workflow. Would it be possible to run only some of the tests? For example I changed the BatMap.ml file and want to check that none of the BatMap-generated tests are failing, without recompiling and launching the others.

Ideally, we could pick any number of file to be tested, and also some of the testsuite/test_foo.ml files as well.

thelema commented 13 years ago

Well, since each quick test has a name (or should, and probably doesn't actually run unless it does), we can probably filter on this name when we run the tests. TEST=map make test could only run tests with "map" in their name. As for the legacy tests... Maybe similar can be done with them too. If I get time, I'll play with this a bit today.

gasche commented 13 years ago

Just a small update: actually what takes up most of the time is not running the test, but compiling the tests. I just found out that disabling native compilation of the testsuite brings major time savings on my machine. More specifically, using make test after a small code change (so one or two files have been modified, and therefore batteries_uni must also be recompiled, but the rest is cached) takes 1m30s of compilation, while BATTERIES_NATIVE="no" make test compiles in 5s (measured by the ocamlbuild interface). Running the bytecode test takes 19s overall, which is much more bearable that the previous delays of native compilation.

thelema commented 13 years ago

In that case, I've reorganized the makefile targets so make test-byte builds only what's needed to run bytecode tests. Commit 995f8e89

gasche commented 13 years ago

This is much better, but it's still not enough. On my machine, testsuite/main.byte alone runs in 5-6s, which I find a little slow but reasonable enough. The qtest/test_runner.byte however takes 11s, and due to the inline nature of the qtests I find that I'm running then much more often (more on that in an incoming mailing-list thread), so this time is an issue.

I think it would really be best if we could run the tests specific to a given module. I will see what can be done for testsuite/, it's all OCaml and with a simple structure so I can hack it easily, but I don't know the qtest/ internals, so if you know them better I would be interested in a helping hand. If you don't, just say so, and I'll take care of it myself, after all I'm the one complaining here.

Finally, I'm planning to instrument the test better to spot those that are taking the most time and taming them (if that doesn't destroy they "testfulness"). Eventually (talking to the buildsystem guru again) it may be interesting to have two levels of testing, some "quick tests" that are supposed to be usable interactively, and some "slow tests" (I'm thinking of the quickcheck tests for example, that generate lots of random input) that would be used less often. With the continuous testing in place, I think it's not a danger if not all the tests are run by the developers during tight developping loops.

thelema commented 13 years ago

It seems that OUnit provides a -test-only command line flag that only runs a certain test. Perhaps this can speed things up? We may have to name tests better to do this.

thelema commented 13 years ago

Currently, to run just the failing BatString.splice unit test, the command is: _build/qtest/test_runner.byte -only-test "Unit tests:16:BatString unit tests:5:String.splice:1:src/batString.ml:604"

One can run just the string tests with: _build/qtest/test_runner.byte -only-test "Unit tests:16:BatString unit tests"

Except for the magic index :16:, this isn't so bad.

thelema commented 13 years ago

I've tweaked the quicktest generator to print the name of the test before executing its components. The result could probably use some \ns for formatting, but isn't too bad at the moment. This should help a lot with the not knowing if your test gets run problem.

thelema commented 13 years ago

This issue should be resolved by commit b63d1efa0006411880f732098052fc30903a0145, which exposes TESTABLE to the user. TESTABLE=src/batString.ml make test will only build quicktests for BatString. All the testsuite tests will still be run. Maybe we can migrate them to quicktests...