Open yallop opened 7 years ago
Yes, I had always intended something like that. It would definitely be useful.
However, the current design supports a particularly handy mode of use, which I'd like to preserve in subsequent interfaces. Given a stateful interface like add_test
, you can compile many different test files into a single binary which runs all the tests, without having to list the complete set anywhere.
In the add_test
style, the single binary has a single test suite that contains many different tests, rather than a series of different test suites run one after the other. This makes it easy to fuzz the resulting test suite, with afl-fuzz
being able to decide which test to spend effort on, rather than doing separate fuzzing jobs for each one.
Another problem with the at_exit
system is that it interacts badly with https://github.com/aantron/bisect_ppx, which also uses at_exit
to dump the coverage reports. The result is that the coverage gets written before the tests are run.
(After writing this up, I noticed something similar-sounding in the TODO, which is probably a good sign. I'm sending this anyway, to add support for the idea, to flesh out the design a bit, and to open up discussion.)
At present the interface to tests is quite stateful: there's a function for adding tests to a global list
and the tests are implicitly invoked by a function installed with
at_exit
.This interface makes it a little difficult to integrate crowbar into larger test frameworks.
It would be helpful to have an ounit/alcotest-style interface, where the user constructs test suites, perhaps hierarchically, and where invoking the tests is the user's responsibility.
For example, here's a basic less-stateful interface that sticks quite closely to the spirit of the current design:
With an interface like this one might write:
Crucially, since this interface separates the operations of defining tests and registering/invoking them, it's easier to select tests (or even decide whether or not to invoke the crowbar-based testsuite at all) based on command-line options or other input.
Is there some reason why it'd be tricky to add such an interface? I wonder if the current approach with
atexit
is just for convenience, or if there's some deeper technical reason why it's better to do things that way.