adolgert / UnitTestDesign.jl

Increases unit test coverage with fewer test cases using all-pairs and other covering arrays.
http://computingkitchen.com/UnitTestDesign.jl/stable/
MIT License
15 stars 2 forks source link

Docs: combinatoric testing because it finds all code paths. #28

Closed oxinabox closed 2 years ago

oxinabox commented 3 years ago

The docs currently say:

We use combinatoric testing because it finds all code paths. Code paths are lines of code chosen by the same if-then decision. Some code has few code paths but many branches

This seems dubious. Appropriate choice of combinatoric elements can find all code paths. But for example

foo(x, y) = x > 1_000_000y ? error() : 1

for (x, y) in all_pairs(
    [(-3, -2),  (2, 3), (-2, 3), (4.999, 5), (0, 1e7)],
    [0.1, 0.01, 0.001],
)
    @test foo(x,y) == 1
end

will not test all paths.

I think this can be contrasted to things that directly generate the test code.

adolgert commented 3 years ago

"Dubious" is an understatement. It should explain that combinatoric testing is an effort to find code paths, but that's complicated, so I got lazy. I'll link to a PR with a more proper, and realistic, explanation.

If you think about the set of parameters as a space of all allowed parameter combinations, then the example you gave, for instance, is two-dimensional and each dimension covers all floats and integers. Combinatorial testing rests on the idea that arguments that are far apart from each other in the input parameter space are more likely to cover more code paths.

But the docs should be less technical than that. I'll add the PR before we close it.