search

AArnott / Xunit.Combinatorial

Adds combinatorial and pairwise testing capability to Xunit tests
Other
180 stars 16 forks source link

Limit the number of generated tests to a random subset #61

Open alrz opened 1 year ago

alrz commented 1 year ago

Given a set of generated test cases, it's useful to be able to select a random subset for each run to reduce the CI time, while keeping the original combinatorial definition intact. e.g.

[Theory]
[CombinatorialCount(30)] 
public void Test(...) {}

There's already a helper to do this Random.GetItem in dotnet 8p1 but I think it's easy to backport.

AArnott commented 1 year ago

I think we typically use the Pairwise attribute instead of Combinatorial to keep the set of generated tests to a reasonable number. When combinatorial results in a test case explosion, randomly picking 30 could be a grossly inadequate coverage, whereas Pairwise (or 3-wise, etc.) provides a more provably confident coverage amount. Have you tried the Pairwise attribute?