search

nanofuzz / nanofuzz

NaNofuzz is a fast and easy-to-use automatic test suite generator for TypeScript that runs inside VS Code
MIT License
36 stars 3 forks source link

How many times should a function without arguments be tested by default? #182

Open ZiyaoWei opened 5 days ago

ZiyaoWei commented 5 days ago

When testing a function that doesn't take input arguments, e.g.,

export function f(): void {
   return;
}

By default it would test it Max number of tests times, which is a bit unexpected since the input only has one possible value. I guess we can say that if a function without arguments is worth testing then it must be nondeterministic, although I feel that should be a separate option users can choose since other functions can be nondeterministic too? E.g., having a "nondeterministic" checkbox that users can toggle to enable testing duplicated inputs and have it on by default for any function without input.

cmumatt commented 5 days ago

Agree that this can be kind of surprising, especially because we state that NaNofuzz lacks support for non-deterministic PUTs.

The backstory for this behavior revolves around the buggy example program random.ts, which is non-deterministic. We used this program in some early pilots but never in an actual study. In very early versions of NaNofuzz, it could test the same inputs multiple times: it didn't keep track of the prior inputs tested, which was understandably confusing to some users in early pilots.

The logic that contains this exception for 0-parameter PUTs is here: https://github.com/nanofuzz/nanofuzz/blob/6474eb5255494b61729e3add9ce784b2afbe6d40/src/fuzzer/Fuzzer.ts#L206

How to support testing of non-deterministic PUTs in a usable way is worth exploring! Initial thoughts are maybe to pursue a hybrid approach where the tool will run some duplicate tests to detect non-determinism in case it's present but the user is unaware but also allow the user to assert via an option that the program is, might be, or is not deterministic. Then, once non-determinism is detected or asserted, it's a good question what to do next to help the user test the program.

Maybe once we know a program is non-deterministic, either via an automatic check or a manual option, we turn off the input dupe check and, within the results grid, group and summarize same-input test examples together?

It seems like our human/example-based oracle should also allow assertion of multiple expected outputs to support non-deterministic programs. Supporting multiple values/ranges/more sophistication in the human validator's expected output is something we should do anyway and independently of when we add support for non-determinism.

Would love to hear your thoughts on this too!