search

dubzzz / fast-check

Property based testing framework for JavaScript (like QuickCheck) written in TypeScript
https://fast-check.dev/
MIT License
4.32k stars 179 forks source link

Sampling a property should give a comparable result to running it #2737

Open nth-commit opened 2 years ago

nth-commit commented 2 years ago

🚀 Feature Request

It would be great if, when sampling a property, it held the same formatting as is used for printing counterexample values.

Sample might not be an appropriate function here (because sample doesn't print), but maybe a fc.print() function could do the sample + format of a property and print the result to the console.

This is what happens in the .NET PBT framework I wrote (https://github.com/nth-commit/GalaxyCheck). I actually go as far as running the body of the test (but muting failures), exactly as if the property was being evaluated. This way, I can throw values out of the sample that would have failed the precondition (like fc.precondition), and you can also get the result of other lazily-enumerated input (streams, functions, even commands in a model - if that's how commands work, I'm not sure).

Motivation

A sampling mechanism that can be dropped-in for an asset is great (IMO) for building trust in the test runner, and also sanity checking your arbitraries.

DTopping256 commented 2 years ago

I completely agree. I was just looking for how I could run fc.sample, so that I could test my arbitraries like the examples in the docs say.

However, in runners like jest. I can't seem to get any nice output when I console log.

I resorted to JSON.stringifying the output of fc.sample, but I'm worried that this would hide keys with undefined values. Would love a way to output the arbitrarys like the test had just failed on that arbitrary as author says.

dubzzz commented 2 years ago

@DTopping256 If you want to log values during the execution of the test, I recommend using fc.context() it exposes an instance having a .log.

@nth-commit The main reason not to do that was performance. Today's version does not need to run the property to be able to sample it. But yes, since the creation of fc.pre we may want to run them. Not yet checked what it would impact in terms of code to add so but at least asyncProperty would be problematics (force to move sample to async).

nth-commit commented 2 years ago

@dubzzz Yeah I feel you there. I haven't implemented async properties in GalaxyCheck yet but I'm hyper-aware of how all these features combine and add complexity to the system.

I didn't know about fc.context. Will use that in the meantime 😊