Give fc.statistics a return value

[ewinslow]

💡 Idea

If fc.statistics returned the distribution of classifications in addition to printing it, I'd have a straightforward way to property test my arbitraries. Would be nice to also have the number of skipped generated values.

Motivation

I've been trying my hand at building custom arbitraries, and noting that bugs can definitely sneak into those. In particular, I'm concerned that the arbitraries might not be well distributed across the input space, and manually printing out statistics is a bit of a pain. So my property tests could be passing but not really finding edge cases (e.g. imagine if my custom X generator just always returns empty X for some reason). Even if I get it right, someone else could introduce a regression and we could silently lose coverage that we thought we had. So I'm looking for ways to property test the arbitraries themselves.

Example

fc.assert(fc.property(_, () => {
  const results = fc.statistics(
    fc.string(), // source arbitrary
    (v) => `${v.length} characters`, // classifier
    { numRuns: 100_000 }, // extra parameters
  );

  return results.classes['1 characters'] > 0;
}));

Thanks for considering!

[dubzzz]

What you are suggesting is close to the feature request: https://github.com/dubzzz/fast-check/issues/342 🤔

[ewinslow]

It is related but I think my request is more focused. Just return the data that is already there :). No need for you to come up with any new APIs for labeling properties or asserting on statistical significance. Others can experiment with those using the data returned from this existing API.