dubzzz / fast-check

Property based testing framework for JavaScript (like QuickCheck) written in TypeScript
https://fast-check.dev/
MIT License
4.25k stars 176 forks source link

Print `verbose: 2` execution summary during the run #5086

Open gruhn opened 1 month ago

gruhn commented 1 month ago

πŸš€ Feature Request

How much trouble would it be to print the verbose: 2 execution summary during the test run, instead of printing it in batch at the end? Using the example from the documentation:

Execution summary:
Γ— [[2147483632,-1868311174,2,-14,-1309756521,948824737,2147483631,1352827217]]
. √ [[]]
. Γ— [[-1309756521,948824737,2147483631,1352827217]]
. . √ [[2147483631,1352827217]]
. . Γ— [[948824737,2147483631,1352827217]]
. . . √ [[2147483631,1352827217]]
. . . √ [[0,2147483631,1352827217]]
. . . Γ— [[474412369,2147483631,1352827217]]
. . . . √ [[2147483631,1352827217]]
. . . . Γ— [[237206185,2147483631,1352827217]]
. . . . . √ [[2147483631,1352827217]]
. . . . . √ [[118603093,2147483631,1352827217]]
. . . . . Γ— [[177904639,2147483631,1352827217]]
...
. . . . . ... . . Γ— [[2,1000000000]]
. . . . . ... . . . √ [[1000000000]]
. . . . . ... . . . √ [[1,1000000000]]
. . . . . ... . . . √ [[2]]
. . . . . ... . . . √ [[2,999999999]]

Presumably, each of these lines can (in principle) be printed as soon as the corresponding test has failed/succeeded. The level of indentation is also known, right? I know there is some effort going on with workers to run tests in parallel? Would that be a problem here?

Motivation

This is just a minor improvement from my view. And I guess for fast running tests, this would make no difference. I have some fairly long running (model-based) tests though. There it would be nice to have more lively feedback. In particular during shrinking. Sometimes shrinking takes many iterations. I noticed that team members, who are not so familiar with the tests, get the impression that the tests are "stuck". Recently, I also gave a presentation on our testing setup with a live demo. I started the tests, then bridged 2 minutes with other talking points, and finally explained the idea of shrinking, once the execution summary was available. No problem but it also would have been slightly nicer to explain the shrinking process as it was happening.

dubzzz commented 1 month ago

I might be doable at some point today by hook something around a property. The more I think of such problems the more I feel I should offer a better set of APIs to offer users with ways to extend fast-check with custom reporters, custom executors...

Not planning to do anything for this topic in the coming releases. But I keep it on the side as a great idea to have in mind for later evolution (including providing an API o extend fast-check in a way simpler way).