search

UnkindPartition / tasty

Modern and extensible testing framework for Haskell
640 stars 110 forks source link

Add option for printing the number of QuickCheck tests and shrinks #431

Closed jorisdral closed 2 weeks ago

jorisdral commented 1 month ago

Currently, tasty and tasty-quickcheck will print a progress percentage, so one can see test progression. However, once a property finds a failure and starts shrinking, then test progression stops. Importantly, it is not clear how the shrinker is progressing, though it would be useful information to show, e.g., to judge whether a test is shrinking too slow, whether the shrinker loops, or whether a shrunk test case hangs. This commit adds a new option to enable printing the number of QuickCheck tests (and shrinks on test failure) in addition to the percentage.

I know this feature has been suggested before here: https://github.com/UnkindPartition/tasty/pull/419#discussion_r1632346098). In my opinion it would be a generally useful feature to include, and it's optional so that it does not use up too much UI space by default. Let me know what you think!

Bodigrim commented 1 month ago

@jorisdral I'm keen to avoid adding more command-line options. The interface is already quite cluttered.

I'd prefer the following behaviour to be implemented unconditionally: firstly show progress from 0% to 100% while testing (progressText is empty, progressFloat is non-zero), then show number of shrinks (progressText is "XX shrink", progressFloat is zero).

Bodigrim commented 3 weeks ago

@jorisdral what do you think about my suggestion above?

No promises, but we might be making a release soon; it would be great if this improvement lands before that.

jorisdral commented 3 weeks ago

@Bodigrim I think your suggestion makes sense. I'll see if I can adjust the PR soon

jorisdral commented 3 weeks ago

@Bodigrim I've updated the PR. Let me know if this is what you had in mind

Bodigrim commented 2 weeks ago

Thanks!