No print output when test timeout

[czhang03]

Hi,

This is a general question. We are using quick-check and tastey to test our student's homeworks. But we encounter the problem that quick-check (even verbose) not been able to print the test cases when the test timesouts.

This makes debugging very hard.

Is this a design desicion? Is it possible to print the inputs before they are send into the test?

[nick8325]

It seems that when you use within, only variables that are quantified outside of the call to within are printed. For example, in the following property, the value of x will be printed but not the value of y:

prop_whatever x = within 1000000 $ \y -> something that loops

So you may be able to work around this by moving within into the property. That is, instead of writing:

prop_whatever x y z = ...
main = quickCheck (within 1000000 prop_whatever)

you should write:

prop_whatever x y z = within 1000000 ...
main = quickCheck prop_whatever

I think this bug was introduced when QuickCheck stopped catching asynchronous exceptions (in order to fix bugs #32 and #144). Previously, the property body would end up catching the timeout exception and so it was treated as any other kind of test failure. Now, only the within combinator catches the timeout exception and the inner part of the property has no chance to add any information to the counterexample.

EDIT: accidentally referred to within as timeout.

[nick8325]

Unfortunately, System.Timeout doesn't export the Timeout exception. Probably we need to reimplement within without using System.Timeout.

[czhang03]

I have also looked into the tasty library. It seems like the bug in on their side: https://github.com/feuerbach/tasty/issues/86

Do you have any idea how to fix this bug? Here is some information about how they are using QuickCheck:

1. The best I can get is that they use Control.Concurrent.Timeout (timeout) to control their function timeout: Here is the line: https://github.com/feuerbach/tasty/blob/2c5a99bb94cc6cd9b8f1554b57c76b9fb7c6f83a/core/Test/Tasty/Run.hs#L161

2. There quick-check linking happens here: https://github.com/feuerbach/tasty/blob/f46ff251f9386bdfa2ab678640582c596360fa7e/quickcheck/Test/Tasty/QuickCheck.hs

My guess is that the timeout function kills the QuickCheck Process, so that quick check don't have a chance to output its output? I am just guessing. Thank you so much for your help.

[tom-audm]

Could we request System.Timeout to export the Timeout exception?

[MaximilianAlgehed]

I think the relevant things are exported from System.Timeout now.

[czhang03]

Hi all, I unfortunately no longer uses Haskell quickcheck, so I cannot test this right now. If anyone can confirm that quickcheck will output the test cases after time out, then I imagine this can be closed.

To clarify this is the behavior I expect:

prop_timeout x: int = <loops forever>

will output something like

`prop_timeout` unsatisfied, time out on input `x = 0`