search

typelift / SwiftCheck

QuickCheck for Swift
MIT License
1.42k stars 106 forks source link

How exactly to rerun failed examples? #281

Open Fryie opened 5 years ago

Fryie commented 5 years ago

Hi, I'm trying to figure out how exactly to reproduce a failure in my test so I can debug it. I receive output like "Replay with 1006624065 8850 and size 2". But I'm not exactly sure how I can generate the exact example that caused the failure.

I'm trying

let args = CheckerArguments(replay: (StdGen(1006624065, 8850), 2))
property("something", arguments: args) <- forAll ...

but it looks like that generates 100 test cases. How can I get only exactly the one that failed?

As a workaround I guess I can assign my condition in the forAll closure to a variable and work with if ... else, but maybe there's an easier way.

(Also, it would be nice if the output could be something like

Replay with "arguments: CheckerArguments(replay: (StdGen(1006624065, 8850), 2))"

or something, because currently it's not really intuitively clear where these values need to be inserted.)

CodaFi commented 5 years ago

You should have the shrunken version of the values that failed your property printed to the console by the failure handler.

Fryie commented 5 years ago

Yes. But if you have quite complicated structures (even after shrinking) with very custom description strings, it can be very hard to take that output and translate it back to code that initializes said value. This is certainly true in my case.

CodaFi commented 5 years ago

Very true. This is a language-level problem I think, and requires a language-level solution. Being able to stringify arbitrary bits of the AST with a good macro system would make this trivial.

In the mean time, as a hack, you could write a modifier type that dumps a value declaration as its description

CodaFi commented 5 years ago

I do agree with your suggestion to improve the replay diagnostic. Especially now that Xcode lets you copy diagnostic banner text.

Fryie commented 5 years ago

I do agree with your suggestion to improve the replay diagnostic. Especially now that Xcode lets you copy diagnostic banner text.

Maybe one could also add another property to CheckerArguments, like runOnlyExample: Int? Then the output could indicate the seed, the size and also this attribute so the exact test case can be easily reproduced.