search

typelevel / scalacheck

Property-based testing for Scala
http://www.scalacheck.org
BSD 3-Clause "New" or "Revised" License
1.94k stars 407 forks source link

Make class declarations public to enable save and replay of shrinked command sequences #143

Open dpolivaev opened 9 years ago

dpolivaev commented 9 years ago

After ScalaCheck generates and executes some command sequences, finds failures and shrinks the sequences to the smallest sequences required to reproduce a bug, it could be helpful to save such distilled sequences which could be used for debugging of SUT and for regression tests. I would like to implement an extension which can capture and replay the findings.

It looks like currently a minor issue stops me from doing it. trait Commands in package org.scalacheck.commands declares private type Commands = List[Command] and private case class Actions(s: State, seqCmds: Commands, parCmds: List[Commands])

Objects of this types are later available in test result communicated to TestCallBack. So the command list could be serialized and replayed.

Therefore I suggest to either make the types public or to create a new public type used in Test.Result.status.args which could be used in implementation of TestCallBack for serialization of the found test sequence.

Regards, Dimitry Polivaev

rickynils commented 9 years ago

We are slowly moving towards reproducible tests in ScalaCheck by making the generators deterministic. In the end, that should make the stateful tests reproducible too.

dpolivaev commented 9 years ago

In my eyes one important advantage of scalacheck is its ability to shrink generated test command sequences and so to find smallest possible tests which still find bugs. The goal of this request is to be able to save a shrinked command sequence in a file so that it can be used as a test case for debugging and later as a regression test.

It is not given just by ability to replay the whole sequence of tests just with the same seed. Therefore I would like to ask you to reopen this issue.

Please let me know what you think and also let me see you have read this post.

Regards, Dimitry Polivaev

rickynils commented 9 years ago

@dpolivaev You are correct, even if the tests are deterministic there is nothing that says the system under test is deterministic. So somehow saving the final (minimized) command sequence would actually be useful. I'll re-open this issue and start thinking about a solution.