search

leanovate / gopter

GOlang Property TestER
MIT License
598 stars 40 forks source link

Best way to capture tests for regressions #68

Closed symbiont-sean-lyons closed 4 years ago

symbiont-sean-lyons commented 4 years ago

Gopter is a great tool for exploring a set of possible codepaths and discovering bugs. However, if I happen to find a bug in my code that comes from an unusual codepath, I would want to capture the sequence of operations that exposed that bug, and quickly turn them into a regression test to live in my suite independent of my normal Gopter tests. In other words, I want to be able to have my Gopter tests run, and then write regression tests in terms of Gopter.

E.g. Let's say my Gopter test for my simple counter fails with the sequence of events

initialState: {0}
sequential=[INCREMENT INCREMENT RESET DECREMENT]

I want some way to write that sequence of commands into a new test that will essentially run those exact commands within the Gopter framework, the same way I would expect them to run when running a more exploratory Gopter test.

Is there any way to leverage the current library to do this?

untoldwind commented 4 years ago

I'd say that the interface of the "commands" tests could certainly have some improvements.

The there are two core functions that (unluckily) are not public:

https://github.com/leanovate/gopter/blob/a9418a5b1253b18c75332bf0899966a611e91a24/commands/commands.go#L77-L84 https://github.com/leanovate/gopter/blob/a9418a5b1253b18c75332bf0899966a611e91a24/commands/actions.go#L41-L53

Especially the latter should contain everything to create a regression test from a given sequence of commands. Would it be helpful to make this a public function? Or maybe all you need is just to make the "action" struct public.

symbiont-sean-lyons commented 4 years ago

Thanks for the quick response!

I think the actions is pretty close to the process I'm trying to hook into. Although, it would probably be best to just expose a simple method like the following:

func Replay(commands.State,
          commands.SystemUnderTest,
          ...commands.Command) *gopter.PropResult {...}

All that's really necessary is a nice wrapper around the lifecycle of how commands are applied. Think of it like this -- I'm looking for something almost identical to a normal Gopter test, but I want to be able to specify precisely the commands that are generated and I only want to run 1 test case.

untoldwind commented 4 years ago

Sorry, I was really busy last week and overlooked this. I created a "devel" branch containing a "command.Replay" function that probably fits your needs (https://github.com/leanovate/gopter/blob/devel/commands/replay.go). If that is working for you I'll merge it to master

symbiont-sean-lyons commented 4 years ago

I think what you have there is a working solution, and is very close to what I've been doing myself to solve this problem. I'd be much happier to lean on this feature from the library than my own codebase.

One improvement I could see is a tighter coupling to the workflow run by commands.Prop so that the two cannot diverge in the future. As a consequence of the loose coupling of your current implementation, I don't think shrinking will be applied to regression tests. That's not a major problem, but I'd think it's sub-optimal to not apply shrinking.

I'm also beginning to familiarize myself more with the codebase, and would be happy to start making some contributions of my own on this front soon.

untoldwind commented 4 years ago

Yes, I see your point. I did a bit of refactoring. Can you give https://github.com/leanovate/gopter/blob/devel/commands/replay.go a try.

And yes, for a regression test a shrinker would actually be counter-productive. Its main purpose is trying to find the minimal sequence of commands to fail the system

untoldwind commented 4 years ago

I merged this version and will close this issue for now.