Run cleanup function with model after each state machine test

[Skyfold]

Like others I'm using hedgehog's incredibly useful state machine functionality to model/test my API during development. However, I realized the model I have built would be the perfect way to ensure the production API is functioning properly. This means I would need to run a cleanup function after each sequence of commands that takes the current model as input so I know what to remove from the API after those commands have completed. If there was a failure I wouldn't want the cleanup to run so I have more to work with to determine what happened.

One way to accomplish this is to append a command to the sequential/parallel generated sequence for cleanup. That way I can also check to make sure the cleanup actually worked. Though, I haven't figured out a way to do this in Hedgehog yet. Let me know what you think.

[jacobstanley]

This sounds interesting, so am I right in understanding that you'd like to only do the cleanup if there isn't a failure?

[Skyfold]

Yes, the cleanup should only run if there is not a failure. My thoughts are: If there is a Command that fails in some sequence/parallel execution of commands then its possible that the model no longer represents the state of what you are testing and as such when the cleanup function is given the model it may not behave as intended. The cleanup function should be written with the assumption that the model it is given is an accurate representation of the system being tested. I think making the cleanup a Command itself seems natural.

[HuwCampbell]

Both PropertyT and TestT are short circuiting monads based on EitherT.

If you can derive your cleanup actions by reversing the list of commands inside the Sequential structure, you should be able to just put it below in the do block and have it work.

[Skyfold]

You are right that a cleanup function would not run if one command failed in the sequence of commands, just like I'm after. The problem is deriving the cleanup actions from [Action m state]. Those Actions are just the functions and data you need to run the commands, not the data you get back from the API. It does have actionInput :: input Symbolic, so I could work out what commands were run, but that would mean the cleanup function cannot depend on what I get back from the API. Unfortunately, I need the unique identifiers (user tokens) generated during execution to know what data I need to delete.

I'm trying to see if I can tweak the process of turning [Command gen m state] -> gen (Sequential m state) so that you always get a cleanup Command running at the end. I think I can append one inside of a modified genActions function by calling action on my cleanup command and appending that to the end of the generated list. I'll make a separate sequentialWithCleanup function that takes the extra cleanup command. (let me know if you can think of a better name)

[Skyfold]

Adding a cleanup Command at the end is not the right solution. If we use a Command the type would allow you to do things you wouldn't want in a cleanup function. You shouldn't have a Require or Update step, nor can the generation of input return Nothing. Instead, its much easier to just return the concrete state from executeSequential. This only requires a small code change, foldM_ to foldM and gives reasonable behavior. If a sequence of commands fail your cleanup function gets the initial model, plus the cleanup function would run on shrinks that succeeded. I'm just not sure how to modify executeParallel yet.