State machine testing multiple results

[ocharles]

I am doing a bit more playing with the state machine testing, in the context of testing work's REST API. For the purposes of this test, we have two end-points:

• /parts?q=<q> performs a search query and returns a list of parts that match the query q.
• /part/<identifier> looks up the part given as identifier.

Internally, I know that performing a search consults an external service, and adds the search results to our local database. Therefore, I can start with a blank database, perform a search, and then lookup any of the parts that were returned. I'm trying to capture this with the following state:

data State var = State { parts :: [Var String var] }

The search command should update the list of parts with the result of the API call:

searchForParts =
  Command
  { commandGen = \_ -> Just (SearchForParts <$> Gen.element sampleSearchQueries)
  , commandExecute =
      \(SearchForParts q) -> do
        response <- lift
          (WAI.request
             (WAI.setPath WAI.defaultRequest (fromString ("/parts?q=" ++ q))))

        return (WAI.simpleBody response ^.. key "searchResults" . _Array . traverse . key "urn" . _String)
  , commandCallbacks =
      [ Update (\(State ps) _query response -> State $ ps ++ response)
      ]
  }

But this doesn't type check! response is a Var [String] var, but ps is [Var String var]. It doesn't look like I will have any hope at reconciling the two if I keep var polymorphic and don't have anything more than HTraversable at my disposal. One vague idea would be to add a codensity-like transformation to Symbolic, so we can at least have a working functor (HFunctor?) for both Var a Symbolic and Var a Concrete, which might be a good start.

Any ideas what to do? Having State { parts :: [Var [String] var] } might be an option, but is not at all satisfactory 😢

[ocharles]

Actually, State { parts :: [Var [String] var] } isn't just ugly, it's equally unusable:

fetchPart =
  Command
  { commandGen = \(State knownParts) ->
      case knownParts of
        [] -> Nothing
        _ -> Just (FetchPart <$> Gen.element knownParts)

Would generate Var [String] Symbolic, but we want Var String Symbolic. Our only option is to take a random String from within the command execution, but then all of hedgehog's output will be misleading.

[jacobstanley]

Hey sorry for the significantly delayed reply, I've been pondering this for a little while and I the best I could come up with was to introduce some commands which don't actually contact the external service, but which can be used to interrogate the Var [String] var.

So imagine if you had something like:

-- Read :: Int -> [String] -> String
data Read v =
  Read Int (Var [String] v)

You'd be able to implement execute easily because that has access to the concrete [String]. For pre and post conditions you probably need a Map (Var [String] v) [String] or something like that, but it's kind of unclear how this might work because you don't really have access to add/remove parts, so I don't even know what the pre/post conditions would be. There's an example in icicle which works a little bit like that.

I understand this is a bit clunky, but maybe we can iterate on the core mechanics of the idea and produce something a bit nicer.

[ocharles]

You'd be able to implement execute easily because that has access to the concrete [String].

This seems to be my second comment, but that seems bad, because as I mentioned:

[This] would generate Var [String] Symbolic, but we want Var String Symbolic. Our only option is to take a random String from within the command execution, but then all of hedgehog's output will be misleading.

My command generator doesn't have access to the [String], so I can't take an element or calculate the length. This is presumably why you mentioned

For pre and post conditions you probably need a Map (Var [String] v) [String] or something like that

But where do the elements ([String]) of this map come from?

I understand this is a bit clunky, but maybe we can iterate on the core mechanics of the idea and produce something a bit nicer.

Certainly, I think iterating on a concrete example might be easier. Essentially, the problem comes down to:

1. A command that returns a list of results.
2. A command that needs to act on an individual result of this collection.

Shall I try and throw together a standalone example?

[jacobstanley]

Yeah that sounds great, I think part of the problem here is stemming from the fact that we don't have control over the results in the system under test. If we could add parts to the system or knew what they should be then we'd be able to populate our model (i.e. the [String]).

[ocharles]

https://github.com/ocharles/hedgehog-scenarios/tree/master/database-constraints is essentially one of the simplest problems that I've got so far.

My external system here has some constraints, but generating commands that respects them isn't possible (in the way I've setup State in that example).

[magthe]

It's a bit disappointing to find this issue, and that it's still open, since I seem to have bumped into it as well -- my question on SO. I'm guessing, based on the age of it, that it isn't something that a lot of people run into :(

[spacekitteh]

Ooof, this hasn't got an answer? D:

[ocharles]

@spacekitteh Maybe not an answer, but https://github.com/hedgehogqa/haskell-hedgehog/issues/459 explains some of my current thoughts on this.