Simulate stateful behaviour in Hoverfly

[JohnFDavenport]

We have debated how to handle state in Hoverfly but have held off as we need concrete use-cases. Bear in mind that we believe middleware will support stateful behaviour but do acknowledge that we perhaps need to add better support and guiadance.

If you have opinions, one way or another please comment here (or add a reference). We are particularly looking for:

• good use-cases, i.e. what tests are difficult or impossible without stateful behaviour
• examples that demonstrate whether there needs to be some different support for unit, integration and end-to-end tests
• what features should be supported in the language bindings such as Hoverfly Java?
• are there specific features that should be added to allow middleware to support stateful behaviour?

[JohnFDavenport]

See request for stateful behaviour here: #417

[JohnFDavenport]

Whereas we might provide in a language DSL something similar to http://wiremock.org/docs/stateful-behaviour/ I suggest that Hoverfly itself it should expose an API to manage a KV store for management of arbitrary data structures (think of a lightweight Redis) that would provide a common 'context' for the test and middleware. This could then be generalised to support some powerful features for population of response templates from data structures accessed by middleware. That potentially would allow developers to adopt whichever approach they found most natural for the problem. We (SpectoLabs) would provide some patterns appropriate for typical problems - FSM enabled mocks in unit-testing supported by language bindings & DSL, testing a multi-domain one-page app using WebDriver, integration or end-to-end tests using a test runner such as SOAPUI and end-to-end tests. Keep it simple.

[JohnFDavenport]

See use-case for stateful behaviour here: SpectoLabs/hoverfly-java#42

[robbiewareham]

re. Good Use Case;

Consider a search API that uses a scroll id. The requests will be exactly the same but the responses different, and eventually return a "completed" response, so that the consumer knows that there is no more data to return.

This can only be achieved with Stateful Behavior, and have already implemented using Wiremock (http://wiremock.org/docs/stateful-behaviour/). However I am interested in other options such as Hoverfly but Stateful is a must have

[JohnFDavenport]

Thanks for the use case @robbiewareham

That sort of behaviour was supported in Mirage (our previous SV product built for British Airways - we gave it back to BA as we decided it was too embedded in the BA environment to be of use generally). Mirage had logic to detect when the same request resulted in a different response. It then built a list of responses. Mirage would always play the list back in the same order. The beauty was it required no additional logic or action on the part of the tester. In contrast, I believe the Wiremock approach requires a fair bit of work as it is based on a strict FSM approach.

The Mirage approach is ridiculously simple but in 3 years I don't remember anyone ever complaining. The developers of Mirage had an approach of 'simplest possible solution that could possibly work' which allowed such changes to be delivered quickly. There was an expectation the solution would be too simple, but in that case, as in most cases, the solution stuck.

I'm not saying we would implement the same simple scheme as Mirage, but I offer it up here for discussion.

[robbiewareham]

I am generating JSON files describing the stub (currently Wiremock format) automatically from a HAR file, therefore any complexity is in code rather than on the tester.

I can't see why the simpler Mirage approach wouldn't work though

[mogronalol]

@tommysitu & @benjih I'm proposing the following for stateful behavior.

1. Matchers have an extra optional field:

"requiresState" : {
    "keyX", "value",
    "keyN", "value"
}

2. Responses have an extra optional field:

"setsState" : {
    "keyX", "value",
    "keyN", "value"
}

3. State is just an in memory map of keys and values
4. For first release, if state is used in a request or response then don't cache it. Then we can solve caching later.
5. The alternative is to have a single value for state, rather than a map. I think it's better to have a map as it will track a state flow better.

Thoughts?

[JohnFDavenport]

A map seems right. No caching seems right. Some initial questions:

1. What is the scope and lifetime of the map (e.g. is it a request/response pair, simulation, or global to the hoverfly instance)?
2. Can I concatenate my keys ... for instance can I concatenate the session variable in a header with something in the body to create a composite key.
3. Is the map accessible from middleware?

Also, so we do not forget, this does not provide a means of detecting statefulness in capture mode. Seeing the same request body elicit different response bodies would probably be enough.

[mogronalol]

1. It's an in-memory map. Will be cleared whenever you switch to simulate mode, or cleared explictely
2. You can just set and require multiple key, would that do what you mean?
3. Not accessible from middleware to begin with, but it could be in the future.

Capturing statefulness should be left for now too, as it will be pretty complicated

[tommysitu]

Why do yo need a key? For requestMatcher, do you think having multiple requiresState values would be enough:

"requiresState" :  ["Applied offer code", "Added an item"]

[tommysitu]

It might be worth to consider this type of scenario: For example, I am testing a long polling function, the response initially showed as PENDING and after a few request, it changes to READY. The same applies for testing some rate limiting UI logic as well. So what I am thinking is that in addition to per-request state transition, you can also define external state transition like this:

{
    "stateToggle": [{
           "current: "Wrong input password"
           "next":   "Login disabled temporarily"
           "condition":  "Count > 3"
     }]
}

[mogronalol]

The problem with a list is that you would keep appending to it without removing any items. So if I had a shopping flow where the states were basket-empty->basket-contains-eggs then changing from one to the other should remove the first state, meaning I need a key which would cause it to be overridden

[mogronalol]

The second scenario seems more complicated but it could end up being built on top of what is initially implemented. However, it would involve introducing an expression language, which would be difficult to manage. That particular example is also solvable by this:

requiresState=never input password,sets state=wrong input once
requiresState=wrong input once,sets state=wrong input twice
requiresState=wrong input twice, sets state=wrong input three times
requiresState=wrong input three times,sets state=login disabled

[robbiewareham]

This is pretty much the same implementation as Wiremock;

http://wiremock.org/docs/stateful-behaviour/

{
    "scenarioName": "To do list",
    "requiredScenarioState": "Started",
    "newScenarioState": "Cancel newspaper item added",
    "request": {
        "method": "POST",
        "url": "/todo/items",
        "bodyPatterns": [
            { "contains": "Cancel newspaper subscription" }
         ]
    },
    "response": {
        "status": 201
    }
}

{
    "scenarioName": "To do list",
    "requiredScenarioState": "Cancel newspaper item added",
    "request": {
        "method": "GET",
        "url": "/todo/items"
    },
    "response": {
        "status": 200,
        "body" : "<items><item>Buy milk</item><item>Cancel newspaper subscription</item></items>"
    }
}

If 'set state' in the final request is set back to the first "requiresState", you create a mechanism that supports searches that use a scroll ids (such as Elastic Search) very well in a repeatable manner.

[JohnFDavenport]

Thank you @robbiewareham

What you are looking for ( @mogronalol and @tommysitu ) from the examples you just posted seems to be a finite-state machine. The nice thing about them is they are simple and deterministic.

Let's do that and leave out generalised state management.

[JohnFDavenport]

State is now in v0.14.0 and it is documented [here]()https://docs.hoverfly.io/en/latest/pages/keyconcepts/state/state.html). If there are further issues please open a new issue for the specific problem.