Fuzzy cassettes

[Stranger6667]

For unhappy path testing, there could be an extra parameter that will mutate recorded cassettes. Use cases:

• Check how the app will work if the format of responses will change
• Validate error handling

Possible fuzzes:

• Mutate response body. JSON - add/remove fields in objects / lists. Change values. Make JSON invalid.
• Completely change response content type
• Change response status code
• Mutate requests during recording??
• Raise exceptions instead of real responses
• Add delays to responses

All these could be combined and generate test cases

[asmodehn]

My 2 cents here:

1. To validate error handling, we could have a set of preexisting cassettes in this package (included in the set used by default) that produce errors as a response. Probably a first simple step to tackle here, and see how it goes...
2. To change the response headers, we can follow the HTTP spec, and define data generator (like hypothesis does for example) for each of the headers.
3. To change the response body, it seems to me that we have two choices :
   • we need some way of specifying the "schema" or structure of that response. So the user would have to specify an API somewhere... which means it might be done by another test plugin, related with a way to specify the API. I m thinking about something for uplink for example, where the format for the response is specified by a marshmallow schema, and fuzzying the data in this schema (with something like hypothesis) seems doable, but probably out of scope of this plugin.
   • we assume the strict minimum (ie. the body is a "string" for example) and we generate random response body using this generic type. Probably the better option here.

That reasoning seems to indicate we should always stick to whatever the HTTP spec authorizes for fuzzying the response. A good example for discussion is status code. Fuzzying to get any kind of string here seems overkill and I would put that in the list of things to test at a lower level of the network stack, but fuzzying 3-digit-string seems reasonable. However only testing the exact list of expected status would not be enough (since some services return "their own code"...)

I think we need to clearly limit what should be expected from this plugin. For instance:

• only HTTP protocol, not another, and not under in the network stack. We do need to test that the code handles the errors that can bubble up from the libraries we rely on however...
• only bare HTTP as per the spec, and not service defined API. Currently it is clear from the README in vcrpy, but not here I feel...

To go under in the network stack, or for different protocols, I would defer to other packages (unless you plan something different ? but it would be much more complicated than just using vcrpy...) To go above HTTP and fuzz with some kind of user API knowledge, we will need "coupling" with a specific way of specifying that API on the user side for the usual application.

What do you think ?