better body assertions: Specify kind of assertion to make

matryer commented 8 years ago

Proposal: better bodies in Silk

Bodies in Silk could make use of the 'formatting' tag thing where not only the data type is specified, but also the kind of assertions that could take place:

``json(mode=exact)
{
  "name": "Silk",
  "released": 2016
}
``

Two back tics are used so as not to confuse renderer
This assumes that the text after the opening code block line is never rendered (needs validating)

The types of bodies could be:

exact - the body must match verbatim
same - the JSON must essentially be equal but fields may be in different orders and formatting (whitespace) is ignored
subset - each mentioned field is asserted, additional fields in the body is ignored

The syntax highlight language specified could also drive which ParseBody function is called.

Cons

When rendered, you can't see the kind of assertion, but that might not be too much of a problem and could always be explained in the plain text surrounding the block.
Alternatives

Other options to consider:

the json language specifier might be enough to indicate the datatype, and we could use flags (when running the file) to indicate what type of assertion to make

dahernan commented 8 years ago

Instead of rely in the user, what I think it makes for sense is rely on sensible defaults. The idea is if the content type is JSON just relay on the JSON non strict equal to assert the response (non strict is not taking into account order on the elements, with the exception of arrays), that should be ok for most of the users.

For advance users, in case of JSON maybe a JSON Schema helps (I don't know the best way to relate a response to a schema)

If the content type is not JSON, the exact match is the most sensible default, with the possibility of implementing more assertions for other content types for example XML

narqo commented 8 years ago

The idea itself looks reasonable, but I think json label should work as json(mode=same) by default. I think that simplicity of the silk's way of written test is one of it's strengths, so one should try to save it.

matryer commented 8 years ago

Agreed... I'm all for simplicity and will always listen to something that strips functionality out, rather than adds it.

Question is, which is the sensible default:

strict matching (i.e. extra fields in response JSON would be a fail)
similar matching (i.e. only mentioned fields in Silk document will be asserted)

narqo commented 8 years ago

JSON isn't biased about the order of fields. I don't remember any use cases where strict matching could make big difference.

The order of fields could change through the time, e.g when API producer switch to another JSON marshaller (nodejs 0.10 and 4 might stringify JSON with different field order). But at the same time, no API consumer would be harmed by this change, as JSON is a scheme-less data structure.