search

cdent / gabbi

Declarative HTTP Testing for Python and anything else
http://gabbi.readthedocs.org/
Other
148 stars 34 forks source link

Faster tests development with gold files #286

Open avkonst opened 3 years ago

avkonst commented 3 years ago

There is a cool method to speed up development of tests. It would be great if gabbi supported it too.

Here is the idea:

  1. a test defines that a response should be compared with a gold file (reference to gold file can be custom configurable per every test)
  2. gabbi runs tests with a new flag 'generate-gold-files', which forces gabbi to capture response bodies and headers and (re-)write gold files containing the captured response data
  3. developer reviews the gold files (usually happens one by one as tests are added one by one during development)

  4. gabbi runs tests as usually

    a) if a test has got a reference to a gold file, it captures actual response output and compares with gold file b) if content of the actual output matches the gold file content, verification is considered to be passed c) otherwise test is failed

This would allow me to reduce size of my test files by half at least.

cdent commented 3 years ago

When I initially created gabbi I specifically did not want it to be for regression testing so left out any functionality that would allow it to be directly used for recording response bodies into future tests.

It does, however, have features that make it easy to read JSON from disk to compare with a full response or an attribute thereof: see near the bottom of https://gabbi.readthedocs.io/en/latest/jsonpath.html

Using some scripting to glue things together it would probably relatively straightfoward to make something that you want: read the existing YAML to run requests, capture the responses to disk, process them to "new" YAML. In fact it would probably be possible using gabbi as a library for the main functionality in the tool.

While I don't have the time to create something like that or something more directly built into gabbi, I'd certainly be willing to look at a pull request that provided the functionality.

avkonst commented 3 years ago

thanks. I will have a look what I can do with existing methods

avkonst commented 3 years ago

you can put it on freeze until somebody else needs it to. The ticket scope then is how to generate these gold files automatically, because checking against the files seems like already available.