Workflow for testing real interactions using httptest

[mfherman]

Hi and thank you so much for your work on this book. I'm just digging in and it's already been a great resource. I'm interested in exploring the workflow for testing against a live API when you have already written tests that work via mocking.

In Section 7.3 you discuss included a folder of "real tests" that hit the API directly and don't use the mocked responses. I'm wondering if you have any thoughts on how best to implement this when there are a large number of mocked tests. Would you advocate for duplicating every single mocked test as a real test as well? My concern here is that it would require a lot of maintenance to ensure that for every new or modified mocked test you also write or change a real test as well.

[sckott]

Thanks for your question!

If you are using vcr it supports environment variables to make this use case easier - So you can have all tests that do HTTP interactions use vcr (so no real HTTP requests after the first recordings) - then if you want to run tests with real HTTP requests set the appropriate environment variable. In my R pkg development, I use Makefile's, so a make command to run tests is

make test
# which is just Rscript -e "devtools::test()"

Then if I want to run all tests with real HTTP interactions I can simply do

VCR_TURN_OFF=true make test

And you can do equivalent of that inside of R, or set the env var on your CI system, etc.

@maelle Maybe you can comment on how it would work with the other pkgs

[mfherman]

Thanks for the reply! This seems super easy with vcr :)

I guess I am thinking about this question in the context of httptest as for my usecase, it will be easier to have mock API responses that I can edit and create by hand.

[maelle]

This is an excellent topic and I think it might warrant its own chapter, I'll probably make a PR later.

Here are some notes, also mentioning vcr and webfakes to keep it exhaustive.

• It's good to have some tests against the real API, it does not have to be all tests I think? See my last point.
• Maybe you can just run the same tests without using the mock files.
  • with vcr as Scott says, this behavior is one environment variable away.
  • with httptest you can create the same kind of behavior https://enpiar.com/r/httptest/index.html#how-do-i-switch-between-mocking-and-real-requests
  • with webfakes you can also create that behavior.
• Now this means assuming all your tests work with real requests.
  • If a few tests won't work with real requests (say they have a fixture mimicking an API error, or specific answer as if today were a given date -- btw you can edit vcr cassettes) then you can skip these tests when mocking is off. With vcr this means using vcr::skip_if_vcr_off(); with httptest and webfakes you'd create your custom skipper.
  • If most tests won't work with real requests, then creating a different folder makes sense. It might be less unit-y as you could view these tests as integration/contract tests. Maybe they could use testthat's snapshot testing (so you could view what's different in the API).

[maelle]

(also, thanks for reading! :pray: )

[mfherman]

Thank you for these notes - so helpful!

One of my winter goals is to implement a test suite into tidycensus, so I'm certain I will be referring to the book through the whole process and will share any additional questions or suggestions as I work through it.

[maelle]

Fantastic, your feedback including questions will be most useful!