Closed maelle closed 3 years ago
with an environment variable (another one?) one could also switch between capturing requests and using the mocks? which might make it easier to record mocks the first time around.
In https://github.com/maelle/exemplighratia/pull/1
This has come up a few times, and I'm open to suggestion for how to make it work. Another approach that I think I like better than env vars is more like what start_vignette()
does: if the given mock directory exists, use it; if not, create the directory and record real requests. Something like:
with_mock_dir(path, {
...
})
This would let you, say, re-record some mocks (by deleting that directory) without having to re-record everything, and have some mock tests that are based on live requests while also have some that are not.
The reason why this isn't built into httptest
already, and why I am less enthusiastic about global solutions, gets to some of the original design principles of httptest
. It doesn't assume that every API mock came from a real request to a real server, and it is designed so that you are able to see and modify test fixtures. Among the considerations:
#rstats
, the specific content will change every time I record it, so my tests can't say much about what is in the response without having to rewrite them every time too.I say this not to argue that there shouldn't be a nice way to have tests that work either with mocks or with live requests: I recognize the convenience and value of that--and that not everyone wants to edit JSON to make trivial API fixtures. I point it out since you're comparing with vcr
and think it's relevant context to explain the different feature sets and design choices.
this is such useful context!! I'll probably work this into the book either before or after the demos.
I was also wondering whether you usually work with good APIs, i.e. APIs that use versioning?
A bit off-topic...
Some conditions (rate limiting, server errors, e.g.) are difficult to test with real responses
I am looking into how to test for the behavior of the package in case of errors. Now, because mock filepaths are based on the request, how do you create two mock files for the same request, one with errors, one with no error? Using https://enpiar.com/r/httptest/reference/mockPaths.html, right?
Actually for errors I see the pattern mocking (general mocking) + fake_response e.g. https://github.com/cran/datapackage.r/blob/0bc1caa1903cf36b2d5f318265426eb072c361d7/tests/testthat/test-profile.R#L68
Yes, one way to have the same request respond differently is to change/append to the mockPaths--this is how the vignette mock setup works. It's trickier if you are testing something that retries the same request internally because you need some kind of hook to know when to switch mock paths. https://github.com/Crunch-io/rcrunch/blob/d1b5cade5e7b0608ddc1ce5de1a576e2d3614d84/tests/testthat/test-progress.R#L28-L37 is one attempt I did a while ago to poll an endpoint and confirm that it kept requesting until it "completed" (corresponding mocks here), but it's not trivial.
For errors, the example you found is a bit odd in that it's using with_mock
to alter httptest
. A better approach IMO is more like I described here: https://enpiar.com/2017/06/21/7-hard-testing-problems-made-easy-by-httptest/#5-rare-or-difficult-to-trigger-server-behavior. You can do a real request and modify the resulting response object to have whatever properties you want (response status etc.), or call fake_response()
and construct whatever you want, and then call save_response()
to write the mock out. (This surely deserves better documentation/how-to guidance.)
Back on my other points from yesterday, one concrete example of when you might want to delete content from response fixtures is with pagination. If you don't want to expose pagination to the R users (sometimes you do, sometimes it's an implementation detail they shouldn't have to care about), you might want an API wrapper that, if it gets a paginated response, makes subsequent requests to retrieve all records. In this package, I wrote a wrapper that did that, but IIRC the server controlled the page size and it was 100 records. I didn't want or need to have that many records in my test fixtures, so I recorded the responses and deleted all but 3 from the first page and 2 from the second. So I could test that the R function made the right requests to consume the multiple pages of results, and by asserting that there were 5 results, I knew that it was paging correctly.
I was also wondering whether you usually work with good APIs, i.e. APIs that use versioning?
I've worked with both good and really bad. Good APIs definitely make the code easier but I'm not sure how much they affect the testing strategy--what do you think? I guess with a well documented API you can get away with being less defensive/paranoid about just accepting whatever the API throws at you and can write more focused tests, perhaps even with mock fixtures that come from the API documentation (as I do in the httptest vignette with the Twitter API).
Thanks, this is very useful and I'll digest the information!
Regarding APIs what I was thinking is that if the response structure changes often and unexpectedly one will probably want to not have to tinker too much manually with the mock files, to be able to re-record them easily?
I think this could be considered closed by #48
Reg real requests in the HTTP testing book, in the httptest chapter I discuss using different tests for real requests, and I mentioned that in passing in #48
Reg https://enpiar.com/r/httptest/index.html#how-do-i-switch-between-mocking-and-real-requests in vcr one can turn off the use of cassettes just by setting an environment variable. This can be useful to have, say, a weekly scheduled CI run of the tests that use actual API responses. I see you suggest some lines to add to helper.R, but why not add this behaviour to httptest itself so the user would only need to set the environment variable?