Workflow advice

[maelle]

So say I have now wrapped all the tests of my package in cassettes.

The tests assess how my package handles responses of the web API at version blabla.

How do I make sure I'm keeping up with web API changes, or even web API failures? Should I have e.g. one build that doesn't use the cassettes on Travis? Do you have any example?

[sckott]

There's a lot of variables here as you know, so it's hard to give one solution.

In the recent version we added https://github.com/ropensci/vcr/blob/master/NEWS.md#new-features re_record_interval setting, so e.g., you could tell vcr to re-record real interactions every 5 days, or 24 hrs, or whatever. This approach is a sort of hack I guess, so that test runs are fast by using cached request/responses, but then more up to date with the current web API than if they weren't updated

Another approach is as you suggested: have some tests that make real HTTP requests. Of course I'd skip on cran these real requests in case they fail at some point. I haven't seen any best practices for this approach. I do this approach sort of by accident in some packages since vcr doesn't yet (but it's almost ready) support when http requests write to disk, e.g., in rnoaa

If you do the re-record interval setting thing, that I guess would take care of any changes in the web API since when re-recording real requests, it should hit upon any problems, right?

[maelle]

Ideal build matrix IMO

• A few builds with vcr turned on
• A build with vcr turned off only for commits within the repo (or any condition ensuring one doesn't run tests with vcr turned off in the absence of tokens etc)

[maelle]

(plus with the re-record you mention)

[sckott]

commits within the repo

does that just mean no PR's?

[maelle]

No PRs from external repos

[sckott]

so PRs from another branch within the repo + commits to default branch?

[maelle]

Yes

[sckott]

what does "A few builds with vcr turned on" entail? How are those builds different? can you list all the builds out?

[maelle]

Ah sorry, one would be enough but often people already have several builds (e.g. a matrix with different OS and R versions)

[sckott]

Should we add some text to the book for this?

[maelle]

Yes but ideally it'd go with an example package repo. I was thinking that example repo could have two GitHub actions workflows, 1 for any push with vcr turned on, and 1 that runs regularly with vcr turned off.

how could one create that example package?

[sckott]

You could use the crandb api or the cranchecks api or similar

[maelle]

I've just added VCR_TURN_OFF to https://github.com/ropensci/ropenaq/blob/master/.github/workflows/R-CMD-check.yaml (but some builds are failing because of dependencies). All jobs are run once a day, I think I might switch to once a week.

[sckott]

thanks for trying that, hopefully it works well