How to make tests work for external PRs in a repo

[maelle]

Workflow idea

• The maintainer creates fixtures with their API key by running a function wrapping testthat::test_package() or so + indicating what's to be redacted. vcr redacts the API key to "foobar" everywhere.
• In all other tests
  • the key is set to "foobar".
  • therefore the tests can use the fixtures.
• The maintainer can update fixtures by running the function.

Problems it solves: one wouldn't need to skip tests based on the presence of an API key.

One could still turn vcr off (and the setting of the key to foobar) for a package check that'd run regularly on a CI service.

[sckott]

thanks @maelle - I like it.

To see if I understand, you're proposing a new vcr helper function wrapping test_package? Or just using existing functions?

[maelle]

I don't know what would make the most sense! vdiffr has a separate function for recording fixtures.

[sckott]

I think (not positive though) that all we'd need to do is document that one could set an env var to whatever (e.g., "foobar") and then tests would run even though a "real" key was not present. does that seem right?

[maelle]

It does, but I'd like for the env var to be set by a line in a test setup file. So the maintainer has to comment out this line to record fixtures, but contributors don't even need to think about changing/setting the env variable.

[sckott]

hmm, okay.

[maelle]

Less work for everyone except for vcr maintainer 😉

[maelle]

I've thought a bit more about this whilst trying to add vcr tests to rhub.

Recording mode

• Everything as usual
• Make sure vcr_configure() has filter_sensitive_data
• Have a look at the fixtures to be sure they do not contain anything sensitive.

Testing mode (in particular CI, external collaborator, when no API key/token present)

• Mock your own package functions so that e.g. yourpkg::get_key() returns. "foobar". Because vcr is on, it's fine the key is false.

It was quite a revelation for me that vcr cannot handle everything that has to be tweaked in the testing mode. I suppose only examples can help here.

[sckott]

we talked about potentially adding to vcr:

• a setup() method, like that for testthat
• a teardown() method, like that for testthat

testthat has fxn testthat::is_testing (just sees if the env var TESTTHAT is true) - could be useful

[maelle]

Related PR https://github.com/ropensci/rredlist/pull/41/

[maelle]

with the PR above, I'm not sure anything other than docs is needed in vcr / the http testing book. :thinking:

[maelle]

this will be documented in the book but it might be useful to mention in the docs here, maybe.

[sckott]

sounds good