Trouble getting tests working on CI

[pat-s]

Hi Scott,

thanks for this package and the amazing efforts around HTTP testing in R.

I am somewhat new to mocking and scratching my head what I might be doing wrong. I followed the {vcr} intro and created a test file with cassettes which runs fine locally (making use of an API key). I committed the cassettes and added the API key as a secret to the GHA build (maybe not even needed?). In any case, the tests fail during CI: https://github.com/ropenscilabs/circle/runs/1073559270?check_suite_focus=true#step:12:99

Here is the {vcr} powered test file: https://github.com/ropenscilabs/circle/blob/21413279be13e1891b06866ed7428607474fdf32/tests/testthat/test-general.R.

Do you maybe have an idea what might be wrong in my setup?

[sckott]

thanks for the issue. don't know what is wrong at first glance. will keep looking.

fyi, looks like auth is sent in request headers https://github.com/ropenscilabs/circle/blob/21413279be13e1891b06866ed7428607474fdf32/tests/fixtures/new_build().yml so are exposed, not sure if thats okay or not

[pat-s]

Not ok, already got automatically invalidated 😄

Can I also filter out the headers in the same way as the API key?

[sckott]

Yes, the string you want to filter out can be anywhere in the request or response.

[sckott]

@pat-s still not sure what's wrong, but a few things to try:

• try removing the parentheses from the file names. that's probably not the issue, but maybe worth a try
• is the R_CIRCLE env var set as a secret on github? make sure to add any other secrets that could show up in the http request or response to the filter_sensitive_data list of vcr_configure()
• is there any reason to suspect that the functions new_build and has_checkout_key would make different http requests on CI vs. locally?

[pat-s]

I thought I found the issue (see below) but the tests continue to fail.

Some functions make use of gh::gh() which requires a GITHUB_PAT for authenticating to GitHub. In {tic} we set GITHUB_PAT to the GITHUB_TOKEN secret from GHA which comes with no relation to the user.

Setting a custom PAT for user authentication and passing the secret down to gh::gh(.token = ) should solve the authentication issue. Should...

[sckott]

Should ...

did you mean to finish that thought?

[pat-s]

= I don't understand why it fails atm. I thought I had it.

Next step is probably to see what differs if a different user calls the function with the same PAT. Or inspect the logs in GHA.

[sckott]

Maybe its this helper file? https://github.com/ropenscilabs/circle/blob/master/tests/testthat/helpers.R#L2

Both lines https://github.com/ropenscilabs/circle/runs/1157281856?check_suite_focus=true#step:12:154 and https://github.com/ropenscilabs/circle/runs/1157281856?check_suite_focus=true#step:12:184 say its the url https://api.github.com/repos/pat-s/travis-testthat that is being requested and doesn't match any recorded requests.

Since you have https://github.com/ropenscilabs/circle/blob/master/.gitignore#L6 i imagine on CI systems that helpers.R file is run which may be causing the issue

[pat-s]

@sckott Wow! Thanks a ton for diving in so deep.

And you were completely right with all your assumptions. Tests passed! 🎉

So I'll continue rewriting all other tests with {vcr} and give the ropensci review a push soon (hopefully).

[sckott]

glad it helped