package tests failing only with covr

[dpastoor]

When trying to covr against packages pulled locally, covr is failing in places that devtools::test() is not. For example, with shiny in the rocker/tidyverse docker image:

> setwd("~/shiny")
> devtools::test()
Loading shiny
Loading required package: testthat

Attaching package: ‘testthat’

The following object is masked from ‘package:purrr’:

    is_null

The following object is masked from ‘package:dplyr’:

    matches

Testing shiny
bookmarking: .........................
bootstrap: ......................................................................
code diagnostics: ..................
garbage collection: ......
get-extension: ....
Parse Shiny Input: .....
modules: ............................
options: ....................
plot-coordmap: ...................................................................................................
reactivity: .....................................................................................................................................................................................................................
Stack: ...............
stacks: ............
staticdocs: ..
stop-app: ...
text: .....................
timer: ......
UI: ...
Update input controls: ..........
URL: .....
utils: ..........................................................

DONE =======================================================================================================
> covr::package_coverage()
Error: Failure in `/tmp/Rtmpyx7sqJ/R_LIBS323b233c825f/shiny/shiny-tests/test-all.Rout.fail`
> library(testthat)
> library(shiny)
> 
> test_check("shiny")
1. Failure: ggplot coordmap (@test-plot-coordmap.R#34) -------------------------
m[[1]]$mapping not equal to list(x = "xvar", y = "yvar").
target is NULL, current is list

2. Failure: ggplot coordmap (@test-plot-coordmap.R#36) -------------------------
sortList(m[[1]]$domain) not equal to sortList(list(left = 0, right = 5, bottom = 10, top = 20)).
target is NULL, current is list

3. Failure: ggplot coordmap (@test-plot-coordmap.R#41) -------------------------
sortList(m[[1]]$log) not equal to sortList(list(x = NULL, y = NULL)).
target is NULL, current is list

4. Failure: ggplot coordmap (@test-plot-coordmap.R#46) -------------------------
m[[1]]$panel_vars not identical to list(a = 1)[0].
target is NULL, current is list

5. Failure: ggplot coordmap (@test-plot-coordmap.R#50) -------------------------
m[[1]]$range$left > 20 && m[

I have tried this on a couple machines, with different versions of R, both on linux, in a docker container, and on windows.

I am perplexed as to how these are failing with covr, yet the tests are all passing. Any ideas what may be the cause, and how to work around this?

[jimhester]

devtools::test() is not the proper comparison, you need to run devtools::check() or R CMD check from the command line to see if there is a true difference.

[dpastoor]

ah! - yes this immediate popped up (an) error.

do you have any advice/thoughts as to testing the cran version of a package in a (reasonably) automated way, given an .Rbuildignore excluding directories/files that cause the package_coverage() to fail?

ideally people would have a github release or some other tag to a cran submission, but I find that few and far between. My immediate thought is to need to find the closest SHA to the submission timeframe and use that, but that feels messy. Hopefully you know a better way.

[jimhester]

The packages sources uploaded to CRAN are on the landing pages (https://cran.r-project.org/package=covr) see Package source:. You can also use the GitHub CRAN mirror for packages https://github.com/cran.

[dpastoor]

But there is no way to link that back to the commit/some mark on github or other vcs? Eg for shiny, the cran mirror has already lost the res/ folder, which causes devtools::check() to fail, since it looks for a file in that folder. To test the shiny version published on CRAN locally, somehow needs to link back to the most proximal commit on the source (eg rstudio/shiny) repo, if accessible, which would allow the ability to run the full devtools::check() against such that package_coverage() will also likely work. Right?