Covrpage reports different results

[llrs]

This issue is related to issue #3 where we were testing the skip branch: There are some differences between local testing, the continous integration report, the coverage reported by covrpage on my machine and your report.

covrpage report:

• my report: 12 errors (and all the test show n=0).
• Your fork get 3 errors + 1 skipped (and the correct number of tests)

ci (travis) : no errors + no fails local (devtools::test): no errors + no fails + 1 skip local (devtools::check): no errors + no fails

[yonicd]

those difference may be expected since the results from each type of test routine is different in practice

testthat::test_check vs testthat::test_dir vs devtools::test

[llrs]

I think this would explain some differences but not the differences between your report and mine if we used the same function of covrpage to generate them (I used covrpage::covrpage).

[yonicd]

session_info?

[llrs]

Session info --------------------------------------------------------------------------------------------------------------------------
 setting  value                       
 version  R version 3.5.1 (2018-07-02)
 system   x86_64, linux-gnu           
 ui       RStudio (1.1.447)           
 language en_US                       
 collate  en_US.UTF-8                 
 tz       Europe/Madrid               
 date     2018-09-06                  

Packages ------------------------------------------------------------------------------------------------------------------------------
 package       * version    date       source                              
 annotate        1.58.0     2018-05-31 Bioconductor                        
 AnnotationDbi   1.42.1     2018-05-31 Bioconductor                        
 assertthat      0.2.0      2017-04-11 CRAN (R 3.5.0)                      
 backports       1.1.2      2017-12-13 CRAN (R 3.5.0)                      
 base          * 3.5.1      2018-07-03 local                               
 Biobase         2.40.0     2018-05-31 Bioconductor                        
 BiocGenerics    0.26.0     2018-05-31 Bioconductor                        
 bit             1.1-14     2018-05-29 CRAN (R 3.5.0)                      
 bit64           0.9-7      2017-05-08 CRAN (R 3.5.0)                      
 bitops          1.0-6      2013-08-17 CRAN (R 3.5.0)                      
 blob            1.1.1      2018-03-25 CRAN (R 3.5.0)                      
 callr           2.0.4      2018-05-15 CRAN (R 3.5.0)                      
 cli             1.0.0      2017-11-05 CRAN (R 3.5.0)                      
 clipr           0.4.1      2018-06-23 cran (@0.4.1)                       
 compiler        3.5.1      2018-07-03 local                               
 covr          * 3.1.0      2018-05-18 CRAN (R 3.5.0)                      
 covrpage      * 0.0.52     2018-08-26 Github (yonicd/covrpage@111cb6b)    
 crayon          1.3.4      2017-09-16 CRAN (R 3.5.0)                      
 cyclocomp     * 1.1.0      2016-09-10 CRAN (R 3.5.1)                      
 datasets      * 3.5.1      2018-07-03 local                               
 DBI             1.0.0      2018-05-02 CRAN (R 3.5.0)                      
 debugme         1.1.0      2017-10-22 CRAN (R 3.5.0)                      
 desc            1.2.0      2018-05-01 CRAN (R 3.5.0)                      
 devtools      * 1.13.6     2018-06-27 cran (@1.13.6)                      
 digest          0.6.16     2018-08-22 CRAN (R 3.5.1)                      
 evaluate        0.11       2018-07-17 CRAN (R 3.5.1)                      
 git2r           0.23.0     2018-07-17 CRAN (R 3.5.1)                      
 graph           1.58.0     2018-05-31 Bioconductor                        
 graphics      * 3.5.1      2018-07-03 local                               
 grDevices     * 3.5.1      2018-07-03 local                               
 GSEABase        1.42.0     2018-05-31 Bioconductor                        
 highr           0.7        2018-06-09 CRAN (R 3.5.1)                      
 htmltools       0.3.6      2017-04-28 CRAN (R 3.5.0)                      
 IRanges         2.14.10    2018-05-31 Bioconductor                        
 knitr         * 1.20       2018-02-20 CRAN (R 3.5.0)                      
 lazyeval        0.2.1      2017-10-29 CRAN (R 3.5.0)                      
 magrittr      * 1.5        2014-11-22 CRAN (R 3.5.0)                      
 memoise         1.1.0      2017-04-21 CRAN (R 3.5.0)                      
 methods       * 3.5.1      2018-07-03 local                               
 parallel        3.5.1      2018-07-03 local                               
 processx        3.1.0      2018-05-15 CRAN (R 3.5.0)                      
 R6              2.2.2      2017-06-17 CRAN (R 3.5.0)                      
 Rcpp            0.12.18    2018-07-23 cran (@0.12.18)                     
 RCurl           1.95-4.10  2018-01-04 CRAN (R 3.5.0)                      
 remotes         1.1.1.9000 2018-07-17 Github (MangoTheCat/remotes@e78d84e)
 reprex          0.2.0      2018-06-22 cran (@0.2.0)                       
 rex             1.1.2      2017-10-19 CRAN (R 3.5.0)                      
 rlang           0.2.2      2018-08-16 CRAN (R 3.5.1)                      
 rmarkdown       1.10       2018-06-11 cran (@1.10)                        
 rprojroot       1.3-2      2018-01-03 CRAN (R 3.5.0)                      
 RSQLite         2.1.1      2018-05-06 CRAN (R 3.5.0)                      
 rstudioapi      0.7        2017-09-07 CRAN (R 3.5.0)                      
 S4Vectors       0.18.2     2018-05-31 Bioconductor                        
 stats         * 3.5.1      2018-07-03 local                               
 stats4          3.5.1      2018-07-03 local                               
 stringi         1.2.4      2018-07-20 cran (@1.2.4)                       
 stringr         1.3.1      2018-05-10 CRAN (R 3.5.0)                      
 testthat      * 2.0.0      2017-12-13 CRAN (R 3.5.0)                      
 tools           3.5.1      2018-07-03 local                               
 usethis       * 1.3.0      2018-02-24 CRAN (R 3.5.0)                      
 utils         * 3.5.1      2018-07-03 local                               
 whisker         0.3-2      2013-04-28 CRAN (R 3.5.0)                      
 withr           2.1.2      2018-03-15 CRAN (R 3.5.0)                      
 XML             3.98-1.11  2018-04-16 CRAN (R 3.5.0)                      
 xtable          1.8-2      2016-02-05 CRAN (R 3.5.0)                      
 yaml            2.2.0      2018-07-25 CRAN (R 3.5.1)   

[yonicd]

I have been testing out this issue on different repos and have seen inconsistent outputs. I will revisit it more in depth when i get a chance.

[yonicd]

i opened a new branch where i replaced test_dir in the main Rmd with test_check in order to replicate what covr::package_coverage and covr::codecov` are running.

Hopefully this will clean up the diverging test results.

[llrs]

Should I test it with a failing test or with all the tests passing? I see that the skip branch disappeared.

[yonicd]

skip was merged with master. you can try with and without failing, i dont have the time today for running your repo.

[llrs]

Without a failing test it works as expected. However with a failing test it doesn't build:

> covrpage()

processing file: _covrpage.Rmd
  |....                                                             |   6%
   inline R code fragments

  |........                                                         |  12%
label: load (with options) 
List of 1
 $ include: logi FALSE

  |...........                                                      |  18%
  ordinary text without R code

  |...............                                                  |  24%
label: tests (with options) 
List of 1
 $ include: logi FALSE

Quitting from lines 16-20 (_covrpage.Rmd) 
Error: testthat unit tests failed

Session_info() 

Session Info

``` Session info -------------------------------------------------------------------------------------- setting value version R version 3.5.1 (2018-07-02) system x86_64, linux-gnu ui RStudio (1.1.447) language (EN) collate en_US.UTF-8 tz Europe/Madrid date 2018-09-12 Packages ------------------------------------------------------------------------------------------ package * version date source assertthat 0.2.0 2017-04-11 CRAN (R 3.5.0) backports 1.1.2 2017-12-13 CRAN (R 3.5.0) base64enc 0.1-3 2015-07-28 CRAN (R 3.5.0) cli 1.0.0 2017-11-05 CRAN (R 3.5.0) covr * 3.1.0 2018-05-18 CRAN (R 3.5.0) covrpage * 0.0.55 2018-09-12 Github (yonicd/covrpage@9cee008) crayon 1.3.4 2017-09-16 CRAN (R 3.5.0) curl 3.2 2018-03-28 CRAN (R 3.5.1) digest 0.6.17 2018-09-12 CRAN (R 3.5.1) evaluate 0.11 2018-07-17 CRAN (R 3.5.1) git2r 0.23.0 2018-07-17 CRAN (R 3.5.1) glue 1.3.0 2018-07-17 CRAN (R 3.5.1) graphics * 3.5.1 2018-07-03 local grDevices * 3.5.1 2018-07-03 local highr 0.7 2018-06-09 CRAN (R 3.5.1) htmltools 0.3.6 2017-04-28 CRAN (R 3.5.0) httr 1.3.1 2017-08-20 CRAN (R 3.5.0) jsonlite 1.5 2017-06-01 CRAN (R 3.5.0) knitr * 1.20 2018-02-20 CRAN (R 3.5.0) lazyeval 0.2.1 2017-10-29 CRAN (R 3.5.0) magrittr * 1.5 2014-11-22 CRAN (R 3.5.0) markdown 0.8 2017-04-20 CRAN (R 3.5.0) methods * 3.5.1 2018-07-03 local mime 0.5 2016-07-07 CRAN (R 3.5.0) openssl 1.0.2 2018-07-30 CRAN (R 3.5.1) praise 1.0.0 2015-08-11 CRAN (R 3.5.0) R6 2.2.2 2017-06-17 CRAN (R 3.5.0) Rcpp 0.12.18 2018-07-23 cran (@0.12.18) rex 1.1.2 2017-10-19 CRAN (R 3.5.0) rlang 0.2.2 2018-08-16 CRAN (R 3.5.1) rmarkdown 1.10 2018-06-11 cran (@1.10) rprojroot 1.3-2 2018-01-03 CRAN (R 3.5.0) stats * 3.5.1 2018-07-03 local stringi 1.2.4 2018-07-20 cran (@1.2.4) stringr 1.3.1 2018-05-10 CRAN (R 3.5.0) testthat * 2.0.0 2017-12-13 CRAN (R 3.5.0) tinytex 0.8 2018-08-30 CRAN (R 3.5.1) tools 3.5.1 2018-07-03 local utils * 3.5.1 2018-07-03 local whisker 0.3-2 2013-04-28 CRAN (R 3.5.0) withr 2.1.2 2018-03-15 CRAN (R 3.5.0) xfun 0.3 2018-07-06 cran (@0.3) yaml 2.2.0 2018-07-25 CRAN (R 3.5.1) ```

[yonicd]

works now. https://github.com/yonicd/GSEAdv/tree/test (notice that the computer committing is travis)

i changed a bit of stuff in your travis.yml file

[llrs]

Great!! Many thanks for your patience and your work it is a great package! I will use it now on for test driven development. Looking forward to see the package on CRAN and being used in Bioconductor!