`skip_on_cran()` reduces code coverage

yonicd / covrpage

Create a summary readme for the testthat subdirectory to communicate with potential users

https://yonicd.github.io/covrpage/

Other

52 stars 7 forks source link

`skip_on_cran()` reduces code coverage #52

Closed IndrajeetPatil closed 5 years ago

IndrajeetPatil commented 5 years ago

Recently I submitted my package and got this comment from CRAN maintainers and so I had to add skip_on_travis() for a lot of tests to reduce the duration of running tests on CRAN machines.

Can't you rather provide a reduced test set of most important tests and 
run less important ones only conditionally if some env var is set that 
you only define on your machine?

But when I re-ran covrpage::covrpage(), my code coverage dropped from 95% to 80%. Is there any way for covrpage to also include skipped tests while computing code coverage?

yonicd commented 5 years ago

they werent actually run.

covrpage runs test_check which is what covr runs.

it better reflects what is being tested.

you are correct when using skip_on_travis, that will run the test in covrpage.

When submitting to CRAN I add for them the skip_on_cran where relevant.

This does create a divergence in coverage in the package cran documentation and the GH/pkgdown coverage, but i think this is ok since the report reflects in each case the system that it was created on.

IndrajeetPatil commented 5 years ago

covrpage runs test_check which is what covr runs.

Interesting. Although I am skipping tests on CRAN, my code coverage on both codecov and coveralls has remained unchanged- https://codecov.io/gh/IndrajeetPatil/ggstatsplot?branch=master https://coveralls.io/github/IndrajeetPatil/ggstatsplot?branch=master

And both of these services, if I am not mistaken, rely on covr. For example, my .travis.yml looks something like this-

after_success:
  - Rscript -e 'covr::codecov()'
  - Rscript -e 'covr::coveralls(type = "all")'

That's why I was a bit confused as to why covrpage code coverage dropped.

yonicd commented 5 years ago

can you check the difference in running the covr addin on your system to see if there is a difference there?

IndrajeetPatil commented 5 years ago

You are right! The local covr addin shows the same code coverage as covrpage.

I wonder what's different about how codecov and coveralls are computing it. Anyway, if covrpage relies on covr in the backdrop then I guess nothing can be done about this.

I'll leave it to you to decide if you want to close this issue.

yonicd commented 5 years ago

i'll leave as is. the report reflects the system it was run on. thanks for checking that.