vdiffr fails cran checks but not travis or appveyor

[njtierney]

Hi Lionel!

I'm in the process of getting visdat onto CRAN for a 0.1.0 release (the cran-0-1-0 branch), but I get a vdiffr test failure when I run the build through rhub.

Here is the error message I get from rhub when using the platform Ubuntu Linux 16.04 LTS, R-release, GCC

* checking tests ...
  Running ‘testthat.R’
 ERROR
Running the tests in ‘tests/testthat.R’ failed.
Last 13 lines of output:

  2. Error: vis_miss creates the right plot (@test-vis-miss.R#10) ----------------
  Graphics API version mismatch
  1: vdiffr::expect_doppelganger("vis_miss vanilla", vis_miss_plot) at testthat/test-vis-miss.R:10
  2: write_svg(fig, testcase, title, user_fonts)
  3: svglite::svglite(file, user_fonts = user_fonts)
  4: svglite_(file, bg, width, height, pointsize, standalone, aliases)

  testthat results ================================================================
  OK: 12 SKIPPED: 0 FAILED: 2
  1. Error: vis_dat creates the right plot (@test-vis-dat.R#10) 
  2. Error: vis_miss creates the right plot (@test-vis-miss.R#10) 

  Error: testthat unit tests failed
  Execution halted

Yet, on my machine, and on travis and appveyor, I get no errors, no warnings, and no notes.

I was just wondering if perhaps I am doing something wrong with vdiffr? Or if you have an idea on what I might be doing wrong here?

[lionel-]

Could you please squeeze something like this in the tests to check which freetype version the machine has:

ver <- as.character(gdtools::version_freetype())
cat(sprintf("FreeType version: %s\n", ver))

[njtierney]

OK! Currently squeezed in, will report back

[lionel-]

I'm going to update vdiffr while porting ggplot2 to tidyeval. It will gain an option to print the whole svg in the test output for easier debugging.

[njtierney]

OK cool, that is very exciting! Will this change much for me?

[lionel-]

It should make it easier to identify the cause of SVG mismatches on different platforms. An end user should never have to use it but we still get portability issues :/

[njtierney]

Hok, so running rhub::check_for_cran() with the inserted chunks in the tests

Platform: | Ubuntu Linux 16.04 LTS, R-release, GCC

* checking tests ...
  Running ‘testthat.R’
 ERROR
Running the tests in ‘tests/testthat.R’ failed.
Last 13 lines of output:
  FreeType version: 2.6.1
  2. Error: vis_miss creates the right plot (@test-vis-miss.R#12) ----------------
  Graphics API version mismatch
  1: vdiffr::expect_doppelganger("vis_miss vanilla", vis_miss_plot) at testthat/test-vis-miss.R:12
  2: write_svg(fig, testcase, title, user_fonts)
  3: svglite::svglite(file, user_fonts = user_fonts)
  4: svglite_(file, bg, width, height, pointsize, standalone, aliases)

  testthat results ================================================================
  OK: 12 SKIPPED: 0 FAILED: 2
  1. Error: vis_dat creates the right plot (@test-vis-dat.R#12) 
  2. Error: vis_miss creates the right plot (@test-vis-miss.R#12) 

  Error: testthat unit tests failed
  Execution halted

Platform: Fedora Linux, R-devel, clang, gfortran**

* checking re-building of vignette outputs ... WARNING
Error in re-building vignettes:
  ...
Quitting from lines 58-61 (using_visdat.Rmd) 
Error: processing vignette 'using_visdat.Rmd' failed with diagnostics:
X11 font -adobe-helvetica-%s-%s-*-*-%d-*-*-*-*-*-*-*, face 1 at size 11 could not be loaded
Execution halted

Platform: Windows Server 2008 R2 SP1, R-devel, 32/64 bit

* checking top-level files ... WARNING
Conversion of 'README.md' failed:
pandoc.exe: Could not fetch https://travis-ci.org/njtierney/visdat.svg?branch=master
no store
CallStack (from HasCallStack):
  error, called at .\System\X509\Win32.hs:56:31 in x509-system-1.6.4-E9zvdZ6nI282vjtOPFYSd1:System.X509.Win32

[njtierney]

Let me know if you need more/less info!

[lionel-]

the cat() call will not show up in the R cmd check output but it should show up in one of the files created by R cmd check (the tests outputs, I don't remember the filename). Can you get these files from R hub? cc @gaborcsardi

[njtierney]

OK, I think it did here though? In the Ubuntu Linux CMD check

Last 13 lines of output:
  FreeType version: 2.6.1
  2. Error: vis_miss creates the right plot (@test-vis-miss.R#12) ----------------
  Graphics API version mismatch

So it looks like the FreeType version is 2.6.1 ?

[gaborcsardi]

I am sorry guys, can you pls. make this simple for me, and tell me what I need to do?

[lionel-]

@njtierney oops, I missed that!

@gaborcsardi all good thanks ;)

[njtierney]

OK, so what do you think I should do from here? Is that version of FreeType OK?

[lionel-]

This version of FreeType should be good. You should probably skip() the visual tests on CRAN for now and release. Could you please post a reprex so I can look into it later?

[njtierney]

OK sure thing - silly question 😞 how do I skip the tests?

I can't quite post a reprex, as I can run it on my machine. Unless there is a way for me to use reprex on the tests and capture the output when run on another machine?

[lionel-]

you can use skip() or skip_on_cran() at the start of your testthat blocks.

The rexprex is the code that builds the plot that is failing on R-hub, it's fine if it works locally.

[njtierney]

OK cool, thanks!

OK, I've got a reprex of the code that builds the plots, I have avoided printint the plots because I'm not sure how helpful that would have been to link them here on GitHub, unless you would like to see them?

library(visdat)

# try out all the options
vis_miss_plot <- vis_miss(typical_data)
vis_miss_plot_cluster <- vis_miss(typical_data, cluster = TRUE)
vis_miss_plot_sort_rows <- vis_miss(typical_data, sort_miss = TRUE)
vis_miss_plot_show_perc <- vis_miss(typical_data, show_perc = FALSE)

vis_dat_plot <- vis_dat(typical_data)
vis_dat_plot_sort_type <- vis_dat(typical_data, sort_type = FALSE)
vis_dat_plot_pal_qual <- vis_dat(typical_data, palette = "qual")
vis_dat_plot_pal_cb <- vis_dat(typical_data, palette = "cb_safe")

# testing code...
# test_that("vis_miss creates the right plot",{
#   skip_on_cran()
#   ver <- as.character(gdtools::version_freetype())
#   cat(sprintf("FreeType version: %s\n", ver))
#   vdiffr::expect_doppelganger("vis_miss vanilla", vis_miss_plot)
#   vdiffr::expect_doppelganger("vis_miss cluster", vis_miss_plot_cluster)
#   vdiffr::expect_doppelganger("vis_miss sort rows", vis_miss_plot_sort_rows)
#   vdiffr::expect_doppelganger("vis_miss show percent", vis_miss_plot_show_perc)
# })

# context("vis_dat")

# try out all the options

## test_that("vis_dat creates the right plot",{
##   skip_on_cran()
##   ver <- as.character(gdtools::version_freetype())
##   cat(sprintf("FreeType version: %s\n", ver))
##   vdiffr::expect_doppelganger("vis_dat vanilla",
##                               vis_dat_plot)
##   vdiffr::expect_doppelganger("vis_dat sort_type",
##                               vis_dat_plot_sort_type)
##   vdiffr::expect_doppelganger("vis_dat qualitative palette",
##                               vis_dat_plot_pal_qual)
##   vdiffr::expect_doppelganger("vis_dat colourblind safe palette",
##                               vis_dat_plot_pal_cb)
## })
## 

## test_that("vis_dat fails when the wrong palette is provided",{
##   ver <- as.character(gdtools::version_freetype())
##   cat(sprintf("FreeType version: %s\n", ver))
##   testthat::expect_error(vis_dat(typical_data, palette = "wat"))
## })